Ultimately, it was the Dark Apostle and his agents on those worlds.
We will eventually learn that the DA was responsible for their eviction in many worlds.
In the case of Nysegard, their original followers were driven to enclaves much like the Siblings people by the Storm Lords. After the fall of Doom, they got even more suppressed by the Storm Lords, however the Siblings had very little to do with that. They simply didn’t have the same resources that the Siblings had. After Doom’s fall, they worked to rescue those that wanted to leave and fortify the remaining.
So there are still a few enclaves on Nysegard, but like most groups, they live in isolated enclaves; their are probably about 1/5 or 1/6 as many NE followers on Nysegard as there are Siblings.
This will actually be coming up at the end of this book/beginning of Next. [SPOILER]Tom needs more people on the Isle of Doom, he is going to invite the Nyjyr Ennead to move followers to the Isle of Doom (either existing ones on Nysegard, or new ones. He will want to strengthen his alliance with them.[/SPOILER]