Excellent point, I was more or less thinking about this exact issue, and it's good that more people should be thinking about it. Gather momentum to remove witnesses whose positions towards the fork are not in alignment with you. Otherwise, it means your movement does not have stake support.