If witnesss don't vote then things just proceed as they do now (slower irreversibility, after a minute or so). Such witnesses should likely get voted out.