Votes are 100 bytes more or less (depends on the size of the permlink), so theoreticaly hundreds can fit in block (64 kbytes). I doubt this will be a major constraint, as very little content even gets that many votes total.

Also, is there a mathematical reason it doesn't make sense to vote earlier than 5 minutes, or is that simply a property of system conditions? Whether the linear penalty curve is really ideal is something I've been meaning to look at and may become more important if the window is shorter.