I think votes should be set to expire after a set amount of time. Currently, witnesses can still retain the votes of deceased persons and users who have left the blockchain or are simply inactive for a very long time. You can imagine how this is going to add up with time.