How is it unfair?

It isn't, really, as long as you accept that downvoting is fair too.

And, ultimately, it doesn't so much matter if it is fair because some people will continue to do it (applies to both buying votes/vote trading/etc. as well as downvoting).