I think there are some ways to avoid this. To be honest I have not completely researched this and neither completely calculated it through because I want community input on these things first before I dig deeper into it.

An easy way to avoid this is to base curation not only on how close someone got to the actual result but also on how big the distance was between after the own vote and the final result.

(After I voted 1$ it got to 1 out of 20$ (final result) which is a bigger distance than after a whale voted 19$ and got to 19$ out of 20$).

However, I do think that the general approach is
a) Less controversial than downvotes
b) More what we actually want.