That's not right. The early curators only make out of there are later and generally larger curators. The later and larger curators are only getting minimal curation rewards to begin with and lose out big time if there are downvotes.
On a particular post, downvotes may not be able to do much but to the extent that the entire balance of voting behavior on the platform can be changed with downvotes (cough, bidbots, cough), then curation rewards to earlier voters go away too.
RE: A Method To Combat Upvote Spam Aimed At Maximizing Curation Rewards