I think downvotes are needed but their effect should not be based solely on Hive Power, but reputation too.
You should have needed to gain some reputation before your downvotes count as anything financial, and they should have limited financial effect until your reputation is really good. This means some users cant just sybil loads of accounts delegate their power and make multiple downvotes. All the power for downvoting needs to be concentrated, and that single account needs to remain in good standing.