Probably for the best, downvotes are likely to be relegated to fighting blatant abuse of rewards, (farming etc and any proper illegal things like id theft and all that) and even that is getting problematic..

It's too contentious as is to downvote to adjust rewards or do so subjectively.