Downvotes have nothing to do with freedom of speech! Downvotes are about reward pool allocation.
Remove it and you will get tons of abusers who will deprive the legit authors of their deserved payout.
My opinion is you are tackling the problem from the wrong side.

...reputation can only go up, in a normal situation.

Tell me about your reputation as a human, if you screw up with your friends, your boss, your family,... Will your reputation go up?
But maybe we should first agree on a definition of "a normal situation".

I do not know of any good example where reputation (i.e. trust) is linked to the level of activity, especially in a limited amount of time.
If this is your only criteria, my bots will always have a better reputation than me, at all times, because they are much more inexhaustible, patient and persevering than me.