My understanding is that downvotes are not to be used to "virtue signal" simple differences of opinion. That's what the "mute" button is for.

Downvoting is free-speech in the exact same way that shouting down a speaker in public is free-speech.