No it wont because downvotes affect payouts not visibility. I wouldnt want someone to reduce rewards just to see less of something. If you dont want to see something you block them and then they cannot see you, you dont see them, they cant comment on your posts, if they comment on another frontend, thats hidden from everyone on Lumen on your post.
The "I want less of this" is something I might build in as a sort of slight adjustment or improvement, but right now for the launch its still not there.