I had a user in my autovote in the past who was mostly running into the automatic and had no engagement in the blockchain. It is said because it was a user that I really appreciated the texts from when I joined. But it is like you said: sometimes we want to help and think that this user is promising, but the years pass, and the user stagnates; let's say.
Downvotes are always associated with drama! And I know that there are medium/big stakeholders who are still against it. I hate to debate, but it is here to be used, right? But one thing that you said in a comment is true, and it can fit in any place in this discussion. Nothing is white and black. And probably one person that I really deserve to be ignored m, maybe in your point of view, deserves a vote and vice versa.
Maybe the main and important thing to think about is to see people who care about the community; even in projects that somehow failed, they can't be forgotten. That's why, in those debates about DHF on Discord, I try to be neutral, even if some projects I really was against since the beginning because I didn't see any sense; I know that people behind them at least are trying to make the community better.