If I was the kind of person who would upvote a single word , don’t u think I’d do it more then once and on all the legit long comments?

I think it was more on the timing of your voting like on Day 6 and having too few on your circle you support that made it suspicious to begin with. While you may have no intention to game the system, the consequences are the same as those who do exploit the system. By changing the voting behavior to upvoting comments within idk, maybe within a day or two after they made would probably be more ok and also not your own or the same set of people, that would be even better. It's still up to you to take the advice or not and there's no guarantee the downvotes will stop. I'm just saying these are the stuff you did that caught some attention.