Because his account is in your account's posting authorities, it means he can do any posting-auth operation on Hive using your account. This means voting, posting, editing. So he's using his auth on your account to vote on his comments using your account. This is why I suggest only keeping the accounts for the services that you use authorized, and removing the rest.