It's better to receive a downvote that can be removed than to lose your account to some scammer.

If you're following a trail please remove your account off it and monitor it for any upvotes or downvotes you didn't cast. If you see nothing unusual, then all good. However, if you are not on any trails you will need to change your keys as that means that the hacker has control of your account.