There should be a confirmation dialog when removing a vote to prevent accidental clicks. Since un-voting doesn't give back your voting power it's really annoying.
steemit.com opens a modal like this when trying to remove a vote. Utopian should do something similar in my opinion:
Tested on Google Chrome Version 62.0.3202.94 (Official Build) (64-bit) on Windows 8.1 x64
Posted on Utopian.io - Rewarding Open Source Contributors