When you delegate, the HP is still yours, it is recorded in the blockchain. The account you delegate to thus cannot transfer your HP to some other account or steal it in any way. The risk is the same as if you were holding you HP in your wallet.