What're your thoughts on buying physical silver to the point that it's not safe to hold at home?

Does paying someone such as a bank to hold your physical metal in their vault, defeat the purpose of if you don't hold it, you don't own it?

How do you overcome this?