What is the drawback of having smart contracts and NFTs on a second layer versus first layer?

I suppose it is less secure or decentralized in some way but why exactly?