Well, not practically.
If somebody compromises your keys they get both wallets if you don't change them.