Of course I am not @netuoso, and I am not an expert in any way, but the way I believe this will work is, that your owner and active key (probably not the posting key, because using the ledger at least a dozen times a day for voting and commenting seems very inconvenient) will be stored on the hardware wallet, and only there.
If you want to change your keys or transfer Hive or Power Up/Down et cetera, you'll need to confirm that operation with your hardware wallet. The Hive will still be on your account, just the keys will be stored in a different manner. (At least that's what I believe)