That's correct, though one of the challenges w/ memo key is that if it is changed, none of the old messages will be readable (unless we store these old ones somehow). And currently, the only private memos we have rely on using transfer memo, which requires active key.