That is right, it's the initial signup where Signal sends an SMS with a code. But, if that SMS is intercepted by a third party, Signal won't know that. Also, during that initial process (correct me if I'm wrong), a key pair is generated, where your public key is stored on their server which it identifies as you, as you explained above. But your description is correct. You are right, I failed to mention that part, but it was well documented elsewhere.
RE: Switching System 7 (SS7) Security flaws