Hmmmn you're right!
It's easy for us to say "let go"
Just as you've said.
the best thing to do is discuss the red flags with your partner, give them time to observe if there would be changes or not, then after that you can take the decision of letting go.