Your reasons are valid. If he really wants you to be his woman he shouldn't be sneaky about it.
Yeah it might hurt, but you'll be fine.