Life is worth experiencing buddy. Don't ever let anything hold you back, your money included. I guess I understand how you feel. Kinda stuck huh? I mean that iis all you've ever wished for, then why this?
The thing is, humans are always on the lookout for what they think will make them happy, only to get that thing and realise that it's not it? I don't really know, but I think you grew too attached to your monetary goals. Take a step back for some re evaluation.