Other people's opinions should never affect the decisions you make. It's your life, not theirs. Do what makes you happy.