You'll find that naysayers do so because they put themselves in your shoes and decide they don't like it and so you shouldn't either. I found that when you surround yourself with people who are passionate about these things, you fair better advice and support. I don't talk about gardening with people I know who have a negative view of it as a income stream. But I have a lot of family and friends who already see the benefits of our house becoming a place they can get cheap plants, vegetables, fruit and even eggs.

Sometimes it's easier to view it as a hobby first, as a labour of love, that will one day grow into a side business. That's the way I view it. Right now, I am dedicating 2-3 hours a day improving my backyard and beginning the process of planting in the next spring. Sure I would like to make a profit from my endeavor, but I know I can at least substitue expenses at the shop or market and lower the cost of living and even trade my produce for other things like an extra pair of hands for a day or some favour or help or item I may need. Any profit after that is a bonus!

Good luck