Personally I kinda disagree. I try all viable models all the time for prompts I like, and choose the model with the best outputs to test prompt derivations. It's not one or the other but always the best and most apt for the task. Some are good for backgrounds, some for anime characters, some for photorealistic characters, some for sci-fi, some for fantasy, etc.

Most of the time, though!

Sometimes I'd just like to go to midjourney because I want a fast image and don't care if it's got that perfect anime style or that perfect dark 60s photography feel. Which reminds me I wanna pay for midjourney but it's so expensive. Sadsad.