You did the right thing. Your well-being should be your major concern. Every other person would be fine.
When you think of it, after or not you consider what other people think, if you do well, they'll clap for you and if you don't, they'll blame you for it.