I can identify with your words, I have lost friendships by saying no. The truth is that nothing you do will change them, so the sooner they get away, the better.
The important thing is that you are clear that there is no error in you. 😊