Too many times, we get faced with situations where saying 'no' would hurt the other party, but it remains the only thing to say to protect our interest or happiness. In circumstances like this, we should apply the principles of self-preservation which implies that "you matter too..."
You should not be so concerned about another person's happiness that you forget yours. It is OK to make sacrifices for others, but learn to live a balanced life where you do not make yourself the sacrificial lamb always.
Learn to say "NO" and not feel guilty about it no matter how much the other party tries to put up an emotional blackmail. As the pilot of your life, before taking any direction, consider your interest and happiness. Self-love, self-preservation does not make you selfish.