That's a really good question. I actually deal with that here at work. I have this reputation of being difficult, and even when I am trying not to be, people still expect me to be. Of course to them, me simply telling them no is considered "being difficult".