Because you have built a certain persona around yourself that you may or may not like. And would rather see it improved upon or remain untarnished. Some people do not care what others think and go on living life as they please , this may be inspirational to others but can also come off as unrefined or brutish to another group of people.
Sometimes it is good to react because you are able to present yourself in the correct light or how you want to be perceived rather than how someone else's perception about you is. In the case of wrongfully accused, their entire lives are turned on their heads and ruined along with their reputation for something they may have not even done. They could be a great person but looked upon differently because of where they were at a certain time or how they dress.
It is hard to just say it doesn't matter what people think because sometimes when you need to reach certain goals you may need to impress the right people .