It is said that our personality is the average of the personalities of the five people closest to us.

Interesting thesis. Since "personality" is not really a solid and final term to define, I tend to say that my intelligence or my ability to navigate a social environment depends on the smartness and stupidity of others and I can choose whether to align myself with the smart or the stupid. Of course, this is also far too simplistic, because intelligence is just as difficult to define as personality.

Perhaps if you tie it to a physical characteristic, it becomes clear what I mean. If, for example, a group of walkers has to walk a certain distance towards a goal, then it can only go as fast as the slowest person in the group goes. If there is only a slight discrepancy between the fastest and slowest runner, this makes little difference to the group as a whole.
If the difference is immense, it affects the overall speed.

This does not mean that the individual cannot run faster.

It is a decision whether to adapt to a person's pace or to ignore it and set your own pace. Since all people in a group never behave the same way, you can trust that there will be someone to keep the slowest person company and that there will always be those who will go the way of the risk-taker or loner. And everything in between. The roles also alternate and are not static.

People around us have a great impact on designing and modifying our attitudes and behaviors.

I agree with that. And why not. It's also vice versa- It's a natural behaviour, I would say, and often it goes unnoticed because you don't think about it all the time, you just live.

What if these annoying people are the only ones around you and you don't have a way out to the kind of people you relate to?? You would finally be modified into ones you didn’t appreciate initially, wouldn't you?

Yes, I would.

It's a very rare and extreme situation. It means that you are physically imprisoned, for example. But as soon as one had the opportunity to escape from prison, one would choose this way out, wouldn't one?

To the extent that one misses relationships on a mental level or no longer feels at ease in a group, one can decide to leave that group. Of course, it is a high price that one must be prepared to pay. Breaking away from the familiar requires courage and a willingness to take risks.

To the extent that one does not feel able to muster this, the natural impulse to adapt is probably the best solution. In fact, this adaptability is one of the most remarkable among us humans and ensures survival. One learns to accept what one is unable to change.

But since this extreme situation is not common, and the annoying people are not the only ones in one's life - often one has a close caregiver in the form of a brother, sister, relative, friend, or colleague - the image of manifested hopelessness is more a mental matter, I think.