I think "toxic" is a fashionable term that someone adopts when they meet a person who actually annoys them, I agree with you.
It is unnecessary to speak of toxic and actually makes the person who uses this term someone who in turn sprays poison.

There were moments of conflict and friction, I had to stand my ground or realise that I could not win a point. To ascribe principled toxicity to people by whom one has been hurt, disappointed or betrayed is rather indicative of a need for retribution. Already this verbal use of " toxicity" and "keeping one's own environment clean of such people" and similar expressions seems to me rather like human beings would be an object to be treated similarly to dirt or vermin.