a lot don't even bother if you photo them

I call it mental health, lol.

If someone takes images with a mobile, everyone thinks - it's super amateur, so why should you bother? And if someone has a photo camera, people feel you are probably doing sort of job. And they are actually right.

you grumpy asking to delete the photo

There is a simple answer in countries where taking images of strangers is legal (in case you want to keep the photo) - "call the police or don't prevent me from working". If they don't leave, call the police.

But I doubt many people would insist on deleting, actually, even in the West. Even most toxic people will leave, I believe, if you shortly explain why you are photographing ("that's my work", "doing a street photography project" or anything), say "sorry", and ignore.