a lot don't even bother if you photo them
I call it mental health, lol.
If someone takes images with a mobile, everyone thinks - it's super amateur, so why should you bother? And if someone has a photo camera, people feel you are probably doing sort of job. And they are actually right.
you grumpy asking to delete the photo
There is a simple answer in countries where taking images of strangers is legal (in case you want to keep the photo) - "call the police or don't prevent me from working". If they don't leave, call the police.
But I doubt many people would insist on deleting, actually, even in the West. Even most toxic people will leave, I believe, if you shortly explain why you are photographing ("that's my work", "doing a street photography project" or anything), say "sorry", and ignore.
RE: People of Pokhara, Nepal: 50mm Street Shots at Prithvi Chowk