When you harvest potatoes depends to some extent on how large you want them to be when you dig them up. Some people like to harvest them when they're a bit smaller, and some people wait until they're as big as they're going to get.
It's better to not harvest them until they flower, but if you want them as big as they will get, wait until the plant starts to die. I usually don't harvest my potatoes until the plants die.
You might find a few of the potatoes have holes in them from grub worms or other bugs, but I just cut that part out when I peel the potato to cook it. It doesn't hurt the rest of the potato.
If you find a potato with a green spot from being exposed to the sun, cut that part off when you prepare it for cooking. The green part of the potato is not good for you.