Tell him, "Sorry Dad, I'd love to see ya, but we're not getting the shot and we don't want to endanger the kids by having them around those who have gotten it. Let's see how things play out over the next year or so before we make a decision. If all goes well, we'd love to have you down here then."
If he can't cope with that answer, then that's his problem.