Well, as far as the specific example you gave goes, i.e. adultery, I'm in full agreement with you! I personally believe it should be punished. To some extent it is, meaning that if you divorce because the other person cheated on you, you have certain benefits in some countries, like you might not have to give them half your fortune or whatever. But generally, I think the main problem is that people cheat so much, that it's impracticable to enforce punishment. It's really not up to the law, but to the culture. In the religious societies you mention, it is a legal crime. That's definitely not the society we want, because in those places being gay too is a crime, or showing your face, or wearing pants, or some other trivial thing.

Generally, as an atheist, I don't see any conflict between non-belief and having strong moral values. Objective values. In fact, it's secular society that has been pushing religion toward the right moral path, e.g. against slavery and such. I might be wrong, but I don't think there's a single case where religion taught humanity something new moral-wise. There's a reason they don't really teach the Bible, or any other religious text, in ethics in philosophy departments.