You're right, he left the kids to the women and he would have probably lost his mind if all the kids were living with him, and he may have stopped having kids if he had the responsibility of taking care of them, because as far as I know, raising kids is a very tasking thing to do.