The Hulk's father was very abusive towards him. The reason that Bruce doesn't have any children is that he is afraid he inherited the "monster" gene from his dad. Brian's alcoholism didn't help matters either.