Your temperment towards the phrase matches the concious response. As I'm sure you know, most people who use it never think about the connotation of the word. I also feel weird saying "baby girl" to a grown ass woman and never used it. But I'm like you, if people get pleasure from it then more power to em. I just doubt they're thinking about why the phrase is what it is and how it came about.
RE: Discussion: Women calling Men "Daddy" and Vice Versa.