Perhaps it wasn't always as dramatic as you descibe it. As you say, the neanderthals perhaps got more absorbed into the sapiens group, rather than being killed by them.
Similar things may have happened in other parts of the world. And on top of that, life was incredibly hard for those guys. Its possible that some groups simply didn't make it over time, and died out on their own.
And we have to keep in mind: we are talking about incredibly long periods of time, and also the world was incredibly large for them in their time. Some tribes may not have met any other humans for many years. Elsewhere may be natural desasters wiped out entire sub species. Too bad they didn't write diaries in those days... :)