Years ago we had a well-paid consultant at work. He came in for a couple of days every week, and we often talked about non-work topics. One day, while sharing personal updates, he mentioned that he and his partner had been out for a walk and had come upon an abandoned kitten that was not doing well. They decided to adopt it. Every week there'd be a kitten update -- it turned out the kitten had extensive health problems and needed a series of surgeries. In the end this kitten needed thousands of dollars of care over the next few months, but was eventually OK.
As he was describing all this, I had a vision of existence from the kitten's perspective. One moment you're wet and shivering under a bush, unsure of what to do, and the next moment there's a vast money hose beyond your comprehension aimed at your well being, flowing from pension funds and wealthy investors to VC firms to a startup to a consultant to a veterinarian and finally to you, and strange magic beyond your comprehension has made your pain disappear.
It's akin to the sort of comfort many religious people feel around a benevolent deity with a special plan for them in particular, or the rosy AGI scenario where a pervasive and powerful artificial intelligence has humanity's best interests at heart, and guides people toward their fullest potential with love and grace.
The problem is that there’s this intervening period where the entity is powerful enough to harm us but not yet at the point where it’s decided to be benevolent (sort of like the period between when mankind gained the power to influence the physical world at large scale but before we reach[ed] the level of consciousness where we decide[d] not to ruin the earth).
A bunch of humans might die during that period, possibly even most humans.