Strengthening oversight and automating systems to protect against these risks is crucial, as this particular failure mode is both catastrophic and entirely preventable.