A big problem with game difficulty is the same as the developers that use stock Unreal Engine for their games.

Lack of optimization.

Developers/publishers don't want to spend the money on optimizing their games. Maybe they don't have the budget, maybe something else took too much of the budget and caused cuts in other areas? I don't know, I just know it is a balance issue that optimizing could fix.

Nintendo was amazing at this early on. It is part of why their games are remembered so fondly by fans. When is the last time you saw a difficulty slider in a Nintendo developed game? Mario didn't have one. Metroid didn't. I can't remember seeing one.

Because they optimized, they balanced, they reworked areas that sucked. The games have valid progression rather than a brick wall on anything other than the easiest setting.