We have downvotes as a check on abusive upvotes, wouldn't it make sense to have some kind of check on abusive downvotes?

They're not equivalent. Upvotes earn rewards and generate payouts, downvotes don't. The sort of abuses are different and while downvote abuse may be annoying to an individual, it isn't systemic. The 25% limit on downvotes (essentially 2.5 full power downvotes per day) means that there are always many payouts escaping downvotes (or at least not being downvoted very much), no matter how much havoc someone wants to cause.

sounds like you don't like having rewards for content on the first layer. Is that a popular opinion among the devs and other large stakeholders you talk to

It is somewhat mixed but very few actually think it is working well, or has ever worked well going on six years.

If so then why wasn't it removed earlier to avoid confusion?

In part because it is a significant task and there have been other priorities, including building out software infrastructure that would make second layer solutions more powerful and accessible. There isn't much support for removing it without better support for alternatives.