A middle ground between removing rewards and the status quo that I've thought about would be a two level system where Hive stakeholders vote to allocate Hive rewards (inflation) to communities and then those communities get to allocate them as they see fit in terms of content voting rules, 2nd layer tokens backed by Hive, DeFi yield farming schemes, or whatever, etc. Of course, that would require a bit of rearchitecting and development at the blockchain level, but I don't think it would necessarily be all that much.
RE: The Downside of Downvotes