One doesn't have to be an expert in Hive blockchain to understand the recent surge in activity mainly due to Splinterlands has posed some unexpected scaling issues.
Quick Scaling Fixes and More Long Term Plans
Some of these issues have mainly been tackled (source 1: Blocktrades post, source 2: CryptoManiacs Interview with Blocktrades), with more optimizations for the "write" operations (broadcasting transactions to the blockchain) likely to be included in the package for hard fork 26 (source 2).
Since dApps - and especially Keychain - have been updated to use asynchronous transactions broadcast instead of synchronous ones (i.e. unblocking instead of blocking), Hive shifted to a higher speed gear. Have you remarked it? And I don't mean compared to the time of the bottlenecks, I mean compared to before. Just go ahead and vote something on PeakD or whatever. See how quick it is now! :) And I haven't seen any issues starting the matches on Splinterlands either since then.
Hive Application Framework (HAF) will be released very soon by Blocktrades team and will also include a number of optimizations and possibly will allow to avoid some of the issues we experienced during the past few weeks in the future, because dApp developers who will use it will likely not have to worry that they might use legacy or non-performant options when coding.
Not Everything Should Be On the Core Level Blockchain
Still... with all the optimizations at the core level, anything that is written to or read from a blockchain takes time.
That was the reason why Hivemind was created in the first place. To act as an intermediary between the blockchain and various dApp which use it, being far more responsive than the blockchain could be.
That is why Leofinance team created LeoInfra on the backend, to act as a caching mechanism that allows a smoother experience for the user even when the blockchain lags. LeoInfra creates a backlog of operations that need to be broadcasted to the blockchain and tries to do it until it succeeds or fails, but the user can go on and do something else in the meantime.
I see Matt from Splinterlands, after the bad experience of the scaling issues they had to battle with while the game has become increasingly popular (as in exponentially more), wants to take a more drastic approach to prepare for even more traffic: stop broadcasting some types of transactions to the blockchain altogether and make them go directly to the Splinterlands website instead.
The issue is correct in my opinion. I don't know about the chosen custom_jsons that will be taken out, if any of them is critical in determining the result of the match. If it is, that's a problem that needs addressing.
Thinking Ahead
But when we talk about thinking ahead a couple of multiples of growth, at some point, with all the optimizations, the blockchain will not handle a high enough simultaneous activity writing to the base layer non-essential information. This is the same reason
why Blocktrades wants smart contracts on the second layer and not on the first layer (one of the reasons, anyway - watch the interview, nice metaphor there with your computer and its operating system).
So why did I mention Project Blank in the title? Well, simply because that's another dApp that will have a significant impact on the blockchain activity, especially if it becomes popular outside Hive too.
It's likely Khal will use his LeoInfra backend for the microblogging app as well, but in the end all short form messages will end up on the chain.
I suppose it depends a lot how he organizes his new data structure he said he designed for Project Blank. Because it's one thing to write to the chain short comments individually, it's another to write a larger data structure at once. This, on the other end, can end up using more processing power/memory to add together/break into pieces. Just throwing ideas here, I don't really know the direction the project is heading.
Final Words
Scaling capability of Hive should not be taken for granted and should not be abused. There isn't any blockchain in the world that can scale indefinitely and little shocks like we had these few weeks should be a reminder.