The democratization of data.
It is a topic I touched upon for more than a year. Over that time, we saw the rise of AI models to a degree few imagined. The race of AI supremacy is on. This is not only the case with countries but also corporations.
One of the keys in this entire discussion, along with energy, is data. It is no secret that data is the one of the centerpieces to these training models.
The search for data is on. Some feel that we are running out of what is available on the Internet. It is forcing some to look into real world data as an option.
Even the data available online is dwindling. While it is still there, we are seeing a lockdown that could cause major issues.
In this article, we will look at the shift that Cloudflare forced upon the world and how it increases the importance of a database such as Hive.
Cloudlflare is one of the most crucial companies when it comes to Internet infrastructure. Recently, it changed its policy, making the blocking of AI crawlers the default setting for all new websites. This means that scraping of sites can occur only with the permission of the site owner.
It was applauded as a win for those creating content, opening up doors to new paths of monetization.
Enter 'pay-to-scrap".
Cloudflare offers the ability for sites to charge, on a per page basis, for a fee. This protects the content from being used to feed AI models with no referral. The numbers of Anthropic and other models referrals are miniscule compared to Google.
The protecting of content was applauded. But it does open up a can of worms.
To start, this blocks all crawlers, including search engine. It also excludes the content from the models. That means the creators could be lost in both search engines or have their data omitted from the chatbots.
Another problem is that some companies simply might ignore this changes by using proxies and other technical workarounds.
The legal environment is behind with no framework which to deal with violators. Obviously, governments would have to take this matter up, a slow process, before we see something that could combat the nefarious players.
Finally, and what I think most important, this simply favors the major players. Those who can afford to pay for the data will. This, of course, comes down to those with large revenue streams. At the same time, many of the bigger players have a lot of data generated on their platforms.
One of the premises of Web 3.0 is open and transparent.
Data that is placed on a blockchain is owned by nobody. There is no way to lock it down since anyone is able to run the software. The database is "owned" by humanity.
Hive offers text storage beyond traditional digital ledger technology. It is censorship resistant and open to anyone. There is no need to get permission to use an API when anyone can set one up.
What Cloudflare did is huge. This changes the landscape of Internet data. Naturally, this is an extension of what already was taking place since many of the top sites were blocking the AI crawlers.
Hence, we are seeing a hybrid system developing.
Some models will contain both open and paid for content. At the same time, many will simply focus upon what is available out there for free.
Hive falls into the latter category.
With this move, the number of sites offering paid data is going to increase. This is a natural outcome if all new sites have AI crawlers blocks by default. Few will manually turn them back on.
To counter this, the amount of data that is democratized, i.e. open to everyone, also has to increase. AI is a very powerful tool. The major risk is that a few entities will be in control. While progress is being made with newer algorithms and better labeling, quantity of data is still a starting point.
Few consider the impact of a digital world where most of the data is fed into the hands of Big Tech. Companies that raise billions for their AI systems are not worried about a few million for data acquisition.
However, this is not the vast majority of newer players. If there is to be AI decentralization, we must have the democratization of data.
Without that, Meta, Google, xAI, and other well financed firms will dominate.
People say data is the new oil. That means we have to seriously question where we are placing our oil.
