Many months ago, almost as if I was already accepting my fate, I began working on a wrapper for S3 technology. My idea was basically to build a custom solution that could scale sideways. Something I could grow or shrink depending on demand, while keeping full control.
It's infrastructure we could turn into a business model, mind you, but more than that, I was thinking about how the Butter Cloud, as I call it, could become the scalable solution for ThreeSpeak.
This part of our somewhat obscure past might be easy to forget at this point, but those who are in the know, are in the know. When ThreeSpeak began, it did so without incorporating IPFS at all. It was an S3 bucket, and it worked beautifully.
Now, I know the promise of IPFS is amazing, and I also know it can be the difference between a robust system and an unstoppable one. But as I've said before on this blog of mine, we're slowly reaching a breaking point with it.
Now, the tricky part is adding an S3 bucket back into our pipeline without adding infrastructure costs, while also having a system that can eventually operate with the Speak Network. Which, of course, is the very reason ThreeSpeak moved to IPFS to begin with.
I've been thinking about this for weeks, and I've even shared some ideas with trusted friends and developers. I think I'm on the right track, and I've begun improving the Butter Cloud for this very purpose.
Not to bore people with too much geek stuff, but I think using IPFS for load distribution and redundancy is still the best way to go. However, the cold and warm storage of videos, along with the giant repos that make our servers suffer, needs to go.
Here's the interesting part. Because IPFS is deterministic, storing the exact same files and directory structure in an S3 bucket, without modifying their contents, gives us a path to bringing that data back into IPFS with the same CID. Provided we preserve the relevant import settings and directory representation, we don't need to keep those giant repositories around just to preserve the original identifiers.
But the kind of hot-cache technology I'm attempting to design has to be a bespoke build. There is no turnkey solution for this, and for good reason.
There is only one ThreeSpeak.
If it all works out, we would be allowing people to back up their own videos with an easy-to-use client app. We would be moving towards our own S3 buckets soon, and within six months to a year, we could begin deprecating the old IPFS repos. The unmaintainable beasts.
In truth, I believe this is the only way we can grow, even through this bear market that seems to never end.
MenO