There's a video somewhere out there, or at least archived somewhere inside my head, of a man teaching people how to squeeze every last bit out of a toothpaste tube. These days, that man is my spirit animal.
As you guys might know, we've cut back our infrastructure quite a bit. Out of necessity, of course. But that also forced us to learn how to optimize everything.
ThreeSpeak now runs on a budget that's a fraction of what it used to be. It has more features than ever, more tricks up its sleeve, and yet we somehow keep adding more.
When it comes to Snapie, my own little corner of insanity isn't much different.
Aside from the witness server, I have two other VPSs my brother kindly sponsors, and those machines are expected to run everything we've been building. Snapie, experiments, databases, prototypes... they just keep piling up.
Sometimes I wonder if I'm pushing my luck.
Every new service feels like asking those poor servers if they could "just do one more thing."
So far, though, they've been surprisingly cooperative. Spending the time to tweak and fine tune everything has paid off. Until now, however, I don't think I'd seriously considered trying something this ambitious.
A local LLM.
Well... local-ish.
The obvious choice is Ollama, and I've started playing with a heavily quantized Qwen model that might just be good enough to pull this off.
So what is this little AI supposed to do?
Read posts.
Not write comments. Not generate generic replies. Definitely not become one of those bots nobody asked for.
Instead, I want it to enrich the database with information about both the blog posts, and the snaps people share on snapie.
Is the post coherent?
Does it look like genuine effort?
Are people actually engaging with it?
Do the tags make sense?
Is it engagement bait?
Things along those lines.
I already have a decent list of questions I'd like answered, although whether this little AI can answer them consistently is another story entirely.
You see, using a tiny model is the price of squeezing every last bit out of the toothpaste tube.
The downside?
Accuracy.
Every now and then Richy, as I've decided to call him, may look at a post about surfing and confidently conclude that it's about politics.
Because... reasons.
So what do I actually want to do with all this information?
Simple.
Personalized feeds.
Funny enough, writing this post is helping me realize another problem.
Since Richy is likely to be tripping on metaphorical mushrooms every now and then, users need a way to tell him he completely missed the point.
Web2 social media has had a "Not Interested" button for years, and honestly, maybe that's all Richy needs.
Just enough feedback to slowly figure out when he's trying to sell flip flops to someone looking for surfboards.
I'm still pretty green when it comes to all of this.
Right now I mostly have documents full of ideas, a proposed tech stack, and what I think is a design that could actually scale.
Scaling...
That's another rabbit hole.
What happens when Richy has to read thousands of posts every single day?
Well... the little LLM that could might eventually earn the letters "n't" at the end of its title.
Irregardless, as my Florida used to say. (I'm serious. Look it up.)
I'm a little too tired to keep arguing with myself tonight.
Tomorrow I'll have more caffeine.
And hopefully, enough energy to lose the argument all over again.
MenO