Disclaimer : This is pre-alpha, in-development software so expect it to crash and/or be unavailable from time to time while I continue to work on it. Thanks for understanding :)
I'm excited to announce the very first rough draft of my Steem Search engine - Steem Shovel. I was frustrated with not being able to find older posts on Steemit so I set out to build my own search engine. I have some experience with ElasticSearch and it seemed like a great fit for indexing Steem posts and comments.
SteemShovel runs continuously indexing new content from new blocks. It will index both comments and posts and make them full-text searchable. Moving forward I plan to allow you to limit your search to specific authors and also restrict searches to certain tags. Right now it's just a full on mega search against all of the fields/text.
The underlying application code is written in Node.JS while the search engine is powered by ElasticSearch. I use the SteemJS module (https://github.com/adcpm/steem) to talk to the Steem process.
The application is load balanced using AWS EC2 servers behind an Elastic LoadBalancer. This gives me the ability to add more servers / horsepower at will if anyone decides to use this app.
The indexing process is currently making its way through all of the Steem blocks. A second process is indexing new posts as they appear but there is a big gap between the two. Because of this it might be a while before you are able to find every single post you're looking for. Hang tight, they're coming!
Well, I'm glad you asked! I have a pretty ambitious set of goals for this project so we'll see how it goes. The main focus now is to work on the indexing and build some more filters that you can use to search. Beyond that I want to improve the look of the site and make sure it's working well on mobile.
Once the basics are covered I have tons of features I want to work on - log in, voting, UI improvements to show threads of comments like Steemit does.
I have also built this to be an API that anyone can talk to. All you have to do is direct an http client to https://steemshovel.com/search/?search={your search term} and you'll get some JSON to do with as you please.
If you have any questions or suggestions as to what you'd like to see this thing do, please comment and let me know.
