Steem is awesome. The Steem API documentation, however, is definitely not. I want to help change that.
A month ago @ashe-oro @crypthoe and I decided to grind on a project to explore deep into the Steem blockchain and see what interesting daily data we could dig up. The end result has been 
Today we will be taking a look at a simple but commonly asked question among Steem developers:
"How do I get find all the posts for a tag in a single day?"
This should be, "in theory", a simple calculation -- but there are particulars to how the Steem blockchain works along with severe limitations on the Steem API that make this way harder than you'd expect.
A new block on the Steem blockchain is produced every three seconds. Over the course of a single day, 28,800 new blocks carrying valuable information on posts, votes, comments, witnesses, exchanges rates and more fly past in a never-ending torrent of data.
This presents an interesting challenge while finding Steem blockchain data in the past, since you are always starting from the most recently generated block. This is a fundamental fact to keep in mind when we are designing efficient code for Steem:
The further in the past your queried data lies on the blockchain, the longer it will take to retrieve it.
Let's say you run a query to find the 100 most recently created discussions from the blockchain for any given tag. You take a break, have a mojito, and return ten minutes later to run the same query. You are highly likely to receive a slightly different array of discussions: there will be a few newer discussions at the head of the results array, along with an equal number of oldest discussions removed from the tails of the array.
This is no big deal for a casual user clicking around on Steemit viewing 10 posts at time; after all, this pagination pattern is fundamental for feeds on social media sites. For developers interested in analyzing time-range data, however, this is a nightmare! If you want to run the same test against the same time range of posts (for example, all posts for 'crypto' created on 2018-02-14), each subsequent round of this test will take longer based on the number of blocks generated between the target time range and the current time.
Therefore any functionality that makes it easier to find just the posts (etc) specific to a time range will help all of us work with Steem a little smarter.
The current Steem API offers the following endpoints related to Tags for our interaction:
getTrendingTags(afterTag, limit, callback);
getDiscussionsByTrending(query, callback);
getDiscussionsByCreated(query, callback); // <-- this is the one we want
getDiscussionsByActive(query, callback);
getDiscussionsByCashout(query, callback);
getDiscussionsByPayout(query, callback);
getDiscussionsByVotes(query, callback);
getDiscussionsByChildren(query, callback);
getDiscussionsByHot(query, callback);
getDiscussionsByFeed(query, callback);
getDiscussionsByBlog(query, callback);
getDiscussionsByComments(query, callback);
Neat stuff, but not super useful towards our particular problem. Of these available endpoints, only getDiscussionsByCreated will return us the discussions (aka posts) for a tag in an order sorted by most recently created timestamp. The other endpoints -- while very interesting in their own respects for data analytics -- do not give us the posts in a strictly time-series oriented order.
No worries, let's just make that query param look for discussions in a certain timeframe... let's say, by using a start_created or end_created param to scope the range. Oh wait, NOPE!! This is the Steem API -- nothing is that straight forward!
Here is what that "magic" query param for getDiscussionsByCreated actually supports...
{
"tag": "steemdev",
"limit": 100,
"start_author": "thescubageek",
"start_permlink": "steem-smarter-vol-1"
}
The following params are required:
tag: tag to search against. Note that, unlike most of the rest of the query params in the Steem API, this param this is passed in as a string instead of an array of strings.limit: number of results to return from the query. The max is 100; any higher and the query will be rejected by the API. Default is 10, which is useful for pagination but not for returning large data sets.The following params are optional and must work together to form a valid author-permlink pair; any discussion for the tag that matches the author/permlink will be the "starting discussion" from which to return results:
start_author: name of author for starting discussionstart_permlink: permlink for starting discussionThese are params are fine if we're building a pagination-based system like Steemit, but severely underwhelming for doing higher-level analytics on a cluster of data (aka our "Sample Set" of all posts for 'crypto' created on 2018-02-14).
...the Steem API supported start_created and end_created for doing native time-based queries!!!</rant>. Unfortunately, that's just not the way things are right now... but with some clever recursion we can solve this problem.
DISCLAIMERS: The Steem API is in Javascript, but as a Ruby on Rails developer I've done this implementation in Rails 5 using the awesome Radiator gem from @inertia for interacting with the API. Naming conventions are pretty much the same between languages (sub snake_case for camelCase when converting to JS, for example), so this should be a pretty easy port from Rails to Javascript or Python.
Here's the high-level pseudocode for what we're going to try to do:
Make sense? Let's see how it works out...
I'm going to be solving this problem in Ruby on Rails. If you want to follow along, you'll need to:
Install Ruby on Rails 5.x on your local machine
Add the Radiator gem from inertia186 to your Gemfile
gem 'radiator'
Bundle install the gem:
$ bundle install radiator
Let's create a simple helper class to handle doing these types of queries:
require 'radiator'
class SteemPostFinder
# the max results we can return for a single API query
BATCH_SIZE = 100
def initialize(tag_name)
@tag_name = tag_name
end
# find posts for a tag in a datetime range
def posts_in_time_range(start_date, end_date)
start_date = DateTime.parse(start_date)
end_date = DateTime.parse(end_date)
posts = []
####### do the magic here
posts
end
private
def api
@api ||= Radiator::Api.new
end
end
Now it's time for the magic. We're going to use get_discussions_by_created from Radiator to pull down and parse a payload of 100 posts from the Steem blockchain. First, we'll create a method inside our SteemPostFinder class to make this a little easier:
def get_posts_by_created(author="", permlink="")
params = {tag: @tag_name, limit: BATCH_SIZE}
params.merge!({
start_author: author,
start_permlink: permlink
}) unless author.blank? || permlink.blank?
# get posts from the Steem API
response = api.get_discussions_by_created(params)
return [] if response.blank?
# format response results has indifferent access array of hashes
response['result'].each(&:with_indifferent_access)
end
Note that I'm using the start_author/start_permlink params optionally. This enables us to retrieve discussions from either a specified starting post or from the head of the blockchain.
From here, we should be able to get our first set of newest posts for the tag:
# find posts for a tag in a datetime range
def posts_in_time_range(start_date, end_date)
start_date = DateTime.parse(start_date)
end_date = DateTime.parse(end_date)
posts = []
posts << get_posts_by_created
posts
end
That's only going to give us the top 100 newest posts for right now, not a specific time range. We need to rewind down the Steem blockchain using recursion to find older posts. One super important fact:
We must remember the author and permlink of the oldest post in our current set of posts in order to find the next set of posts
In Computer Science speak, the tail of the last array will be the head of the next array. Let's see how that works out:
# find posts for a tag in a datetime range
def posts_in_time_range(start_date, end_date)
start_date = DateTime.parse(start_date)
end_date = DateTime.parse(end_date)
posts = []
# this is the equivalent of a do-while loop in other languages
loop do
# find author / permlink info if last_post exists
last_post = posts.last
last_author = last_post.try(:author)
last_permlink = last_post.try(:permlink)
# concats the posts array results to the existing posts array
posts << get_posts_by_created(last_author, last_permlink)
# makes the array 1-dimensional and removes duplicates and empty elements
posts = posts.flatten.compact.uniq
# exit the do-while loop under the following circumstances:
# 1. no posts are found in the response, or
# 2. the posts returned the same list of posts as the previous response
# 3. the last post was created before the start date
should_break = posts.blank? || posts.last == last_post || DateTime.parse(posts.last.try(:created)) < start_date
break if should_break
end
end
Do-While loops are typically a no-go in my coding practice, but this is one of the rare cases where its use makes sense. The loop will continue to rewind backwards over the blockchain from a starting point until it runs out of data to consume or goes past the start date for the search.
This code gets us all of the posts from right now to the start date. Now we need to cull out the unwanted posts that fall after the end date. We will add this to the end of our posts_in_time_range method
posts.reject! do |post|
created = DateTime.parse(post.created)
created < start_date || created > end_date
end
And here we go! We should have everything we need to find all the posts for a tag in a time range. Let's take a look at the solution...
require 'radiator'
class SteemPostFinder
# the max results we can return for a single API query
BATCH_SIZE = 100
def initialize(tag_name)
@tag_name = tag_name
end
# find posts for a tag in a datetime range
def posts_in_time_range(start_date, end_date)
start_date = DateTime.parse(start_date)
end_date = DateTime.parse(end_date)
posts = []
# this is the equivalent of a do-while loop in other languages
loop do
# find author / permlink info if last_post exists
last_post = posts.last
last_author = last_post.try(:author)
last_permlink = last_post.try(:permlink)
# concats the posts array results to the existing posts array
Rails.logger.info("#{posts.count} posts found so far...")
posts << get_posts_by_created(last_author, last_permlink)
# makes the array 1-dimensional and removes duplicates and empty elements
posts = posts.flatten.compact.uniq
# exit the do-while loop under the following circumstances:
# 1. no posts are found in the response, or
# 2. the posts returned the same list of posts as the previous response
# 3. the last post was created before the start date
should_break = posts.blank? || posts.last == last_post || DateTime.parse(posts.last.try(:created)) < start_date
break if should_break
end
# reject all posts out of time range
Rails.logger.info("#{posts.count} posts found, removing posts out of range")
posts.reject! do |post|
created = DateTime.parse(post.created)
created < start_date || created > end_date
end
Rails.logger.info("Final results: #{posts.count} posts found")
posts
end
# gets posts from Steem API
def get_posts_by_created(author="", permlink="")
Rails.logger.info("Getting #{BATCH_SIZE} posts starting at #{author} - #{permlink}...")
params = {tag: @tag_name, limit: BATCH_SIZE}
params.merge!({
start_author: author,
start_permlink: permlink
}) unless author.blank? || permlink.blank?
# get posts from the Steem API
response = api.get_discussions_by_created(params)
return [] if response.blank?
# format response results has indifferent access array of hashes
response['result'].each(&:with_indifferent_access)
end
private
def api
@api ||= Radiator::Api.new
end
end
lib/steem_post_finder.rb
Download the gist from Github
To search a time range for a given take, ie 'crypto' from February 14-15 2018, you would simply use:
SteemPostFinder.new('crypto').posts_in_time_range('2018-02-14','2018-02-15')
Once posts_in_time_range finishes crunching the data from the Steem API, you'll have an array of hashes for posts in that time range. This can easily be converted to JSON as well.
Thanks to @morning for the Javascript solution using recursion, @klye for an overview of the API calls, and @inertia for creating the Radiator gem for Ruby
So you've pulled down a big array of post data from the Steem API. Now what? How do you use the data inside the posts API payload to find out anything meaningful about that post, including its payouts, author, commenters, voters, and more?
In our next post, we will take a look at Exploring the API Post Payload and how you can extract all kinds of interesting data from the Steem blockchain.
Don't forget to upvote and resteem this post if you enjoyed it, and follow both @steemsmarter and @thescubageek to keep up with our latest developments!
Until next time, keep on Steemin' Steemians!