When people ask me what their project should focus on, or how they could improve their project, I often point out how hard it is for an outsider to engage with their project (front end/app/game/whatever). ## Motivation for wanting to process non-hive accounts You may just not want to support them, as a dev who implemented them, it's a real pain, you have to engineer a lot of things (auth, signup, profile settings, making sure your whole app is compatible with multiple types of accounts etc) when it's usually just handled by the chain. Here's why we did it: Steempress is a wordpress plugin. Nowadays most of our efforts have been directed towards the comment section. One of the best ways to get hive to be known is not by catering to people that are already using hive. It's to try to get new people from the outside. That's really fit for a comment section powered by hive. We can easily "hiveify" any website, they install the steempress plugin, and in an instant they get a hive powered comment section with everything that comes with it (decentralized, uncensorable, powered by rewards etc), and they become a new place where people can find out about hive. you can view an example on this blog: https://www.blogrevenues.com/photography/how-i-look-at-art/ ![image.png](https://files.peakd.com/file/peakd-hive/howo/ILJJTHGd-image.png) If you are a blog owner you don't care about your excuses like "oh sorry people can't post on your blog unless they have a hive account", they want their comment section to be as easy to sign up and comment on as any other comment section (usually disqus). So assuming you don't support non-hive accounts, if someone wants a hive account his options are: - buy an account ? No way, no matter how cheap, that's just not how people are used to use websites, especially not if it's to comment/like/whatever - sign up for an account and wait for it to be approved: most people just won't bother to come back even in a day to put the comment they wanted to put - Sign up for an account instantly but you have to give a ton of info (something with a phone number etc) that won't work either, few people will actually care enough to put that much info, especially if it takes more time to put all the info than to write a comment. No, what people expect is give username + email + password -> confirm the email -> publish comment.(Ideally you wouldn't even confirm the email) or even better click "login with google/facebook/twitter) and instantly have an account to publish their account with. And for that you have to handle non-hive accounts. # Architecture: ## Sign up flow: ![image.png](https://files.peakd.com/file/peakd-hive/howo/RBhbCqLm-image.png) Our sign up flow is like this: - You enter display name (username), email, password. - You hit sign up, we tell you to check your emails to confirm your account - You get an email with a confirmation link, you click on it and it redirects you to the blog. (improvement path here: you should be logged in when you are in the blog) Or you can just click on sign up with google, and you'll be logged in. The whole thing takes moments and it's very familiar to what people are used to, which is what we need to aim for: web 3.0 with the ease of 2.0 ## Actions flow: ### Votes: Those don't get carried over to the chain because there's no point, we simply have a db table that stores votes so we can render them on our comment section. ### Comments: We carry them over using a proxy account: #### on hive.blog ![image.png](https://files.peakd.com/file/peakd-hive/howo/qTBhf8bn-image.png) (as a side note looks like a forgot to replace a "steem" for "hive" in the code) #### on the blog ![image.png](https://files.peakd.com/file/peakd-hive/howo/mWKPqDNP-image.png) One limitation with proxying is the fact that an account can only comment once every 3 seconds, but it's quite easy to have a fleet of proxy accounts and make sure there is always one that's available for commenting. The whole idea is that from the user perspective it's all transparent. ## What to do with the rewards ? We earmark all the rewards and whenever a user earns more than x hive, we let them "claim" a hive account, and send them all the rewards they earned so far. This is a great way to incentivize people to join hive and have an actual account. # let's get to the actual programming: ## Database: We have a local db used for storing non-hive user data with those tables Use this as a reference sheet when reading the implementation logic. ### authors: Authors contains all the data related to the non-hive user: most notable fields are ``` regular_author_uuid => a unique uuid to identify the user display_name email password profile_image => direct link to the image about => about/website/location match the usual stuff you see on hive profiles website location verify_id => the code that will be sent to the user to check his email blog_origin => url of the page of the blog he signed up from (useful for redirecting) type => how he signed up (email / google / twitter etc), it's important to know how to authenticate an user ``` (it's not called users for legacy reasons in our infra) ### comments: ``` id <= auto increment id author_uuid <= foreign key to author permlink <= actual hive permlink body <= the text of the comment parent_author <= author he's replying to parent_permlink <= permlink he's replying to root_author <= the author post of he's putting his comment on root_permlink <= the permlink of the post he's putting his comment on date <= date payout <= current payout likes <= votes is_paid_out <= boolean, useful for calculating rewards for the user ``` root_author/permlink is useful for rendering the comment section, I can just query all comments by root author/permlink to fetch all the non-hive comments related to that post. ### votes: ``` author <= post/comment to vote on permlink <= post/comment to vote on voter_uuid <= foreign key to author percent <= 1000 or -1000, it's set the same way as hive for easier rendering root_author <= same as comments root_permlink <= same as comments date <= date ``` ### sessions: used for managing returning users (so they don't have to enter their key for every single action basically) and providing a token that can be stored in the cookies where they can be stolen and it's relatively okay. both user_uuid and type are set as primary key because a single user may be signed in using different methods on different devices ``` user_uuid (pk) <= uuid of the user type (pk) <= type of the user username <= hive username, display name for regular users token <= auth token that we issue private_key <= encrypted posting key of the user or hivesigner token, empty for regular users expire <= date to define when the session expire in order to auto purge keys when they expire ``` # Implementation logic: FYI, I am only going over the happy path, and unless it's an important design decision, I am skipping over all of the sanity/security checks, those are usually obvious (always double check what the front end tells you, encrypt/hash stuff etc) and not worth going over. ## sign up ### sign up using email ![](https://i.imgur.com/mDmwNCO.png) Check if a user already exists with this email, take into account that an user might want to sign up with different methods but with the same email. For instance I could register with `howo@gmail.com` using my email, then do a google log in with the same google account, you should check for this in your flow, and set the email field as unique in the db. As for the flow: - We generate a regular_author_uuid using a uuid v4 (https://en.wikipedia.org/wiki/Universally_unique_identifier) - Then generate a unique verification id - We store the email, hash the password - Set the user type as `regular` (so we know how that user was created and how he should be authenticated) - Set the blog_origin as the exact page the user registered on, it's useful for redirecting and for statistics. - We leave the optional data (profile_image, about, location, website) empty Then we send an email with the verification id to the user, which is your basic "confirm your email" email. When the user clicks on that link, we set the verification id field as null in the db, marking that the user has verified his email. We could have went with a boolean or a status thing to mark if the user had verified his account or not. But we didn't see the use for now, so we saved the extra column by just setting the field as null. After all we have no use for it anymore. We then redirect the user to the page on which he signed up using the blog_origin field. ### Sign up using google Since you don't "sign up" per se, we just sends data straight to our backed, signing in and login in with google is the same endpoint. So I'll save the sign up process using google for later ## log in On top of what I usually return (described at the end of each authentication method), I return the user data in user json metadata: for me that would be ``` { "name": "", "about": "Software engineer by day, Blockchain developer by night. SteemPress co-founder. @steempress witness", "website": "https://brokencode.io", "location": "", "profile_image": "https://images.hive.blog/u/howo/avatar", "type": "hive" } ``` it's useful to do it that way because then it's less of pain for the front end if both regular authors and hive authors use the same format. by the way, on hive there are two json metadata for a given user, one that is edited with the active key (json metadata) and one that is edited with the posting key, nowadays most apps use the posting one, but some people still use the active one (which is bad practice, please migrate if this is you). So in order to support both here's a neat function: ``` function parseUserJsonMetadata(user) { try { return JSON.parse(user.posting_json_metadata).profile } catch (e) {} try { return JSON.parse(user.json_metadata).profile } catch (e) {} return undefined } ``` ### hive log in check if username + type `hive` exists in the db, if it's not there it means it's a first login (or a relogin of an user that expired) first login: We just generate ids, and store everything in the sessions table - Get the posting key and username - Generate a user uuid - generate a login token - Set the type as "hive" - store the hive username - Encrypt the posting key and store it - Set an expire date as now + 2 weeks Send back the login token + user_uuid, it'll then be stored by the front end as cookies and used later on for auto auth returning login: fetch the user uuid and token, set the expire date as now + 2 weeks. Send back the login token + user_uuid It's important to keep the same token instead of generating a new one because otherwise if your user is loggin in from two different locations, he'll keep sign himself out by loggin in on another device: - User logs on device A, gets token "AAA" - token "AAA" is stored in cookies and user can interact with everything on his browser and the "AAA" token works - User logs on device B, the token changes to "BBB" - user tries to do things on device A, device A has the "AAA" token stored which gets refused, he has to login again which will create a new token "CCC" which invalidates the token on device B This is an easy mistake to make and will make up for a bad experience for your users. ### hivesigner login pretty much the same, except we encrypt the hivesigner token instead of the private key side note on hivesigner tokens: even though they are not the posting key. They should be treated as such, often they have almost the same priviledges, so if someone were to gain control of the hivesigner, he could benefit from the same powers as a posting key until it expires or gets revoked. ### log in using email - User sends us email + password - We check if it matches our records of an user of type `regular` and get the corresponding user_uuid - we generate a login token - set the type as "regular" - set an expire date as now + 2 weeks - leave username, private_key as null Send back the login token + user_uuid ### log in using google Most of the logic related to google itself is done on the front end, basically google opens a popup, user authenticates there, then it sends you back info on the user (email, name, profile picture etc) and a token to authenticate the user which we forward to the backend. First we check the email to check if it exists in authors and if it has the `google` type (if not we tell them to login using said type) if it didn't exist, it means we need to create a new author: #### New google user: - We generate a regular_author_uuid using a uuid v4 (https://en.wikipedia.org/wiki/Universally_unique_identifier) - We store the email - Set the user type as `google` - Set the blog_origin as the exact page the user logged in on for statistics. - We leave the rest as null (profile_image, about, location, website, password, display_name) empty We leave the rest empty because the source of truth for those is google not us. You want to end up in a state where if the user changes his profile picture, his profile picture should change for your app as well. then we proceed with the login flow, since we know it's a new user we create a new session: - generate a login token - set the type as "google" - set an expire date as now + 2 weeks - leave username, private_key as null Send back the login token + user_uuid (which we just generated) #### Returning google user: Then we just refresh the expiration time and return the login token + user_uuid. ## returning login Via the sessions table, we can just authenticate a user using user_uuid + token and do actions based on it's type. ## expiration a simple cron job runs every 5 minutes to expire sessions if `expire` is in the past ## displaying regular comments We haven't implemented the fleet account logic yet because we don't process enough comments for it to matter (worst case if two people try to post at the same time, they get an error message to retry and then the comment goes through). The logic is quite simple, when I get the request to get all the comments for a specific post, I fetch the comments (I'm skipping pagination logic, but it works the same way), then fetch all of the regular comments for that post using the `root_author` and `root_permlink` fields in the `comments` table then for every comment made by `steempress-io` (the proxy account making the regular comments on the chain), I find the corresponding db entry in `comments` using the permlink, then I replace the content, author etc and then the front end can do the work to display them correctly. Keep in mind that it's important to mark them as regular comments and not hive comments so you can still reply/comment on it, so when you do actions on it you replace the author name with `steempress-io` because the chain is not aware of that. # more on the fleet accounts First of all, this is a great problem to have, it means your app has a ton of engagement, hats off to you 👍 So if your front end ends up needing a fleet of accounts, here is how you should do it: don't create a bunch of accounts manually and hope those will be enough to handle the load, do a specific program to do this: have a fleet of accounts, record when they are in cooldown and when they are not, and have a runner check every x seconds the ratio of available accounts for commenting vs the amount of accounts that are in cooldown. If the ratio is too low for some time, (let's say less than 20% for 5 minutes) then it means you need to create more accounts and add them to the fleet. Keep a "mothership" account with a bunch of hive power and claimed accounts, to be able to create new accounts and delegate enough hive power to them so they can comment at least once every 3 seconds. That way the fleet can auto expand and you won't end up in a situation where your app is just stuck just because you are asleep and your app blew up overnight. # Conclusion That's it ! I didn't describe in details more than the authentication and sign up because I think that part is the most important and also the implementation starts to be use case specific so I just put a high level description on how we solved our specific problems, and you can probably guess using the database schema. Also I don't claim I have the best solution, this was bundled together during multiple dev iterations. This is really a "this is how we do it, use this to think of a better solution" that I wished I had when I started to architecture it. As there are a lot of things you can overlook when designing something like this. If you are making a new app, please factor the fact that you **will** most likely need to support non-hive accounts in the future, so keep it in mind when designing everything. I didn't and had to redesign a big chunk of the code for this feature. If multiple apps use some kind of regular author thing, I think it is becoming more and more important to have it in a common database (which is decentralized) like hivemind. This is another subject entierely and I don't have time to talk about it, but at some point I think it'll be important to store the user info in the json_metadata of the comments like display_name, avatar, app, etc so the various front ends can display them (and perhaps with a little logo to know on which front end the regular commenter came from) Sorry if this is quite long and for repeating myself a bit, sometimes it makes sense to explain something twice. If you liked this writeup, please consider voting on our witness: ![](https://i.imgur.com/oPJ63jA.png) ^{You can vote for our witness directly using Hivesigner here.} By the way, I know steempress may seem like an abandoned project between the lack of updates and me (the main dev) working on the core development team. But it really isn't, @fredrikaa and I are hard at work almost every single day. We are in the process of making an enormous update on top of the rebranding (don't want to have steem in the name for obvious reasons) and we want it to be perfect which is why we are taking so long to release it.