Hi everyone,
As mentioned previously, there were some issue with SteemWatch that were pretty noticeable for people and they were complaining. I tried to address these issues and SteemWatch should be much more stable now. What has been improved follows.
The input fields were not being validated properly, which lead to people inserting crazy values, i.e. using a single input field to insert hundreds of usernames separated by commas. This created very long values in the DB that prevented me from creating indexes since there is a hard limit on the string length in MongoDB if you want to create an index. So first of all, I had to fix input validation and then write a script to fix the database. This is all done now.
There were some issues with go-steem/rpc considering the WebSocket transport. When trying to reconnect, it was not possible to close the transport and exit. In the end I decided to completely rewrite that package.
When I was at it, I also added support for setting multiple RPC endpoint addresses, which are being rotated on reconnect using round-robin. This makes it possible to survive one particular RPC endpoint crashing by passing multiple addresses to the package.
I actually started digging into SteemWatch because it was lagging far behind the last block produced. This made SteemWatch pretty much useless since the main point is to watch content as it is being published. So I started measuring.
It turned out that SteemWatch was so slow because it was using just a single thread to process blocks, and processing a block meant that the steemd RPC endpoint may be called even multiple times. I am not using a local steemd node, but rather the public nodes that are managed by other people. This makes every roundtrip a bit expensive.
So, the solution to the performance issue was to spawn multiple threads and process blocks concurrently. So instead of a single thread connected to a single RPC endpoint SteemWatch now uses tens of threads connected to various public RPC endpoints.
There is still room for improvements, though, because when a call to any of the RPC endpoints times out, SteemWatch restarts itself. That was the fastest solution, but the best solution would be to keep trying forever I guess.
I also spent some time updating libraries to the current version. Particularly I started using the current version of Echo and got rid of some ugly hacks considering session management. I am pretty happy for that.
SteemWatch should be ok for now. I am not receiving any warnings any more that it is lagging behind the current block.
