Project Information

This bug is related to the Github repository:
https://github.com/steemit/steem-python

I'm currently using the Steem Python Package version number 1.0.1, but I have reproduced these errors on 1.0.0 as well. I'm using Python version 3.5.2

Expected behavior

I expect that when instantiating an instance of the Steem class using Steem-Python that if an address to a steemd websocket server (RPC node) generates a server error (such as 503 Bad Gateway, or 404 Not Found, etc.) that:

A more informative error is produced by the Steem class. Rather than simply giving the error "TypeError: 'NoneType' object is not subscriptable" an exception that informs of the server error would be more helpful.
If an address to a websocket server produces a server error and is present in the list of nodes given to Steem during instantiation, steem should ignore the bad address and continue attempting connections using the next address in the list.

According to the documentation at http://steem.readthedocs.io/en/latest/steem.html

If you would like to override the official Steemit nodes (default), you can pass your own. When currently used node goes offline, Steemd will automatically fail-over to the next available node. ~Italics mine

nodes = [
    'https://steemd.yournode1.com',
    'https://steemd.yournode2.com',
]

s = Steemd(nodes)

Actual Behavior

When instantiating an instance of Steem using an address to a websocket server that is currently generating server errors Steem produces unpredictable behavior and exceptions that cannot be handled in a meaningful manner.

I can reproduce these errors on all three of my Ubuntu 16.04 servers:

When running the following script:

#!/usr/bin/python3

from steem import Steem

nodelist = ['https://steemd.minnowsupportproject.org', 
    'https://steemd.privex.io', 
    'https://gtg.steem.house:8090', 
    'rpc.steemliberator.com', 
    'steemd.pevo.science', 
    'steemd.steemgigs.org']

s = Steem(nodes=nodelist)

acct = s.get_account("ned")

print (acct['sbd_balance'])

The following error is encountered:

  File "test_connection.py", line 7, in <module>
    s = Steem(nodes)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/steem.py", line 60, in __init__
    steemd_instance=self.steemd, no_broadcast=no_broadcast, **kwargs)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/commit.py", line 96, in __init__
    self.wallet = Wallet(self.steemd, **kwargs)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/wallet.py", line 61, in __init__
    self.prefix = self.steemd.chain_params["prefix"]
  File "/home/ger/.local/lib/python3.5/site-packages/steem/steemd.py", line 70, in chain_params
    chain = props["current_supply"].split(" ")[1]
TypeError: 'NoneType' object is not subscriptable

Depending on the type of server error the exception may be a never ending loop of exceptions. Below is a small snippet of a never ending loop.

Traceback (most recent call last):
  File "/home/ger/.local/lib/python3.5/site-packages/steembase/http_client.py", line 202, in call
    response = self.request(body=body)
  File "/usr/lib/python3/dist-packages/urllib3/poolmanager.py", line 162, in urlopen
    response = conn.urlopen(method, u.request_uri, **kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 630, in urlopen
    release_conn=release_conn, **response_kw)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 610, in urlopen
    _stacktrace=sys.exc_info()[2])
  File "/usr/lib/python3/dist-packages/urllib3/util/retry.py", line 273, in increment
    raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='rpc.steemliberator.com', port=80): Max retries exceeded with url: / (Caused by NewConnectionError(': Failed to establish a new connection: [Errno 111] Connection refused',))

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 137, in _new_conn
    (self.host, self.port), self.timeout, **extra_kw)
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 91, in create_connection
    raise err
  File "/usr/lib/python3/dist-packages/urllib3/util/connection.py", line 81, in create_connection
    sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 560, in urlopen
    body=body, headers=headers)
  File "/usr/lib/python3/dist-packages/urllib3/connectionpool.py", line 354, in _make_request
    conn.request(method, url, **httplib_request_kw)
  File "/usr/lib/python3.5/http/client.py", line 1106, in request
    self._send_request(method, url, body, headers)
  File "/usr/lib/python3.5/http/client.py", line 1151, in _send_request
    self.endheaders(body)
  File "/usr/lib/python3.5/http/client.py", line 1102, in endheaders
    self._send_output(message_body)
  File "/usr/lib/python3.5/http/client.py", line 934, in _send_output
    self.send(msg)
  File "/usr/lib/python3.5/http/client.py", line 877, in send
    self.connect()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 162, in connect
    conn = self._new_conn()
  File "/usr/lib/python3/dist-packages/urllib3/connection.py", line 146, in _new_conn
    self, "Failed to establish a new connection: %s" % e)
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0xb6f4046c>: Failed to establish a new connection: [Errno 111] Connection refused

Coincidentally, if I intentionally enter a spelling mistake into a node address such as "https://steemd.p_rivex.io" it creates the exact same error as above.

This is unexpected behavior.

Even if "try" and "except" statements are used to attempt to catch this error, instead of showing a never ending loop the system simply hangs forever.

The issue can be reproduced by running the script that connects to a steemd websocket server that generates an error. For example, today if I visit "steemd.minnowsupportproject.org" in a browser I get a 502 bad Gateway Error. Running this script with just "steemd.minnowsupportproject.org" as a node:

#!/usr/bin/python3

from steem import Steem

nodelist = ['https://steemd.minnowsupportproject.org']

s = Steem(nodes=nodelist)

acct = s.get_account("ned")

print (acct['sbd_balance'])

produces this error:

Traceback (most recent call last):
  File "test_connection.py", line 7, in <module>
    s = Steem(nodelist)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/steem.py", line 60, in __init__
    steemd_instance=self.steemd, no_broadcast=no_broadcast, **kwargs)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/commit.py", line 96, in __init__
    self.wallet = Wallet(self.steemd, **kwargs)
  File "/home/ger/.local/lib/python3.5/site-packages/steem/wallet.py", line 61, in __init__
    self.prefix = self.steemd.chain_params["prefix"]
  File "/home/ger/.local/lib/python3.5/site-packages/steem/steemd.py", line 70, in chain_params
    chain = props["current_supply"].split(" ")[1]
TypeError: 'NoneType' object is not subscriptable

However if I run the same script using a known good websocket server such as "https://steemd.pevo.science" then the script runs as expected.

Visiting this node in a browser displays:

11 eof_exception: End Of File
stringstream
    {}
    th_a  sstream.cpp:109 peek

    {"str":""}
    th_a  json.cpp:479 from_string

And the output of the script is as expected:

5753.498 SBD

To make matters worse, it seems that if a node server that generates an error is in the node list handed to an instantiation of steem, then an error is generated, regardless if there are working nodes in the list.

For example, if I run the script:

#!/usr/bin/python3

from steem import Steem

nodelist = ['https://steemd.minnowsupportproject.org', 'https://steemd.pevo.science']

s = Steem(nodes=nodelist)

acct = s.get_account("ned")

print (acct['sbd_balance'])

It generates the same errors as described above, seemingly getting stuck on the "bad" server and never attempting the next server in the node list.

Temporary Work-around

To fix this issue temporarily I am using urllib to test each server node address before using it to instantiate the Steem class. The following code works even thought the first node gives a 503 Gateway error:

#!/usr/bin/python3

import urllib.request
from urllib.error import HTTPError
from steem import Steem

nodelist = ['https://steemd.minnowsupportproject.org','https://steemd.pevo.science']

def goodnode(nodelist):
    for n in nodelist:
        req = urllib.request.Request(url=n)
        try:
            handler = urllib.request.urlopen(req)
        except HTTPError as e:
            pass
        else:
            return n

s = Steem(nodes=[goodnode(nodelist)])

acct = s.get_account("ned")

print (acct['sbd_balance'])

No recording of this bug or behavior

I made no recording since the previous description and code boxes should be enough information for a developer to reproduce this behavior.

Possible related issues already on GitHub

I did see two issues on GitHub that could possibly be related, but because the issues reported could not be reproduced and lacked sufficient detail it's uncertain.

https://github.com/steemit/steem-python/issues/224

https://github.com/steemit/steem-python/issues/236