All credits to: Nuriel's Blog - Original post: https://goo.gl/wpiUtk
TL;DR
Enable firewall, change SSH default port, disable root SSH access, use SSH key authentication and disable password authentication.
In addition, disable any unnecessary commands on the API port (e.g. add/remove/getNeighbors and attachToTangle if you don’t want people using your node to perform PoW).
Head on to the bottom of the blog: I offer some links to security hardening instructions.
Intro
The IOTA network is growing very fast. Every day new nodes are being deployed. It is hard to estimate how many nodes are currently on the network, but there must be thousands at least.
For example, the last count I’ve heard of, Nelson nodes alone had more than 2000+ nodes on the network (Nelson is a software created by Roman Semko that runs on the full node and manages automatic neighbor peering). I would imagine at least 8000 non-Nelson nodes running along side those, if not more.
There are several installation guides out there meant to help the new node owner getting started. Rarely do any of those guides focus on the security aspects.
Most node owners are not tech-savvy, and simply find the easiest and fastest way to get a node onto the network, for any number of reasons: to connect their light wallet to their own node, offer others a node to which they can connect their light wallets to, help the network’s confirmations rate, etc.
Security plays a very important role. Unfortunately, this is often completely overlooked.
The Public Node
A fullnode owner can choose to make her/his node public. That means exposing the API port so that clients (e.g. wallets) can connect to perform transactions, check balances etc. It is not a requirement, as one can simply let the node be part of the network without exposing it as a public node.
The benefit of having one’s node publicly accessible is for the greater good, as it provides more options where wallet owners can connect to (taking off the load from “central” public nodes). In addition, one can choose to list their node in one of the public nodes’ listings so that they can easily be found by wallet clients.
Exposing the API port exposes the node to new types of threats. Some can easily be mitigated, and others are still unknown (just as any other system, this is a never ending task — to improve security as time progresses).
Not a Public Node
Have you chosen not to have your node publicly listed? Great! That’s fine.
However, these are no longer the early days of the IOTA’s network, when most IPs and host-names have been hidden. It is very simple to automate and collect nodes’ IPs and host-name should anyone wish to do so. You are never guaranteed that your node’s IP or host-name will not be leaked via your neighbors.
If you are using Nelson, it is rather futile to try hide your IP or host-name. Your IP will be present on multiple nodes within a short time. (For the record: I find it perfectly reasonable that node’s IP/hostname is known. If the node operator has taken some basic steps to secure her/his node, this shouldn’t be a problem).
Once you purchase and deploy your VPS, it is already exposed on the internet with a public IP. An attacker can scan your node’s ports and try to access services (SSH, webserver etc). This doesn’t necessarily have to be an attacker which is directly focused on attacking the IOTA network, but script kiddies who are trying to brute-force their way onto servers.
Just leave your SSH port on the default 22/tcp and you will notice brute-force attempts when viewing the server’s logs. These are just automated scripts trying to find weak nodes with password authentication where they can try to brute force weak passwords.
Recent Attacks on the Public Nodes
Recently, the IOTA network has been under a DDoS attack. It has been suggested that these attacks came in parallel with the stolen funds (due to the online seed generators scam). I won’t go into the exact details. If you want, you can read Ralf Rottmann’s detailed blog about those events.
I want to talk about the attack, and what happened which stalled almost all of the public nodes. The first reports of nodes being under attack appeared on IOTA’s Discord #fullnodes channel (where I spend a lot of time helping node owners). Very quickly did we realize that the nodes being attacked are public nodes and are listed on one of the fullnode listings.
I have been spending hours trying to figure what the node owners where experiencing. Unfortunately, my nodes are not public, and were thus left unaffected by the attacks. Having my nodes publicly listed would have helped me to inspect those attacks first hand.
I eventually managed to pin-point how the attacker(s) were getting nodes out-of-service: with much gratitude to the node owners who gave me access their nodes, I was able to inspect several nodes during the attack.
Discovering the Problem
Most nodes are not fitted with a reverse proxy (nginx or HAProxy) where you can decide to log the HTTP request body. That would help very quickly identifying which API calls are being made to the nodes which cause them to stall.
Luckily, our good ol’ friend tcpdump came to the rescue.
Having it dump the traffic to the API port, I was able to see the HTTP calls made to a node. I then proceeded to run those commands on my own server, trying to re-produce the error.
Most calls were not interesting: just asking for getNodeInfo and other commands which would receive a reply in no-time. Finally, I caught the problematic command:
1
'{"command": "getTransactionsToApprove", "depth": 3, "reference": "GFCCGTGSUEVBZYFK9QNAHOVHXUMEATUFLGBZLDKMOGWCXVUZDVEJVCLLCJJ9NQOIMOLMUMFMGZSI99999"}'
This command stalled the node’s API port, it just kept processing. Having done “CTRL-C” didn’t seem to change anything, as the IOTA (IRI) software was still busy processing it in the background. New connections to the API port have gone into a queue and stalled as well. There was thus no need to hammer the node with a high-rate of calls, but simply run the above command, and any new connections would just get stalled.
What happens is, that those commands keep adding up, keeping the ports in CLOSE_WAIT state. After a while, you would find the node with 4000+ orphan connections. A Linux system (user’s namespace) has by default 4096 open files limit. Once it has reached this limit, it cannot do much more. Network connections add up to this. When IRI reached 4000+ open files, it stalled. Take into account those connections made to the database, for example. This attack would render your node unusable.
I contacted one of the core developers and reported to him about my finding (after having repeatedly tested this is in fact the problematic API call). The reason this command was problematic is because it was referencing a very old transaction. The developer has replied to me after a short while:
The api is stalled when calculating the weights for the random walk – this was overlooked from a DoS perspective.
This problem has been fixed within a day after having reported it.
Aftermath
The developers acted swiftly and came up with a fix quickly.
These events have prompted the community to step up the overall security, and become more aware of potential threats.
One example, is the work done to add HAProxy with security policies to run as a buffer between the API port and the clients. This is a very powerful basis on which new policies can be written and tweaked further. For example, invalid headers can be dropped, PoW (attachToTangle) can be globally rate-limited, specific regex’s rejects and so on.
Security Considerations
Once your node is out there, it is out in the wild. And you should probably consider taking some steps to make it more secure.
The iri-playbook includes some default security precautions related to the fullnode’s software and overall Linux OS. For example, running all the processes as unprivileged users, enabling restrictive firewalls and offering to run the API behind a load-balancer(HAProxy) with security policies enabled (e.g. rate-limiting).
In addition, the playbook includes an extensive chapter how to secure the system (SSH key authentication, disabling root, changing SSH default port etc.)
I encourage you to visit those links, get inspired, and take a pragmatic approach to securing your node.
Thanks for reading!
Donations
CSSFHHDBUQDGAUGYUHTENLBJ9JMTUFFLYLJZKTLRZVLLDCZZOQHOUXJOVDKXOLXGCJEMXJOULDIKADBHWMGVALMAUW
