Support Proposal 303 on PeakD
Vote for Brianoflondon's Witness KeyChain or HiveSigner
This is a value for value post: see the explanation in the footer.
On Friday 11th October my systems alerted me to something strange going on with v4v.app. A bunch of outgoing payments didn't seem to match up with the proper notifications from my back end.
I had a bad feeling so I jumped out of bed and very quickly shut down the API server and the Lightning node and started looking at log files.
Toward the end of this post I'm going to tell you later the broad type of attack which caused me to loose Bitcoin but not the very specific details.
Fortunately I keep quite good logs and I could see I had a problem quite quickly. I don't want to explain exactly how this happened in public but this was a genuine security failure on my part.
The problem was I did not think like a hacker when I built the KeepSats functionality which I introduced a bit too quickly earlier this year.
Friday morning was a particularly bad time for this to happen. This year the Jewish holy day of Yom Kippur started on that Friday evening and I knew I did not want to rush fixing the site. I took the difficult decision to turn off most aspects of v4v.app. I would not be able to work on Yom Kippur.
I set to work on Sunday morning fixing things. I worked on this all day Sunday and put in place some changes which I thought might take care of the problem. Unfortunately this first set of fixes wasn't quite good enough. It slowed the rate at which the site could be exploited but the hacker came back and was able to extract a few more sats before I stopped him (satisfyingly I did stop him just after he deposited 2m sats on the site and before those could leave).
The next few days I spent re-writing a significant part of the API system which is public and which my v4v.app website uses behind the scenes. That was where the vulnerability lay.
The broad outline of the problem was that Lighting invoices take a few seconds to pay and in special cases can be set to take many minutes or even 24 hours to pay. Once my Lightning Node has started to pay one of these invoices, there's no way to interrupt the process.
The mistake in my code was that I wasn't properly deducting an outgoing payment from a user's balance whilst this payment process was in progress. The attacker first deposited some sats on my system with the KeepSats function. The attacker then used custom code or curl commands to send multiple, simultaneous Lightning invoice payment requests. My system jumped into action and started using the same balance to pay multiple outgoing invoices. He would send in 1M sats and then get 3M out. This happened over the course of a few minutes until I shut everything down.
In the end I lost somewhere around 0.13 BTC or 13m sats. I'm not even going to do the maths on the dollar value, and this was a few weeks before the current massive highs we're reaching now in the BTC price.
I hope it is fully fixed now. The answer lay in imposing very strict filtering on the API system which pays invoices around timing and simultaneous requests. This has no impact whatsoever on anyone using the v4v.app front end, but it should block any abuse behind the scenes using custom code to copy what my front end site does.
@v4vappIt means that the
On the other side of the ledger is the DHF so I have received enough to swallow this loss. Mentally I'm looking at this as a security audit fee. I've done nearly all the dev work on this project myself paying out almost nothing to others. This was an unwelcome expense, but that's how I have to view it.
For the last few months while building
If you appreciate the work I do on and around Hive, you can express this directly: upvoting posts on Hive is great. Also consider a direct donation (there's a Tip button on Hive or a Lightning Address) on all my posts.
Support Proposal 303 on PeakD
Support Proposal 303 with Hivesigner
Support Proposal 303 on Ecency
Vote for Brianoflondon's Witness KeyChain or HiveSigner
