Support Proposal 303 on PeakD
Vote for Brianoflondon's Witness KeyChain or HiveSigner
This is a value for value post: see the explanation in the footer.
V4V.app was hacked
On Friday 11th October my systems alerted me to something strange going on with v4v.app. A bunch of outgoing payments didn't seem to match up with the proper notifications from my back end.
I had a bad feeling so I jumped out of bed and very quickly shut down the API server and the Lightning node and started looking at log files.
Toward the end of this post I'm going to tell you later the broad type of attack which caused me to loose Bitcoin but not the very specific details.
Logs
Fortunately I keep quite good logs and I could see I had a problem quite quickly. I don't want to explain exactly how this happened in public but this was a genuine security failure on my part.
The problem was I did not think like a hacker when I built the KeepSats functionality which I introduced a bit too quickly earlier this year.
Taking the site down
Friday morning was a particularly bad time for this to happen. This year the Jewish holy day of Yom Kippur started on that Friday evening and I knew I did not want to rush fixing the site. I took the difficult decision to turn off most aspects of v4v.app. I would not be able to work on Yom Kippur.
Fixing the site
I set to work on Sunday morning fixing things. I worked on this all day Sunday and put in place some changes which I thought might take care of the problem. Unfortunately this first set of fixes wasn't quite good enough. It slowed the rate at which the site could be exploited but the hacker came back and was able to extract a few more sats before I stopped him (satisfyingly I did stop him just after he deposited 2m sats on the site and before those could leave).
The next few days I spent re-writing a significant part of the API system which is public and which my v4v.app website uses behind the scenes. That was where the vulnerability lay.
The Problem
The broad outline of the problem was that Lighting invoices take a few seconds to pay and in special cases can be set to take many minutes or even 24 hours to pay. Once my Lightning Node has started to pay one of these invoices, there's no way to interrupt the process.
Double Spend
The mistake in my code was that I wasn't properly deducting an outgoing payment from a user's balance whilst this payment process was in progress. The attacker first deposited some sats on my system with the KeepSats function. The attacker then used custom code or curl commands to send multiple, simultaneous Lightning invoice payment requests. My system jumped into action and started using the same balance to pay multiple outgoing invoices. He would send in 1M sats and then get 3M out. This happened over the course of a few minutes until I shut everything down.
The Loss
In the end I lost somewhere around 0.13 BTC or 13m sats. I'm not even going to do the maths on the dollar value, and this was a few weeks before the current massive highs we're reaching now in the BTC price.
Fixed?
I hope it is fully fixed now. The answer lay in imposing very strict filtering on the API system which pays invoices around timing and simultaneous requests. This has no impact whatsoever on anyone using the v4v.app front end, but it should block any abuse behind the scenes using custom code to copy what my front end site does.
What this means for @v4vapp
It means that the @v4vapp system has cost me much more to operate than the total amount of the very low fees the system has collected up to now. The total fees I've collected are in the 250,000 sat range so a loss of 13,000,000 sats is clearly well beyond that.
On the other side of the ledger is the DHF so I have received enough to swallow this loss. Mentally I'm looking at this as a security audit fee. I've done nearly all the dev work on this project myself paying out almost nothing to others. This was an unwelcome expense, but that's how I have to view it.
Value for Value
For the last few months while building @v4vapp I was generously supported by the DHF. Going forward I have a much more modest support which covers direct server costs and a little of my time.
If you appreciate the work I do on and around Hive, you can express this directly: upvoting posts on Hive is great. Also consider a direct donation (there's a Tip button on Hive or a Lightning Address) on all my posts.
Support Proposal 303 on PeakD
Support Proposal 303 with Hivesigner
Support Proposal 303 on Ecency
Vote for Brianoflondon's Witness KeyChain or HiveSigner
I really appreciate your transparency here. V4V is such a win for hive utility. I’m thankful you are committed to continue. I’m sending you a few token sats in solidarity!
Edit** i scanned that QR code with my lightning wallet... it kept saying it was a receiving wallet not a sending wallet?? Not sure if im doing something wrong
The one in my signature works for me, not sure what happened, would need to know what wallet was trying to send from. Lightning is still a mess.
Ouch, learning the hard way!
That's why I do intensively try to break my own code. Me kind of Dr Jekyll and Mr Hyde 😈🙃
👀
No good deed goes unpunished.
That's the truth
That really sucks. I'm glad you caught it quickly enough. Love the service keep up the good work! !PIZZA
Yikes! That’s not good but you’ve taken the right course of action and especially when you run this system solo, it takes a lot of nerves and patience to do so, that I do not doubt!
sorry to hear this. best of luck with fixing everything.
Hope you somehow can recover because your service is needed on Hive.
There are a lot of people out there hacking these types of services because they can earn some money.
Since the dhf is paying you so handsomely why are you collecting fees?
Geez so many wrongs about this post and yet all you get is support messages.
A loss of 13,000,000 sats is no less, but then as you said, we have to be brave and move forward. Thank God you quickly swung into action, to prevent it fast, usually the hackers always choose a time, when you will be busy, so Jewish holy day seems to have been planned.
The loss must hurt, but at least you hopefully fixed it! Hang in there!
0.13 BTC are more then 10K USD at present and considerable an big amount, I m sorry about the problem and hope you have fixed the same now.
I am not holing a small hive power but still supporting the proposal and witness.
I hve used the service in past and will use in future also ...Keep doing good work.
I was really sorry to hear about this hack.
You have put in so much effort and created such valuable tools for Hive users.
I hope you make it back somehow.
You lost 0.13 btc worth $10387.49.. that's a big amount of money. Good thing you understood because your system was able to alert you. Hackers are really bad
I am a frequent & happy client of v4v.
Sad to read this.
Probably the best way to look at it.
I hope you stay motivated enough to keep it up and that it eventually becomes more profitable.
Hackers think different indeed Brian. I hope there are no more hidden mistakes in the system but well done on acting fast. This could have been much worse, especially if it was a group of hackers.
So, maybe you should hacker test your system one day and see if you got more open cracks.
It is really sad to read the los you made, but don't give up and keep on pushing forward. Thank you for all of your hard work.
Ouch 🤕.
Sorry for that man. It's hard to code thinking on all exploits.
Thanks for sharing that. V4V is a very useful project which I just recently started to use.
Are you maybe looking into getting some development help with the project? Particularly the front end part could be much more user friendly and elegant.
Hey, Brian, app development is a bumpy road. When you are faced with these problems you just have to strategize and face the problem head on. Keep up the great work on Hive.
I'm sorry about this sudden problem and your lack in $BTC, that inevitably slowed down the development of the application. I hope the system will soon return to the active, with the imminent Bullrun many will want to use the application at best
Fingers crossed you fixed it but I trust in your judgment. Good you noticed it quickly and got it solved, Brian. Well done. 👍🏻
Although I'm not a programmer, I understand how difficult it is to deal with such problems. I salute you
0.13 BTC there is a lot of money to be losing at this time, but then, I wouldn't want to also be doing the dollar conversion as it can. E further depressing. I hope you can actually get the system back up in ways which it cannot be exploited again like the hacker did. Sorry for the losses
I am happy not to be coding for financial apps as it is so hard to foresee every vulnerability. You are a brave man to offer this service and fortunate that you could detect this issue fairly quickly. I hope you are able to continue.
Damn, That sucks buddy! :O
At least you got a small amount from the exploiter to cover a bit.
Glad you got it sorted before it got too bad!
!BBH !DUO !hiqvote
Oh no! @caspermoeller89, you are out of DUO!
Go acquire more Stake to increase your DUO standings.
(We will not send this error message for 24 hours).
Learn all about DUO here.
Sorry by your lost I will try to uses your lightning from a connect between this and chivo wallet.
@caspermoeller89, the HiQ Smart Bot has recognized your request (2/3) and will start the voting trail.
In addition, @brianoflondon gets !PIZZA from @hiq.redaktion.
For further questions, check out https://hiq-hive.com or join our Discord. And don't forget to vote HiQs fucking Witness! 😻
@brianoflondon! @caspermoeller89 likes your content! so I just sent 1 BBH to your account on behalf of @caspermoeller89. (2/20)
(html comment removed:
)
!DUO
You just got DUO from @misterc.
They have 1/1 DUO calls left.
Learn all about DUO here.
!PIZZA
$PIZZA slices delivered:
danzocal tipped brianoflondon
@nastyforce(1/15) tipped @brianoflondon
Congratulations @brianoflondon! You received a personal badge!
Thank you for participating in the Leo challenge.
You can view your badges on your board and compare yourself to others in the Ranking
Check out our last posts: