So, crypto is a weird place. Today was supposed to be one of my favorite days in crypto. @eonwarped took code from @harpagon, adjusted it, and got our P2P system working. We've been silently running an internal test net and the goal for today was to announce a public test net.
How this day actually went is that in the middle of the night I started getting texts. Bad news is that we were hacked. The main node for Hive-Engine was directly accessed and the balances changed.
This is especially ironic, because if the P2P network which we are/were planning on launching today had been been active then this hack wouldn't have been successful.
What was lost
500,000 HIVE
1.05 BTC
Then what...
So, we close up all the access points and redo the security on the servers. We get a backup replaying so that we can get correct balances showing. It's literally just hive-engine balances that were effected, so no one else should notice anything. We'll switch to the backup shortly.
Then I'm sitting here trying to figure out what the fuck I'm going to do. Because 500k Hive and 1.05 BTC is a lot of money.
Hive Engine makes money, but I spend more money than what I make to turn this thing into something exceptional. So, a $100k hack is enough to really fuck with this project considering it's losing money every month while we're developing the core features.
So, my mind is flashing through all the options: tell people, hide it, close it down, personally guarantee it all, flee to Mexico... The whole panic thought line goes through my head, but ultimately I know the right thing to do is fess up ASAP and figure out how to secure this thing so funds are SAFU.
So, I'm getting ready to write up a post talking about how we're still ok and I'm gonna cover the damn thing personally. I go to start a power down on @aggroed when I see my aggroed account has 477k of Hive which has been deposited.
Now, I'm not a computer crypto forensic guy, but I feel pretty good assuming the gray/white hat that just kicked my ass has returned most of the Hive.
I'm made more confidant in that because they left these messages:
BTC
We'll see what they do with the BTC. Hopefully they return most of it. I'm in the process of getting some BTC into the system to make sure it's all covered.
More irony
So, the reason we were able to quickly detect and diagnose the problem is that the P2P system is actually quite nice. We're adjusting it slightly based on this extreme test case, but feel like we're in a good position with it.
So... What's this all mean?
- The server has been resecured. We're replaying a backup we'll switch to so that the correct balances show.
- Hive has been restored and BTC should be there by end of day.
- Aggy lost some money, but funds are SAFU, and I'll call the Hive difference a bug bounty.
- Surprise! We have a functioning P2P network that has worked great internally and we'll share more details this week about it going public. It's useful to prevent hacks just like this one!!! FML...
Crypto is weird
HIVE on!
if the gray hat is a white hat feel free to return the btc to 3Czgjc7tcPvp2xsEPFN6sWXnLTWzSH62FU.
Very sorry to hear this, but glad to hear the Hive was returned at least, and hope the BTC will be as well.
Another "incident" that casts a shadow over HIVE. Do you have a "damage control" press release planned explaining how this does not affect the integrity of HIVE?
Our base product is the cryptocurrency itself; if there were a loss of confidence, that would be very harmful. It's important to be proactive in these kinds of circumstances.
I'm sure you don't need me to remind you that if that foundation fails, everything else that is built on top, for as wonderful it may seem, isn't worth the time of day. ;-(
I just read your message and I don't get it. This seems to have happened on the Hive-Engine Note Level. That's pretty far away from a HIVE Blockchain "incident".
That reads, Note Server security problem to me.
We know that. Does the outside world know that?
Perception is everything, and there's nothing like misunderstanding to screw perception. That's what I'm referring to when I say "proactive".
(And that's not even to mention whether everyone even understands what you just said, even here on HIVE.)
The name is Hive-Engine. Like I said, we know how it's related, but everyone who is not on HIVE, do they know?
And those who don't know, what do you think they'll assume? That it's an integral part of HIVE, or something completely divorced? I think the name association clearly suggests the former of the two.
Do we care if the outside world knows exactly what happened, and exactly how HIVE is totally insulated from Hive-Engine? Is that important to us? Or shall we leave it up to everyone's imagination (again, referring to those who are not "in the loop" here on HIVE)?
Tough question. Or is it? We've got everything to gain by being proactive - it's an educational opportunity, an opportunity to actually brag about how safe HIVE is. I'm sure everyone here is familiar with the concept of "spin". The "spin" is totally positive for HIVE, and it's totally true too! That's win-win if I've ever seen it.
And what do we have to gain by doing nothing?
Or should I ask, what do we have to lose?
Don't worry no one outside of Hive talks about Hive.
Also, I don't think a hack on a side project is a big blemish in Crypto there are ETH scams every day..
2nd layer/side chain mate and the world does not care about hive anyway
It will get lost between news about btc and ETH
Did you even think about that before you wrote it?
If you don't have HIVE, you've got nothing.
And if you don't know how people make more this of kind of news, how they distort things, in order to hurt you, you haven't been around the wild, wild west of scammy wammy crypto for very long. The association exists, and it's up to us to be preemptive to avoid having people think it was something worse. I've said it so many times my finger hurts. :D
Think I'll leave it at that.
BTW, the Leo Fianance hack was another high visibility "HIVE blockchain related" incident. They add up, don't they? Even if a totally misunderstood and incorrect perception of incompetence were created, wouldn't you say that would be worth trying to preempt and avoid? If you were a high level dev on HIVE, wouldn't you want to distance yourself from that kind of negative perception (or even the possibility of)?
What are you suggesting?
Probably to hide in a cave somewhere
It is very related in some sort, but there is also a variety of reasoning on the chain which has lead to where we are. Aggroed is not a Top 20 witness even with some of the most significant projects. He's doing stuff and he's taking the risk, good for him - I hope. This was not a specific HIVE 'Source Code' Problem, more a within the Community and Projects Sphere of it.
That of course does not invalidate your main point, but it bends over to something else. Let me try to build this from the ground. Every entrepreneur has this Sword of Damocles hanging over his head, sometimes you're more aware of it and sometimes less, but it's always there.
Now when we look at exchanges, that's what I would call a very risky business to be responsible for. Especially in Crypto, we all know that for years already, exchanges have always been somewhat dangerous places for lying funds. There have been countless incidents so far. How comes? Most attacks against them have also been ordinary IT Problems, such as this one. This leads to the real question here, how secure are the note servers, and how much effort is being put into that. How trustworthy are the admins and how suited is the system to lock itself after a breach? Those are fundamental questions I'd be interested in.
... But then again personally not really. It's a few thousand bucks for me and I trust @aggroed so far that I'd be willing to lose them if he really fails. He's not a con artist and he's doing a fantastic job so far.
My Conclusion:
A very capable Hacker wanted to let us know, that he thinks that Aggy suxx. Well, ok, Message received and thanks for returning the funds.
Or maybe he just wanted to remind us that it's not DECENTRALIZED?
If it's not decentralized, it's not trustless, and not crypto. As you've correctly said, you have to trust him.
Either way, I'm not speculating on motives or talking good or bad about any particular people. I'm talking about HIVE's reputation, both real and perceived, and what we're doing to preserve it.
I hope you're beginning to understand the difference.
That's a very consequential statement, something larger than life. Too bad we're still trapped inside this fallen world where everything ain't black or white but shades of grey :)
Safu?? More like
SNAFU.
Posted Using LeoFinance Beta
Yeh, I don't like the term SAFU either. It sound so Justin Sun . . .
SAFU, yes, I also dislike that. Nothing is save in life, literally nothing.
its CZ
I can't read the thread, the UI is borked now, but thanks for the comment!
SNAFU 🤣
Posted Using LeoFinance Beta
Lol guess someone got rekt by your "game" and decided to show you their discontent.
Glad most of the funds were returned.
I hope that makes him think a little bit about this...
you do great work but please get this shit together, if this was not a white hat hack it was be so devastating to the ecosystem.
And someone took it and returning as well ? Insider ?
hello sir nice to meet you. i have sent you a request on your linkedin please do check.
Posted Using LeoFinance Beta
Rough morning. On the bright side, what doesn't kill us makes us stronger.
Dementia doesn't
That's dark
It seems it's someone who knows the Hive ecosystem well.
Also, memo
104624678used by @nightowl1 to cash out was used a lot on Steem (and probably on Hive too), over 1.5M Steem transferred since 2019-05-26, at the start from @nrg and @jamesc1, even @trevonjb but from many other accounts also, so it might be OTC exchange.hmmm interesting
Sorry to hear. Hope it's somewhat of a wake-up call to invest in security. Do y'all run any kind of pen tests and vulnerability scans? I know hiring security professionals can be costly but may be wise moving forward.
Ohhhh no. I'm glad you got the Hive back at least. I was wondering if something was up with H-E as I could buy/sell various tokens but haven't been able to buy swap.ltc since yesterday. I am guessing that's because you had to put things on pause regarding external tokens?
I hope they return the BTC to you as well. What timing!
crazy story, thanks for being honest about it!
I don't know i should be happy or sad, but this will make us more strong. I am curious to learn more about p2p system. So will be patiently waiting for it.
There comes the point of decentralization, i hope you will now focus more towards it.
Best Regards
Sorry to hear this, glad they gave some back hoping for the rest to be returned. Happy to hear about the P2P network, excited to see it
S'up, nice to see you down here in the comments :))
Most important question, was the vulnerability fixed?
How'd they get in? That is the pertinent question. Anything with access to crypto keys should be locked down with encrypted private keys to get into the box. :/
Between the unexpected and the unknown. There really isn't much white or gray in between. };)
yep. "should be"
Glad that you got the Hive back and hope that the BTC will come back as well. Would suck if you would need to close down because of that.
1BTC and 500k HIVE is a lot, fortunately the hacker at least returned the HIVE almost completely - let's hope the BTC will come back too..
!gif hacked
Via Tenor
On reading this post, the first thing I did was to check my HIVE wallet. Of course, there should be nothing from this that should affect the integrity of my HIVE wallet, and everything is powered up, so it seemed like a fairly silly exercise, but I did it anyway.
Interestingly, I had rewards claimed yesterday THAT I DID NOT CLAIM.
I claim my rewards once every 15 days - for accounting reasons - and I'm very aware of when I do that and when I don't: the 15th and the last day of the month. But my wallet shows they were claimed yesterday???
Could I have hit the button accidentally? Of course, I could have, but I didn't even access my wallet yesterday. Sometimes it comes up in notifications, sporadically, don't know why, but I could have hit the button accidentally. Isn't there a confirmation notice when we claim rewards?
Very weird. I mention it so others can check too.
not sure what frontend you are using. but peakd has an option for notification when your rewards pass some number of hive to claim. And there is a button to claim it, so it could be that you click it.
I'm using Hive.blog.
crazy, but good that at least the Hive has been returned, have you found and fixed the vulnerability that caused the hack?
so let me get this straight. He hacked the hive engine, took 500k of hive, 1 btc, the returned the 500k hive but not the btc? (but he will? or we hope that he will?)
and all this just a little bit before P2P network is up which would have prevented that?
Posted Using LeoFinance Beta
that's pretty close
Glad you got the hive back and hopefully the bitcoin too. 🤞
Honesty and transparency are the best course of action and I respect the fact you came straight out with it and didn't try to cover up a problem. It counts for a lot more in the long run.
Lessons were learned and it could have been a. Lot worse.
They've said they'll return the rest also: https://peakd.com/farmfarm/@farmer.farmer/i-ll-send-you-the-rest-back-but-later-first-i-have-to-farm-farm
https://hiveblocks.com/@farmer.farmer
An account created today, where is the creation data??? 🤔
Strange, looks like a bug @roadscape
Here's the account creation tx https://hive.ausbit.dev/tx/ae904e2be58f281f80a2b5a876a0b7a03a0f712a
Thanks 👍
Was this account the attacker? Contact me if you need more information about the account.
@aggroed
Thx @doze - got in touch with him.
👍
Sorry to hear that, I hope the BTC gets returned too.
Happy to hear about the P2P, and willing to see it.
Sounds like a HE Note Security incident then, well that has happened to all of them at some point. An Exchange is an Exchange, has always been and will always be. Of course, it would be good if HE could take a 100k hit at some point, without needing your private funds.
I've been reading through all of the comments so far and I wonder why some people believe their Hive Wallet might have been hit. Are they out of touch with the reality of this technology or have plaintext Keys been leaked?
Take a close look at my case. Tx show I was commenting and upvoting in a 3 min span on either side of my "claim" rewards. I jumped from the page I was commenting on to claim rewards only to immediately jump back and upvote the post I was commenting on? Sounds logical as hell.
3 min time span: comment > claim rewards > upvote post.
And as I said in my comment, it wasn't on my "to do" list, nor do I remember doing it.
Must have done a one minute "sleep walk"?
BTW, how does this post start? crypto is a weird place?
Instead of belittling, maybe you might treat things a little more seriously. Or a 100 grand hack is no big deal and any other "weirdness" is out of touch with reality? You tell me.
EDIT: serious consideration wouldn't relate the two directly, rather indirectly: if someone was hacking HE, why not try to do the same on HIVE while they're at it? Or do you discard that as absurd too?
There are Hacks and Hacks, one can use a specific exploit and cause damage or just gain access to part of a system OR even open pandora's box.
I don't spend time on 'belittling' people, I try to check my understanding of reality by comparing it with others.
Hive Engine is not really decentralized, even if it's already open source (besides deployment scripts I'd guess). That means full control over the Main Note, should be close to full control over the running system. Meanwhile, Hive is super decentralized, so if you gain full control over one of the main HIVE Notes, you are still just one of many and can't just change things.
The problem is that when there are people who are so defensive about coming out and clearly stating that in public as you just did, it makes people wonder if that's really true. As I've said, you and I know it, but the rest of the world doesn't, and the rest of the world is going to assume the worst, the opposite of what you just said, unless leadership here on HIVE actively starts proclaiming what you just said to the outside world.
I said it in few words to begin with: it's an opportunity to brag about how safe HIVE is (as well as to clearly educate on the difference between HIVE and HE, or any other 2nd layer token for that matter).
Hmm, you say Leadership as a call-to-action for maybe all the Top Witnesses?
How about you tell us what you did wrong so we can learn from your mistakes instead of giving us a horseshit lullaby.
Thanks.
That's the understatement of the year. Most hackers would've responded like this:
Especially if they were from TRON/CCP
Maybe after they fully test to make sure it's actually fixed and can't happen again
So sad this happened. Thanks for your work and effort @aggroed.
wow. silver lining I guess. Best of luck for next time.
Glad you did not lose your cool. Stay blessed and I pray that the BTC is returned to you.
Posted Using LeoFinance Beta
Should we laugh or should we cry?
Posted Using LeoFinance Beta
That sucks but it’s crypto. I’ve been really into the hive engine lately though, so much cool stuff of there...
Hope you get the BTC back too.
Sorry to hear this :/
One of your friends did this to you.
what the hell. glad you got the hive back
Yeah, spin some good news out of this...
The good news is that I suppose you can cover a paltry 1.05 BTC
Good thing they returned the stuff no one wants, the Hive... 😬
that will make it easier on the exchange.
Posted Using LeoFinance Beta
Stay strong brother
Yikes! It looks like this was a best case scenario then... I hope that your lessons learned and subsequently adjustments to H-E security stop all future hacks like this one. Thank you for being honest with the community. It helps if we are all keeping both eyes open!
I am very sorry this happened
How to hope on people working together.
Best wishes for rolling on your work forward an on and on .
Thanx.
This was just a free card from what I see... a push to get things decentralized. Someone that knows your weak points.
@aggroed,
Sorry to hear that, but nice to hear it became a white gray hat & hope the hacker will send that hacked BTC as well. Think it's time to move to decentralization isn't it?
!wine
Cheers~
Cheers, @theguruasia You Successfully Shared 0.100 WINE With @aggroed.
You Earned 0.100 WINE As Curation Reward.
You Utilized 1/3 Successful Calls.
WINE Current Market Price : 10.000 HIVE
We won't judge you for self voting for a while.
Sorry to hear this.
Potentially see about investing in Weedcash and Weedminers, if you hold shares its easy to curate extra value as a vote bot for cannabis content.
I ranted in my blog posts today. This post and news makes me feel much better. Thank you for the update.
I think you will get much more problems that just a "hack" in the future.
But i am sure, you know what you do.
The flee to Mexico option might be a problem given COVID travel restrictions.
Hacking the HIVE is bad enough, but the BTC hack is a disturbing revelation to those of us hodling SWAP.BTC
Posted Using LeoFinance Beta
Excelent