This article was written in collaboration with Eric Björk of Sw/eden, Eugene Luzgin of EOS Tribe, Jae Chung of HKEOS, Igor Lins e Silva of EOS Rio, Michael Yeates of eosDAC, and @jemxpat.
Motivation & Concerns
The EOSIO software is both groundbreaking and revolutionary. It has the potential to bring a new era of blockchain applications and create new paradigms in blockchain usability, scalability and governance. Thousands of people, including the people involved in this release, have invested huge amounts of time and money to ensure this potential is realised. This is all culminating in an expected launch of an EOS mainnet on June 2nd/3rd 2018.
However, the EOS mainnet is a unique target for attack. With a year long ICO and so much money raised, large community involvement, huge expectations, it is reasonable to assume that there are malicious detractors who wish it harm. We feel it prudent to consider the nature of possible attacks, and the defenses and procedures expected to be in place against such attacks.
The bootstrap methods implemented in the eos-bios program, result in severe vulnerabilities, allowing any attacker to damage or destroy producer nodes or the entire blockchain. The excessive information about the producer network published directly on the blockchain will allow all types of attack, from protocol injection up to volumetric DDoS attacks.
While it is never possible to perfectly secure any system, it is irresponsible to not create some reasonable defenses. In this message, we detail some of the attacks possible, demonstrate some tested attacks against the insecure boot chain, and then suggest our preferred method of bootstrapping in a more secure manner.
We deeply regret any discomfort and angst caused by this. We have tried to explain the problems and suggest solutions in public and private Telegram chat rooms for over two months. We have encouraged modifications to the proposed automatic bios code. Alas, changes to the security model have not been implemented, and have been actively resisted. We feel that we are left with no choice but to publish a few precise and demonstatable methods of attacking and damaging such a blockchain, and show how to mitigate such risks. If we do nothing, the risk of failure is too large to accept.
We did not publish these concerns earlier because we needed to investigate them extensively and verify that they were not just speculations before sharing them to the community. The writing that follows consists of facts supported by evidence.
Current insecurities discovered in the eos-bios boot process.
Much of the functionality of eosio is provided by plugins. They allow the system to produce blocks, connect to other nodes etc. The base eosio software includes many plugins. A block producing node should not run any unnecessary plugin on a live network, because some of them allow the chain to be hacked. Some of these plugins should never be exposed on a publicly accessible node.
One particular plugin we have noticed is the net_api plugin, which provides an api which can be used to control the plugin remotely. If this API is exposed on a public network then anyone with the ability to connect to this network can tell the block producers to disconnect from, or connect to, any other machine. This does not require anything close to a botnet. Without even being a part of the eos-bios network, a mac mini could disconnect every peer in the network.
Allowing this vulnerability on the live network could mean that an attacker can stop the chain from function properly and even partition the nodes so that eventually they stop. The Ghostbusters team noticed the vulnerability early on in the design of eos-bios and reported it to EOS Canada but no changes were made. This is one reason that we cannot support the launch using eos-bios.
How to hack a chain knowing one peer node running net plugin
Run command:
cleos -u [peer-url] net peers
Returns you a list of responses like following for each node:
{
"peer": "",
"connecting": false,
"syncing": false,
"last_handshake": {
"network_version": 1206,
"chain_id": "0000000000000000000000000000000000000000000000000000000000000000",
"node_id": "b92b2b7d8835e46e2fed97f5eebda31faea63fd07cc40ad52f132254f22cac8e",
"key": "EOS1111111111111111111111111111111114T1Anm",
"time": "1527171594944050635",
"token": "0000000000000000000000000000000000000000000000000000000000000000",
"sig": "SIG_K1_111111111111111111111111111111111111111111111111111111111111111116uk5ne",
"p2p_address": "reach.me.example.com:9876 - b92b2b7",
"last_irreversible_block_num": 57,
"last_irreversible_block_id": "000000391a9439aaa2864d9807965bc84865cdd15c4a0a3d0ae3c7e54a85a38f",
"head_num": 300,
"head_id": "0000012c9826a4f37db5e5c8b4790acad70c01188aee8e7b7330790937a93cdd",
"os": "linux",
"agent": "\"EOS Example\"",
"generation": 2
}
Next get a list of peers only:
cleos -u [peer-url] net peers | grep peersAnd disconnect each peer:
cleos -u [peer-url] net disconnect host:portOne could eventually force the connection to any other peer using, also increasing risks of network instability:
cleos -u [peer-url] net connect host:port
The Recommended Solutions
So should we be panicking and throwing out all of our tokens? Definitely not. As much as we have investigated and tested the potential security risks associated with launching the network, we have also experimented and discussed about solutions to the problems above.
Wireguard Private Mesh Security
Given the fact that EOS software has not been securely vetted and BP node stores sensitive producer private keys, we have to take extra measures for securing those producing nodes. Ideally no public Internet access should be allowed to BP nodes while allowing meshing between BP nodes. Hence we propose using secure peer to peer communication between BP nodes via point-to-point secure tunnels using the open source WireGuard kernel based VPN software.
Each BP node will have 1 or more full nodes that it establishes secure connection to. Those full nodes will be exposed on open Internet protocol and available for access for overall Blockchain network. BP node should not expose it’s non-VPN IP address to the open Internet access. Also for fault-tolerance BP node should establish several trusted connections to other nodes.
- Layer 1 (Block Production Layer)
- 2 Producers: 1x Producer node & 1x Stand-by node
- Producer Control Switch: a machine to monitor and enable failover switching
- Layer 1 nodes could be connected to other trusted producer or full nodes via VPN (WireGuard)
- Nothing extra installed other than Producer API, whose access is restricted to the producer control switch
- Layer 2 (P2P Layer)
- Full nodes to relay blocks
- Connected to the Layer 1 nodes via direct tunnels (WireGuard)
- Securely meshed to trusted BPs via P2P VPN (WireGuard)
- Only uses History API and Chain API, restricted to the proxy servers on layer 3
- BPs are encouraged to make public full nodes available for external access for example, exchanges, portals, new BPs…
- Layer 3 (API Layer)
- Web servers to support HTTP endpoints
- Layer 3 nodes should be pure web firewalls and have no blockchain information
- Connected to our Layer 2 Relay nodes via HTTP (wireguard)
- Uses Patroneos to prevent against basic DDoS and application layer attacks
- Layer 4 (Public Layer)
- Global BP Load Balancer (all traffic goes into this single point)
- Volumetric Attacks gets handled at ISP level
- Routes all HTTPS (SSL) traffic to our Layer 3 web servers
- Configured to DNS (optional)
- L4 can be as simple as DNS round robin with short TTL.
Conclusion
We are 100% committed to launching, and supporting, a stable and resilient EOS mainnet. Given the facts stated above, and insecure nature of eos-bios boot process, our community of BPs is strongly advocating against using at as standard launch procedure as it exposes a complete chain to an attack by anyone.
We, as a community, consider it our duty to ensure that the EOS mainnet is secure and resilient. Using the eos-bios process will create unnecessary risks for the EOS Blockchain launch and ultimately all EOS token holders. Also, any negative press on insecurities in EOS Blockchain launch or failed attempt to launch the Blockchain will have a negative impact on EOS price and reputation.
Therefore we believe it will be in all stakeholders interests to bring down any insecure and unstable network, using above methods we have highlighted, before any malicious third party takes advantage of such insecurity. We will simultaneously endeavour to introduce security improvements into launch and operational processes.
The security imperatives cannot be ignored and, given the stakes, must viewed as the paramount consideration.
- BP Teams Involved:
- Sw/eden
- EOS Tribe
- HKEOS
- EOS Rio
- Block Matrix
- eosDAC
- Eosmeso
- AcroEOS
The eosio::net_api_plugin was made optional for this very reason. No producer or public endpoint should be running with this plugin and it is a simple config file change.
I feel like the intro to this post is overly alarmist way to support an otherwise worthwhile effort at securing block producing nodes.
Yeah, especially that all those are pretty much basic and obvious do's and dont's (don't run plugins you don't need on a witness node, do not expose API to public that you don't want to be public, etc, etc. :-) ) and everyone who is familiar with this family of blockchains already knows that
Well, at least those running for the office should ;-)
(no, I'm not, too busy with Steem)
We are sorry you feel that way @gtg
The effort was honest and others in the Trinity group are saying it isn't resolved this easily. Let's let the dust settle and see how these groups resolve this issue before discounting anyones work on strengthening the EOS network.
To add to the points of @dan and @gtg, just curious why was the public disclosure not made littler earlier since you are considering the plugin bug a serious issue ?
Personally I think the answer for this post is
RTFM.
Aside from BOOT BIOS group not being able to stand up the chain after 3-4 days, there were still dependancies required for the plugin in question to still work.
From my observations, a lot of politics and back-channeling was done to dissuade people just like you, that's all the happened.
Same exact stuff that happens at big companies when IT tries to communicate this stuff. No one listens and then when there is any disclosure it's met with this type of tribalism because no one wants to take blame.
Now these guys are the people launching the chain in case you didn't know. Doh!
All I was saying that we should be as transparent as possible. I am not mad about EOS or any project as such but support FOSS and free knowledge sharing. So all I meant to say is that any vulnerability after giving reasonable time for the developers/companies to fix, should be released to the pucblic. In this case since there is no production network even now, your disclosure was pretty much on time. When you posted, I was under the impression that the methods and procedures for Etherium snapshot, validation and deciding on the initial token supply and main net launch all are tested and frozen and the staging network which will be used to performance - pentest will be ready in less than 24 hours time. I was not aware that these procedures are just getting planned or tested. In a nutshell, since you sounded like the production mainnet is just 2 or 3 days away from the date of publication of this post, I said it could have been done earlier. Thats all.
Sorry guys but eat some humble pie on all the judgments that some put on this group for trying to bring all this up prior to the launch. No one listened.
Now here we are 2-3 days later and BOOT BIOS didn't deliver. All that hype. All that political BS, pomp and circumstance, etc... and in the end, they feel flat on their face.
Funny the difference that just a few days makes. ;)
In the end, the very people you put in question ended up being the talent that will launch the chain.
Are you aware of the Ghostbusters project?
here's where Alex explains that net_api module was added for testing only and he warns it has to be taken out:
Yeah, that wasn't what it was about. The plugin in question still had some dependancies that required it to work predictably/easily for the majority of users using it - some of which were not capable of doing it any other way. Regardless of whatever spin people try to put on it, these guys were just trying to help and got sandbagged for it.
Anyone who knows anything understands that's what really happened with all this. Read through the lines people, don't believe to the first thing you hear. This is a political race and we should expect people to push their own narrative, especially if it dissuades popular opinion in their favor in the eyes of token holders and the larger community.
Thanks for the update, heard about this but hadn’t looked into it yet...
so EOSCanada ignoring security concerns for 2 months straight? way to become a BP
now, EOSCanada has always been very transparent about any problem, let's wait for them to tell us their version of things before rushing to conclusions...
They failed to launch the chain because of the very things the Trinity/Ghostbusters group was saying all along - for literally months. There is no excuse. The serious security and systems guys were right, a lot of the others were flat out wrong. Maybe the next time something happens like this people might not dismiss it so quickly.
Thanks for the information! Launching the EOS mainnet will require a great deal of coordination between Block Producers. This involves configuring secure connections to prevent attackers from scanning the entire network topology looking for nodes to shut down.
We will need to plan, communicate and work closely together with other Block Producers to ensure the network is secure and efficient.
We have identified this requirement in our Technical Roadmap
The vulnerability you are pointing to is in the net_api plugin, not in the EOS-BIOS software.
I'd rather see a transparent, consensus oriented, well orchestrated launch, than one obscured by a VPN system among a group of BP's.
I've never been an advocate of "Security by Obscurity" especially when BP's are suppose to act with complete transparency.
But I would wait to see EOS Canada's debunking of this issue before making further comments.
The plugin in question in fact had some dependancies at the time of announcement though. A time when they were only days away from launch.
The Ghostbusters/ Trinity group had serious concerns that this could slow down the launch of the chain ultimately - which ended up actually happening. Now the Ghostbusters/Trinity group are the CORE group launching the chain.
That says everything you really need to know about all this.
End of story.
why I am not surprised. first thing they have to do is to behave like professionals and act securely. bps must realise they are not 'talking' to the community only, nor the other bps; they are talking to their future hackers. bring down the youtube videos sharing screens and commands please and stop being careless. if you care about the community be secure. and if you care more, be more secure. you'll never tweet, share your name or your email because that would be insecure. and you would reset your router each day like a pro.
Relying on hackers not knowing how to hack your system (also known as "security by obscurity"), doesn't work. You can safely assume hackers to know and understand your system well. Hence you can not hope for hackers not attacking your system.
What this group have published is nothing secret but standard functionality.
Besides these concerns were raised in smaller group conversations for some time now but no actions were taken to remedy security concerns.
And while marketing was winning technical side of this argument - we had to make these facts known to a wider community while keeping in mind it may make some people unhappy.
Ultimately our goal is successful launch of EOS Blockchain and a duty to EOS token holders.
Really is a breath of fresh air to see a talented community with a clear. The combination of this security bug and the consensus algorithm flaw i think it's safe to say mainnet isnt ready for launch and the fact that the community is aware of that and not trying to rush it is a great sign.
Exactly!
Great work Eugene and team!
It's a good article. You should edit it to take out the alarmist parts now that you know...
But that's the thing - the Ghostbusters/Trinity group was right about all their assessments. The BOOT BIOS group was not able to stand up the chain because of some of the very reasons the Ghostbusters/Trinity group outlined. And the plugin in question ended up having some dependancies and wasn't as easy to use without them - so some people in the BOOT BIOS group couldn't hang when it was game time.
Not trying to be confrontational, but that's the final word on all that.
Fair enough then if the tone of the article was about right in the end. To me it initially sounded overly political. Always good if you can end up coming back and say "I told you so"! Good work ghostbusters
Go get em son we aint no verge
@remind.bot 3 hours
Mentioning and replying to you here on Mon, 28 May 2018 21:46:10 GMT to remind you of this post.
Please consider upvoting this comment to keep this service running.
Reminding @cmichel of this post as requested!
Please consider upvoting this comment to keep this service running.
Thank you for putting together this article, all of you are now on my list of BP candidates to most likely vote for.
Thank you for the information. The way i understand this EOS and other current ETH based tokens are currently safe (relative term) in registered MEW or simaler wallets. However tranfering them to the EOS main net upon launch may be risky unless security measures are implimented. Please confirm.
you don't really transfer those ERC-20 token into the EOS blockchain, is more like you get a copy of them on the other blockchain, if it makes sense. They will still be in your eth wallet AND in the EOS blockchain.
You don't have to implement particular security measures beside storing your priv keys in a safe way
good job!
This is a good article with a lot of good security advice, but I suggest that the Ghostbuster team and the Canada team and other core teams can communicate deeply, make up for each other's shortcomings, and finally reach the same starting point, such as reaching a compatible solution that will not cause the community to split up, only in this way can we start a main network in the whole world.
The following figure is a deployment diagram that I suggest for each BP. Each BP provides two types of nodes, some node communication with other BP's nodes through VPN(WireGuard), some use public networks to connect directly to other BP nodes， The details are decided by BP himself
Congratulations @eostribe! You have completed some achievement on Steemit and have been rewarded with new badge(s) :
Click on any badge to view your Board of Honor.
For more information about SteemitBoard, click here
If you no longer want to receive notifications, reply to this comment with the word
STOPDo not miss the last announcement from @steemitboard!
What an article! Resteemed to my Cyber-Security blog!