Did you notice issues using steemit.com today? How about powering up? Logging in? How about trying to transfer SBD this morning to blocktrades only to receive a message like this one:
Or 
How Did This Happen?
You Do The Math!
- We do not have enough seed nodes to handle all activity of the steem blockchain at its current growth rate without limiting users bandwidth. Period.
- over 90% of currently active seed nodes and ALL of the full RPC nodes are top 20 witness owned and operated either directly or via proxy thus creating a single point of failure (i.e. api.steemit.com ) for the network. Basically, api.steemit.com+unavailable=Dude where's my dough?
- 90% of users on the steemit network are essentially (and this is an assumption) in the "minnow" or "dolphin" class which makes us the majority however we tend to carry considerably less bandwidth.
- There is zero monetary incentive to buy, build, manage and maintain the seed nodes which empower our network (however the top 20 witnesses do get paid, so they're encouraged to also run seed nodes because they are assumed to be able to afford them though not a requirement)
- The steemit blockchain is growing at a phenomenal rate!
After Reading articles like the one posted by @drakos who is also a witness (See The Bandwidth Limitation - Problems and Solutions ) and others like @gtg Call For Steem Seed Nodes! , I Began thinking of ways that not just myself, but we as a community could contribute to fixing addressing the issue with available seed nodes er.. The lack thereof. At the same time trying to figure out how to add value to the network (As if building a witness and 2 seed nodes myself out of pocket wasn't enough) So I decided to start a project called Steemit Forever!
For the past two weeks, I've been working on this and other steem related projects to empower the steem blockchain+steemit network as well as strengthen the integrity of it's infrastructure. I took a few approaches on addressing some of the current obstacles that we are facing technology wise as this is an area where I will probably add the most value. I created a witness server which you can vote for simply by clicking here, and two seed nodes which can be found at steemit.chainchopper.com along with the current active list of nodes operated by top 20 witnesses. The alpha seed node is a donation from chainchopper.com . I will maintain it as long as I have funds to do so till this witness can afford it on its own. I did this in good faith with the presumption that most people on steemit are generally good and want to see the network succeed as do I. But that's just the beginning . . .. .
The Mission - More Math
"To serve and to protect the integrity of the community by empowering it's technology and it's supporters"
So you're probably thinking to yourself "The hardware contributions and technical expertise that you bring is all great, and it's good to see you come bearing gifts, but what will this mean for me?" Firstly, Steem nodes are what make this network function. This is something every single user will benefit from. Keep in mind that there are no payouts or monetary incentives for running seed nodes (which is probably why we have so few). Especially for people like us who are minnows here on steemit and cannot afford to run them or lack the time to learn how to do it properly. By now you're probably wondering " how and why should we pay for the seed nodes if they won't make us any money?" Well, the idea is that once people are informed and grow to understand the need for seed nodes along with the importance behind an initiative to remove centralized points of failure on a platform that is in it's own right cash-centric, we will empower a witness that's purpose is to stand for and protect the blockchain by providing the needed technology as well as carrying out civic contributions in the form of delegated SP. Decentralization will be promoted by separating services both within our local groups, as well as remaining separate from current node groups thus providing fast and efficient redundancy in case of failure. . But if you're asking seriously how this project can work for you, then i believe this can be achieved in a few ways starting but not limited the following:
- I propose to create two new seed nodes for every 500 witness votes for @justinadams. (1 full RPC node + 1 standard seed node built to proactive CPU/Memory/Storage specifications) This process will continue as needed and may very well mean that I will have to continue to fund these out of pocket as witness votes do not guarantee that it will receive rewards from blocks.
- 50% Of the rewards produced by @justinadams will be used to fund and maintain the actual seed nodes and RPCs
- Give It Back! Voters of the witness @justinadams will become eligible to become recipients of a percentage of delegated SP from a fund held by @chainchoppers as outlined below.
- Separation! Separation! Separation! .... Did I say separation? Share the work load of the steem block chain processing with current nodes whilst remaining independently hosted. This will create an effective "Fail-Over" contingency making us less vulnerable as currently there are none in place. (who would have thought?)
The Blame Game
Who's fault is it? Well, It's yours. . Let me explain why. A year ago, @gtg published a post calling for more seed nodes (See link above). Granted, this call was seemingly more directed to top 50 witnesses (and rightfully so). A week ago, I'd been noticing issues with node availability while setting up a witness and seed node. This is evidence that even someone with as much influence as @gtg can be brushed off and summed to a no-action (It works right? what's there to fix?). I published a post called "Steemit Forever!" Over the course of the past week, I noticed at least 5 of the top ranked witnesses had made posts regarding their server upgrades and availability. This is great, and this is the type of action we should be taking, however, This is a problem that the top 50 witnesses cannot address in their current positions. Inherently, they are a minority by design. This, in theory, is to promote competition among witnesses according to The Steem White Paper In an effort to encourage them to basically work together to eliminate fraud and I quote " The elected witnesses are presumably trusted by those who have a vested interest . . . . By paying those who are elected, Steem creates market competition to earn the right to produce feeds. " Now, Proof of Brain would suggest that in order for the top 20 witnesses to succeed that they must indeed unify. That's not the problem. That's actually probably "smart". The problem, is that ALL of the seed nodes and RPCs are owned by the top 20 witnesses... Now do you get it? So you see, there is no way for the current witnesses as owners of the infrastructure to fix this problem with seed nodes without further contributing to what will become an even bigger problem in the future due to the fact that many of them use the same service providers and technical teams. This created a single point of failure for us today, with no failover. But alas, it is not their fault.. It's not their fault at all because running seed nodes isn't what they were elected to do nor is it officially supposed to be their job. At some point, witnesses inherited the task. Some out of necessity, some do it because they are just good people that want to see steemit succeed and give back to the community. At the same time, It's the fact they inherited the task that causes problems like the one we faced today. This is due to the unilateral structure of witnesses. Many of us have voted for witnesses for many many different reasons, so why not let them do what it is they do without the burden of worrying about the availability of seed nodes? Quite honestly, I'd rather sit back and watch from the bench rather than to sit my seeds in the middle of a cat fight :)
The Solution
Let's build community owned steem seed nodes and full RPC nodes. These will be called "Forever Nodes"
-- The nodes that this project builds will indeed benefit everyone and these nodes will forever belong to the community for it's use. Hence the project moniker "Steemit Forever". In the case that this project does succeed and the witness begins to earn, I have set up a steemit account @chainchoppers which will hold 50% of payments earned from the @justinadams witness. This will be called "The Fund" . The other 50% will be designated to operational costs for the witness and forever nodes. Funds distributed to the @chainchoppers (The Fund) account will be converted to Steem Power. It will be disbursed (delegated) evenly to 25 voters of @justinadams witness in the form of SP for a total of 50 voting users per month. These voters will be chosen at random regardless of their reputation or current SP and account value. This process will repeat bi-weekly commencing on the first day of the first month following the first successful block written by @justinadams which can be checked on steemian.info. Over time, this gives 50% of rewards back to the steemit community, while giving everyone a chance to earn curation rewards from the SP while they have it thus giving back to the community. The delegates will maintain the SP they received during the 14-day period and will keep all curation rewards during this period. At the end of the holding period, the SP will be retracted, and redistributed to another 25 randomly chosen delegates from the voting list (Kinda like a lottery that you didn't buy a ticket for) . . This can change as needed, but it is the only way I could think of to prevent someone from actively cheating by voting from 100 different accounts and unfairly saturating the payout pool with monster bot votes. If anyone has any suggestions regarding this, please feel free to discuss it in the comments below. Lastly, In the event that I am unable to maintain the nodes due to any reason health or otherwise, the rewards pool will be disbursed evenly among ALL current active voters in the form of SBD during a period not to exceed 90-days giving ample opportunity for server keys and ownership to pass for a proper dissolution.
What Can We Do About It?
Vote This Witness and you're all set!
You may think it's a coincidence that this happened while I was in the middle of writing this article, but I have been screaming ever since I joined steemit that this could/would happen only to get ignored. I'd forecast an issue could be eminent if we did not do something about the availability of seed nodes and divide ownership of the network more evenly, however with such low sp and rep, it likely fell onto deaf ears. Even more shocking is that even though @justinadams has already contributed 3 nodes, it only has 3 witness votes including myself (wow). Needless to say, to whom it may concern, "I told you so". Hopefully with today's outages we can learn something about ourselves and the responsibility we have to the community. Most people before today probably even thought that this didn't concern them, but what about now? I am doing my part the best I can. With your help, we can prevent this type of service denial and interruption in the future!
Community Action - If anyone wants to consider setting up nodes themselves (please do) and need help, I usually hang out with the other witnesses on steemit.chat or PAL on discord. If I'm available, I'll be signed in under Chainchopper or Justinadams, and can answer questions regarding setup etc. I'll also setup seed nodes or witnesses for others in special cases where we could come to an agreement. If you're interested in checking the status of the Forever node(s) , as well as all Full RPC nodes and seeds owned by the top 20 witnesses, Head over to our secure website steemit.chainchopper.com where it will show who's online and available and who's not. And if you're feeling froggy, Donations are always an option.
Steemit Forever!
I think I have to take issue with one of your core axioms – that this sort of redundancy maintenance isn't part of what the witnesses signed up for.
On the contrary, it's exactly part of what the witnesses signed up for and, in particular, the top witnesses have a vested (pun not intended but certainly appreciated) interest in making sure that the seed node architecture is distributed across multiple platforms and multiple pipes.
After all, a failure of the system like we had yesterday is, front and center, first and foremost, a problem for the top witnesses. They have failed in one of their primary purposes, to keep the blockchain supported and stable. Even if we discard the idea that they should be interested in the stability of the platform for its own sake, there is an even more motivating push:
If nobody can use the blockchain, they don't get paid.
As such, I think it's telling that you are not only asking for support for a series of seed nodes but as a witness.
I would actually be more interested in seeing a breakdown and analysis of what servers got saturated and why it was possible for api.steemit.com to be overwhelmed.
I reiterate, it is in the best interests of the current witnesses to want a solution to this problem as soon as possible, because nothing erodes trust like failure – and if the behavior of the system yesterday doesn't count as a failure, I'm not sure what would.
This invalidates your original point four where you state, "there is zero monetary incentive to buy, build, manage, and maintain the seed nodes which empower our network." Of course there's a significant monetary incentive to maintain the seed nodes. The witnesses and prominent whales have the greatest incentive to maintain the seed nodes, because without them, as you put it almost immediately before, "dude, where's my dough?"
If the folks who have a significant stake in the system and its stability can't be bothered to see to it, I'm not sure what tiny amount of support you're likely to get from a slice of minnows and plankton is supposed to do.
Wow when you break it down like that, it definitely hits harder! Remind me to be less modest in my next post. It's hard not to step on anyone's toes with this one considering the funding is expected to come from a space that is quite competitive. I definitely have to stand by you on the fact that contributions from those that stand to lose the most will go a long way in resolving instability in our networks. Thanks for your comment man it is extremely thoughtful indeed. Also as a side note, you can use steemit.chainchopper.com to see the availability of all of the Full rpc seed nodes as well as ours .
I'm not exactly known for being the most gentle of conversationalists. And I have a significant preference for being frank and upfront, which probably makes me a relatively hard-to-find entity around here.
If you are concerned about stepping on anyone's toes, you're never going to change the status quo. The toes are upholding the status quo, with all of the disturbing imagery that creates.
Arguably, you're taking the wrong tack with this. Rather than trying to create some parallel architecture for seed nodes, you should simply be campaigning to become a highly voted witness because you are interested in maintaining multiple paths and multiple locations for services, along with more seed nodes. After all, again, I maintain that doing such things is one of the core competencies of those who would like to be highly rated witnesses.
If that is insufficient to bring you to the top of the pile, then you know exactly how much the community values such things. After all, it's a lot easier to get a witness vote out of a large number of people than it is a continuing commitment for funding out of a large number of people – even if it does end up being the same thing in the long run.
If you can successfully make the case that witnesses have a responsibility to maintain the architecture of the blockchain because it's in their own best interests and can exemplify all of that, then you can actually make a difference to the situation. If you can't – better to know quickly and waste as little effort on the try as possible.
Fail fast, fail cheaply, after all.
That was very well put. With all due respect however, this isn't the first time that someone has stepped up to try and do something about node availability. The response was that more witnesses (top 50 albeit) started creating them and setting them up (or having their friends do it for them) which placed a sort of band-aid on the issue and also set them in the cross-hairs when things go awry. I'm sure several of the witnesses wouldn't mind supporting the initiative as well. One of them has actually voted for the @justinadams witness today (Thanks @ura-soul) and that's a big step. Hopefully other witnesses (especially top 50) and whales will do the same as that type of support will show that they do recognize the issue and they are willing to go above the call of duty. Not to mention PR never hurts anyone as many of them have their own business floating atop the blockchain currently so yeah plenty to lose . If we had to count losses yesterday, I wonder what they would look like on a pie-graph considering we had to deal with an already weakened sbd at the time when it happened? Maybe no one even noticed it and we can keep dreaming?
P.S. You should be my campaign manager btw lol. Such a way with words.
Unfortunately, I'm well aware of what has happened in the past regarding node availability.
Mind you, having more witnesses starting to create them and maintain them is what we want. There is no ultimate solution, especially when you're talking about denial of service attacks and distributed denial of service attacks – there's just too much computing power lying around doing nothing and being poorly secured.
What we really need is for interest in seed node traffic capacity to be something that someone rates, tracks, and develops a solid metric for – and makes sure that the witnesses in place know that they are being observed and graded on that particular axis among others.
There's a much bigger problem at play, in that even at some of the top levels the assumption is that any blockchain technology and in particular the steem blockchain is "sufficiently distributed" to largely ignore this kind of issue. In fact, there are regular posts about how the blockchain is nowhere near maximum capacity in transactions per hour and that everything is blue sky if everything goes to the moon.
We both know that's ludicrous. The problem is that it's almost impossible to prove that is ludicrous because there's just no interest in actually figuring out what makes for a good witness at a technical level, grading and rating that capacity, and communicating it to the people at large. Instead, being a witness is largely a political position, even though the top witnesses must actually provide part of the technical infrastructure that supports the blockchain.
There have been occasional nods toward the technical underpinnings, but by and large I'll bet that you couldn't ask the average user on Steemit what a witness actually does and get anything like an accurate answer. Worse, even as someone with the technical inclination to understand it and having spent a good chunk of time working it out – I'm not sure I could give you good answer.
Really taking a long view, this is deeply unsettling and if you're not unsettled by it (in the referring-to-the-audience sense of "you"), you need to become more aware of how things actually work on the blockchain.
I hesitate to even think about trying to quantify or qualify value losses yesterday, because there's just so much smokescreen and general crap maintained with a straight face when it comes to the steem blockchain that I just don't believe that I have enough understanding of the underlying numbers to make a stab at it. The already weak SBD may have actually been beneficial by reducing the magnitude of potential losses.
People noticed, but how much it'll matter in the long run? Very much up in the air.
(Believe me, you don't want me to be anyone's campaign manager. I have far too underdeveloped a sense of self-preservation for anyone to want to tie their political aspirations to my butt.)
It's so funny that you use words like "smokescreen" and "political positions" because these are things that I had to deal with while trying to decide how to get these nodes funded so that they could be a real possibility. The issues are indeed broad when it comes to the technology behind the blockchain, and there are also other issues that I'd like to address. major ones as a matter of fact. This is what i see needs to be rectified first however because none of the other things matter if the network isn't in a consistent stat. I have to ask though, If there were a bar or standardized guideline to be set defining what exactly a witness was and how one would qualify as being able to actually do it, then what type of oversight would be in place to make sure that those standards were upheld? For the most part, The guidelines set forth in the whitepaper, Especially the price quotation, is expected to be regulated among the top 20 witnesses +1. I started to raise a bit of noise about the accuracy of that, and even how to stabilize the feeds as others have been able to manage it, but most have not (@justinadams witness maintains a stable price i'll add - see steemian.info/witnesses). A few people actually agreed, but just the same, a few people held resistance. Something about me being new and trying to change things. . . I kindly responded with the steemit logo proudly displaying it's "Beta" signature... This thing is still in the testing phase. Tests are designed for a reason. If change isn't accepted then how will we ever move beyond beta and pass it?
There are a number of issues that exist within the steem blockchain which is working directly against everything that you should be focusing on in order to maintain a clean and overly connected network of people who want to see what each other are up to.
However, the system was established with a clear disdain for and active work against setting up a central authority who would be capable of determining whether you have made a good decision this day or a negative decision. The lack of a central authority, beyond that of the witnesses, whom they themselves don't understand but from which depends their success is the core, is the very life supplied of what everything's about.
The problem, if we must to put our finger on one in particular, is the fact that we, as users, already except that the beta is implicit. No way in or out exists except through the beta. They've called for access to the financial side of things far, far too quickly. All of this would probably be easier to deal with if people weren't already counting on it to give them part of their rent. That's why all sorts of cunning manipulations may be in the offing once the weather opens up
No one on the blockchain has the power or the authority to make a declaration of how things are going to be and the power to make it stick in fact, most of the serious ones have already planned how to get out, it just needs a certain impetus. In the absence of an authority to go to, you get to see the worst parts of human society work themselves back together from nothingness. Congratulations, it should be amazing.
Luckily it does stay amusing.
Being a Steem Witness with not much following is a big pain in the a**. Another way to help witnesses who are dedicated would be to temporarily give them SP delegation, so they can get a few blocks, pay for the servers and keep this system running. At present much of the power is in the top list's hands. Which is not bad but considering if these outages will recur, people will go eventually off from this platform...
Support your witnesses, not only the top ones, they have enough votes and are generating more income from witnessing. Push up those who are willing to take the witness job and enthusiastic about Steem.
That's exactly what we are trying to do here. Thanks very much for your comment. Though my upvote is not worth much, your comment was very heart felt. This Steemit Forever! initiative is not to produce money for myself , it is simply to help everyone on the network by contributing technology, skillset and security to the network that will help prevent outages like we had yesterday. It's purpose isn't to wrest power from other witnesses or threaten the hierarchy in any way. It's simply to put more nodes out there for us all to use that will be available no matter who is in top 20. So this is not an assault on witnesses. It's awareness to help people understand that they have a task now that they honestly probably don't want and shouldn't have yet alas it is there. Keep in mind, the seed nodes do not write blocks and do not generate rewards as a witness node can, but we need them and they have to be paid for so that's why this witness @justinadams was created. We cannot afford to sit on our asses with this issue. With alternative networks popping up every day, and the price of steem being threatened by the likes of google and facebook, resource allocation is the last thing we need to be concerned with. We just want to express ourselves like the awesome creators and authors that we are, but we cannot do that if the network that allows us to do so remains unhealthy. To be frank, I have already lost $1300 in developing this project because I believe in the power of the human spirit (still) and the ability of mankind to ultimately do the right thing. Hopefully the people that stand to lose the most when the site crashes or is unavailable, will be at the forefront of supporting this initiative. Thanks for using one of your 30 votes to support @justinadams btw!
@justinadams I'll support your initiative and I like your attitude. I just don't see what good any of it will do. But I will support it anyways. Just show me how to vote for you as witness.
@xsorbik I don't really blame "leaders" or "people who run things" because like you said, its all about having a following, right? I think that main issue is that human immaturity is getting epidemic. No human system can function properly without a mature human race. We don't have a mature human race. Compassion, kindness and empathy are considered as weaknesses. Ruthless sociopathic behavior is viewed as strong. And humanity has an addiction to fighting and division, not unity and cooperation. I've basically lost faith in the entire human race, Steemit is merely a microcosm. The only thing I can do is keep pointing out these truths even knowing fully well that not only will they get rejected by most people, but that its not exactly going to make me any friends or make people happy with me. But thats what trendy-ism is about, right? The illusion of approval. People pining for others to be happy with them and approve of them, driven by the fear of abandonment. Well, I don't really care about how many people approve of me or not. I just can't in good conscience deny that humanity is on a sinking ship. This issue here on Steemit is only but one of many proofs in this world of the fact that humanity, pending a collective change of mind, is kind of screwed. Not because it has to be, but thats what people have decided they want. The herd has spoken.
Thanks for sharing
Perhaps the competition for witnesses has only just begun.
I hope you are successful in your quest to get some more nodes up.
Voted you, though my vote is considerably smaller than many.
Thanks for your vote @geekpowered . Everyone should remember, the agenda of this witness is in favor of the entire community and not just witnesses or sharks or whales or dolphins (the cutest IMO). Voting for it does not negate your current witness votes in any way nor do they diminish your voting strength . This is not a bid for power or a campaign for politics, it's simply a recipe created to perform one main function. Strengthen and empower the integrity of this network's foundation by providing the necessary technology it needs to run securely and efficiently. I am a minnow just like most people on here. I do however possess the skill set needed to complete this task as I have already contributed 2 nodes + a witness server 100% on my own. With support, I believe this mission will succeed. Thanks again for your witness vote. EVERY VOTE COUNTS!
Well, anyone can say that they have good intentions. I have plenty of witness votes left. I voted you because I'm willing to trust you for now and give you a chance. A lot of the top 50 witnesses don't deserve to be there IMO. In the coming months, you'll need to prove more and more that you deserve to be there. If you can, you will get there, as there are plenty of spots currently filled by people that shouldn't be there.
Good luck.
I'm up for the challenge. I absolutely never make promises that I do not feel I can keep. One thing for certain is that I'm not the guy who says "Yes, I can do it!" and then types in google and youtube tutorials. The things I propose are definitely within my right lane.
Have you written anything about how to run a node? Might be interesting if you did.
I don't plan to in the immediate future, but perhaps in time.
Well, you have my vote and my upvote, but other than that I don't have a lot of equity here. I am behind anyone who is here to help for the good of the platform because it seems that there are enough people here for themselves.
Thanks
Thank you very much. Every vote counts so don't be discouraged. It's amazing to see people come together and do something positive. Bringing things into existence that were not there before is a power that we all possess. It's what separates us from the rocks and the trees :P But jokes aside, It shows that we are loyal to this community and dedicated beyond just using it as a source of income, And THAT is what this is all about. Thanks for your vote
You got a 5.16% upvote from @postpromoter courtesy of @justinadams!
Want to promote your posts too? Check out the Steem Bot Tracker website for more info. If you would like to support the development of @postpromoter and the bot tracker please vote for @yabapmatt for witness!
Congratulations! This post has been upvoted from the communal account, @minnowsupport, by chainchopper from the Minnow Support Project. It's a witness project run by aggroed, ausbitbank, teamsteem, theprophet0, someguy123, neoxian, followbtcnews, and netuoso. The goal is to help Steemit grow by supporting Minnows. Please find us at the Peace, Abundance, and Liberty Network (PALnet) Discord Channel. It's a completely public and open space to all members of the Steemit community who voluntarily choose to be there.
If you would like to delegate to the Minnow Support Project you can do so by clicking on the following links: 50SP, 100SP, 250SP, 500SP, 1000SP, 5000SP.
Be sure to leave at least 50SP undelegated on your account.
8.78% @pushup from @justinadams
Congratulations @justinadams, this post is the second most rewarded post (based on pending payouts) in the last 12 hours written by a Newbie account holder (accounts that hold between 0.01 and 0.1 Mega Vests). The total number of posts by newbie account holders during this period was 3300 and the total pending payments to posts in this category was $2026.90. To see the full list of highest paid posts across all accounts categories, click here.
If you do not wish to receive these messages in future, please reply stop to this comment.
nice your post
You got a 74.81% upvote from @upmewhale courtesy of @justinadams! Earn 90% daily earning payout by delegating SP to @upmewhale.
Great article @justinadams! Followed!
Can you tell me something....is this @byresteem service actually worth the cost or is it just another ploy to squeeze money out of people?
I would like to talk to you one on one if at all possible my friend. Are you on Discord? LuckyDev#1969
I am on discord chainchopper#7547
Haha.
Good..
I hope direction from you..
Because i'm new a steemit.
good advice
thxnfor the upvote love ur posts
please like my pictures
Nice
I voted for you. I like that you're doing something different and supporting your voters in return. Reciprocity goes a long way. I also followed chainchoppers.
I was just reading how someone was running a seed node on Amazon Web Services fairly inexpensively. Now I'm curious what it would cost to set up a witness node. I read somewhere about $100/month. Is that about accurate today as well? So all total for both about $150/month?
@justinadams - Great post, and incredible project. You have my vote[s] already but I really hope this goes far, and you pick up steem ( :P ). I'm sure there is a lot on your plate already, but I hope you can write posts more frequently, even if short, so I can resteem your posts.
Thanks for putting in all this hard work and let me know if there is any way I can help.