Hi everyone,
In the past few days it should've been made clear to everyone that the Steem blockchain can't continue as it is right now. Things have to change!
In a worst case scenario, if Steemit Inc completely fails (which should only be an edge case scenario), most applications on this chain will go down with it, due to the fact that almost all apps use Steemit Inc's public rpc node cluster, https://api.steemit.com. So while we may have decentralised apps, they run off a single point of infrastructure failure.
This alone should show how centralized and dependent this blockchain still is and if we don't act to change this, $0.32 Steem will certainly not be the bottom.
But, in this case, we can't make Steemit Inc or Ned out to be the culprit because in truth, it is us, the community, which so heavily relies on Steemit's services and delegations. We have become complacent and reliant on the free infrastructure handouts at the expense of Steemit Inc which could rather repurpose those funds to something more beneficial like developing SMT's which are now delayed. In this below timestamped livestream video, Ned Scott the CEO of Steemit Inc debates whether they should continue absorbing these costs on behalf of third-party developers, so the longevity of the Steemit nodes could be in question and we as a community need to start proactively debating alternatives.
What BuildTeam offers:
BuildTeam is aware of this responsibility and thus, Minnowbooster as well as our other BuildTeam services (The Delegation Leasing market, Steemvoter, GINAbot, Quicksteem) have been running for a long time using our own private RPC node cluster in a self sufficient and community responsible manner; which means that even in the case of a catastrophic failure of Steemit Inc's infrastructure, our services will continue to run.
Up until just a few months ago we even offered nodes for the general public which we had to decommission due to the increased cost involved with running it and lack of delegation from whales who chose to delegate to other bot services that are not as conscientious.
What has to be done:
That's why now is the time that those, who most benefit from the Steem blockchain environment, not only extract wealth from Steem but also contribute to the decentralization and stop relying on Steemit public nodes. This will not only help Steemit inc to decrease costs but also help distributing the load and decentralizing the chain.
Let us help you help Steem:
BuildTeam would like to offer all interested app service owners an open door to discuss this topic and use our experience managing these servers to formulate a community coalition rpc node solution or a managed solution that we can likely provide if there is enough interest. We also welcome any other technical questions you may have regarding nodes, and invite you to use this post as a platform for knowledge sharing.
We invite any interested parties to contact us in comment on this post or join our Discord channel and contact @thecryptodrive in DM.
Thank You for your attention!
Stock photo credits: pixabay.com
BuildTeam is one of the first among a small number of service providers on Steem that runs and maintains its own RPC servers. Maintaining this infrastructure is expensive. If you want to support the growth of Steem network outside of Steemit.com, consider investing in one of our services, this helps to pay the bills, keeps the machines running and helps us incubate great developer talent within BuildTeam. The easiest and most effective way to support BuildTeam is to delegate to Minnowbooster or enable vote-selling within the https://minnowbooster.net user settings. For other options to help BuildTeam secure and grow the Steem community, contact Steem witness @thecryptodrive.
 |  |
We are always looking to improve @minnowbooster and try to keep you up-to-date whenever something changes. If you have any suggestions about what we should change or add to @minnowbooster, then please drop by our Discord server and let us know!
@thecryptodrive (witness), @cryptomancer and @reggaemuffin (witness)
steemit is our home @minnowbooster is our father xD
lol :)
Resteemed. Thanks for helping direct the conservation in productive ways. There is no growth without challenges.
Great discussions here on themes that we will surely be reading more about... These are the times that I wish I had more technical knowledge to help engage with the community to support the future of the ecosystem. I look forward to continue supporting this ecosystem and appreciate when leaders like Buildteam spearhead the thought process.
Posted using Partiko iOS
I can certainly appreciate the costs associated with all of these processes we take for granted that come for free. What I would hope for but don't know about is that when times are great, is Steemit Inc. investing their money in ways that benefit the long term success of the platform or are they unfortunately squandering it. I know some people in business which are far smaller and less complex but they are not investing and saving for when things get difficult.
The BuildTeam I am not familiar with but I do appreciate its willingness to help out the users on here. I am still new to the minnowbooster project but enjoy the concept and goals of it. We can only hope that the price of Steem is not so tied with the price of other crypto in the downturn. With how good we are at linking apps to the chain, I would think that our prices would be independent of the price of other coins but I find it strange that it goes up with it and down with it while some other coins are maintaining their prices or gaining small amounts.
been with u guys since ages, good team
I definitely think it would be great if there were a managed node solution where projects which need node use but can't justify a full node could lease some portion of one. I'm not sure I'm very much the market for that right now, unless it came in very small chunks, but yaneverknow.
What would you define as "small chunks" @tcpolymath?
I don't know, I'm not even sure how to count my API reads. On public nodes right now I run two autovote streamers, one bot that reads two dozen or so recent account histories every 50 minutes, one bot that does 7-21 reads every half hour and occasionally votes, and one bot that makes maybe a hundred custom jsons per day. So very much a minor hobbyist level.
I don't really have a sense of how expensive streaming is, though, since I've always gotten it for free. What does it actually cost on a node level to run a single basic autovote streamer like this one? I haven't the slightest idea.
I try to be a good citizen and limit my reads, but I'm sure I'd be better at that if I were paying for them.
I could imagine node-as-a-service providers offering a free tier with limited capacity (like AWS) or low cost plan for low volume usage.
Likewise some services (anything requiring the follower and/or tags plugins) cost a lot more than others and would have to be priced higher.
All of this can get worked out if we have a sensible and sustainable market.
Agreed, still waiting to hear from anyone willing to pay for such a service, no takers so far. Maybe a community blacklist showing bots and apps revenues and whether they use own rpc servers or not will start getting action.
Good idea we need something like Karma points high score free use low score pay more.
Anyone running these services should also be accepting brave payments as I'm sure many users have some free tokens to use.
The one that does 100 custom jsons per day is very hobbyist as you say and the Steemit nodes won't even feel it. Regarding the streamer bot, I'm not sure of bandwidth/resource use so I am mentioning @gtg and @drakos in this comment for insight. If anyone else has an effective way of testing the load an app uses I would be interested to know, we are currently looking at bandwidth use monitoring solutions, but don't have anything concrete yet.
I'm running all of these from home and it has no noticeable effect on my residential bandwidth, so I'm sure from a node perspective it's collectively almost nothing. But what tiny amount of resources I'm using is something I'd be very interested to know, if only to satisfy my own curiosity.
I think it comes down to more the server cpu load and tps more than how many GB per month it uses.
That’s far from accurate. You can make queries that take very little network but hammer the shit out of a node.
I don't disbelieve you, but I have no idea how to characterize what such things would be.
It would probably be useful if some of you who have an idea how this works would document it for the people out here who are using the nodes but don't. Right now we're pretty much doing whatever we want because we don't know any better.
Shouldn't it be the task of the witnesses to provied this infrastucture, since they get paid for it?
No (somewhat complicated question but that's really the answer).
Anyway, as a practical matter, at current or potentially even lower STEEM prices in the future, there isn't enough revenue for witnesses to do it. All witness rewards added together are only about $600K right now which is far less than what Steemit Inc. is spending on its node cluster. That is ALL witness revenue, and no it can't all be used to provide API nodes.
There needs to be a better business model that involves actual revenue to support the servers. Providing unlimited usage for free doesn't scale and isn't sustainable whether witnesses or Steemit Inc is doing it.
When steem was new (price=few cents), how witnesses were managing costs?
Posted using Partiko Android
Witness rewards were about 5x higher pre-HF16 as I noted elsewhere. Costs were also much lower (smaller blockchain), and the initial witnesses were all early adopter whales who didn't really need the rewards. In case you don't know power down used to take two years so the rewards (which have always been 100% SP) weren't even all that accessible to cover costs (plus for a while, no exchanges). Witnesses covered costs out of pocket, though again the costs were much lower (any cheap server would suffice).
Ok I didn't knew they had so many nodes
Posted using Partiko Android
@smooth drops mic :) booya!
Difficult question, a High-level witness should already be running decent witness clusters with redundancies which come at a decent price attached already. I believe the people who should be running these are those who mainly use these APIs as well and benefit from them.
The problem is that you can't provide a free application if you have to pay such a node, so the solutions would be:
Posted using Partiko Android
as suggested in the livestream, idea would be to focus on "fixing" the software as to so that running full nodes would not have such high RAM requirements, drastically lowering node expenditures. IMHO this is truly a way forward. With bitshares there is a "node selection" setting based on latency. This could be a way forward as more nodes become available and there would be a static "discovery file" for those nodes (hosted on github) which could be integrated in third party apps or proxies. In Steemit.com / settings one could manually pick a node, or get the node with lowest latency automatically connected.
https://wallet.bitshares.org/#/settings/access
good one - sounds feasible and a realistic outlook
I mean early on you can rely on small free nodes which for example witnesses or steemit inc could make available. But as soon as you move bigger amounts of data you would be forced to switch (Like it is with any other system as well). On other blockchains you have to send gas or crypto with every action you do on the chain.
I agree, the problem would then be how you decide if a project is now to big.
For the seconde point, isn't it the selling argument for Steem: Free and Fast Transactions
Posted using Partiko Android
Yes, but not unlimited free and fast transactions.
Witnesses generally provide block producing/validating node services, seed nodes and backups thereof as well as governance participation and decision making at chain level. RPC nodes are much more specialised and costly, only a handful of witnesses can run full nodes.
As far as I know every top 20 should earn enough to Run one
Posted using Partiko Android
They should also review the code.
Test changes
Make pull requests
Delegate all their stake to communities
Fight spam
Running $1,000+ /m full node
$500-1,000+ witness cluster
Test on test net
Provide witness server for test net
Be available 24-7 for outages
Do all nighters for hard and soft forks
Answer witness surveys
All for $36,500/year (which isn’t even liquid) as a top 20 witness. I’m #30 and make 1/8th that.
Just to put in perspective, a blockchain dev that is capable of reviewing code and making patches is worth $100,000-$250,000/year if all he did was code.
This reward is only so low because of the price, they made over 500k some time ago when the price was high.
Posted using Partiko Android
It's not only about earning enough for the hardware, it is about the admin overhead to do it. It also sets a bad precedent that is not scalable, Steemit Inc has many times more stake than the witnesses have and they are struggling, one should not increase witness overhead so that bidbots can be more profitable off of free infrastructure. Basically witnesses provide core blockchain functions, much like local town councils provide basic amenities communities require, such as roads, refuse removal, town planning, bylaws etc, but they don't provide electricity for free to businesses and households, which wouldn't be sustainable.
For bidbot this is true they can pay such servers, but there are a lot of free services here that rely on this free infrastrustur
Posted using Partiko Android
Agreed, the issue is that when you provide a public rpc node there is no easy way to identify and limit or exclude users, it's kind of all or nothing. The minute you make a node public all the freeloaders pile on and the node is bogged down to a crawl and the only recourse you have is to constantly tweak rate limiting or add more nodes to the cluster as Steemit does.
The key is to educate both the services and the consumer to switch buying habits, ie to not use services that freeload off of public infra when they can afford it, much like you would not purchase products where animals have been harmed in testing or products that have non-recyclable packaging.
I wonder if app operators, especially those benefitting stake delegations would consider beneficiary for node access.
And how that would work out financially. Because the “benevolent” model doesn’t work freely. Inflationpool is of course an issue in a beneficiary model but would definitely be worth doing some math on to check the finances.
I am not an expert in this so I will believe what you said, but this reeducation is not that simple, you see it IRL, not everyone is buying the bio Or fair trade stuff.
Posted using Partiko Android
There is no such thing as free.
But are these services economically sustainable then?
Shouldn't they develop a business model which allows this?
Advertisements, Freemium, Premium models?
In my point of view (as a customer) I prefer a service that is down 3 days in the year then a service I have to pay for. And so the people will always go to the cheapest one.
Posted using Partiko Android
So then, in that model, you get a total of 20 nodes. That's not enough.
Is busy.org independent?
Could you guys possible offer a website interface if Steemit goes under?
That‘s a great offer for other app services and of course us as a whole community!
May I ask what the running costs for a node are? I have absolutely no idea and am curious.
Posted using Partiko iOS
It can vary depending on the app and what the overhead is, a very small app which doesn't need resource/RAM heavy plugins could run a server for $100 per month, while apps requiring expensive follow and Tag plugins along with full history will need RAM intensive nodes which could start at around $750 pm. This doesn't however cover the cost and complexity to run the nodes from an admin perspective which is often more than the hardware cost itself, also doesn't give redundancy/failover so you have to double that for redundancy, which is important during Hardforks when you will need two servers so you can upgrade and replay one while the other is still running. A managed solution with usage based billing would benefit the community, so you only pay for what you need essentially.
@minnowbooster you guys are great. Thanks for all you do!!!
Resteemed.
One thing that I keep thinking, over the many months on Steeem, is that using crypto earnings to fund real world expenses can be a problem. Was great when STEEM was $8, but not so great now.
The most obvious solution is to allow advertising, but this was generally shot down as being against the spirit of decentralisation and independence. We already have a form of advertising in post promotions. I have blogged on websites whose whole income came from advertising, and it can be integrated in such a way as to not look intrusive or ugly.
I feel this topic needs revisiting, especially as it would require some coding to insert ads into a platform.
And what are the chances of Steemit Inc getting bankrupt?
Greetings mate of steemit! Excellent post, full originality and creativity. Total support from Venezuela. Many blessings and successes!
i love @minnowbooster . i havn't been as active on here but always enjoyedyour service
Congratulations @minnowbooster! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :
Click here to view your Board of Honor
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Is there an economic incentive for setting this node infrastructure?
Resteemed! Hope many take you up on your offer!
Posted using Partiko iOS
Congratulations @minnowbooster!
Your post was mentioned in the Steemit Hit Parade in the following categories:
I wouldn't say only one.
Thank you!
Huh? What am I missing? Why will Steemit Inc. fail in the worst case scenario? Huh? 😱 Is this abt the cryptos going down?
Posted using Partiko Android
I run @we-are that streams the IRREVOCABLE blocks as well as making a few API calls to update state, claim rewards, claim accounts, respond to memo commands and upvote posts. I also have a clerk bot that makes API calls to get status of account and then transfer it about. I recognise that this takes a lot of resource and I'd be willing to to a provider that could guarantee an upto date blockstream (privex load balancer gradually drifted behind). @we-are is charity, but I don't expect anything for free.
@thecryptodrive - give me a ballpark figure and I'll see if I can budget for it. Heck, even better if you can tell me how many BUILDTEAM tokens I need to stake in the name of my bot to cover the costs wink
Bookmark to read later