steem-python
Users of the Python Steem library should upgrade to 0.18.93 as soon as possible. This version comes with new fixes, one of which prevents undefined behavior when all steemd nodes fail. It will now throw a SteemdNoResponse exception after exhausting automated failover and all of 200 retries (10 node failovers, 20 retries per node).
SteemData
I have been receiving a fair amount of complaints about SteemData state falling behind.
There are two ways to remedy this issue:
1.) Write a high performance http -> websocket proxy [1], as well as perform internal
state reconstruction to avoid querying steemd.
2.) Deploy a dedicated, self-healing steemd cluster.
Needless to say, both solutions are time and resource intensive, and unfortunately, I am not
abundant in either right now.
Big thanks to @gtg for letting me use his personal steemd node(s), which, when online, keep SteemData in sync.
Call for help
Full rpc [2] steemd nodes are a challenge to run. Currently, they require expensive servers with 128GB of RAM, and regular baby-sitting. To my knowledge, there are no open solutions for managing steemd clusters .
As rpc nodes became harder and more expensive to run, community powered nodes disappeared.
Aside from @gtg's node, I am not aware of any public nodes. As far as I know, two of the best community developers, @jesta and @good-karma run their own. Steemit's nodes are generally a bit slow, and occasionally unreliable [3].
If we'd like the Steem ecosystem to support third party apps and developers, we need better infrastructure.
We need to create a node cluster that is:
- worldwide distributed, with geo-location based load balancing
- scalable, openly monitored, and self-healing (trough automated remediation workflows)
- capable of running on all servers, and not locked down to a single vendor like AWS
There many talented people in our community, and I hope we can tackle this challenge.
[1] Steemit is already working on one, jussi
[2] Nodes with all plugins/api's enabled, serving at high throughput.
[3] Based on my experience with extended usage of their nodes for SteemData and Witness Tools.
https://rpc.steemliberator.com is a full RPC node ran and managed by yours truly.
So there is an issue after all with running a node? @elfspice was complaining about having problem with his.
It depends on what kind of node. Seed, wallet and witness configurations are generally working well.
The full rpc nodes however are expensive and somewhat troublesome to run.
@furion, your witness support would help me ensure I have the ability to keep my public full RPC node online indefinitely. I am going to pay out of pocket for whatever I don't raise on SteemIt, but something as simple as witness support can make a huge difference.
Btw, thanks for this kind of awareness post and thanks for SteemData.
So RPC node is just all these three combined. That doesn't sound too bad. But isn't running even one of the three getting increasingly harder as blockchain bloats? I hope Steemit Inc distributes funds towards this "issue of the tomorrow".
They're simply expensive; Full RPC nodes also work well, otherwise this site wouldn't be functioning.
Thank you very much for the update @furion. And great job on the decentralized video app for Steemit! Looking forward to using your service for uploading videos instead of having to rely on YouTube. Keep up the great work - as long as there are talented people like you in the community that are constantly improving it, Steemit will always be in great hands!
Thank you for supporting us at the early stage of our journey.
I would upvote this myself furion but i am trying to instead give out tokens for it.
Please email me your bts addy bro
Hey @fuzzyvest, would it be crazy to ask for your witness support so I can more easily pay for the full RPC node I have been running? Up to you kind sir.
You're very welcome @furion!
Hi @furion I've published a post about you, check it out if you can, thanks.
30 Best Steemit Bloggers Of The Day To Follow 3rd August 2017
https://steemit.com/steemit/@jzeek/30-best-steemit-bloggers-of-the-day-to-follow-3rd-august-2017
good jobs
So, we need a healthy and well organized community run network.
In my opinion, Steemit Inc. should have a top priority in establishing this network. With planning, support, providing resources and rewarding participants. Full package!
Ps: self-upvoted to push the comment up.
Which?
We already pay for and run a high-availability cluster of steemd nodes, at https://steemd.steemit.com.
What exactly are you proposing we do?
What I do miss are clear and precise instructions / documentation on how to set up and run a community node. Hardware resources needed, recommended, software needed, set up procedure, day to day operations needed. The n the why 's of running a node - benefits, not only financial.
Perhaps all this is already written and published somewhere?
All this to distribute the load from your cluster to the community.
Next, looking into the future - one word comes to mind - sharded.
If it really is a requirement to have 128 GB of RAM to run a node then that's way too much. Somehow the load should be lowered to come into a reasonable range. I am not telling you anything new, I guess.
OK, I realize that my suggestions are not very constructive. I can't propose an out of the box solution. I can only voice my concern. I believe that you are working on a long term solution to keep the platform functional and scalable.
Good luck!
I'm starting to understand the architecture more (I think), and am feeling more confident about the long-term future than I was, but here are some of my thoughts about what you could do:
a) Allow it to be run to provide only the last n days worth of vote, comment and transfer data. This is all I would need for running my services on it, and I expect many others could also manage with less historic data. A smaller index surely reduces the RAM requirements significantly.
b) Allow it to run in a 'ledger only' mode, where the index could be smaller (as it would exclude all other operations). This is almost all the exchanges would need I think.
I realise this will need some work on steemd, but over time, these increasingly demanding running requirements are centralising the network unnecessarily, making it more vulnerable, stifling community innovation and leading to doubts about scalability.
Thanks for your time.
They are working on several pieces of infrastructure to address these issues. I hope that the community can also help, and add some resilience to it.
Thanks for the update. I'll have a look at jussi.
For how long will 128GB be sufficient RAM for rpc steemd nodes?
I am currently watching vids on View.ly!!!!! Thanks for all your great work on it!
My concerns for Steemit continue to grow, so I hope much support comes your way so that you can address these issues.
resteemed.
Way to seek help it's awesome to see the call for the community to rally on this.
How openly dare we discuss this, for fear of being tarred with the FUD brush? I get the feeling like @elfspice that this may be a very serious situation, and it almost feels that if the community isn't strong enough to be able to be open about these issues, it is too weak to survive.
Your feelings are based on his FUD-spreading nonsense;
steemdruns fine. If it didn't, you wouldn't be seeing this comment.@gtg, @netuoso and @furion have been very helpful in allowing me to better understand the situation. Thanks.
It's not the community, the choice to show our metal is being taken away it seems by the politics at the top. I think the way Steemit.inc have behaved is very bad.
Couldn't agree more lol!
Yup, this :D
argh its already posted
Yes, I noticed that yesterday when trying to create a witness account. Kept getting errors until I restarted from scratch by installing steempy again and saw some updates.
So this applies to all the newbie witnesses too who are just setting up. thanks!
good job :)
Thank you very much for this article @furion ! Keep up the great work, as I see a lot of people like you in this community.
@murda
Hi @furion! You have just received a 0.5 SBD tipfrom @cardboard. Noice!
@tipU - super easy steemit tips and sharing profit. Click here if you wish to learn more.
By upvoting this comment you support the service - thanks!
thanks for sharing @furion
this is the useful post for steemian
keep steem on @furion
Excelent work for all Steemit comunity..thanks @furion to share it
nice post @furion
i like this post
Yes!!!! Thank you @furion
its a great work looking forward to it.
Really beautiful .. thanks for the information .. hopefully we can cooperate .. follow me @riansteem .. may be a friend
Thanks for sharing this important information.
thats explain it all.... thanks for sharing
great post,awesome work
That's so timely @furion and the update could not have come at a better time. Any good work needs to be appreciated. Kudos
voted is done :)
please vote for my post
https://steemit.com/cameras/@asmaasema/8-tips-for-ladies-to-protect-themselves-from-hidden-cameras
On October 10, a new exchange of crypto-currencies will be launched.
The main advantage - the commission is 10 times lower than that of competitors.
Also, 50 WCX tokens (~ $ 5) are issued free of charge to all registered users, for which dividends will be accrued monthly at a rate of 20% of the exchange's profits. Not a bad opportunity to participate.
Https://wcex.co/?ref=aLQ81Ap
thanks for informing us ..
keep posting like these post we are waiting for u, great post keep it upp broo...!!
Thank you very much!!!!
Thank you friend Ali for this update and wish more prosperity and development for steemit
Thank you for your work @furion. Tip! 0.5 noad
thanks for giving nice information
Do you have any suggestions for hosts that can dish out 128 GB while accepting reasonable Bitcoin payments? I haven't seen anything that's affordable and provides this sort of RAM.
👍👍👍👍
A very beautiful post @furion
I like it
Vote and follow me @ayuwandira
It's like 80% of comments are:
I'm part of them. ^u^
But kidding aside, i'm not tech savvy and not much knowledgable about what you are saying, but it deserves to be resteemed. As you have said:
"There many talented people in our community, and I hope we can tackle this challenge."
And to resteem your post so that many others can take heed of this call is the only way i can help.
More power to you!
And keep on Steemin'!
(____/)
( ͡ ͡° ͜ ʖ ͡ ͡°)
\╭☞ \╭☞
anything i can do to help?@furion
i like this
I read the post about problems with steem nodes , can you say aomething about the issue with exchanges ?
This ^
Is this contributing to the lack of exchanges?
I have no idea on the post i read was sayd exchanges like Poloniex and Bittrex have problems because of Steemit .
I agree that I want to know the answer to the question. I am told that the only exchange currently open to Steem is Blocktrades.
This is of great concern. Not to me, cuz I have neither much Steem nor any intention of powering down, but without exchanges Steemit will lose it's allure. I like Steemit, and don't want to lose it.
This isnt true Bittrex is working :)
Is working now.
It was not working for days possibly prior.
It has not been reliably working for a while.
Hopefully they have sorted it out though :)
Blocktrades is not an exchange.
LOL
I am outed as ignorant =p
I use no exchange, nor even a bank. Honest. Please forgive my lack of diligence regarding the name of the service. Perhaps it was Bitshares?
I plead ignorance of the correct name, due to my own complete (as possible. Money remains in my hands from time to time, as trading beaver pelts for food seems to have gone out of style) boycott of financial edifices, that has caused me to have limited exposure to such institutions.
My point was, that there was but one exchange that was trading Steem, at least for a while, and this concerned me.
I think that wasn't an incorrect statement.
Edit: abhorrent grammar
amazing post dear keep it up
I like it
good post
gracias por informar, hablare con mi supervisor para que actualice la biblioteca a su ultima versión, Saludos
This is welcome news for the steemit community, knowing that we have steem developers who are actively tackling the challenges being experienced as steem continues to expand and broaden its horizons, is a warm and welcome feeling, and also openly talking about these hurdles to surmount, and calling for help from all quarters to scale over is also a good sign of the transparency, openness and fairness to all users under which steemit operates.
I commend you @furion and other steemit developers... Keep up the good work. You guys are awesome!
STEEM ON !
Hola Furion, Fíjese en esta iniciativa y dime que te parece usar STEEMIT para estas cosas! Si esta bien o mal!
https://steemit.com/spanish/@rojassartorio/4x4uhc-ayudame-a-ayudar-a-esta-familia-que-lo-perdio-todo-solidaridad-leelo
Your post is very great @furion, and I am very interested to read it. Many benefits for the steemian of your post thanks for the post you share. I like your content.kamu a great writer. Please meet with mu.please help me to be great like you .my babyak lack in writing post.
Suppose I had a VPS mainly working as a webserver. Would that be able to run steemd? Could it impact the web traffic on the server?
Currently have a LOT of spare bandwidth on the server, run using less than half the ram, and only 51% diskspace used. I'm currently working to transition a lot of the sites to use steem based posts rather than a mysql database, so plan to free up a lot of diskspace also.
Wait, no, I'm going to answer my own question. Definitely nowhere near 128GB of RAM on that VPS. So 'No', I won't be running a steemd node anytime soon...
You can run a consensus node in just a few gigabytes of ram. There are several different modes.
The "full node" config required to support e.g. a website like this one requires a lot of indices on very fast storage (we use a ramdisk).
Useful to know. A consensus node might be more within my capability. Another thing to put on my 'to do' list. :)
Thank you for this very timely and illuminating post @furion
Allow me to thank you on behalf of @elfspice as well.
Thanks for the update. Thanks you @furion.
Thanks for the update