Run, boy, run! They’re trying to catch you.
In the previous episode of the Steem Pressure series, we checked how much time we needed to get a fully functional steem consensus node from scratch.
We also took a closer look at the steemd version v0.19.2 resync and replay performance for 20’000’000 blocks.
It was back in the pre-appbase era with the HardFork 19 rules.
Time flies, and so do blocks.
Recently, our chain has grown beyond 30’000’000 blocks, and this offers a good opportunity to look at the performance once again, especially as we have also seen many important changes in the code.
Video created for Steem Pressure series.
Time needed for replay
We won’t be measuring --resync performance this time, because we want to avoid the bias introduced by external factors such as network latency and the performance of 3rd party nodes.
We will focus on the --replay part, using block_log file, which we can get either from a previous --resync operation or we can download it from a another instance or public source.
If you want to find out more about the difference between --resync and --replay, feel free to read the previous episode.
Test setup
We have exactly the same machine as in the previous episodes:
An entry-level dedicated machine with Intel(R) Xeon(R) CPU E3-1245 V2 @ 3.40GHz on an Ivy Bridge with 32GB DDR3 1333MHz RAM and 3x 120GB SSD.
That’s not the configuration I use for my witness nodes, because as a consensus witness that takes part in every round, I’m not going to risk losing blocks sporadically due to high latency.
However, that is a perfect test configuration to check the performance of steemd for the purposes of public seed nodes, backup witnesses, state providers, private RPC endpoints for wallets, or even exchanges, as account_history plugin with just one or two tracked accounts has a negligible footprint on the performance.
But why?
If we have all the blocks downloaded to our disk, we have all the data about all transactions that were performed on the Steem blockchain. But at this point we still can’t say how much Steem Power Alice has or how many upvotes Bob’s most recent post about cute kittens received.
We can’t until we check (replay) every block since The Genesis.
That’s the only way to validate the current state of Steem.
Every three seconds, one of the witnesses signs a new block (head block) with new transactions making the whole process increasingly difficult.
Speed does matter
When you can’t replay fast enough to catch up with the current head block in a reasonable time, you are doomed.
Of course, you can rely on your state providers, snapshots, etc., but whenever you run out of reliable copies of the reasonably recent state (due to software or hardware failures), or whenever new features are introduced, you have no other choice but to replay.
When something unexpected happens
Back in September, after an unexpected event caused by a bug, we were forced to replay with the fixed code.
That’s where the ability to replay the whole blockchain from scratch in less than 4 hours and multiple high-end servers at your service pays off. Especially when you need to do that multiple times.
I started producing again at block 26038153 (one hell of a block!), soon after we got back to a fully operational status.
There were no transactions going on between 12:47:00 and 19:56:51 UTC.
And it could have been much worse.
Benchmarks
v0.19.2 - pre-appbase, 20M blocks, 2018-02-19
In the previous episode, we were able to replay 20M blocks in 145 minutes.
v0.19.12 - appbase, 25M blocks, 2018-08-12
There were a number of optimizations to improve reindex performance.
With appbase based steemd v0.19.12, we were able to replay 20M blocks in 121 minutes and reach the 25M block mark in less than 5 hours.
v0.20.9 - Current version, 30M blocks, 2019-02-02
The current version of steemd is even faster. Despite “Reconstructing Block Log Index..” itself takes over 6 minutes longer, we were able to replay 20M blocks in 115 minutes, reach the 25M block mark 25 minutes earlier than v0.19.12, and replay 30M blocks in less than 8 hours (7h38m)
Replay speed
Replay speed, measured at 100k block intervals never drops below 324 blocks per second, and the last 1M blocks replay at an average speed of 454 blocks per second.
Summary
| Replay | v0.19.2 | v0.19.12 | v0.20.9 |
|---|---|---|---|
| 20 M blocks | 145 min | 121 min | 115 min |
| 25 M blocks | n/a | 297 min | 272 min |
| 30 M blocks | n/a | n/a | 458 min |
Paying attention pays off
By paying attention to performance, we are able to identify and resolve issues that, if underestimated, may prove silent killers and endanger the reliability of our platform in the long run.
One of such incidents I detected during appbase development was a witness plugin that caused significant resync performance degradation
Upcoming improvements
We are nearing 50GB of state file size for a consensus node.
As you can see, with the current architecture, that is difficult for servers with “only” 32GB of RAM.
I’m looking forward to the MIRA release.
My preliminary benchmarks of MIRA branches are far from being optimistic, but Steemit Inc team updates say that MIRA jest under heavy development / optimization phase; a lot of effort is being put into making it replay in reasonable time, but the advantage would involve robust access to the state in low memory conditions.
Previous episodes of Steem Pressure series
Introducing: Steem Pressure #1
Steem Pressure #2 - Toys for Boys and Girls
Steem Pressure #3 - Steem Node 101
Steem Pressure: The Movie ;-)
Steem Pressure #4 - Need for Speed
Stay tuned for next episodes of Steem Pressure :-)
Bonus: Inspiration for the title
“Run Boy Run” - Woodkid, The Golden Age
If you believe I can be of value to Steem, please vote for me (gtg) as a witness on Steemit's Witnesses List or set (gtg) as a proxy that will vote for witnesses for you.
Your vote does matter!
You can contact me directly on steem.chat, as Gandalf
Steem On
@gtg, have you considered using auto voter instead of sitting on all that VP?Or perhaps delegation? I could really use one of the 2. I am a fulltime steemian mostly focused on dtube, steemhunt & steempress. I know we are not acquainted (yet), but would you please put me into consideration?
Sincerely, Elsie.
All I have is... wow
Is that a good or bad wow? I can't tell if you're being sarcastic.
Its a snarky non-comment. Prolly best to ignore.
Gtg is a top witness. The top 5 people he votes have a couple million SP together and SP is what moves you up the witness ladder.
The top 5 people you vote have 60k SP all together.
Dont let others call you selfish based on stats because stats can be explained in any what way.
Just like @gtg called you selfish i could say he generally votes high SP holders so they can vote him up as a witness.
I could say: "Just wow" to @llfarms and say she mostly votes fellow echo chamber members..
Stats can be explained any way you want.
Too late lol
True. But this has escalated and diverted from what I intended for it to be. So much negative energy around here.
Hey @elsiekjay, sorry if my “wow” was seen as negativity. Like I said, good for you for having the self confidence to ask.. I guess possibly the way you asked came off differently than you may have intended. Maybe a lost in text sort of thing and I apologize if me saying wow was offensive.
As far as the rest here.. butterfly and I have a history where he likes to bash and insult me and I respond. To someone not knowing any of this, I guess it looks quite bad. 🙂
I believe the first time I commented on your post was to try to help you with the confusion with the Steem Alliance group, where I linked you some videos I thought may help and tried to correct a comment claiming it was “government”.. where I was essentially told to shut up.
So, again.. this is me stating I meant no offense.. I was just taken aback by the comment. Then butterfly came in to make things worse. In his defense, he was trying to “defend” a friend I guess, but i was not attacking. So I apologize for inability to ignore him.. it’s a bad habit of mine I’m working on. This was nothing against you personally.
Have a good one.
Thank You Justin for the clarity! I Appreciate
Have a great one too.
xo
Some people are very conflict prone and/or like to act holier then thou.
It s inevitable to run into those kind of folks.
Nah, it was a pretty straight forward wow. Glad you are here to defend the self entitlement though.. and my stats do speak for themselves.. even though I don’t do much curating lately 🙂
Is “echo chamber” something different than your circle jerk? I’m not sure of the terms you kids are using these days...
My circle jerk? Id love to have one. 😟But not with her. She only has a couple k SP. Thats boring. Maybe if i could get a whale to vote me.. Maybe a sexually unfullfillled psychotic one... That would be great. 😁
Im here to "defend" someone that wrote an honest question and does steem for a living. And youre here moralizing over someone that didnt do or say absolutely anything wrong.
Take some of that moralizing to the guy that doxes people and threatens to kill them...
Posted using Partiko Android
What whale votes do I get from a “sexually unfulfilled psychotic one” 🤔
There was nothing to defend here, your smart ass response actually made it worse, as usual.
Not sure what them doing Steem for a living has anything to do with anything? That’s awesome, I bet many wish they could do so as well.. what’s your point?
I said wow as I was surprised by the comment (I was tagged and it’s what drew my attention to it).. your rude and attacking response to trying to defend where there was no defending needed made me elaborate... so yeah, the entitlement of the comment was shocking.
Maybe Steeming for a living should be backed up with working for a living too? It’s what most of us do. Again, no idea why “steeming for a living” was your basis here for anything.
I said wow, I didn’t bash the girl 😉 and I’ve been moralizing those that actually do wrong all morning. What have you done for the Steem ecosystem lately?
You always seem to come in and really bring a lot to the conversation, great job.
Except that it would be a lie, that can be easily debunked by data written on the blockchain.
2/5 are not voting for me, including the most voted one which is a non-profit curation initiative.
That "couple million SP together" comes from one guy, or rather his proxy votes, but I don't think he cares about my votes much.
All those contribute a lot to the platform, and there are plenty more people that I should vote their content for if I would like to earn witness votes that way.
TL;DR: Voting for me as a witness is irrelevant.
(by the way, I just set wise.vote rules for @elsiekjay, feel free to help her finding relevant content)
I dont think you got my point. It doesnt matter if it is a lie or not. I could have drawn whatever conclusion i wanted from the stats. As i did... Maybe i try it again?
Maybe you vote them because you still havent secured their vote?
I know what you do for the platform, ive been here long enough..
My point is that you probably shouldnt pass judgement simply based on a few stats online..
When you do that as a top witness you attract attention seekers and generally toxic individuals that are looking for targets and everyone ends up having a bad time.
And im glad youre giving Elsie a chance. Shes one of the good ones. ;)
Is that a good or
Bad wow? I can't tell if you're
Being sarcastic.
- elsiekjay
I'm a bot. I detect haiku.
It was a wow at pure amazement directed at the attempt to come to someone’s blog and ask for them to put you on auto voter or give you delegation as you feel you deserve it. Good for you for your self confidence I guess. For the record.. there is nothing wrong with an account sitting at 100% SP.. and gtg is one of the only larger accounts that still manually curates.
Why are you so bitter?o.O
It seems like you like picking fights all over.
First of all, my comment wasn't direct at you. You could have easily ignored but here we are.
2nd of all, I find No Shame in Asking for help, It's actually up to Gtg to decide if I deserve it or not, either way, I'd be good with either outcome.
3rdly,
I never said there was.
Stay on your lane next time. TIA.
I said wow, you asked for me to explain.. so I did. I was tagged in the above exchange, which is what brought me here. I’m not bitter and there is no lane to stay in. Have a good day 🙂
Sorry I just came along this conversation as SWM was tagged as well.. Its really sad to see this kind of disrespectful conversations in our community.
I am with you @llfarms on this one.. you got tagged so of course you have an opinion on it getting driven to this comment. Just as I landed here somehow.
Of course it is brave to ask @gtg something like this but seeing his answer showing your voting behavior @elsiekjay I seriously doubt you know how to use it better then he does for the community.
Nevertheless I think it is great @gtg is even considering this and maybe you can prove us wrong and learn to be a better curator changing your voting behavior according to his rules, which would be a great outcome for everybody.
No reason to pick on @llfarms though. I dont know your guys history here and I really dont want to know, but I know she's done A LOT for this community working hard, so show a little respect for that.
Cheers, Liz
Wow, great thing! How it goes using wise with GTG VP?
Posted using Partiko Android
Damned, I got infected by a Steem virus. Sorry to !sneeze at you 😪
Try to use fron-end agnostic links there, i.e. Instead of using
[King of Disease!](https://steempeak.com/@king-of-disease/beware-the-steem-plague-is-here)that explicitly takes user to steempeak, you can use:[King of Disease!](/@king-of-disease/beware-the-steem-plague-is-here)so users will stay where they are and most likely are logged in too.
King of disease is an initiative from @suesa, not me. Anyway, I will pass your good hint forward to her.
Good job! I like the way you feel about work. Unfortunately, I didn’t understand so much, because I don’t understand much, and there are still problems with English)))
Good job, buddy !!!!!!!!
Ahahaha are you watching Umbrella Academy too? Pretty sure that track was used there
What is the umbrella academy? Been told by multiple people to watch it.
Netflix series. Weird one. Per @geekpowered review:
Not yet, we are watching third Fargo currently. :-)
@gtg, long time no see. 😊
Thank you for following and upvote!
I look forward to seeing you again.
Posted using Partiko iOS
I was just reviewing list of accounts that I tracked with my SteemFest3 app :-) Apparently I wasn't following many.
Fixing it now :-)
I can see that replay speed, as important as it may be now, is going to be even more so as we get into 100 million blocks, not to mention 1 billion, etc. I know it's a ways off at our current rate (I guess things would change if we had a significant increase in user activity), but is that something these current modifications are going to help with, especially when our rate of block creation is running at twice or thrice the speed it is now? So, basically, we'll have storage issues unless we can continue to compact things, and/or speed replay issues, right?
Also, I'm wondering, if I'm reading this correctly—in v. 0.20.9, there appears to be a difference of 157 minutes between replaying 20 million blocks and 25 million blocks, but it's up to 186 minutes between 25 to 30 million blocks. Did the blocks get heavier with more transactions between 25 and 30, or is there something else at play?
Content of blocks matters, both amount of transactions within a block and types of operations (there are differences between resources needed to process them, which is reflected in their Resource Credits costs)
Thanks for doing this @gtg - very cool to know that replays are faster than ever before.
Wondered what you meant when you said that the block was one hell of a block. Checked it out on steemd. WOAH, (https://steemd.com/b/26038153), 70K virtual ops.
Thank you @gtg for improving this great blockchain
Posted using Partiko Android
Enjoyed the song :)
Great fight by the way! :-)
Hello there, @gtg!
I've seen you around, helping some people who were unjustly attacked...
I'd like to ask for your assistance recovering this this new user, who was flagged for no reason whatsoever.
https://busy.org/@maxsieg/protecting-free-speech-on-steem
https://busy.org/@maxsieg/nazi-volkswagen-und-seine-heutigen-konzentrationslager
https://busy.org/@maxsieg/silencing-journalists
https://busy.org/@maxsieg/us-politics-is-victim-to-bad-journalism
He joined 3 months ago and he's already being attacked simply for reporting on sensitive topics. This way, Steem will be left behind.
Hello @trincowski :-)
Unfortunately I can't - physically can't - get involved in every story that goes around, especially that in order to make a fair judgement it would involve a lot of research for which I simply don't have time.
Quickly looking at first post of this user I see a lot of misinformation about Steem that I can't agree with.
Saying that flagged users are being robbed is absolutely ridiculous.
Stakeholders have every right to assign their reward shares as they like, whether it is up or down vote.
I really don't understand why people so easily accept "random" upvotes, and yet are so furious when it comes to "random" downvotes, even though downvotes barely exist platform wise.
And that is a lie.
Ability to post on the blockchain is one thing, and even a top witness could not block you from doing that. However, getting rewards is a completely different matter, and upvoting is all about getting rewards.
For instance, while I acknowledge that there are people who believe that the Earth is flat, and they are on Steem, I see no reason for upvoting them just because of diversity or because they were "falsely" flagged.
Upvotes and downvotes are subjective, and from the perspective of voter there's no such thing as "falsely".
Ok, but in that case, Steem is a perfect Troll Magnet for rich kids.
And if that's so... then, once next Bull Market comes, many of us who were investing in Steem for the long term will take the chance to get our money back and won't be investing another cent in this network. I know I will.
I was buying Steem every month and I stopped doing that 4 months ago because of these kindergarten fights.
If the top witnesses are OK with all the senseless bullying that goes around... then it's obvious to me that this platform has little to no chance of surviving, even with all the amazing Developers who are giving sweat and blood for this Blockchain.
It's not smart, from an investor point of view, to keep planting crops in a land full of moles.
If the moles stay... then I'll go elsewhere and I'll take my money with me. I'll look for more fertile ground. And I know for a fact I'm not alone on this. Many are simply afraid of speaking out. But you'll understand that, once you see the number of Power Downs that will be triggered once Steem goes a bit higher.
I came here thinking this would be a better place than Facebook or Twitter, and in many ways, it is... but if the Top Witnesses don't care about their investment, why should we? This starts to feel like a Ponzi Scheme to me, where a few whales lure in new users, get them to buy Steem and then destroy their accounts. What a disappointment.
Also, I was really working on my blog, trying to get as much quality as possible... but now, I'm not trying so hard. And I'll start trying even less. I'll just milk the rewards pool with the least effort possible, if that's what you Top Witnesses want.
Sorry for the long rant, but it pains me to invest so much money, time and effort in a place and see that the top players can't care less about what happens to it.
I'm NOT OK with bullying and you should already know that.
I simply disagree with that author's view on Steem and refuse to upvote it. Is that really a reason for this tirade directed at me or other witnesses?
Being a witness, is in this case, is not relevant at all.
I think this user having his entire blog flagged is a case of bullying.
He wasn't flagged because someone else disagreed with his content, he was flagged simply because people who are involved in a flag war resteemed his article... and a lot of authors these people resteem end up flagged, as collateral damage.
In fact, let me give you a few more examples of people who were flagged for the same reason:
To me, that is nothing but bullying.
Why do I direct this rant to Witnesses? Because I realise now that some of the witnesses are directly involved in this abuse of flags... and most of the others are seeing it happening for months now, and they say nothing or do nothing to stop it.
This is the reason all of these witnesses lost my vote recently:
@gtg I got notified of the mention here and would like to thank @trincowski for his consideration.
I believe the problem is serious with a small group of users, I think they're the same user with multiple accounts and no proper contribution of any value to the system here.
That user mentioned initially here whom you checked his post and disapprove his views, I believe he wrote so out of frustration.
Here's an example of what happened when I got flagged by the same user bot with his multiple accounts because that user targets someone who resteemed my resteem of @ahmedsy post which got flagged then when I wrote a separate post about it, my post got flagged by the same user's different accounts. You can see the other users getting flagged for whatever they write evn their comments get flagged by the bots users - main user name I guess is 'berniesanders'.
I had to ask that user who resteemed my post to not resteem any of my posts because he's under attack by the bots.
Here's again the link: https://steemit.com/informationwar/@arabisouri/how-low-someone-should-be-to-downvote-this-post-dsxow2az
Posted using Partiko Android
Thank you, yes that's exactly what happened and I don't even know what those people wanted until I saw your post you quoted above about my post which got flagged.
Posted using Partiko Android
This post has been included in the latest edition of SoS Daily News - a digest of all the latest news on the Steem blockchain.
I like your post,steemit noline income is very good line.
please 11 dolr give me .
@gtg, long time no see. 😊
Thank you for following and upvote!
I look forward to seeing you again.
Posted using Partiko iOS
Dear gtg:
We are SteemBet, the next generation STEEM based gaming platform. We are honored to invite you to join our first fantastic dice game, which is just the beginning of SteemBet game series. Our dividend system has now launched. The prize pool has already accumulated 2,000 STEEM and more than 60 players have participated in staking mining token SBT. A huge reward of 40,000 STEEM is awaiting! Join us NOW with other 500 STEEM users to loot HUGE dividend reward!!
SteemBet Team
Official Website
https://steem-bet.com
Discord Server
https://discord.gg/95cBN3W
Telegram Group
https://t.me/steembet
Thanks and I've just voted for you :)
You have been infected by the King of Disease!
Will you quarantine yourself?
Or will you spread the plague?
Congratulations @gtg! You received a personal award!
Congratulations! You've been among the most infectious people during the April Fools' Steem Plague in 2019! organized by @suesa
You can view your badges on your Steem Board and compare to others on the Steem Ranking
Vote for @Steemitboard as a witness to get one more award and increased upvotes!
Sir give me 100% upvote in my post to helping homeless children
Posted using Partiko Messaging
What do you mean exactly?
How upvoting your post could possibly help homeless children in Bangladesh?
Why don't you bring some value here, instead, get rewarded for that, and then help them?
ProTip: posting copied content doesn't bring any value to the platform.
Thanks a lot for this report, I also wrote a report about how I've reached the reputation 70. I included you as well because I'm voting for you as a witness all this time. Thanks for being so valuable to steem.
Interesting post!
A long term goal of mine is to set up a Witness server as well. lol, tho its a log ways off for me at this time... for not its just slow going. I do enjoy however reading about others who are Witness's and what they are doing.
Keep up the good work and I hope one day we can Witness together! YEA~
Excelente amigo, gracias por subir cosas como estas...