The top 20 witnesses are getting around 1 block per round (about 1 minute), as usual, but the schedule of block production for smaller witnesses is messed up, lower rank witnesses (outside top 20) have not received blocks for around 24 hours already.
Before the fork
I can say for myself, before the Hardfork 26 I was getting around 2 or 3 blocks a day, but since then me and other witnesses around my level are not getting any blocks assigned for production. That is, we are not producing blocks, but we are not missing them either because we are not being assigned any, the blockchain is not asking us to produce blocks, only the ones are being allowed to produce.
The top 20 are getting their blocks regularly, once every 60 seconds
In my almost 2 years as a witness I have only missed 16 blocks during maintenance, and the 16th block I have missed was a few monts ago, so I did not miss any, but did not produce any either, because the blockchain is not assigning me (nor other low ranks!) blocks!
Low rank witnesses, like me, have not been assigned a block since the hardfork
We are not missing blocks though, some low rank witnesses have missed none in their life time, yet, it has been about 24 hours without being assigned blocks!
I remember that on the last hardfork during 1 week there was a bug that allowed bots to snipe the post rewards during the first week. This hardfork is depriving low ranking witnesses and not assigning us blocks.
I am hoping this gets fixed. I feel bad and I am worried about that since the hardfork. My witness node never game me profit, it costs $100 a month but the return is about $60 or $70 a month (which I don't cash out, I leave it powered up), so I lose a couple dozen dollars a month, which I consider a risky investment, but now without being assigned blocks I am losing 100% of the infrastructure investment because I am getting no blocks to produce!
Emotional damage
But that is not the worst part, small witnesses are used to producing blocks and supporting the blockchain at a financial loss, we don't get many blocks and we need to invest in infrastructure. So the worst part is the worrying about the hardfork, worrying if our node is running correctly, it is not knowing if we messed up, if our node is running properly and we are being good witnesses.
My last block before the fork
The last block I was assigned before the fork can be seen on the blockchain:
I am monitoring anxiously, but seems like me and other low rank are not being able to produce anymore after the fork!
Hopefully you’ll get some blocks soon, need to cover at least your server costs
It never covered the costs, but it would be nice to turn the server costs into Hive Power, because when we produce blocks we get HP. I would consider that running the server would be actually me buying HIVE
It is a known issue and devs and top witnesses are looking into it. The fix will be issued at some point soon (hopefully).
From what I know is that there is a problem with scheduler where it "randomly" picks one backup witness and stuck it into a production loop, while everyone else is almost being ignored. It is not dependent on the witness rank (we see witnesses from rank (20-30) to a rare case of rank 100-ish being stuck in a loop, and doesn't pose any security risk.
Good eye for spotting this one!
So it possibly will require another node update after a fix is found?
Yes, but I don't think there will be need to replay the chain (I might be wrong tho). :)
If you want to accompany the development, here is the link.
Unfortunately it seems that the developers haven't seen this yet. The closest I've seen of this issue being discussed in in this post.
They did see, I know a few devs. I talked to them and they are working on it, but I don't have much more information than just that they are working on it.
Nice to know. I don't interact much with people, just read their Gitlab Boards.
Yep... this seems to be the case
You are way above me on the witness rank and I saw it is happening to you too! That is quite distressing, isn't it?
While it is not a pleasant personal experiance, I just try to keep it cool, especialy since I'm quite new in the game. The big picture and a main question is this temporary or not. Becouse if it is not, than it is an overall big disadvantage for the whole system, and a risk, becouse the tail of backup witnessses will be cut off, no matter who is in those spots.
Overall I expect the issue to be adressed sooner or later.
It's being addressed as we speak.
Great!
good to know - I consider building a node and this would be a big bummer
Yep.. wait until it's fixed. We are NOT producing blocks from 12 days!!!
okay thanks for the update!
Interesting.
I'm sure that it will be fixed once the main Devs know about it but it's great to have the smaller witnesses like yourself putting in the time and money to help the ecosystem.
Keep up the good work and in time the price of hive will make it worth the effort.
Just hope that the fix doesn't require another hardfork
I hope you will get some blocks soon. We appreciate your work on Hive.
Thanks. I hope so too, at first I thought it was something wrong with my server, maybe I had misconfigured the node after the maintenance and the fork, but after looking at other low rank witnesses they seem to also not have been not missing but also not producing any blocks since the hardfork, so it feels like we are simply being left out of the blockchain, apparently
I hope this gets resolved. No one will make a node if there isn't any blocks for lower-down witnesses.
I hope things to get to normal and you get blocks as usual or even more.
Yeah, while this is not fixed Hive is getting a little bit more centralized since only the ones at the very top are being able to keep producing blocks
That's not actually correct. The bug is just causing non-top 20 witnesses to frequently be scheduled more than once after they get scheduled. So the same overall rewards are being handed out to non-top 20, just not as uniformly at a particular point in time. The issue has been discussed and analyzed in more depth in the witnesses channel on mattermost, probably you should join there for more up-to-date info as we work on and test the fix.
Where do I join it? I would like to join to follow more closely
It's run by @guiltyparties so contact him.
Hope it will get fixed soon :)
I have voted for you before the fork. I not sure if I need to revote. Or everything is still normal !
I think it doesn't affact votes. Thank you for the support by the way!
You're welcome, thanks for reply !
Look at the virtual time scheduled. There were not that many decimal places different between the first runner witnesses and the rest. Usually there were only a couple zeroes more, not that much.
https://hivetasks.com/witness-schedule
I did not know about problems and costs small witnesses have. For decentralization it would be of utter importance to keep all witnesses no matter how small they are up and running at all times. There should be a fair witness proposal in DHF covering the costs of all running witnesses. (Just an idea this might piss of a bunch of people I don't really know if its stupid)
I will vote your witness! I will have a look at all my votes and reshuffle them a bit.
Greetings from Vienna, see you at next HIVEFEST the latest.
I don't mind not turning my witness node into a profit, there would be too much incentive for abuse if every witness were profitable, what is hurting is my feelings, I used to produce 3 blocks a day, now it has been 3 days and I have not yet been allowed to produce a single block since hardfork 26
You are not alone... even worse we still have the 1.25 tag displyed on peakd even if i upgraded the witness because we did't sign any blocks... 😔
It is a big deal that only affects people without a voice, like us, small witnesses. The ones at the top are printing money like never before because of this
Has the situation changed already? Just wanted to follow up. Did you produce a block since then?
Reading rapidly thought it seems a known issue that will be fixed soon!
Me too, it was weird that I didn't get a block reward, but your post was helpful. Hope it gets fixed soon.😀
I hope it's fixed soon, thanks so much for keeping your node running :)
This post has been manually curated by the VYB curation project
@igormuba I don't really understand much my friend but what I can observe is that you are in the sixteenth position and that's great because you are in the top 20 and that's very good I congratulate you.
I am in position 90, nowhere near 16 and nowhere near the top 20
Yes, I have already checked and I apologize, but with a lot of hard work and help you will be very close.