So yesterday I noticed that my @hextech node and the @town-hall node missed multiple blocks in a row, which is usually a sign that something is wrong and the operator has to go in there and tinker with it to get it working again. Best case scenario is usually just resetting it off/on like ya do but occasionally the fix is more annoying than that.
In this particular case the issue is systemic.
Multiple nodes on the Hive blockchain have been burned by this.
Looking at our witness list this is my guess as to which nodes are affected.
Based on having not minted a block in "one day":
- @hextech (me)
- @town-hall (@l337m45732, @anomadsoul, @taskmaster4450)
- @ura-soul
- @aliento
- @actifit (maybe?)
- @apshamilton
- @r0nd0n
I personally contacted three of these accounts to let them know what happened as they were not aware of the situation at that time. I shut down my node after missing something like three blocks... and then someone immediately unvoted my witness... which is super annoying because they probably wouldn't have if I had just left it on to continue missing blocks like everyone else :D... but I guess that's a whole other can of worms wrapped up in weird incentives.
What are the implications here?
I think we have to assume that someone was doing something illegal on the shared virtual server we were all using and it got pulled. These things happen. On top of that Privex is a privacy-centric hosting service which can be a magnet for criminal activity because privacy isn't the standard (even though it should be). For more context on this idea go read my post on the CyberBunker documentary.
As a reminder, in order to prevent this from happening in the future we will have to provide data to local authorities when requested, so use encryption.
I'm not 100% sure why a Hive node-runner would need to mask public data but this definitely is best practice especially for privacy-centric infrastructure and generically addressing all customers at the same time. At the same time when you run a server like this you have to obey both local and international laws and we all know how ridiculous that can get. As a noob I don't really know how to do this yet and I'll just be scrambling to get my node back online over the next however many days it takes.
Annoyance or Opportunity?
It's both really. This is the perfect time for me to upgrade my node to a dedicated server that has exponentially more resources. This is a really frustrating experience because I'm offline with my dick in my hand but at the same time it forces me to push forward and get more involved with the chain. I'm definitely going to need a refresher course to get this thing back up and running. Luckily I wrote a few posts on the process that I can look back to as a resource, in addition to various pinned Discord discussions.
Still profitable
Even at these low Hive prices and the increased cost of a dedicated server my node will still be profitable at the current rank #41 spot. Maybe I can even grab a few more votes by offering better infrastructure to the chain. Perhaps I'll go on a targeted campaign trail (aka begging rich people for upvotes, kek). We'll see.
I do have to mention that I have a lot of respect for those who aren't as fortunate as I am. Kudos to everyone running a Hive node in the red at this time. I can imagine it would be very hard to justify keeping it going even though it doesn't pay for itself.
Ultimate Future Goals:
I still very much would like to create a profitable business on Hive that revolves around gaming and potentially a connection to DEFI. It's almost embarrassing to mention a long-standing goal like this because I feel like I should have made it happen years ago but always seem to fall short in terms of motivation and access to resources. "I can't afford that until number go up," seems to be a real thing in crypto... and then when number goes up we ask, "But what if it goes up more," during the FOMO. There is definitely some personal psychology I need to resolve if I want to make progress going forward.
Decentralization in action.
It's also nice to realize that an event like this shows us just how impossible it would be to take down the Hive blockchain. The top 20 have an extreme financial incentive to keep their nodes running (most with backup contingencies). Hive would be exponentially harder to take down than Pirate Bay, and Pirate Bay is still up and kicking even with way less decentralization and incentive to keep it going.
Conclusion
Alright well this situation is not ideal but it is what it is. I probably should have switched to a dedicated server a while ago, so perhaps a blessing in disguise. I certainly don't envy Privex right now for having to deal with law-enforcement when their business model revolves around privacy.
These things happen.
Tick Tock Next Block.
Hi mate, it is a real pain in the butt to have to recover from this kind thing.
Before I started to build my node I spend a lot of time looking into the terms of these hosting companies and they all have the power to shut you down.
The best bet is to own your own server and host it in a reliable safe location, sure it more Short term pain but for a long term gain. Plus the decentration of hive witness nodes is important, this is a example on one hosting provider effecting several witnesses.
Then get another server as a backup so you can switch over if your primary has any issues. Nice to have for sure 😊 but a must for the top 20 in my opinion.
Anyway as you say its a good to to refocus your efforts, and I wish you a speedy resync.
PS. Thanks for your support for my witness, I am probably the only one running in Adelaide.
We are already looking at other options like Hetzner, but that will take time to set up, so we have already started another Privex and are syncing with the chain as we speak.
We also decided to do this to use the credit that Privex gave us.
Nice!
Any chance you want to help me with mine I'm hopeless right now, lol.
My Discord is @edicted as well.
Oh crap you just reminded me I forgot to mention that Privex gave a 2-month refund to all those affected.
Thanks for letting me know.
I should look at running my own infrastructure again.
But just have to find the time.
The police when they realize it's all just shitposts and complaints why the shitposts don't make more money.
How dare you make 5c for this shitpost Acidyo! 😡
Also if it were me shitposting I should make 20c per shitpost.
...I think that's the mentality, right?
You forgot to add a sprinkle of "if you make more than me I'll downvote"
🤣
Ah heck, I knew I wasn't being sassy enough! 🤣
I've thought about running a node myself from time to time, but I don't think I would be able to get enough votes to make it profitable. I also don't think I would be able to get away with it on my current resources. I'd have to use work equipment and that would be an ethics violation among other things. It's too bad that this happened to so many people.
If Hive were to go mainstream I'm guessing the only users running a node would be the ones to need to have access to a node they control due to the other services they provide to the network. The entire vibe is very much "still early" sentiment.
I personally don't understand how the node running and servers go about. Yet I will advise a compulsory backup server. I don't know how expensive it can be but something shared and also out of your control can go nuts anytime.
Sorry for the inconveniences.
A server is a server so a backup server costs twice as much.
This isn't really viable for a lot of block producers unless they are actually in the top 20.
By definition anyone not in the top 20 is already classified as a "backup" by default.
Some backups have a backup but its definitely not "compulsory".
Missing a couple days as a backup witness isn't a big deal as far as I can tell.
Ok, got more clarity here. It means you were not over pressurized that's the angle I was thinking from.
What's crazy is many people suspected this that too many nodes were being run out of one central location. While VPS's have their advantages they also have their downfalls as well. Nothing beats running a system out of your own home or somewhere you control. The chances of that being taken down seem far lower as most wouldn't even know about it.
This is why I run mine from home haha, I can't be arsed with dealing with this kinda crap happening. xD
I had read about privex having server seizures a few weeks back but I think those ones didn't have hive nodes on seems like they have quite a problem at the moment.
It's like most witnesses run on either hetzner or privex at the moment, the former for value the latter for ease of setup. I'd always encourage people to diversify if possible or host at home.
I do have an interest in a home setup but people keep telling me it's not reliable enough.
I just need to buy the hardware.
Thanks for noting that this is indeed a thing.
It does come with it's own risks, Home internet can be.. dodgy, sometimes you can get buisness internet at home addresses and such but mileage varies, Powercuts can happen.. misshaps like brakers popin.
It's why I've got a UPS battery backup, And I'm running on some second hand server/workstation hardware I got for £400, it's all over 12 years old at this point and.. well it's fragile but works flawlessly as long as I don't touch it xD
Sounds like hardforks are not fun for you :D
Well it's more physically touching the server, one CPU socket is already missing a screw, I've already accidently chipped some resistors off (thankfully nothing I use broke) and the plastic is so brittle that if I took the ram out it would never go back in cause the slot would disintegrate haahah.
Software wise it's perfectly fine.
I am building a fund ready for the inevitable failure of the hardware, preferably I'd replace it all or have a second server with 'newer' stuff in it before it gives up.
lol nioce
I noticed that a lotnof witnesses were off and even thinking to msg some of them but this explains it. I'm also runing a witness on Privex, seems not affected for now :) Fingers crossed. Wishing you a speedy recovery!
never a dull moment, ser
the migration has begun
Thanks for letting us know!
I'm not even sure if I vote yr witness, I'll double check and give you a vote if not once you're back up and running, I know a number of witnesses rely on privex, so good to diversify out of it!
Diversity is important but also Privex could completely blow up and we would all just find another solution.
I think the centralization concern about Privex is highly exaggerated.
Also just vote for it now you're gonna forget again lol :D
Done, I trust you'll get it back up sooner rather than later!
I just had to buy another one (will have to cancel the first) because it was going too slow heh.
Good luck getting it sorted!
gonna be synced in... 10 hours hopefully
Good to know cheers!
I wonder if it has something to do with this news this evening from Finland.
https://yle.fi/a/74-20112197
oh... my... god... lol
fucking Richard Heart
wen prison?
kek
No idea - they don't know where he is apparently.
Maybe it would help if they spelled his alias correctly.
lol?
Maybe they should ask the SEC!! LOL
Wow that is crazy!
No matter how much money you make, sometimes it is not enough.
Thanks, mine is now disabled on the blockchain until I replace it. For anyone else having this issue - one thing I learned is that you need to cancel your existing server inside the Privex system before you get the full 2 months credit.
Take away: sometimes shared servers can be shut down... like blowing up an apartment building because one guy is suspected of selling drugs on the third floor. These things happen.
Well that sucks.
I don't think that happened to me before when running a node on ethereum, but certainly could have...
--- sometimes this stuff is just luck of the draw and other times just luck.
...It's crazy luck for @daltz to be on the same service and not be effected, but they most likely have multiple independent services.
I have often thought of running a node on hive, but lost money on previous operations during the bear, so I didn't do it...
Plus..I would have to put on my computer geek hat and review.
But ..I imagine technology has marched on.
So it may ..be easier then I thought and with luck for you too.
Nearly 2 years running in red. It's alright; we need to help decentralize the chain.
Thanks for clarifying.
Sometimes coincidences do exist :D