@techcoderx witness node crashed [Update: fixed]

in #witness2 years ago (edited)

witness update log 27.png

Just an update post for the record that @techcoderx Hive witness node have crashed unexpectedly, and right now in the process of replaying the blockchain up to the head block.

The witness node is currently hosted with Privex using the pre-replayed VPS located in Sweden, and I have been using it for over a year without issues until this got hit 2 days ago. For reference, the previous downtime was scheduled on 7 November 2021 when the disk had to be resized. It is the only other downtime with this provider.

The pre-replayed image was 3 weeks old hence it needs to catch up with the network, and unfortunately this is running slow as it replays at a rate of ~10b/s. As expected, this is due to disk I/O bottleneck as the VPS only has 8GB of RAM to work with. The CPU scored poorly on Geekbench 5 as well.

This remains an ongoing incident and will update here as the story develops.

In the worst case scenario (and probably long term as well), I may have to move this server off to a dedicated instance that might cost more, and likely spin up a backup hived node on a NUC that I bought for another blockchain. It might even help with the ridiculously high block time offsets that I often see on techcoderx.com node even with v1.26 optimizations.

IMG_0148.jpg

Update 1 (19/8/2022 08:48 UTC): Caught up with the network, price feed is also now up to date.

Update 2 (19/8/2022 09:25 UTC): Subsequently a block has been produced successfully.

Update 3 (20/8/2022 04:47 UTC): It is now techcoderx.com's turn.


Server resource statistics

hived (v1.26, d821f1aa, all plugins)

block_log file size (compressed): 323 GB
block_log.artifacts file size: 1.6 GB
shared_memory.bin file size: 20 GB
blockchain folder size: 345 GB

HAF db

Output of SELECT pg_size_pretty( pg_database_size('block_log') );
Database size: 2,976 GB

hivemind (v1.25.3, a68d8dd4)

Output of SELECT pg_size_pretty( pg_database_size('hive') );
Database size: 533 GB
Compressed: 206 GB

Overall Postgres database

RAM usage: 4.5 GB
Compressed disk usage: 1.4 TB


Hive witness footer 2.png

Sort:  

Thank you for your witness vote!
Have a !BEER on me!
To Opt-Out of my witness beer program just comment STOP below


Hey @techcoderx, here is a little bit of BEER from @isnochys for you. Enjoy it!

Learn how to earn FREE BEER each day by staking your BEER.

Congratulations @techcoderx! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s):

You distributed more than 22000 upvotes.
Your next target is to reach 23000 upvotes.

You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Check out the last post from @hivebuzz:

HiveFest⁷ Meetings Contest
HiveFest⁷ - Participate in the Balls of Steel tournament and get a new badge
New badge - LEO Power Up Day - September 15, 2022
Support the HiveBuzz project. Vote for our proposal!