Last month I released the Hive witness tool I created to keep an eye on my witnesses and make sure they are still in sync even if they are not the primary witness. You can read about and download it here.
One of the requirements to use my tool is to have logging enabled for your witness node. I highly recommend you do this reguardless as it is the only way to really troubleshoot problems that may arise. These log files can get big quickly at around 20 Mb or more growth daily.
Fortunately, there is a native tool for handling log files on Linux that makes this really easy. Logrotate will automatically rotate log files by breaking up your logs by whatever criteria and compressing older log files.
When logging from Hived (typically through screen) you have one log file that just keeps growing until you run out of disk space. Even if you already know how to use logrotate, you might run into an issue with hived where it no longer writes to the file name specified and this will trip up Montior Witness causing false alerts.
This happens as log rotate will move and rename log files automatically but some processes will continue to use the same file handle and if you were writing to witness.log, you will end up writing to witness.log.1 causing Monitor Witness to fail because it cannot detect the last block processed.
You can just cut/paste my logrotate config that handles this situation without the false alerts. The big change to a standard logrotate config is I use copytruncate rather than create. This will introduce a very brief split second gap in the logging while it makes the cut over, in most cases you will never notice it, at worse case one line might be missing.
To configure this, all you need to do is create the following file: /etc/logrotate.d/witness as root, and enter in the following:
/home/ubuntu/witness.log {
su ubuntu ubuntu
daily
copytruncate
missingok
rotate 30
compress
delaycompress
notifempty
}
You will need to change the first line to be the file path to your log file and the second line to be your username and group you want for the permissions (so it isn't owned by root). This configuration will rotate your log file daily, compress them, and keep 30 daily logs. It of course will use copytruncate instead of create which is important to avoid false alarms.
There is no service to restart and it will take place immediately upon the next time it would usually do a log rotate. If you check back in a few days you should see something along these lines.
If you haven't checked out Monitor Witness, I highly recommend running it on your witness node(s), especially if you have at least one backup node. Without it, you will never know if your backup nodes are in sync prior to switching over to them and finding out it fails.
Considering a witness node can take days to resync, knowing when a node is out of sync before it is used to sign blocks is a huge help.
If you are having trouble with any of this, just leave a comment below or hit me up on Discord.
Can you describe the process of cutting over to your backup node, presumably it will have your keys already installed in the cli_wallet
sorry, I meant to respond to this yesterday but I got distracted.
You just need to broadcast the new key, you can do it via CLI Wallet, Beem, or even Witness Essentials.
No problem, I have been thinking about dipping my toe into the game and have finally pulled the trigger and registered my node, I left it running for a few days and keep an eye on the syncing and it was very stable, so went ahead. Now I will begin to learn.
Thanks for sharing! I bookmarked it and will have a closer look over the weekend!
Thanks for sharing. Nice to know about this.
Rotating logs is a great option to keep you getting out of disk space. On top of that some logging systems like LOG4J also allow compression, so I suggest that if not already done.
Good to know this.
Wao! This is a great tool you created and I can see it impact and how it has really helped you keep track of your witness. Good one friend 👍
Nice, I always use logrotate too:
@tipu curate
Upvoted 👌 (Mana: 53/63) Liquid rewards.