I've been hanging around in the https://steemit.chat/channel/witness chat for a week now, after I started my own witness node. I've seen many new comers like myself bringing their various contributions to the witness community and wanting to setup their own nodes. Many are following @someguy123's guide on setting up a witness server. It's an excellent starting point if you have no experience with linux. It's written by an experienced IT guy who is a witness himself. I've followed his guide personally and I've been up and running in no time.
However, there are a few recurring concerns being voiced in the channel by users during their setups. I would like to address them here, especially for the panicky types. Also I'll share a few simple linux tips for managing and monitoring your node.
Witness Considerations
First and foremost, a witness node is a very sensitive process, it's highly recommended that you do not run anything else on your box, or you will risk missing blocks. Dedicate your box to running the witness only, with the recommended minimal specs. A decent CPU and a stable bandwidth are a must. A Raspberri Pi with 4GB RAM ain't gonna cut it!
Currently the blockchain data is 16.7GB (not a big problem)
For the shared memory file, the current miminal requirements are
- Seed node (p2p mode): 5.5GB
- Exchange node: 16GB --> that's us (we can comfortably run with 12GB)
- Full node: 65GB --> that's the heavyweights like @gtg
For the rest of this post, I'll assume you followed @someguy123's setup guide and you're on Ubuntu.
Monitoring the Witness Logs
The guide included the command ./run.sh logs which is an alias for the command:
docker logs --tail=30 witness
This will show the last 30 lines of the log. If you want to monitor the logs in realtime, add --follow (or -f) option like this:
docker logs --tail=30 --follow witness
You can stop the log monitoring with Ctrl-C. It's a universal linux command that terminates a process running in the foreground.
Monitoring the CPU and RAM Usage
There are two commands that allow you to do that with the commands top or htop. They have different outputs, so use whichever you prefer.
You can install htop on Ubuntu with sudo apt-get install htop.
Common steemd Errors
A common startup error is:
[33m2263080ms th_a main.cpp:133 main ] Error parsing logging config from config file /steem/witness_node_data_dir/config.ini, using default config
It happens with @someguy123's method and it's nothing to worry about.
During the replay, you will see many errors like this:
[33m2416593ms th_a steem_evaluator.cpp:85 do_apply ] Wrong fee symbol in block 3208405
Again, these are normal.
When full synchronization is reached, your node is ready and you will start seeing messages like this:
585054ms th_a application.cpp:507 handle_block ] Got 16 transactions on block 13549520 by timcliff -- latency: 54 ms
588382ms th_a application.cpp:507 handle_block ] Got 27 transactions on block 13549521 by xeldal -- latency: 382 ms
591143ms th_a application.cpp:507 handle_block ] Got 20 transactions on block 13549522 by witness.svk -- latency: 143 ms
594427ms th_a application.cpp:507 handle_block ] Got 24 transactions on block 13549523 by roadscape -- latency: 427 ms
597057ms th_a application.cpp:507 handle_block ] Got 19 transactions on block 13549524 by jesta -- latency: 57 ms
600292ms th_a application.cpp:507 handle_block ] Got 18 transactions on block 13549525 by good-karma -- latency: 292 ms
603441ms th_a application.cpp:507 handle_block ] Got 20 transactions on block 13549526 by anyx -- latency: 441 ms
If you're running the replay for the first time (./run.sh replay) and you have an hour to stare at the monitor while you drink coffee, I recommend you follow the logs in realtime just to be sure everything is working OK.
The steemd daemon is single-threaded so don't be alarmed if only one CPU core is operating at 100% during the replay. And don't be surprised if it's taking ages on a Atom CPU!
After the replay is done, the CPU usage is pretty minimal. What's really important is the RAM. If you don't have enough RAM, the replay will freeze.
Once you're synchronized and you broadcasted your witness (update_witness in the cli_wallet), don't forget to run a price feed. You can use Steemfeed-JS. Although be advised, sometimes the price feed will sporadically crash with the following error (you guessed it, monitor it with docker logs --tail=30 --follow feed)
SteemApi error: get_required_signatures [ { ref_block_num: 0,
ref_block_prefix: 0,
expiration: '1970-01-01T00:00:00',
operations: [ [Object] ],
extensions: [],
signatures: [] },
[ 'STM11111111111111111111111111111111111111111111111111' ] ] {}
(node:17) UnhandledPromiseRejectionWarning: Unhandled promise rejection (rejection id: 2): Error: SteemApi error:get_required_signatures[object Object],STM11111111111111111111111111111111111111111111111111{}
It's a known bug https://github.com/svk31/steemjs-lib/issues/8 and the workaround for now is to crontab it to restart every two hours, with the following command:
crontab -e
Then add this line to the end of the file and save:
0 */2 * * * docker restart feed
Then cd ~ and start the feed again in the background with:
docker run -itd --rm --name feed steemfeed-js
There was a little mistake in @someguy123's guide, he wrote:
docker run -it --rm --name feed steemfeed-js
which will run the process in the foreground, so add the 'd' to run it detached (in the background).
Setup the Time Synchronization
One last crucial step is to set the server time synchronization (it was mentioned in another guide, don't remember which one). We need to keep in synch with the rest of the nodes!
Open a root shell (I hate typing sudo everytime):
sudo -s
Set your timezone, for me it's EST (optional):
timedatectl set-timezone America/New_York
Install NTP (Network Time Protocol):
apt-get install ntp
Open the config
nano /etc/ntp.conf
Add these settings to the end of the file and save
minpoll 5
maxpoll 7
Or, you can do this quicker with one line:
echo "minpoll 5" >> /etc/ntp.conf ; echo "maxpoll 7" >> /etc/ntp.conf
Then:
systemctl enable ntp ; systemctl restart ntp
Check everything is good:
timedatectl
Should output something like this:
Local time: Sun 2017-07-09 23:27:57 EDT
Universal time: Mon 2017-07-10 03:27:57 UTC
RTC time: Mon 2017-07-10 03:27:57
Time zone: America/New_York (EDT, -0400)
Network time on: yes
NTP synchronized: yes
RTC in local TZ: no
And finally, exit the root shell:
exit
Conclusion
If you have other troubles with your witness setup, you can join us on https://steemit.chat/channel/witness, there are plenty of cool people who can help.
Join us on https://discord.gg/GpHEEhV
If you really like it, resteem.
If you love it, follow.
Don't be shy, click that banner and enjoy my blog (gif animation by @justcallmemyth)
🗳 Check out my Steem witness application and vote for me. 🗳
Thank you.
Thanks for the tips.
I've found that the crontab above does not always work because there are cases where the container just vanishes so a docker restart would do nothing.
So I changed the cron like this:
0 */2 * * * docker ps | grep feed && docker restart feed
*/5 * * * * docker ps | grep feed || (cd /opt/steemfeed-js && docker run -itd --rm --name feed steemfeed-js)
True. The docker container may crash sometimes. However, I monitor my servers and processes daily, so if anything goes wrong I fix it :)
I do the same, but it stresses me out when I wake up in the morning and see a Gina bot message saying my feed did not update for two hours some time during the night...
It's fine, you only need to publish a price feed once per 24h. I do mine every 6h, less than that is overkill.
Ah right
Excellent tips. Greatly appreciated.
Thanks alot for the help in steemchat and this wonderfull fix
This is all new to me, thanks for adding to @someguy123 setup guide, great info!
Really helpful informations for newbies, it's a little too complex for non tech savvy people so this is a good heap of information, thanks for sharing!
There's tons and tons of helpful information on becoming a witness on here. Be careful though some of them are not up to date or are not complete. The witnesses and DEVS are constantly updating it to make it almost fool proof. Some basic knowledge will however go a long way here @kryptokayden.
Thanks for the advice, I'm not planning on running a witness anytime soon, i just want to understand it's relevance and who i should give my votes for.
I am planning to setup witness node. Was surfing over the steemit and got your post on hot section. Its really helpful...Thanks @drakos
Where was this a week ago when I was setting up @drakos?! :p . Thanks for informing others though man. You get my vote always
It was still in my head lol. Good luck with your witness :)
This post received a 3.1% upvote from @randowhale thanks to @yunkzilla! For more information, click here!
@drakos.. very helpful posts here for people trying to setup their witness great Job!!!
Thanks
Hi @drakos, loved this post and hoping you will get what you wish to attain! I am trying to find you on steemit chat, could you tell me how. Or are you on the discord channel by @aggroed ? I have some questions and I would be so thankful if you could help with it?
Hoping to hear from you!
Yes I'm on discord and steem.chat.
very helpful my friend :)
Thanks. Saves me from manually restarting steemfeed every hours :-)
There's also conductor, by @furion, it's an alternative to steemfeed-js and doesn't crash like it.
it always shows ./run.sh: line 168: docker: command not found
please can anyone tell me what is the problem?
Install docker:
./run.sh install_docker