In my May witness report I gave you a teaser for this technical article. I will explain how I use a combination of tools to keep tabs on my witness nodes and give you the practical benefit of what I've learned in putting this into practice.
This explanation is not intended to be an exhaustive tutorial on each of these tools, however it will cover the elements not so easy to find documentation for and that I found difficult and time consuming to figure out. I will also provide a link to my github repo where you can find the scripts I used.
Summary of Features
1 - Automates startup of witness node upon system restart
2 - Automates starting cli wallet when witness becomes synchronized with blockchain
3 - Optionally unlocks cli wallet to support bts_tools price feeds
4 - Creates a multi-windowed tmux session to view witness log file in real time
5 - Starts bts_tools monitoring and associated web interface with price feeds
6 - Creates a shell window for running builds or for general use
7 - Attaches multi-window tmux session upon logging in to server via ssh
8 - Provides administration of services through systemd (start, stop, status, journalctl etc)
An image of the tmux session is shown above. Below is a screenshot of the web interface provided by @wackou's bts_tools which I have been using since I first began producing blocks in 2015. His tools are built with python 3, use a flask template system for the UI and served to the web via nginx through a uwsgi interface. Web access uses ssl encryption and password authentication to provide "at a glance" monitoring from anywhere the web can be accessed. I couldn't recommend @wackou's bts_tools more highly.
We had some grand plans to work together and establish an infrastructure of seed and witness nodes that has never come to fruition. Back in the day I wrote a python extension to his bts_tools to distribute delegate (i.e. witness) pay to a set of accounts specified in the config file. I used it for the duration of the BTSX but it was never updated to work with graphene. Since graphene was released there has been little need for it. It wouldn't take much to get it working with graphene should anyone need that functionality.
Components Used
Subsystems
o - Graphene witness_node -- blockchain software
o - cli_wallet -- Command LIne interface for witness_node
o - bts_tools -- Nicholas Wack's marvelous tool suite
o - nginx -- web server to support bts_tools (and anything else you might want to serve up)
Scripts
o - witnessSysD -- installs a SYSTEM scoped systemd unit to manage witness node service
o - tmuxSysD -- installs a USER scoped systemd unit to manage tmux service
o - startTmux -- initializes tmux session, starts bts_tools and cli wallet, monitors witness RPC socket
There are a few other items in the github repo you may find useful but these are the main elements involved. Note there are a few lessor items I use that will make your life much easier such as virtualenv and virtualenvwrapper that provide a "sandbox" for python scripts. There are also a number of almost forgotten bumps in the road you may encounter such as how to get different python versions to coexist in harmony. It's also worth noting that some items such as uwsgi can be installed globally OR via python pip in a virtualenv sandbox. I recommend the later as it will isolate and encapsulate the dependencies of bts_tools.
SystemD
As I mentioned before, I had great difficulty getting systemd working for a USER scoped session. It was much easier using systemd for SYSTEM scoped service units. A unit in systemd parlance is a unit of work or a service to run. It can be as simple as a script or it can get very involved as to dependencies and order of execution. I started using only SYSTEM scoped units and later migrated the tmux elements to a separate USER scoped service unit as that was oriented towards user interaction. I also had difficulty with tmux in running the witness node and user items which them separated eliminated (I never got to the bottom of why tmux thought I was trying to run nested [tmux] sessions. I believe it should be possible to run everything in a systemd USER session, but that is not where it is today). All USER scoped systemd unit files reside in the user's home directory under ~/.config/systemd/user.
The primary obstacle I had to overcome to allow systemd USER sessions was to get a USER scoped dbus process fully started. Although systemd did start a user session and create a /run/user/<uid> folder, it was totally empty. This is where the dbus sockets live for the USER session (actually /run/user/<uid>/systemd/notify & private). Usually this is accomplished by the operating system via the X Windows startup process. However when that is absent some other means is required, for example when logging in though an ssh connection. When the user session is properly initialized two environment variables are set:
- XDG_RUNTIME_DIR=/run/user/<uid>
- DBUS_SESSION_BUS_ADDRESS=unix:path=$XDG_RUNTIME_DIR/systemd
I had always configured my sshd_config script to disallow password authentication as well as disabling PAM with UsePAM = NO. I still disallow password authentication, however PAM (Password Authentication Module) must be enabled so that libpam-systemd will be used to initialize the user's systemd context, which includes creating a separate dbus user process, in memory message sockets and setting environment variables so applications like systemctl --user can communicate. Another element I had to find was that on some platforms (I primarily use Debian and Ubuntu operating systems) the dbus sockets were still not being created. I noticed that on platforms that did work properly there was no /etc/pam.d/atd file present. I'm not sure of what that file does, but removing it resolved the issue and the systemd user session was properly established. I suspect atd is used to delay the start of "at" (i.e. cron style) jobs, but I am not certain of that.
Another aspect hindering the continuous operation of USER scoped units after logging out is a systemd setting, whose default behavior changed with version 206. The new default behavior is ALL user processes are terminated on logout. It was a highly controversial change and you can find many discussions about it online. There are several solutions, the one I use is a script command loginctl enable-linger <user> which reverts the default for the user account specified.
To automatically switch to the tmux multi-window view upon logging in you will need to add the line tmux attach-session bts-node to your .profile script, or whatever your shell of choice runs when it starts.
I think that covers the main points, and along with the scripts I published in github and with bts_tools provided by @wackou you have the essential elements to reproduce these interfaces.
verbaltech2 and baxters-sports-witness
are always appreciated! Thanks for your time and attention
@full-steem-ahead Always above and beyond!
upvoted and re-steemed
Thanks so much o-u-p-a-t for you help & generous, supportive spirit!
Namaste...
He's a real trooper!
Great work on these Bitshares tools @full-steem-ahead! :)
Very technical (as you said) but very helpful for those doing this. Congrats on cracking the code! Upvoted for sure!
Wow, I am deeply honored you took the time to venture into geek land to comment on such an article. Thank you so much Terry!
Namaste
I like geek land. I have much to learn and enjoy it. Thanx for letting us all know!
Terry
@full-steem-ahead,
Great technical review! And really helpful for the community.
Thank you very much for sharing such useful information!
Cheers~
Nice post!! But it is too much complicated to me because i am new in codding field.
Thank you. All it takes to succeed is the desire and perseverance. If it's important to you it will happen!
GREAT JOB!!!
Thanks for the update and namaste :)
I went through the post but seems like I have to be spme guru to understand this post. Nice one.
I did say it was a technical piece, but hopefully it will be useful to others of the "geek" ilk. Thanks for commenting!
Great post, A little too complicated for me, but I like that! Happy 4th of July.
Thanks, and I had no exposure to systemd or any of the things I use now, but I learned and so can you, if you put your mind to it. It all starts with your intention. Good luck!
Great technical review,, very helpful
Great work @full-steem-ahead thank you so much sharing your valuble knowledge with us..i like yours ideas..and i can get more details it....keep it up post...GOD BLESS YOU resteem and upvoted this post
Glad to know it was very helpful to you, and thx for the promotion!
great post.. thank you for sharing
Awesome post. Thank u for the information. Have a nice day.
very usefull, thanks @full-steem-ahead
Hi @full-steem-ahead
VERY interesting. Would like to see a follow-up article on your blog post. Followed and voted.
Well done!!!!
Thanks for the post. Very good information.
Always a giver. Keep up the good work. You are voted from @skapaneas the sooner I get to my active keys at home.
Upvote and resteem. Being a coder I realize how much time and effort goes into such accomplishments.
Abow out..
what a nice explain , thanks for sharing post like this , always welcome the freshers , cheers follow you @full-steem-ahead
Resteemed.
Really amazing ,I was looking for something similar, and here you go ,much respect for this helpful sharing, upvoted ,resteemed. No problems at all , more articles like this. @full-steem-ahead
Great technical article And great information..
very good information and very technical , but helpful post
great work on these Bitshares tools.... Congrats on cracking the code!