This post is a quick update about my current witness downtime on June 18th 2018, as both my main-server and backup-server are replaying the blockchain.

What happened?

I'm currently in Switzerland for a big blockchain / cryptocurrency event (more on that in another post) and before I left home for the airport, I wanted to make sure that both of my servers were up-to-date in regards to the Ubuntu packages. So i ran sudo apt update & upgrade

While the updates went completely fine and in the past updates like that never caused any problems & my time was running short - I went on my way.

But just when I was in the train to the airport I got a notification on my mobile-phone that my witness missed a block. Well, you can imagine how perplex I was, but luckily I've prepared for such an event.

My Reaction

The first thing I did was to login remotely into both servers (while still on the train), where I found out that both steemd instances stopped working.

It seems that the updates for Ubuntu packages were so fundamental, that somehow the docker-image stopped working - which caused my witness to miss a block.

Now, as soon as I realized that both servers weren't able to produce blocks, I disabled my witness, restarted both servers and started to replay the blockchain.

However, this process takes a lot of time - the older the blockchain gets.

The more blocks the blockchain has (a new block is getting added each 3 seconds), the longer it takes to get to the current block-number (replaying is always starting from 0).

Right now, my server with the highest sync-progression has reached about 82% sync with the blockchain.

My estimation is that the blockchain should be fully synced in the next hours, when I'll activate the witness again.

What I've learned

Now, I'm a big believer in honesty, transparency and learning from everything I do.

Out of this experience, I'm taking away two core things:

1.) I should always check double - even if out of 100 times, the same action never caused any problems.

2.) I should take my time for important things. Updating packages for ubuntu isn't a great thing to do, when the train leaves in 30 minutes.

While these 2 lessons are important, I'm also proud about my reaction time. I successfully disabled the witness fast enough, even while being on a train-ride to the airport, to miss not more than 1 block.

Steem is a huge priority in my life and I'm always giving my best!

I will give an update as soon as my witness is activated and producing blocks again (should be within the next hours).

Wolf

Update 1:

Both replays stopped. One with an error and one without. Changed some things up and restarted replay. Let's see how it goes this time.