DevOps is not easy
I'm writing this post purely to inform anyone interested in what it takes to keep services on the web running. When I started my journey I wasn't a professional developer or any kind of systems administrator. I just had a bunch of skills gathered over 40 years of tinkering with computers and the ability to search the web and watch YouTube videos.
Today I find myself running a service which a growing number of people are using on a regular basis and for which I've been compensated for building by the Hive DHF. Here's a little story about a glitch which stopped @v4vapp v4v.app from working properly for a couple of hours last night.
Support Proposal 244 on PeakD
Support Proposal 244 with Hivesigner
Support Proposal 244 on Ecency
Vote for Brianoflondon's Witness KeyChain or HiveSigner
Sweden Network
My @v4vapp service is run mostly from 3 servers hosted by @privex who I'm very happy with. They're small, independent and Hive focused. They take payment in HBD which is a huge bonus. They should also accept Lightning but they don't and I'm working on that.
For some reason I woke up at 4am my time. I know I shouldn't, but I can't help myself, I looked at the internal messaging system I use for monitoring my services and got a screen full of horror. My server which watches the Hive blockchain was unable to see Hive. That means that if someone sends me some Hive to pay a Lightning invoice, nothing will happen as the processes will not start.
Panic
I knew this wasn't good but I could see that the system I wrote to detect network problems was working to some extent. However because I was getting these error messages I knew that the servers were still online.
Fortunately I had the presence of mind to check on @privex Discord and there I found the likely cause of my problems. Fortunately I seem to have woken up pretty soon after the last message.
Less Panic
What then follows is a period where I wait to see if the error detection and re-start code I wrote actually works. My design is that it should just keep trying until it starts to work. And that is exactly what happens.
Lightning Invoice expiry
My system is working between Hive and Lightning. Hive is a blockchain as you may know, but Lightning isn't. Nevertheless my Lightning Node is running all the time and so the idea is that if something like this goes wrong and my service is unable to read Hive blocks or connect to my Lightning Node, it should just wait until it can. As soon as it can see Hive again, it will pick up reading blocks where it stopped.
That's exactly what happened and a few minutes after the data centre was back to normal, my system started picking up Hive messages which came in during the outage. There was one particular attempt to pay a Lightning invoice which failed because the Lightning invoice expired. Exactly as my system is designed to do, it returned all the Hive.
Everything Normal
Basically all this is to say, I did nothing and everything is working again. But that's only because of a large amount of work getting to this state. There are things I would change if doing this again, and things I will change in my push to open source and get the service documented to a point where someone else can run it.
Show some love for the people who keep the web running!
All this is to say that these services which we all use take a lot of work to keep going and to build in resilient ways. I have a lot of respect for the people who run systems especially those much larger and more heavily used than my own!
Once again the only reason I'm able to do this and keep this stuff going is because of the continued support of the DHF so I thank you all again for your votes and look forward to seeing more of you use @v4vapp .
Support Proposal 244 on PeakD
Support Proposal 244 with Hivesigner
Support Proposal 244 on Ecency
Vote for Brianoflondon's Witness KeyChain or HiveSigner
Such things happen and businesses are affected when DDOS attacks are performed. Great it ended well and all services are back running as usual.
👍
Those routing loops can be tricky to resolve. All it takes is 1 redundant patch cable to confuse the network. Some switches are smart enough to prevent that from happening though if you configure them properly. I believe the protocol is called STP. @privex
While our switches and routers do use Spanning Tree Protocol, and no network changes were made to trigger the routing loop - unfortunately it still happened.
Strangely using STP debugging commands and checking the logs of each switch - it wasn't even clear that it was an STP loop, we only managed to determine that it was a routing loop through trial and error, by turning ports off until the network came back up.
It's unclear why the issue was triggered, given no network changes were made for a long time, but nevertheless, we've resolved the issue :)
Just keeping my home network running is hard enough!
Congratulations @brianoflondon! You have completed the following achievement on the Hive blockchain And have been rewarded with New badge(s)
Your next payout target is 27000 HP.
The unit is Hive Power equivalent because post and comment rewards can be split into HP and HBD
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPTo support your work, I also upvoted your post!
Check out the last post from @hivebuzz:
The service you provide is so great. You even check it regularly, you have all my support. I have voted for you !
Thank you!
When there is a great effort there will be an improvement, I really appreciate your time on this project. Keep the good work
Good to hear you got this fixed very quickly. 👍🏻
And btw…FountainApp also has issues from time to time when Sats streaming doesn’t work. That sometimes takes days to fix.
Everybody who runs something like this has issues from time to time.
This is really encouraging and motivating to see relentless efforts been put to stuffs like this and I appreciate every part of it.
Congratulations @brianoflondon! You received a personal badge!
Wait until the end of Power Up Day to find out the size of your Power-Bee.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking
Check out the last post from @hivebuzz:
This is really quite impressive and I appreciate the fact that you keep improving in what you do which I believe a lot of people can learn from the effort.
👍
This is a great advancement, kudos bro
Click on the badge to view your board. Click [here](https://hivebuzz.me/rankingWC) to check your ranking.
Thank you to our sponsors. Please consider supporting them.
Check out the last post from @hivebuzz:
Congratulations @brianoflondon! You received a personal badge!
See you at the next Power Up day to see if you will repeat this feat.
May the Hive Power be with you!
You can view your badges on your board and compare yourself to others in the Ranking
Check out the last post from @hivebuzz:
Click on the badge to view your board. Click [here](https://hivebuzz.me/rankingWC) to check your ranking.
Thank you to our sponsors. Please consider supporting them.
Check out the last post from @hivebuzz:
Hi @brianoflondon
I wanted to start using v4v.app to send HBD to BTC via lightning, and from there to an exchange. I installed Atomic Wallet, but I can't figure out how to create a lightning invoice, or if it is the right wallet at all...
I have added LIGHT as a custom token, but only ERC20. I guess I will need it on BTC, for which I couldn't find anything...
Noobs questions? For sure! Maybe you can point me in the right direction.
Lightning is not another token built on Bitcoin. Lightning is something completly different from ERC20 tokens on Ethereum.
If you want to take Lightning out as BTC look for wallets called Muun or Breez. Both of these will receive Lightning (with relatively low fees) and then let you send a Bitcoin transaction.
This process isn't free but it is relatively low fee.
Lightning is best when it stays as Lightning and is sent around that way but ultimately every Sat which is lightning exists as Bitcoin.
Thanks for the answer, I start to understand ;)