A few months ago, I decided to take the plunge and start getting involved with development for the Steem blockchain. I have learned a lot through the process so far. In this post I will share some of my "lessons learned".
Blind Leading the Blind ;)
Please keep in mind that I am still very much a "newbie" when it comes to programming for the Steem blockchain. To a large extent, this post will be an example of the blind leading the blind :)
Still, I hope that there is at least enough useful information in here that anyone planning to work on changes themselves can learn from my experience and have a slightly easier time.
SBD Changes
I decided to work on two issues that were a high priority for many in the community, but Steemit was not working on due to other priorities. The changes I decided to tackle were: 2140 and 2022. The specific details of the changes are out of scope for this post, but for those interested you can read about them here: SBD Print Rate (2140) and Beneficiaries Payout (2022).
Building Consensus
Developing code for the Steem blockchain is quite difficult. I would go as far to say that building consensus around an idea is even harder.
If you are planning to develop a change, it is extremely important that you have sufficient support for your change in order for it to be accepted. If you develop something and there isn't support to accept it, then it will have been a waste of time.
Assuming the change will require a hardfork, then you will need at least 17/21 of the witnesses (20 primary plus one backup = 21) to vote "yes" for the change.
A good place to start is with a post (or series of posts) to discuss the idea with the community. If there is sufficient support, then a good next step is to reach out to various witnesses and let them know you are considering coding whatever the idea is, and find out their level of support.
If you want your code to actually make it to the "finish line" then it is your responsibility to get it there. That includes doing whatever convincing needs to be done in order to get it accepted.
Getting Steemit Development Team Buy-In
I know many people dream of a day when the community will be self-sufficient enough to code up our own hardfork changes, get them accepted by a majority of witnesses, and then get all the exchange + developer nodes to upgrade to the new version without Steemit's support - but the reality is we are not there today.
If you are planning to develop a change for the Steem blockchain, you are most likely going to need a lot of support from the official dev team. If you don't get their help, it will be very difficult (if not impossible) to get your change to the finish line.
A big key here is that you have to understand their role in the process, and be respectful of their time. They are already super busy working on the changes that they are required to do (Hardfork 20, SMTs, etc.) and whatever change you are planning to do will take time away from their required duties.
A good place to start is to open a GitHub issue to describe what you are planning to work on, and make it clear that you are planning to do the coding work. It is also good to demonstrate the amount of support that there is for the change.
Before you spend too much time coding, you should try to get some assurance from the dev team that they are on board, and will accept the change (assuming you code it properly).
Hardfork vs. No Hardfork
An extremely important part about developing changes for the Steem blockchain is determining whether a particular change will cause a "fork" - i.e. a change in consensus.
One way to think about this is that the blockchain data is basically split into two parts: there is the data stored in the blocks, and then there is the state data that is derived from the data in the blocks.
If you are changing the data that is stored in the blocks, then it is likely going to be a hardfork change. If you are only changing the state data, then it is likely not a hardfork change.
Handling a Hardfork
If your change is going to result in a hardfork, then it is important that it is coded in the correct way.
First of all, there are several changes that need to be made (in addition to the "actual" changes you are planning to make) just to make the hardfork to occur properly. This includes incrementing the version number (i.e. 19 -> 20), setting the time that the hardfork will occur, and including all of the necessary logic to trigger the hardfork at that time.
If you look through previous hardforks in GitHub, there are plenty of examples where this is done. (For the changes that I worked on, I did not complete this step myself - since I coded them to rely on the hardfork 20 logic that Steemit was already implementing.)
Once the framework for the hardfork has been setup, then all of your changes to consensus (what gets written in the blocks) need to be wrapped in proper conditional logic. What this means is that when the code executes it tests to see if the hardfork has occurred yet. If it has, it runs the 'new' code; if it has not, it runs the 'old' code.
Here is an example from my pull request that updates the print rate start/stop percentages after hardfork 20 occurs:
Here is another example from my pull request to pay out beneficiaries using the author's payout setting after hardfork 20 occurs:
Testing Changes
I found that testing to make sure all my changes actually worked as expected was actually significantly more difficult than coding them up. I won't be able to give sufficient information to explain how to test every possible change, but here is some general guidance on how to proceed.
Test Basic Node Functionality
Verify that a node can successfully resynch up to the current block, and continue processing new blocks with the changes applied.
Test ALL Paths
Find a way to get it to run through all the possible paths of the code. When coding a hardfork change this can be quite challenging without a dedicated testnet, as there isn't an easy way to test what happens after the hardfork takes place - which is likely the most important code to test.
One option is to launch a testnet. An alternative is to create "parallel code" that runs in addition to the "actual code" and compares what is currently happening (pre-hardfork) with what will happen (post-hardfork).
To give an example of parallel code: when I was testing the print rate changes, I setup additional variables to hold "temporary print rate" values. I copy and pasted the code that would run at hardfork 20, and had it set my "temporary print rate" variables to the "post-hardfork" values when the "pre-hardfork" code ran. Then while the pre-hardfork 20 code was running I could see what would happen once the hardfork occurred by looking at what was in the temporary values.
Debug Output is Your Friend
One of the main ways that I tested to see what was happening as the code executed was to add a whole bunch of extra debug output into the code. I set the debug output to whatever values I needed to see, and then scanned through the log file to see what the values were as the different portions of code were executed.
Here is an example of the logging syntax. It is fairly easy to copy/paste this, and then tweak it to output whatever information you need:
Automated Testing
In addition to the "live node" testing, it is also important to setup automated tests for your code as well. There is some information on their automated testing system here: https://github.com/steemit/steem/tree/master/tests with a bunch of examples inside the "tests/tests" folder.
You can run the automated tests by using the sudo docker build -t=steemit/steem . command.
It is extremely important that you run the automated tests and verify success before submitting your changes. If you don't and your change ends up causing a test to start failing, then your PR will not be able to be merged into the official repository, because it will show failing tests.
The first thing you will need to do is update any tests that may have broken as a result of your changes. After that, you should setup any new tests that are needed in order to verify that the outcomes you expect actually occur after the conditions that cause them to happen occur.
Here is an example of the tests that were run to ensure that beneficiaries payouts are split properly:
When you are running the automated tests, you may want to see debug output from those. The way to output debug info from the automated tests is using BOOST_TEST_MESSAGE. (There are plenty of examples of this in the code.)
In order for the 'boost test' debug output to actually show up though, you will have to edit Dockerfile and add a line: ENV BOOST_TEST_LOG_LEVEL=message.
Converting Values to Strings for Debug Output
One of the challenges that I ran into was trying to figure out how to get values that I wanted to see output to the debug logs converted to strings. Special thanks to @blocktrades for helping me to figure this out!
There are a few different tricks to get things to convert:
- You may need to add
.valueto the end:db->get_account( "alice" ).reward_vesting_steem.amount.value - You may need to use the
boost::lexical_castconversion tool:boost::lexical_cast<std::string>(db->get_account( "alice" ).reward_vesting_steem.amount.value) - You may need to use the
fc::jsonconversion tool:fc::json::to_string(gpo.current_sbd_supply)
(This may not cover every possible variable, but it at least worked for all the values that I wanted to output.)
Verifying You Are Running the Right Code
A few times I ran into a situation where I made changes and compiled, but then when I ran my code to test - it wasn't running the latest version.
There are steps that I used to ensure that I knew I was running the latest code:
- Make sure all instances of steemd are killed
- Delete the compiled executables that are there from the previous version:
sudo rm -rf /usr/bin/steemd
sudo rm -rf /usr/local/bin/steemd
- Update the capitalization of the "Transactions on block" debug output in
libraries/plugins/p2p/p2p_plugin.cpp. - Compile
- Run
- Verify that the
Transactions on blockdebug output in the log has whatever weird capitalization I used (i.e.TraNsacTioNs On blocK).
I'm not sure if this is the right/best way to do it, but it got the job done :)
Miscellaneous Items
These are a few other random things that I learned.
Global State Variables for Parameters
Steemit is moving towards having a lot of the global constants defined in dynamic_global_property_object. Here is an example of how a variable is transitioned into that implementation: https://github.com/steemit/steem/pull/2571
Producing Blocks in Automated Tests May Break Pointers
When I was working on the automated tests, there was a line of code that I thought could be removed:
It turns out that it is needed, since when the db_plugin->debug_generate_blocks call is made, the pointers may become undefined.
Conclusion
Well, that's it :) I can't guarantee that anything in here is actually "good" advice, but I hope that if anyone else is planning to write code for the Steem blockchain, this guide at least helps.
Kudos for jumping into blockchain programming. It's way beyond my league. I did one small PR for
steemand closed it shortly after, that's it, lol. Although blockchain development is different from other types of projects, your experience with it is generally what is encountered in other environments. For example, testing and debugging take most of my time. The code may seem simple at first glance, but testing it while covering all possible angles is a challenge. I usually step back and ask myself questions. What happens if:Then, I try to simulate each condition while fixing and optimizing the code.
Another important thing for me is efficiency: how much time does the code take to execute? My code is full of performance checkpoints and benchmarks, you may have noticed some of them in small font on my websites (Processed in ... ms).
People often judge and bitch at the developers too quickly, especially when something breaks (how many times have we seen it with Steemit?). It's not easy to cover all angles, the projects are extremely complex with sophisticated integrations. Despite all the efforts to avoid bugs, there will always be some lurking and waiting to happen. That's where the beauty of open source lies; it brings up the best in many developers to fix and improve projects. I wish people would understand why Steemit is still in beta, it's because that stuff takes time.
If steem has a heartbeat (which it does).. it is very organic...
I am proud that you're part of that heartbeat Tim.
You, yourself, involved brings confidence to me that good people are involved in this project.. and knowing you're there.. at the helm... doing what you do... makes me sleep comfortable every night.
I believe in steem. I believe in you.
I only wish I could be as involved as you are... and maybe one day I can be...
But until then... I'm proud to stand by your side, and point...
That is the guy -- right there --- @timcliff -- investing his life into this chain.. that we all need to thank and acknowledge.
What Tim does... isn't easy. It involves dedication, perseverance, devotion, and a significant supportive partner in his life that understands his mission.
Hats off to you sir... You will never hear it enough... as much as you deserve...
...but this is exactly what I want to say...
THANK YOU
Thank you so much, Tim. I've been looking forward to this post for some time now. It's as detailed and as helpful as I was hoping it would be and should give anyone who wants to jump in a really good head start based on all the work you've already done.
I really appreciate what you've done for this place over the years.Since your arrival you have continually thought of the ways you could bring value to Steem(it). And have respectfully related your views on a variety of subjects without trying to force your views on others.
Maybe it doesn't need to be said, but I immediately though of the considerable pre-requisites the following the lessons in this post.
Spending a considerable amount of time gaining the respect of the community:
that's not meant to be an exhaustive list.. but I wanted to mention this stuff because that's how my mind works.
Thanks for the peek under the hood, Tim.
You perked my interest in knowing how the 21st witness is choosen to participate in the vote?
If at the time of the hardfork, 17 of the 21 witnesses in that block are voting yes then the HF is approved. If the backup is voting no and there are only 16 votes then it can still be approved in following rounds if a new backup is chosen that votes yes.
Ok, so it is all software version triggered? For example, if 17 of the witnesses install the new fork that is how consensus is reached? Sort of like the version triggers you mentioned in the post?
Yes
I don't know how I missed this excellent post. Thanks so much for sharing this. I'm not a developer but managed the development of an app for my company some years ago and reading this brings back fond memories. Yes, actually GOOD memories. LOL. (It was a turning point in my life.)
I'm so grateful for everything you do for this community. You're genuinely my favorite witness.
Thanks :)
For as much as you seem to do on here I figured you had been a developer for a while. lol
Thanks for explaining how things work. It sounds like a lot of work but it's good that the blockchain seems to be working pretty well now.
This is perfect. Soooo well written Tim. You made this feel approachable even to non-coders like me. I feel like this really encapsulates one of the conversations we had a few months ago on my Dear Littlscribe show!
One part I am a little unclear on:
My question is, if you've got 17 out of 21, then core dev really doesn't matter, does it? Wouldn't that just kind of seal the deal?
Technically yes. If 17/21 voted yes then it would fork and it would become the new consensus rules.
There is more to it though than just forking. A huge part of it is making sure that it is a safe/correct change. The main point I was trying to make is that we still need the official devs to be involved in order to help ensure that everything is coded correctly and won’t cause unforeseen issues. At least for now, I would not feel 100% safe accepting code that I wrote unless one of the official devs had looked it over first.
Also there is a lot of involvement with getting exchanges to upgrade. Steemit currently does all of that. There is also the fact that a lot of exchanges will only upgrade to versions that are from the official repository, which means if we want all the exchanges to pick up the fork that Steemit needs to review/approve/merge it.
Cool. That makes sense. Thanks for the explanation.
I admire the great work you are doing in the platform dear friend @timcliff, congratulations for your interez in knowing more development of Steem's blockchain, without a doubt all your questions and changes will be favorable for the steemit and the people
I wish you a great day
@timcliff, i am not an Developer, Coder or Programmer but this post is great to go through because this will spread an message regarding the weightage of work which has to cover by the Developers to create an effective aspect.
And we know what kind of work you are doing and how effective it is, in my opinion your thoughts will help others if they plan to write the Codes for sure.
And yes, Steem will move towards the super boost position because in my opinion we have some great projects ahead which will for sure going to fill the fuel for the Long Run.
Wishing you an great day and stay blessed. 🙂
I had to read each line at least twice in order to try and understand a bit of this post.. I'm a "newbie" truly when it comes to blockchain development. I still have to read more and more to get a clearer understanding.. Your post is educating @timcliff. I would still love to read more of your posts on this topic.
Thanks a million times.
Nice work! The only change I can think of for Steem is to give resteems 100% of the curation they generate. Would be a hard fork for sure. I think curation needs a major revamp though.
You deserve high kudos, Tim! About a year ago I was looking into possibly becoming a witness but I'd be an empty vest, if you know what I mean. Without the technical chops, it would be very hard to understand what's really required and I admire you for taking these steps. @timcliff community witness
Gaea info... @timcliff thanks for sharing information..
Really very good material, it is really from your work I would like to learn about the blockchain.
Officially speaking... You are not a newbie! Wether in blockchain programming or not.
Your post is so detailed... It got me totally lost.
But at least i have an idea how complex the work of steem devs and witnesses are.
Kudos (2 elbows!!!) to you guys for keeping the steem engine running... You guys are responsible for making steemit the economic fortress of the internet it is destined to be...
So Dont let up. Nice one mahn!
He is a newbie to coding for the blockchain. He's only officially been doing it for a few months. Up until recently, the only blokes coding for the actual block have been core dev. Anything outside of that has simply been accessory for the most part, from what I understand. He's actually coding hard fork changes to how the payouts and voting structure might work of the actual coin. This takes tremendous research, vetting, reconfiguring, testing, more testing, and support-finding. Not something he's had a chance to do much of for real since recently, when he was able to retire from his day job!
Also, you're right. It is detailed, but I feel like the first half is extremely accessible to everyone serious about the blockchain, and the second half is for those who actually hope to code for it.
@timcliff, i am not an Developer, Coder or Programmer but this post is great to go through because this will spread an message regarding the weightage of work which has to cover by the Developers to create an effective aspect.
And we know what kind of work you are doing and how effective it is, in my opinion your thoughts will help others if they plan to write the Codes for sure.
And yes, Steem will move towards the super boost position because in my opinion we have some great projects ahead which will for sure going to fill the fuel for the Long Run.
Wishing you an great day and stay blessed. 🙂
I agree @chreerocks! I think we take for granted what goes into coding for the blockchain, let alone getting a consensus from the top 20 and an OK from the core dev team.
That's true, and the job is really tough because at the end of the day we have to create something which is Practical, Producing Value, Creating an Economy. So development and coding is most integral part because we can call them Technological Architects.
Wishing you an great day and stay blessed. 🙂
I have only respect for you trying to get into this. Thanks