Below are a few highlights of the Hive-related programming issues worked on by the BlockTrades team since my last post.
Hived (blockchain node software)
Hived nodes can now directly exchange compressed blocks over p2p network
Hive nodes can now directly exchange blockchain blocks in compressed form. Previously each node would receive new blocks in uncompressed form, then compress them locally before storing them to their block_log file. In our most recent changes, the p2p network protocol has been enhanced so that when a node connects to a new peer, the two nodes inform each other of whether they can accept compressed blocks (only nodes compiled with the new code can accept compressed blocks, older ones can only accept uncompressed ones).
So new nodes will send blocks in compressed form to other new nodes, saving the recipients from having to compress the blocks locally. In other words, only one node in the network has to compress the block, and all the other new nodes who receive that compressed block to will benefit from the work done by that node.
Eventually, once the network is only running with new nodes, all blocks exchanged over the network will be compressed. This not only reduces network traffic, it will also reduce the overall CPU cost for supporting compressed blocks, because each block will only need to be compressed by one node in the network (i.e. the witness that produces the block).
The update to the p2p negotiation protocol was also designed to allow for interoperability between nodes in the face of further enhancements to the compression algorithms, where we could once again see a situation where we have old nodes and new nodes that compress by different methods.
Finished up storage of block ids in the block_log.artifacts file
We completed the changes for storing block ids in the block_log.index file (now renamed as block_log.artifacts to avoid confusion with the older-format files) and integrated those changes with the full_block/full_transaction code discussed in my previous post. This optimization reduces the amount of CPU work that hived nodes need to do. It also speeds up get_block API calls and sync speeds between peers.
We changed from our initial algorithm for creating the new block_log.artifacts from an existing compressed block_log, reducing the time to generate these files on a magnetic hard drive (HDD) from 126 minutes down to 72 minutes. Note that this task is now only IO bottlenecked by the time to read the block_log file backwards, so on a set of 4xraided NVME drives it takes as little as 7.5 minutes.
New compress_block_log tool
We also have a new utility for compressing/uncompressing a block_log file. This tool should be especially useful for existing hived nodes that will likely want to compress their existing block_log file to save disk space and serve up compressed blocks more efficiently on the p2p network (of course, there’s also an option to just grab a compressed block_log from someone else instead of compressing your own).
Optimizations to Hive’s transaction and block processing speed
When looking at the performance of the Hive network, it is useful to look at two different performance metrics: 1) the speed at which transactions can be added to the blockchain (transaction performance) and 2) the speed at which data about the blockchain can be served to Hive client applications (block explorers, hive.blog, gaming apps, etc). These two metrics could also be viewed as the “write” speed and the “read” speed of the blockchain network. Transactions write data to the blockchain, and API calls allow apps to read this data.
A lot of our time this hardfork was spent on improving the read speed of the network: mainly by creation of the HAF framework, but the creation of the block_log.artifacts file was another example of this effort and it helps speed up get_block API calls. Read speed is very important, because it puts an upper limit on how many people can see what is going on in the network, and in a blockchain network, more people are reading data than writing data, especially when that network supports social media functionality.
But our work during the last month has focused on the “write” speed, the ability to process more transactions without breaking a sweat or slowing down the ability to read the data being written.
This year, during periods where nodes were receiving high bursts of transaction traffic (typically due to bots playing splinterlands), we’ve seen hived nodes put under stress when a bunch of these transaction build up that can’t be included into a block (either because of lack of time to process the transactions or because the block was simply full). These non-included transactions were then re-applied to the node’s state after the block was generated so that they could be added into future blocks, but previously this reapplication of these transactions delayed the send of the newly generated block to other nodes.
So in one example of a flow change that we made as part of recent optimizations, new blocks are now sent to the p2p network for distribution prior to the reapplication of these non-included transactions. The upshot of this is that nodes can handle much larger bursts of transaction traffic better, because we can insure timely delivery of new blocks to other nodes, even when we have a lot of non-included transactions due to burst traffic.
After all our recent transaction/block optimizations, we ran benchmarks with the new code exposed to transactions levels 20x higher than current traffic levels. These tests were performed both with 128K blocks (2x current size) where lots of transactions don’t get included due to limits on the block size, and 1MB blocks (16x current size) where all transactions typically did get included into the larger blocks, and in both cases the new nodes respond without even a hint of a hiccup.
Completed optimization of OBI (one-block irreversibility) protocol
We resumed work on OBI optimizations at the beginning of this week and completed them as of yesterday. Currently we’re writing tests to verify the performance of the optimizations.
Hived testing
We continued creating tests for the latest changes and found a few more bugs related to the new transaction serialization of assets. In a lot of cases, this is just a continuation of the long term task of creating a complete test suite for existing hived API calls. Now that we have tests for more API calls, it’s easier for us to detect when a change breaks something, but we still don’t have a comprehensive set of tests yet.
In related work, we’re adding more transaction functionality to the CLI wallet to enable it to serve as a better testing tool, but this task isn’t in the critical path for the hardfork, so we will pick it up again after the hardfork is past (or if a dev comes free in the meantime).
Hive Application Framework (HAF)
We’ve begun re-examination of scripts for backup of HAF database and HAF apps in light of changes to HAF since scripts were created. Need to check on the status of this task still.
HAF-based block explorer
We’re currently creating new tables in the block explorer database schema to track votes for witnesses. These tables will be used to serve up various forms of voting data for the new “witness” page of the block explorer. We’ll probably also create something similar later for a “DHF proposals” page on the explorer.
HAF-based balance tracker application
While working on the HAF-based block explorer, we found it would be useful to have vest balances for all accounts in the block explorer’s database, so rather than rewrite such code, we’re incorporating the balance_tracker prototype app into the block explorer as a sub-app.
Originally the balance tracker was created as a simple “tutorial” app to show new HAF devs how to create a HAF app, so it wasn’t fully tested for accuracy, but now that we’re going to be deploying it for production use we undertook to do such testing and we found and fixed a few bugs due to some of the more esoteric events that occurred during the history of the blockchain (for example, splitting of vests at hardfork 1).
HAF-based hivemind (social media middleware server used by web sites)
We fixed some bugs, did some code refactoring, and worked on CI changes for HAF-based hivemind. Last remaining big task is to create and test a docker for it to ease deployment. This will hopefully be done in next week.
Some upcoming tasks
- Test enhancements to one-block irreversibility (OBI) algorithm and merge it to develop by this weekend.
- Merge in RC cost rationalization code (there is a small possibility this change doesn’t get into the HF if we encounter too many issues in the next couple of days).
- Continue testing and performance analysis of hived on mirrornet.
- Test hived release candidate on production API node to confirm real-world performance improvements.
- Tag hived release candidate early this coming week (probably Monday or Tuesday).
- Finish testing and dockerize HAF-based hivemind and use docker image in CI tests.
- Test and update deployment documentation for hived and HAF apps.
- Complete work for get_block API supplied via HAF app
- Continue work on HAF-based block explorer
- Collect benchmarks for a hafah app operating in “irreversible block mode” and compare to a hafah app operating in “normal” mode (low priority)
- Document all major code changes since last hardfork
When hardfork 26?
We found a few more errors related to the changes to transaction serialization of assets than we initially expected, so we’re going to take a more comprehensive approach to make sure we’ve eliminated any potential problems from this change and this will slightly delay the hardfork date. A formal announcement of the date should follow shortly from hive.io account.
in theory, every post on hive is an NFT, maybe it's something marketable. Because opensea and friends make a lot of revenue with a centralized inefficient NFT game.
I mean in theory it needs some link somewhere ( IPFS or centralized database) and a way to track ownership. That could be in theory a post do with some update.
Maybe to go beyond art nfts and open another way to build on hive.
I know it's not scaling or making hive better, but may be valuable. Special it could be used for something else to have some "proof of ownership" with transferable "posts".
Hi @urun I did a little write up on this very topic back in early April (topic of Hive posts as NFTs). Mostly, it was thinking out loud. I'd love to hear your more on your views on this.
The compression feature is awesome, should have been done a long time ago. However, it's optional isn't it? Because you mention
Eventually, once the network is only running with new nodes, all blocks exchanged over the network will be compressed, which might not be the case if some/many witnesses choose or neglect to use compression. The network would still be serving uncompressed and compressed blocks, correct? Therefore, I think it should be recommended that everyone move to compression so the network/witnesses/endusers would benefit.No, even if you have an uncompressed block log, your node will compress old blocks before sending them as sync blocks.
So this means your new node will do more work whenever it to provides sync blocks if you don't compress your block log (because it will have to compress the blocks each time they are requested). But the rest of the network will be running optimally in either case.
So the recommendation is definitely to compress your block_log, but mostly for your own node's benefit.
if i understand it right, if all is compressed it could slowdown the network?
On the contrary, compressed blocks are smaller to transmit over the network, so it's faster. The block is compressed by the block producing node (witness) and it takes a very tiny fraction of CPU processing. It's well explained in paragraphs 2-3 of this post.
got it, its an efficiency update to handle more traffic :) Was a long day and read it super fast :D
For whatever reason, I thought about compressing old blocks.
20x higher traffic x 16x block size = 320x or 1.5 trillion transactions/sec, give or take a few.
Take that Merge Vitalik.
Good to see the compression. I presume that will increase the overall efficiency of the nodes since data is compressed once, shared by many.
Scaling for blockchain is like anesthesia is to surgery: very painful if you dont have it.
Great to see the continued work being done.
September it is then
The rewards earned on this comment will go directly to the people( @hivetrending ) sharing the post on Twitter as long as they are registered with @poshtoken. Sign up at https://hiveposh.com.
Will the RC rationalization make it so that Splinterlands bots need more RC in order to play as much as they do?
I'm not sure yet of the full impact of RC rationalization on custom_json yet, because the real costs of operations will have changed due to recent optimizations. As examples, we have speeded up the json parser, which will make custom_json cheaper. On the other hand, we now account for signature verification costs, which will make custom_json more expensive. My guess is that, overall, custom_json will be more expensive in terms of RC than it was before. But I won't know for sure for at least a few days.
Good job as always. Thank you.
Congratulations @blocktrades! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s):
Your next payout target is 1780000 HP.
The unit is Hive Power equivalent because post and comment rewards can be split into HP and HBD
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPSupport the HiveBuzz project. Vote for our proposal!
Nice post
Hello @blocktrades. As always, fascinated with the work you have done for this wonderful blockchain.
I have been waiting for news about the implementation of filtering functions for HAF nodes. I can't afford a server spec for a general purpose HAF node because I just want to experiment with basic dapp that uses small custom json data.
Have filtering functions been implemented?
If not, do you have an estimated time for this?
Yes, there's two kinds of filtering available (they can be used together, also): one filters by accounts of interest, the other filters by operations that match a regex.
This post has a little experimental data on results using filtering: https://hive.blog/hive-139531/@blocktrades/th-update-of-2022-on-blocktrades-work-on-hive-software
Excellent! Thank you very much. I am going to try to install a local server to experiment with basic dapp.
Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with the following badge:
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPSupport the HiveBuzz project. Vote for our proposal!
Reducing it from 126 minutes to 76 minutes is a great achievement. Congratulations and success. what's your new goal?
Thanks @blocktrades for this useful information you have done good to all especially me, I just came to realize some things which I would explore soon as i leave this comment.