Below are a few highlights of the Hive-related programming issues worked on by the BlockTrades team since my last report.
Originally, I thought we would be able to release updates for a bunch of different tools by now, but everything took longer than expected, although in some cases the delays weren't always for bad reasons: we discovered several ways to dramatically improve performance of HAF, so we decided to delay the release to incorporate those changes and continue to add features to hived and HAF apps in the meantime. As for the "bad reasons", we've basically been overhauling the entire approach to deployment of HAF and HAF apps to make it super simple, and we ran into lots of issues along the way as we determined the best practices for what I'm now calling HAF 2.0.
What is HAF 2.0?
HAF is a library and service for developing specialized Hive-based APIs and applications. It acts as a backend server for web-based applications that want to utilize the Hive blockchain network.
We started working on HAF more than 2 years ago, and released the first production version about a year ago. Since then we've steadily been making incremental improvements, but in the past year we've been steadily increasing the number of developers working on HAF and HAF apps as it is the foundation for all our future apps including our layer 2 smart contract processing engine.
HAF 2.0: Completely overhauling deployment and maintenance
With HAF 2.0, one key focus has been on easing deployment and maintenance of HAF servers and HAF apps. The recommended deployment method has completely changed with 2.0, as our goal is to create an ecosystem where any API node operator can quickly and easily install or uninstall any selection of HAF apps they want to support on their server(s).
Perhaps the best analogy (well, for programmers) is we're building something like a packaging system for HAF and HAF apps. Another way to look at is we're making HAF apps available as appliances that can easily interact with each other.
Another deployment improvement has been to standardize on a ZFS dataset layout as a method of delivering "operation ready" snapshots of HAF servers that don't require a replay of the entire blockchain at setup.
This is also extremely beneficial during development of HAF apps: you can take snapshots of your server at any point in time and then later rollback to that same state again, making it easy to: recover from database corruptions that occur while developing your app, reproduce bugs, etc. I've found the ability to easily reproduce bugs and performance problems to be especially useful in my work (in one case I rolled back to the same snapshot about 20 times while analyzing a performance problem that occurred in a specific block range).
HAF 2.0 uses docker compose scripts to deploy the entire stack for an API node
HAF 2.0 also includes a full deployment stack with all the apps needed to run an API node. Previously an API node operator had to not only setup a HAF server, they also had to deploy various other apps such as nginx, jussi, varnish, haproxy, caddy, etc to setup an efficient server that caches API responses, allow re-routing of traffic to different services in the stack, and manage rate-limiting to prevent DDOS attacks. Varnish is a "newcomer" to our stack and is used to provide caching for the new REST-based APIs offered by our new HAF apps such as balance_tracker and the block_explorer. In practice, we've found these offered better performance than the older json-rpc based APIs (these are still cached by jussi).
With HAF 2.0, you can deploy and manage all these services with docker compose scripts all configured by a single .env file. More about this setup can be found at https://gitlab.syncad.com/hive/haf_api_node
HAF 2.0: improving performance and decreasing hardware requirements
In the past few weeks, we improved massive sync replay time by 33% and live sync update performance by 45%. And looking at benchmarks over the past three months, a full replay of HAF used to take around 30 hours to process the entire 80 million blocks of the Hive blockchain and it is now down to 17.5 hours (14 hours for replay, 3.5 hours for creating HAF indexes). At the same time, we recently cut CPU usage by 10x (this doesn't show up as a 10x speedup because the code is heavily multi-threaded, but it does mean HAF can be run on a cheaper computers with fewer cores and consumes less energy or alternatively those extra cores can be used for running other processes like nginx, jussi, haproxy, varnish, more hived nodes, etc).
Disk space storage has also dramatically been reduced. First by storing blockchain operations as binary data, and second by using lz4 compression via ZFS. These methods allowed us to cut database storage requirements by more than 2x.
HAF servers also require much less memory to operate now. Databases almost always like more memory, and HAF was originally targeted towards servers with 64GB memory, but by keeping hived's shared_memory.bin statefile on an NVME drive, we've found that a HAF server can quite comfortably operate as a production server with 32GB of RAM.
It is worth mentioning we're not yet done with performance improvements for HAF: during our recent work we came up with a few more ideas for speedups, but we just ran out of time to fit those improvements into the upcoming release.
Other projects
We're also working on a host of other tools (some of our devs have already made posts about Clive, for example) and it has gotten to the point that I think it makes more sense if I post about the projects I spend the most time on and leave posts about the other projects to the devs involved in them (otherwise my posts would start to get really long nowadays given the number of devs involved on different projects).
So I'll leave it to them (hopefully) to make posts after the holidays about some of the other projects such as the block explorer, Denser, Helpy, WAX, Clive, beekeeper, etc.
Final testing started for HAF 2.0
We started full replays of HAF servers with the latest code a couple of days ago. Although HAF only takes 17 hours to replay, hivemind takes considerably longer (something on the order of 80 hours I think) as we haven't had any time to further optimize its performance yet.
We should have several servers fully replayed by Tuesday, at which time we'll start directing production traffic from api.hive.blog to some of these servers as a final test of performance under real world conditions.
Barring any problems, we'll tag passing versions as final releases and setup downloadable snapshots for api node operators who don't want to wait on replays.
[UPDATE] Sharing one more benchmark from our new fastest system:
"speed king" s16 (AMD 7950X with 64GB DDR5 6400Mhz and 2x4T CT4000T700 nvmes) full replay with ramdisk reached livesync in 11.1h, built indexes in 3.2h, and ready to serve up data after 11.1+3.2=14.3 hours!
Appreciate what you are doing and hope this ultimately makes it easier for people to use Hive. Have tried onboarding dozens of people and most get confused and just give up.
Truth. I've set up accounts for people and they just walk.
Sometimes I think we are destined to be only a niche community, but maybe thats ok.
It's ok to be a niche community, but it isn't OK to have poor UI/UX, definitely something to be improved, especially when it comes to the onboarding process. That's however a task for the front ends.
Indeed, any improvement will be a definite plus!
Thanks for all your work this year. I look forward to great things from Hive in 2024.
Wow, thank you so much for this blog. We can really gain a lot from reading your blog. Please continue with what you're doing, and you'll teach many more bloggers through your posts. Very informative.
Thanks for the post! I need to study some of the terminology and learn more about the blockchain. I love the fact that we are updated about what is going on and the projects that are coming.
Although I don't fully understand what you mean by HAF but I feel you guys are doing great to take hive to greater heights. Thanks for your hardwork.
However, if yiu wouldn't mind ,I wish to understand the whole concept of HAF
Great work! @blocktrades
damn... Memory speed does really make a huge impact...
But nice to see that rebuilding indexes got way faster. I am soon going to try HAF 2.0 finally on my new mobo! Then once I am satisfied, I will grab more NVMes to speed it up.
It's probably not the memory speed, so much as the next generation AMD processor (7950 vs our "normal" 5940s) plus those CT4000s (those are the super-fast PCI 5 gen nvmes). This was also run using Ubuntu 23.10, which has higher performance mitigations of CPU vulnerabilities.
Hummm, good point about ubuntu 23, will give it a try! Here in NZ, the 990 PROs (samsung) are more cost effective, price/capacity wise, but lets see (waiting for the gen5 to stabilize the prices here, and then making the choice, for a 4 to 5 NVMe ZFS setup. My aim is to run everything HIVE+HAF+HE on the NVMes and then snapshot to my disk array, for regular "backup" solution.
For the most part, our systems use 980s and a couple of 990s. We got this one system a couple of weeks ago specifically when we needed to run replays as fast as possible.
Curious about how fast can local replays go on the 7950X with the T700s? I am doing some tests myself too (different gear) and will share them once I have compiled enough conclusions.
On the Ubuntu 23.X, had to add
#include <cstdint>to two header files. But so far, looks interestingly different.We build/run dockers in ubuntu 22 still. You only need 23 for the "host" that runs the dockers, so there's no need to compile on Ubuntu 23 if you don't want to.
Ah yep! Got so excited trying a few things on the system that forgot about that. 😅
Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with this rare badge
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out our last posts:
$WINE
Great work!
great!
I cannot post anything from the hive blog due to continuously downvoting by @themarkymark @buildawhale @thevil @punkteam @gogreembuddy @freebornsociety
It's bad enough that I've been maliciously excluded from airdrops only because I'm Korean, so I'm not sure what the point of mass downgrading is. Is Hive a racist organization?
Your actions are why you were excluded from the air drop, not your race. Those who voted to support Justin Sun's hostile take over were excluded from the air drop, it was a very specific action that determined this that was determined programatically.
I didn't do anything. I didn't support Justin Sun and even didn't know what happened at that time.
Stop shadow-boxing
I'm sorry that people are still downvoting your account, but I guess even after all this time there are people that are still upset about what happened and not willing to forgive the misguided. You could try to contact the downvoters and discuss it with them, but your tone is a bit aggressive, so I'm not sure you would get very far with them. Worst case, you could always create a new account and avoid linking it to your old one in your posts. You would essentially have a "fresh start" with a new identity.
I am not that aggressive, I am just saying that I am treated wrong, I never blamed those people. I just reports what they are doing. I am not describing those badly as well.
Also, do you think I can just use and write articles and communicate with others with zero HIVE power? That must be the main reason prior STEEM members moved to HIVE with 1:1 HIVE airdop, but I am one of the few members who didn't get it. As you know after the several disccusions, I never supported Justin Sun and never supported the main Korean members who were against you guys, but I was excluded without any immediate recognition at that time. I think HIVE blockchain is super fast and so convenient for using as one of blog users, but still I cannot use it without HIVE powers. I can make a lot with pennies spending my time and efforts, but still your people not allowing it. That's why I bore shame to writing a reply again to your article.
This account does not originate flags.
The flag from this account is a dust amount.
I recommend ignoring it.
I cannot ignoring @themarkymark @buildawhale @thevil @punkteam @gogreembuddy because they just delete all of my rewards. Also, before ignoring your downvotes, I want to ask what is the reason of your action? Since only one that I can assume is just because I am one of the Korean community members, I used the word 'racist'.
The only one bringing race into this is you. The air drop was calculated programatically based on on-chain actions, not by choice by any person.
I have zero tolerance or sympathy for those who supported the hostile takeover.
I didn't support anyone you are talking about. The on-chain actions must be done only to the hostile takeovers, but it was applied to the whole korean community members. I didn't bring you and I even didn't know about you before you started attacking me.
https://peakd.com/muhrewards/@freebornsociety/this-account-does-not-originate-flags
If you don't like how hive is managed, power up more hive.
my power was more than 600,000 but they get rid of my power when they move to hive from steem.
Ouch.
I sold my steem one to one for hive, maybe you should've, too?