Below is a list of Hive-related programming issues worked on by BlockTrades team during last week or so:
Hived work (blockchain node software)
Further optimization of sql_serializer plugin
The sql_serializer plugin is responsible for reading blockchain data and pushing that data to a HAF server’s Postgres database. As a practical matter, this means the speed of this plugin sets an upper limit on how fast a HAF app can operate (especially during a replay of a hived node), so it is important that this process is as fast as possible.
After our latest optimizations (and bug fixes), we’ve brought the time required to do a full replay with the sql_serializer plugin down to 5.5 hours (previously about 7 hours, so about 17% speedup). Note that both the old and the new benchmarks were performed on one of our fastest systems (a Ryzen 9 5950X with 4 NVME drives in a raid0 configuration).
The latest changes for the sql_serializer are now merged into the develop branch.
Cleanup display of results for several hive CLI wallet commands
The hive CLI wallet is a command-line interface for creating hive transactions. It is mostly used by Hive apps, exchanges, and Hive “power users”.
The most recent improvements include minor fixes to the display of wallet command results, such as case consistency, indentation for tables, headers and other misc. display fixes for get_withdraw_routes, get_open_orders, get_order_book, etc. There were also various improvements and fixes to CLI wallet internal docs.
Miscellaneous changes
We eliminated some false errors generated by GCC 9.3.0 linter and fixed a Cmake-related issue (compilation of the sql_serializer plugin requires Postgres dev libraries as a dependency):
https://gitlab.syncad.com/hive/hive/-/merge_requests/254
https://gitlab.syncad.com/hive/hive/-/merge_requests/255
Hivemind (2nd layer applications + social media middleware)
The most recent hivemind work was focused on optimization of the hivemind indexer (the process that builds social data from the raw blockchain data).
Reduced peak database disk usage (20% reduction in peak storage requirement)
We noticed that postgres database size temporarily grew to around 700GB during the massive sync phase (this is when a hivemind server is first being initialized with old blockchain data). We determined that this temporary increase occurred during record cleanup resulting from the reputation calculation process, and by changing from using DELETE calls to DROP TABLE and TRUNCATE calls we were able to eliminate this temporary peak usage requirement, resulting in a 20% reduction in peak storage requirements (approximate 141GB less storage used at current headblock). We’re also hoping this will lead to further speedup in the post-massive sync initialization of table indexes, but we haven’t had a chance to benchmark that yet. https://gitlab.syncad.com/hive/hivemind/-/merge_requests/509
Elimination of post’s active property speeded up hivemind sync by 20%
We discovered that none of the Hive apps used the active field for posts (this field indicated that a post was still actively competing for rewards from the reward pool) so we removed the field from the database schema and eliminated it from API responses. This turned out to be surprisingly beneficial to hivemind’s full sync time, at least according to our latest benchmarks: we completed a full sync in 10.5 hours, approximately 20% faster than our previous time on the same system. https://gitlab.syncad.com/hive/hivemind/-/merge_requests/511
Completed optimization of update_post_rshares for Postgres 12
We’re planning to move to Ubuntu 20 (from Ubuntu 18) as the recommended Hive development platform soon, and this also entails a move to Postgres 12 (from Postgres 10) because that’s the default version of Postgres that ships with Ubuntu 20. So we’re working thru performance regressions associated with differences in the way Postgres 12 query planner works. Most recently we changed the query for update_post_rshares to fix a performance killer and changed when we executed some vacuum analyze calls:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/510
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/512
Condenser (open-source codebase for hive.blog, etc)
We tested and deployed various fixes by @quochuy to https://hive.blog:
https://gitlab.syncad.com/hive/condenser/-/merge_requests/259
https://gitlab.syncad.com/hive/condenser/-/merge_requests/260
https://gitlab.syncad.com/hive/condenser/-/merge_requests/261
https://gitlab.syncad.com/hive/condenser/-/merge_requests/262
Hive Image Server
We’re in the process of doing a major upgrade to the hardware that runs the image server for Hive. The new system has a LOT more disk space, faster drives, more CPUs, more memory, etc. Currently we’re in the process of moving over the huge amount of images to the new server.
Hive Application Framework: framework for building robust and scalable Hive apps
Optimizing HAF-based account history app
We’re currently optimizing and testing our first HAF-based app (codenamed Hafah) that emulates the functionality of hived’s account history plugin (and ultimately will replace it). Our initial benchmarks at 5M blocks had good performance, but we’ve seen a slowdown in indexing performance when operating with a fully populated database (i.e. 50M blocks) so we’re working now to optimize the speed of the queries. We’re also preparing benchmarks for the speed of the API calls.
Database diff tool for testing HAF-based apps
We're developing a new testing system to test the port of hivemind to a HAF-based application: essentially it is a "database diff" tool that will allow us to detect differences (via SQL statements) between the tables created by the old hivemind app and the upcoming HAF-based hivemind.
Upcoming work
On the hived side, we’ll be adding support to the sql_serializer for directly injecting “impacted account” data. After that, we can compare the relative performance of this method of inserting the data into postgres versus using the C-based postgres extension for computing this data, in order to make a decision about the best design alternative. We’ll also likely merge in the first hardfork-related changes that 1) allow more than one vote per three seconds by an account and 2) don’t kill curation rewards entirely when someone edits their vote strength on a post.
Our HAF work will continue to focus on our first example HAF apps (account history app and HAF-based hivemind implementation).
Good work. Optimisations should pay off as Hive grows and I think we will see a lot more users soon, not just from Splinterlands. It has to be robust and responsive.
Thanks!
Yes, these optimizations are key to Hive's future, in my opinion. Scalability had been the Achilles heel of all blockchains.
Hive seems to be coping with the extra load from the new Splinterlands players. It does seem to be pretty efficient.
Yes, so far Hive is handling all the traffic without even breaking a sweat. I think we have headroom for at least 10x current traffic without any further increases to our existing infrastructure. And I'm hopeful that upcoming changes will increase scalability by many orders of magnitude.
Very happy to hear that. I wish I could get involved with development, but it's not really my field. I am having fun with some HiveSQL and beem scripts. There's lots more I could do with that.
Thanks for all you do. Hope we can meet at a real life Hive Fest some time.
@blocktrades,
Do we have any plan to develop 2nd layer something like SMT old days?
Thanks for the updates!
Cheers~
Yes, it should be relatively simple to support SMT-like functionality (and much more) with the 2nd-layer smart contract platform we're planning.
Looks like a lot of good work.
Good ones :) also the tech stuff looks really cool, makes everything faster right?
Faster and less storage, good combination of optimizations.
Exelente!!! , Es decir que habra una mayor coherencia del tiempo, en cuanto a los registros de resultados.
Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with the following badge:
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out the last post from @hivebuzz:
Wow awesome!
Hello, @blocktrades Would it be possible to transform the HBD into an algorithmic stablecoin like the ones on Terra? In this case, Hive would perform the same function as the Luna coin. Cardano is also going to develop algorithmic stablecoins. If the Terra software is open source, it should be relatively easy to copy/paste or adapt it to the Hive blockchain. What do you think?
HBD is already an algorithmic stablecoin (arguably one of the first, but the original algorithm had a weakness: no peg on the high side).
But after the changes in the last hardfork, it has the best algorithm of any of the algorithmic stablecoins I've reviewed. I'm very happy with its performance since the hardfork and I think the new economics associated with it have been very beneficial to the entire Hive ecosystem.
Edit: note this is Dan from "blocktrades", but I'm responding from my personal account on Hive.
Hello, @alpha Thank you for your answer, but HBD has been fluctuating in the last couple of months between $0.88 and $2.50, while the fluctuation on TerraUSD (UST) has been between $0.96 and $1.04. If HBD has the best altorithm as you say, how is it possible that UST is holding the peg much better?
First, not sure where you're getting your data, but it's not accurate. HBD peg has held well since HF 25 when the Hive->HBD conversion operation was added. I've had bots trying to buy it at 0.99 or below off and on since the HF, with very little success (I've been lucky enough to buy a couple thousand at 0.98-0.99 range) .
If you were trading on HBD/BTC market, you might have got lucky and got some small amount at a cheaper price when BTC took a rapid dive before orders could be removed by HBD sellers (one guy mentioned he managed to buy about $40 USD worth at a better price, maybe around 0.92 USD). But trades like that are purely short term anomalies that aren't really meaningful to the overall stability of the coin.
Now on the high side, the price was around $1.20 for a substantial amount of time. But if there had been any large Hive stakeholder with a strong desire to quickly stabilize it down to $1.00, it could have been done: most just found it more profitable to slowly sell to whoever was trying to rally the price. And ultimately that buy pressure capitulated, as you can see in the graph below.
This graph shows the price of HBD since the HF. The small black arrow shows when HF25 triggered and basically put the brakes on how far HBD can rally:
src link
[EDIT] my arrow didn't show up but HF25 was on June 30th, killing that first spike you see (although it was already starting to die, June 30th is on the tail end of it).
Generally speaking, with current trading behavior, I think you can expect to see HBD trade anywhere between 0.99 and 1.06. If it starts to exceed that, you can expect that traders will start doing Hive->HBD traders to profit from the arbitrage opportunity. And the @hdbstabilizer puts pressure on the price in the 1.01 to 1.05 range and puts associated profits back to the decentralized hive fund. Finally, the other operation, HBD->Hive conversion, keeps price from dropping much below $1. Interest payments also add some upward pressure to the price, which encourages it to stay a percent or so above $1.
Trying I don't like. Pls customer careing service upgrade doing.
Thanks!
Wow, interesting post
It is pleasant news all the advances that are being made to improve the Hive ecosystem, I am happy to hear about the project to improve the image server since sometimes the system gets slow when uploading images to publish ... success
Buen trabajo. Las optimizaciones deberían dar sus frutos a medida que Hive crezca y creo que pronto veremos muchos más usuarios, no solo de Splinterlands. Tiene que ser robusto y receptivo.
Congratulations @blocktrades! Your post has been a top performer on the Hive blockchain and you have been rewarded with the following badge:
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOPCheck out the last post from @hivebuzz:
I agree with you in general, but I am not sure what that system would look like. If the reputation system meant something, maybe authors with high reputation would be rewarded with longer payout windows. The problem is, it opens the door to abuse.
Sounds like we should discuss this idea more thoroughly. The 7 day window has always been a problem. Peakd tried to fix this with a tipping option that is however not used very much.
@tipu curate
To be fair out interface for tipping was a bit lacking and we hope to improve it to make it more obvious and easier to use and way faster. Also easier to do on mobile.
Recently someone suggested a really simple solution for this. Frontends have to do some work to better navigate the users to take advantage of this possibility. It's all very simple - when a post's payout has already been paid, you don't vote on the post, you instead make a comment and set the post author as your comment's payout beneficiary. Then you upvote your comment. This will have the same effect as you voting on the author's post. Plus it will leave a comment on their post, even better.
Does this meet your requirements?
Increasing the post payout period would increase memory requirements for a hived node. In fact, there was a time during Steem days when posts paid out in two separate payments (7 days and 30 days), and the 30 day payment period was eliminated (for performance reasons, I've always assumed).
Also, practically speaking, most votes of economic significance are cast in the first few days, because the majority of voters with substantial stake are voting on an "attention" basis as posts appear in their feeds, on ranking lists, etc. So I don't believe that rewards would change much for posters if the reward period was increased.
In fact, some recent analysis of voting patterns done by @arcange using HiveSQL showed that most votes are cast in the first few days of the posting and this likely means that the reward period could be reduced to 3 days without causing any significant change in reward distribution. But no analysis has been done yet as to what performance benefits that might bring.
I think that the idea mentioned by @borislavzlatanov seems like a good workaround for "evergreen" content, and your idea for potentially automating this process via frontends also seems reasonable, but the frontend dev teams are all pretty busy right now, so I'm not sure where such an idea would fall vs their other priorities.
As a further aside, I think that authors are to some extent indirectly rewarded for past posts when they make later ones. This is part of a reputation building process. Authors who consistently make posts that are liked by voters often end up getting placed on automated voting lists, so in a way, they continue to be rewarded for those earlier posts. This is part of why long term posters often achieve some of the highest post rewards.