Last week we continued work on post HF24 optimizations. Below is a summary of the work done last week and our plans for the upcoming week.
Hived work (blockchain node software)
As mentioned last week, we’re currently creating a hived plugin that can directly write the needed data into hivemind’s database during hive reindexing and normal block reception. Most of the data being provided by get_block_api is of no interest to hivemind, so using this API to get the data is unnecessarily wasting cpu, in addition to slowing down hivemind. I expect that using the plugin approach will lead to significant speedup in the initial sync time for hivemind (my guess now is 2x at least) and it should also reduce normal hivemind live-sync write time. Work is ongoing here:
https://gitlab.syncad.com/hive/hive/-/commits/km_live_postgres_dump/
Hivemind (2nd layer microservice for social media applications)
Most of our Hive devs continued to working on hivemind last week. Below are some of the merge requests incorporated into the develop branch of the hivemind repo:
various bug fixes:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/381
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/382
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/372
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/387
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/401
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/399
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/404
enable decentralized muting:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/385
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/392
enable reputation calculation:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/393
new tests:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/368
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/361
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/388
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/389
test system improvement:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/386
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/390
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/397
performance optimizations:
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/394
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/395
https://gitlab.syncad.com/hive/hivemind/-/merge_requests/377
Setup an additional system for api server testing
We bought and setup one of the new Ryzen 9 5900x systems to see how it performs as an API node server. It has good single-threaded performance, so it will be interesting to see how it handles hived reindexing and hivemind full sync (neither of these tasks are particularly parallel right now and they take a fair amount of time).
What’s the plan for next week?
- Finish hivemind and condenser decentralized list changes.
- Create a hivemind database dump for other API node operators.
- Continue creating hivemind tests.
- Run hived/hivemind full sync tests on new server.
- Continue work on speedup of hivemind full sync via hived plugin as the slow sync time has a big impact on the speed of hivemind CI (which sets an upper limit on how fast we can validate changes).
- Write a post with plans for HF25.
You guys are always so busy!
Work work work, don't you ever have any fun?! :P
Programming work is fun! Which is fortunate, because otherwise we wouldn't often work on the weekends.
Thanks for these updates, I see that a votes is taking a refresh of the page to show it registering, is this a known bug or is it an issue with Hive mind? Looking forward to the ideas for HF25.
Posted using Dapplr
It's a known issue with hivemind. Hivemind runs two blocks behind hived to avoid microforks, because it doesn't know how to handle fork resolution properly (it never has).
Part of the change we're working on with the hived plugin mentioned above should allow us to enable hivemind to support fork resolution properly, which will allow hivemind to finally serve "real-time" data. This will also be very important for 2nd-layer apps in general.
2x is promising outcome and plus have instant data availability. So, we are gradually moving all social logic to 2nd layer, right?! My only concern was scalability of postgres, if we were to move most social logic to it.
Generally speaking, I've found scalability of postgres to be much better than that of hived for social data. And it's also a much friendlier/safer environment for programmers who aren't aware of all the delicate issues associated with changing hived.
Oh right rocksdb vs postgres of course and I agree, postgresql has wide adoption. But it is bit complicated to scale it, with our current use, it will be ok though. Later replicas and rearranging tables might help...
I did some analysis a couple of weeks ago, and most of the slow down in hived API response time isn't really related to rocksdb itself. The first issue was related to locking, but we improved that a lot already.
In my limited tests, there were usually two remaining causes for slow API call response times: 1) computation of results that could/should have been cached and 2) time to serialize output data (often supplying data that wasn't even strictly needed for most applications).
But the schema changes required to fix some of these issues will be much easier to do in postgresql. And I also see some nice potential benefits with postgresql having built-in interpreters (more about that later).
Excelent good job
Music to my ears
that would be greatly appreciated. thanks
You lost me at update
I guess my question is... WTH does a this mean to a new user?... How can we make Hive... more user friendly? You know... not everyone is a programmer or computer savvy. They just want a username and simple password. I understand that is difficult with $$ being involved... somehow that needs to be simplified.
That is just my 0.02 coming from a blue collar worker...
These progress reports can be read at two levels. I try to put in some information in that can be understood by everyone, but a lot of the content is aimed at other Hive programming groups, so they are aware of where we're at now, and what features will likely be available in the near term for their use.
Some of the things we're doing now do impact new users (e.g. optimized API calls make for more responsive user interfaces), and will sometimes even impact the user-friendliness. But user-friendliness is mostly a function of front end apps. When the BlockTrades team does front end work, we usually do it on condenser (hive.blog). There's also a couple other devs that work on that (most notably @quochuy). We didn't do any work on condenser this last week, although you can find work we've done in previous reports.
I am working on hiring some new frontend developers and later re-tasking some of our existing frontend devs to Hive work (once they free up from current projects).
Much of our current work has been focused on enabling Hive to scale to large numbers of users without requiring prohibitive cost and enabling other programming groups to execute on their own plans for new 2nd layer Hive apps. Both of these will also have important impacts on user experience in the long run.
I hear ya, I was being little sarcastic. When it comes to explaining the ins and outs of Hive to someone it gets pretty confusing.
A new user might benefit from knowing those of us who were here in the beginning and over the course of years, when this project ran under a different name, were often kept in the dark. It might not make sense but the main takeaway is progress.
There's plenty of pull in several different directions though, yeah. So much confusion. Are you using Keychain? That might be the simple password solution you're looking for.
Did you try https://ecency.com ? Let us know if you feel little less confusing in terms of user interface and easier onboarding.
When will HMT be formed? It didn't be built from 3 years because it is so difficult for hive developers
HMTs aren't that difficult, IMO. The issues in their development were mostly issues of software management.
Our team hasn't done any work on HMTs since we started working on Hive, as we've been focused on higher priority issues first. Now that we've about completed that work, we'll be reviewing HMTs and other competing options soon.
Good news
Every body will now wait for your post on new HF. Appreciated your last week work!
I'd love to see more governance fixes and improvments in HF 25 personally.
Those are definitely part of the plan.
You nerds get to have all the fun! Please let us know how it performed in the next update.
THAT IS YUUUUUGE
Yes, I think we can revolutionize Hive app development with some of these changes.
Nice
Is there a roadmap for the next 12 months ( or some timeframe)?
Congratulations @blocktrades! You have completed the following achievement on the Hive blockchain and have been rewarded with new badge(s) :
You can view your badges on your board and compare yourself to others in the Ranking
If you no longer want to receive notifications, reply to this comment with the word
STOP