SteemData now has new, multi-threaded blockchain indexers. This is an long-overdue update, bringing better scalability, performance (one order of magnitude increase) and consistency guarantees.
Further, SteemData has been re-synced from scratch to address past inconsistencies.
Syncing Status
- Operations (100%)
- Accounts (100%)
- Account Operations (99.9%)
- Posts and Comments (93%)
The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.
Real-Time Updates
The last open concern is state change propagation in near-real time.
For example, a single vote operation should trigger cascading updates of all affected Account, AccountOperations and Posts/Comments entries. This is because the vote affects the voters state (voting power), voters history, and the post/comment the vote was cast on.
As the Steem Blockchain usage grows, the amount of dependent state that needs to be updated with each block grows exponentially. Scaling SteemData infrastructure to support this naive model might be tricky in the future. I am currently evaluating 2 options: Using an indexer, which is more efficient and offers better consistency guarantees, vs using a distributed task queue which offers higher capacity (scalability). I hope to have some time this weekend to reach closure on the matter.
hey man,
pardon the off-topic comment…
I made a post today regarding a large-scale idea to advance Steem’s development, and it was recommended to share with the witnesses to help get this in front of the audience who’d be in the position to do something with/about it:
The $1 Billion Steem Development Fund: How Steemit Inc.'s Stake Could Be Best Allocated To Grow A Thriving Network Of Applications And Users...
would be cool if you could have a read, and IF you feel it’d be a great idea that’d serve the community, forward to anyone in particular you know who might be in a position of influence to advance the discussion.
either way, I thank you for your continued service to this community. 🙏
cheers,
Rok
Thanks for giving your service to this community! I truly appreciate that you do this, and I'm happy to support you in any way I can. Even though I don't understand it at all I know MANY depend on it, and that ultimately enhances my own steeming experience. Thanks for doing what you do so that I can come here and play!
Maybe this post can clarify you some questions. https://steemit.com/steem/@lightproject/recap-why-steemdata-is-so-important-for-steem
Yes! Thanks for bringing that to my awareness, @lightproject! This community is so helpful and resourceful!!!
Good job!
Excuse me @furion, accounts Collection seems to be not 100% synched. Been checking "lastpost" timestamps and are not updated. Also found couple of account that are not indexed. For instance, cannot find in Account collection account "vcelier".
Interesting...I'm looking into it.
Just for the record. Accounts that I found missing seem to be indexed now. Timestamps tho, seem not to be synched yet. Thanks a lot in advance for the efforts @furion.
Sounds great and great project!!!
I have just tried to make some Queries. if you get the account "cljg918" ( {name: "cljg918"} )
It says that it have 0 (zero) comments in "comment_count" but if I get comments by this author there is 700 and if I do the same query in steemDB it returns 1386? Do you know why?
HI, your web-api ( https://webapi.steemdata.com )
returns: 503 Service Temporarily Unavailable
Thanks for this update...SteemData very useful great work..
The Charts app is very insightful as well!
The charts app will be restored next week. Also, the Notify app is due for overhaul, and I have 1 unreleased app pending from the Steemfest 2 hackaton.
You are the best writer @furion
oh my god
your post is so amzing and really helped me
thank you very much
please help me to vote my post :) :)
good pos
Good to hear. Steady improvement is the key for long term success
still searching for the best steemit app to use.. the current one running on my android doesn't give me notifications and it always have a blank/white screen when i launch it.
I love what you are doing @furion .. i hope this update helps develop a better application for Steemit
Keep Steeming!!!
Nice! Any Viewly update regarding release?
This is an interesting read, quite interesting and good content. Thanks.
Great updates!
NICE good to know thanks for the update bro!
Thank you for the update! :) Looking through a bit this seems interesting! :3
Good
When you repeat the same comment on multiple post you sound like a bot! If it walks like a bot, squawks like a bot, it may be flagged for being a bot!
Your Reputation Could be a Tasty Snack with the Wrong Comment!
">The database is currently @ manageable 150GB. You can see the status of the indexers and other basic stats here.<"
Does database grows linear with growth of members? Say membership grows from 0.5 Millions ten times to 5 Millions, would database increase to 1.5 TB?
Would then every witness need to have 1.5 TB space reserved for Steemit? I' just curious how is DB scallability working, what happens when membership grows say 100 times.
It all depends on the user activity
For example In 'Accounts' collection objects I see arrays like 'tags_usage' or 'transfer_history'
So 1 user using a lot of tags and doing a lot of transfers takes more space than 100 inactive users
I think the size of the DB will mainly depend on number of posts and comments no matter if they will be done by 1000 users or 10 very active ones
But I'm not @furion so he would need to validate this
Great job
I still see missing posts hope the sync will finish soon
DB response times even for bigger queries is great
You either did a good job with Mongo perf tuning or have nice hardware for this :)
I am not in the "Accounts" list. All other i checked were inside. Any clue?
I need all people post plenty
SteemData seems much fitter overall, just made a quick check and many TimeStamps are synching!
Hi @furion I can see you good heart and hardwork to support steemians. That's why I upvoted you as one of the good witnesses. May you continue to support us! @jejes
How long to live mysql base? :)
Congratulations you are a true writer @furion thanks for the information
Good post
My name is muhammad rizal
I'm a new member of steemit
Nice to meet you
Please followback and upvote my post https://steemit.com/picture/@riskiakbar/drawing-of-3-dimension-8c267d06ca597
Thank you for sharing @furion ...
This is very helpful ....
voted and followed you :D
Thank you @furion for your great support.
hello@furion, i am @princessjoyesto from philippines. I just want to inform you that I voted you as my witness thank you in advance, have a great day ahead.