As a former witness operator, I have had intimate dealings with the software that runs the Steem blockchain. Since I first started working with it, back in November, and gradually getting worse until now, the rate at which the blockchain replay takes place (necessary to get an RPC server running, which is necessary to run Steem based applications), has slowed down at an alarming rate.
Note that facebook uses massive clusters of a specialised database that has the ability to break the database apart, and runs queries entirely in memory, in giant data centers. Steem's success necessarily will cause this database to grow beyond the capabilities of graphene, and even, eventually into the size of a database like Reddit or Facebook (at least, circa 2-3 years ago).
Because I was curious to understand why, it turns out that the database system, Graphene, a graph database, is very unsuited to storing the data of a forum like this. The memory utilisation becomes enormous and at a certain point, the process of creating search indexes out of the transaction log (blockchain) basically goes on for eternity.
It already goes very slow. 6 weeks ago I was trying to get an RPC up and running so I could do some development of Steem based applications. 2 weeks later, I still was unable to get one running. Yes, I stopped the process a few times through that, but I should not still be waiting for an RPC replay to finish 3 days after I start it. You can imagine how inconvenient that is with a computer basically doing nothing productive for this long, and no certainty it will ever finish doing anything. I certainly was not going to be able to do any kind of programming that required access to the database.
If something is not done soon, everything will go offline. Steemit.com, all the exchanges, and all the steem apps.
People think I am just being a chicken little about this, but just look at the situation. Poloniex trading in steem is down now for over a month. Changelly ditched steem some time ago. Only blocktrades, one of the founders, and bittrex are still functioning. The frequency of outages at steemd.com and steemdb.com is becoming alarming.
I wish my upvote was worth more, because I am very interested in your response. Since I'm not that technical that meant something to me, but I'm still a bit unclear.
How about this. Let me ask you: What would you do, if you had the ability in order to overcome this obstacle?
just guessing I would say replace Graphene with something less resource demanding. But is that like asking a mountain to move?
Thank you for your honest opinion, it means a lot to us.
Doesn't hurt me at all. My node doesn't need near as much memory as a service that needs to track account balances like, you know... an exchange.
If you want to see Steem stick around and even potentially make money with it, what you're doing is so short sighted it's like a dog eating its own leg because it's hungry.
Errrrr....Bittrex unavailable for 48 hours now...Automated Maintenance...
I have been here for more than a year, posting, commenting and curating, encouraging newbies, organised Steemit meetups....but finally decided to power down and call it a day. I only hope thete will be somrthing to trade in the next 13 weeks...I have put in a lot of work for that SP. Would like to get something out
They not hurting, so no big incentive.
Is Status not something similar? https://blog.status.im/tagged/introduction
This is very valuable information @loki/@elfspice. I've been following you from way back and have always valued your honest opinions even though they don't praise Steem always( neither do mine ). We have to be critical and not just bury our heads into the sand. There are only few people with as long history with steem as you, and even fewer who really know the technology and can critique it. I know I can't do that.
So a big thank you for making this comment! I've also noticed the increased mainteance rate in Steemd and have wondered what the hell is the issue with exchanges and Steem. I'd love to hear more about your opinions and findings on this.
You might not at times feel welcome here, but nevertheless, you're an valuable member that has the best interest of whole rather than few in mind. Sadly that isn't rewarded here. That's at least mine perception of you, hopefully I'm not wrong!
I'm not native english speaker nor a professional in any way at computer science so I don't understand all the details and terms you're trying to explain and use. This is something I'll try to fix with further education and I can always come back to this answer to fully understand it at some point in time.
I did however understand(or did I?) that it's not wise to store every post as plain text data and it rather should be coded into a string that could be searched,processed and stored efficiently.
Thank you for the detailed answer!
Interesting. So majority of people isn't informed enough to see this problem at all and those who know aren't voicing it.
It's starting to feel Steem was just hastily put together using tools available and without that much thinking for the long term (+5 years and beyond).
Doesn't really make me want to put money to EOS at all.
How familiar are you with Ethereum? I'd like to hear your view on it.
I remember reading about pruning and ethereum in a same sentence before, they must be working on it.
Only time will tell, but there's really no returning back for Dan if he leaves EOS shortly after, but at that point he can buy an island or two for himself so it'll be fine.
And yes, Dan always mentions how he can spit out these different projects out fast and then he leaves the rest to fix them or to keep together. Bitshares seems to be holding fine though.
So do you think there's any blockchain that's doing "it" right now or have we yet to see such a project? Will any of these projects be running in 10 years.
What you said in the last paragraph is where I think Dan is going with his "Blockchain OS" concept. You raise some great points and are obviously no spring chicken to coding and systems architecture.
Having a solid foundation upon which to build higher layers is an important architectural principle. As things change in the upper layers they impose new requirements on the lower layers that test how flexible and comprehensive the foundation layers were designed. Sometimes a redesign of the foundational layer is more cost effective or quicker to implement than trying to retrofit changes (refactoring) to meet new requirements.
I'm reminded of the way the systems of the Voyager space probe were designed, both hardware and software, and b/c that design was so good the probe had a much longer lifespan that was ever anticipated. This depth of engineering and innovation is what we need for systems whose aim is to make mainstream institutions obsolete. We need long term thinking motivated by the value to humanity, not limited by greed or personal gain. An attitude of altruism but marketed in a way that provides support.
I want to read more of your ideas to motivate coders via 'monetisation of code', but on the surface it doesn't address this altruistic / innovation aspect for the long haul. It does address personal rewards for the coders, but it's divorced from the goal and purpose of the coding effort. It's not easy to anticipate future needs and build flexible systems than are adaptable to those needs in a cost effective way. How that type of innovation is incentivized and rewarded is the distinction I'm getting at.