Hi there!
For those who have checked out the repo of the steem smart contracts tool (SSC) you may have seen that the storage layer is powered by LokiJS, a Javascript in-memory database. This library is great, the performances are awesome but that's thanks to its nature of being hosted in RAM... but we know what happens when we store data, too much data in RAM... it becomes expensive to maintain nodes and solutions need to be found in order to cut hosting costs.
When I started to work on SSC, I was just trying to "copy" ideas from different "blockchains" and integrate them into the tool, while I saw having a in-memory database was a must have to ensure performances I then figured out that, well, is that really necessary? I saw what Steemit Inc. had to do to solve that problem and now I don't want to run into that issue in the future so I decided to work on replacing LokiJS.
LokiJS is a no-sql database that uses an API that is very similar to what we can find on the very well known MongoDB database engine, so guess what? I decided to change the storage layer of SSC to MongoDB, that wasn't really painful... it took me a few days to figure everything out but it's working and it's working great!
One cool thing with MongoDB is that you can use different storage engines with it, WiredTiger, the default one that's shipped with MongoDB, but other companies worked on integrating other engines and so you can for example find RocksDB as one of the engines shipped in the fork of MongoDB maintained by Percona (they also have the in-memory engine if you really want to keep your database in RAM...)
One major drawback with moving to MongoDB is the replay time, I didn't calculate exactly but that's wayyyy longer...BUT you don't need to replay when you use MongoDB because you can simply get a dump of the database from another node an import it into your database...then you just need to run your node and it will only have to process the blocks that weren't included in the dump.
Moving to a well know database engine also comes with benefits, MongoDB offers solutions for scalability (replica sets, sharding) and depending on the performance you want to achieve with your node, you can choose the storage engine that best fits the hardware you have; you have a crazy server with tons of RAM because you know your node is going to be highly solicited then go with the in-memory engine, you're short on RAM, go with WiredTiger or RocksDB, that's up to you!
I've fired up two nodes to test this migration out, you can interact with these them via these two links:
- https://harpagon210.github.io/steem-engine : MongoDB 4.0 with WiredTiger
- https://harpagon210.github.io/steem-engine-rocksdb : MongoDB 3.6 with RocksDB
You can try to browse around and check the data against https://steem-engine.com/ and it should (surprisingly?) be the same... (except for the block hashes which are calculated based on the database hashes which are different between LokiJS and MongoDB)
Thanks for the update @harpagon. I wish I knew what 80% of this said but it sounds like you made a break through.
It is good that you are trying to head off scaling and memory issues off before they get started.
You work is a blessing to this entire ecosystem.
what are you seeing for cost difference with these? lack of ram use is going to save monies?
as of now, it's not going to have an impact on the cost of running an SSC node, but in the long run it will definitely! so yes, people who run SSC nodes will save money later on.
Hi, @harpagon!
You just got a 0.04% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.
Congratulations @harpagon!
You raised your level and are now a Dolphin!
Do not miss the last post from @steemitboard:
Vote for @Steemitboard as a witness to get one more award and increased upvotes!