Repository
https://github.com/Cajarty/Seed
Overview
The following development changes were based upon the design article Transaction and Block Storage.
Whether a cryptocurrency is based upon blockchain or directed acyclic graphs, there is one aspect that is certain; Blocks and transactions are meant to be stored. An app is not meant to stay open constantly, as users should be able to close an app without worrying about losing the data. Losing data would mean requesting more data over the network whenever a user tries to log back on. We want to minimize network requirements, especially if Seed is to achieve its goals of being a high-throughput, low-latency blockchain solution.
This article goes into the implementation of storage across various environment requirements, the implemented "database" schemas used, and storage compression.
Storage
The "Storage" class handles the high level concept of saving/loading transactions and blocks. This class is accessible through the LLAPI, and can be used by clients to request saving and loading.
The implementation of saving and loading is abstracted away due to differing requirements from the differing environments.
Browser vs. Desktop Constraints
Due to the Seed LLAPI being environment agnostic, it cannot be assumed whether a user is running Seed on a web browser, a server, or an Electron app. As such, the storage implementation may or not have access to differing storage options, such as the file system, LocalStorage or Cookies.
Modular Storage With IDatabaseInjector
The differing environments added the constraint of modular storage. The act of "storing a block" or "storing a transaction" must not care about which platform is being used, however the implementation behind that action must be able to choose which database provider suits our current platform.
This led to the creation of the "IDatabaseInjector' interface. Although JavaScript is a dynamic language and does not have true interfacing built in, we can still create objects with the same function names, and then use them as if they were objects sharing an interface. This "interface" is more of a pattern to follow, for which the Storage object will expect for usage.
The expected functions for all IDatabaseInjector implementations to have are the following:
readBlockAsync(generation, storageName, callback)
readBlockSync(generation, storageName)
readBlockchainSync(generation)
readBlockchainsSync()
readTransactionAsync(storageName, callback)
readTransactionSync(storageName)
readEntanglementSync()
writeBlockAsync(storageName, storageObject, generation, callback)
writeTransactionAsync(storageName, storageObject, callback)
removeTransactionAsync(transactionName, callback)
removeBlockAsync(generation, blockName, callback)
Following an interface pattern for the storage implementation helps decouple the saving/loading logic from the environmental constraints.
Data Compression
Data compression can be enabled or disabled upon the creation of a Storage object.
Compression has a time and place. DApps that are not computationally exhaustive, such as a cryptocurrency Wallet, care little about performance. These types of DApps would see a lot of benefits from compression, as their stored data can be leaner in size without affecting the DApps usability. However, DApps which have high performance requirements, such as a live updating video game, may prefer to sacrifice the storage size gain from compression, in order to use less computing time saving/loading from storage.
Both types of DApps are encouraged to exist on our platform, therefore the act of data compression is optional.
zlib
For compression, the Storage object uses zlib, a compression library built into NodeJS. This library is both a top performing NodeJS library, while not adding another dependency to the project. For these reasons, it was the obvious choice for a compression library.
Implementation
The Storage exports expose the three primary functions that clients will desire using, while the Storage class has more functions for more advanced usages.
The three primary exported actions are loading the initial state on launcher, saving a transaction, and saving a block.
Loading Initial State
When loading the initial state, the order of execution may matter for applying transactions and blocks. For this reason, blocks and transactions must be read in as close to proper order
as possible.
First all blocks are read in by the database injector, then sorted by timestamp. They are then checked for validity and added to the blockchain, applying its changes to the ledger.
Once all blocks are processed, all transactions in the saved Entanglement are read, then sorted by their timestamps. These transactions are added to the live Entanglement
one-by-one, validating that each transaction meets all validation checks.
Save Block
When saving a block, all transactions in storage which belong to that block must be removed by the database injector first. After removing all of the transactions belonging to the block, the databse injector will write the block to storage.
Save Transaction
Saving a transaction must be done when creating a transaction to propagate, or receiving one from the network. These transactions are saved to the stored entanglement by the database injector.
IDatabaseInjector Interface
The IDatabaseInjector interface abstracts away the implementation of the act of storage, from the higher level logic surrounding storing. A different implementation will be needed for each environment to be used, such as LocalStorage for web apps, the File System for Electron apps, or potentially even a MongoDB or other REST storage solution.
Implementation
The expected functions for all IDatabaseInjector implementations to have are the following:
readBlockAsync(generation, storageName, callback)
readBlockSync(generation, storageName)
readBlockchainSync(generation)
readBlockchainsSync()
readTransactionAsync(storageName, callback)
readTransactionSync(storageName)
readEntanglementSync()
writeBlockAsync(storageName, storageObject, generation, callback)
writeTransactionAsync(storageName, storageObject, callback)
removeTransactionAsync(transactionName, callback)
removeBlockAsync(generation, blockName, callback)
The implementations may change, however many implementations may be very similar. As such, the IDatabaseInjector.interface file outlines a basic implementation for what is expected, with comments where code would need to be added.
FileSystemInjector
The FileSystemInjector implements data storage through the file system's read/write as multiple files in folders. Due to relying on the operating systems IO, this implementation of the IDatabaseInjector interface cannot run on all environments, such as certain browser web apps. This implementation was built primarily for the Seed Launcher and other Electron DApps, but should work on all systems which have file system read/write access.
Storage Schema
Blockchains
Blockchains are organized by the generation of blocks within the chains. Within each chain is an array of blocks named by their block hash. Within each block is the data it represents.
This relationship can be easily represented with files and folders. In the base folder, /data/blockchains/, will be strictly subfolders named by which generation of blocks reside within it. If all our blocks are first generation blocks, there would simply be one folder, /data/blockchains/1/.
Within each subfolder would be a bunch of files. Each file is in the .json file format, representing blob storage of an object. These files contain a single block, where that file is named after the blocks hash.
Entanglement
For the Entanglement storage, transactions are simply stored in a /data/entanglement folder. Each transaction is a .json file containing the transaction in JSON format, with the file name being the hash of the transaction.
Implementation
The FileSystemInjector implementation can be found in the fileSystemInjector.js file. Reading and writing from the file system was accomplished through relying on the "fs" and "path" modules built into NodeJS. Although these libraries are built into NodeJS, they cannot be used on all systems, as some platforms may not have access to the file system.
Once created, the FileSystemInjector will ensure the proper folder structure is ready for use. The following base structure should be added to the clientSrc folder during runtime.
LocalStorageInjector
The LocalStorageInjector implements data storage through a passed in LocalStorage object for reading and writing. Due to relying on the LocalStorage object, this implementation requires the environment to have DOM access, and is intended to be used for web apps.
Storage Schema
Blockchains
As LocalStorage follows a key/value pattern for data storage, the subfolders approach taken in the FileSystemInjector is not viable. However, blocks must still be organized by generations, in order to be organized by blockchains.
Blocks are stored following a naming convention for their keys. The key is the block's generation followed by the block's hash, being separated by an underscore. For example, "1_BlockHash1" would be the key for storing a block who's hash is "BlockHash1" as a first generation block.
Entanglement
Transactions for the Entanglement are stored with their key being their transaction hash. Due to the underscore not being a character found in base56 encoded characters, we know that if a underscore is found in the name, it's a block. Otherwise, if its a valid hash it has to be a transaction.
Implementation
The LocalStorageInjector implementation can be found in the localStorageInjector.js file. Reading and writing from LocalStorage was accomplished through relying on users to pass in the "LocalStorage" object. For users to have a LocalStorage object, they must be in an environment with DOM access.
Launcher Implementing Storage
The Seed Launcher is an Electron app, and as such, Electron has access to the computers file storage. Therefore, the Launcher uses the computers file system rather than local storage. as its database injector implementation of choice.
Implementation
When the Main process has finished its startup cold, the launcher's main.js file accesses the Seed LLAPI, requesting the creation of a Storage object. It passes in a FileSystemInjector, created by the LLAPI as well. Now, whenever the system adds a new transaction or block to the Entanglement or Blockchain, it will store it to the file system.
Load Button
In the Electron Launcher, a button was added to begin loading from the file system. This would read all transactions and blocks from the /data/ folder at the root directory of the /clientSrc/, allowing us to test the storage feature manually.
Demo
The demo will consist of showcasing how transactions and blocks get stored in the file system while invoking the unit tests. Afterwards, a new instance of the Launcher will load the data from file storage. For the purpose of showcasing data within files, compression will not be enabled, however both options are fully functional.
Create Seed Constructor
After manually invoking the Seed module constructor in the Seed DApp (keeping all default values, as the unit tests rely on the default values for construction), a single transaction is added to the Entanglement.
By opening the transaction in a text editor, we can inspect the transaction, and confirm it is our constructor.
Run Unit Tests
In a new instance of the demo, we run the unit tests from the beginning. Upon completion, the file system will have multiple transactions, with zero or more blocks. For our demo, we had one block trigger, compressing multiple other transactions.
Loading Data
The following video showcases a newly launched Seed Launcher loading in files from storage.
The files from the "Run Unit Tests" section above can be found on the left side of the video. In the Launcher, the "Load From Disk" button is clicked, triggering loading the files into the launcher. In the terminal, various logs are printed confirming various transactions and blocks. Upon completion, the final logged data is the ledger's end state after loading. It displays that there is 975 SEED in circulation, among other data. This "975" is the final amount in circulation after running the unit tests, confirming the data was loaded properly.
In the Seed launcher, we can see the "Seed" total changing from a zero to a 850. At the end of the unit tests, the default user ends with 850 Seed, confirming that the client can properly read the newly read in data.
References
GitHub
https://github.com/CarsonRoscoe
Thank you for your contribution. Again a very well written post, I really like the way you have added all the information in the pull request. The commits are descriptive as well as the comments in the code.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thank you for your review, @codingdefined!
So far this week you've reviewed 2 contributions. Keep up the good work!
Hey, @carsonroscoe!
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!