For anyone doing work with Steem data, having a flat file of the entire Blockchain is very convenient for fast, bulk access.
The only dependency is the Official Python Library, which we can install with:
pip install -U steem
The script
import json
import os
from contextlib import suppress
from steem.blockchain import Blockchain
def get_last_line(filename):
if os.path.isfile(filename):
with open(filename, 'rb') as f:
f.seek(-2, 2)
while f.read(1) != b"\n":
f.seek(-2, 1)
return f.readline()
def get_previous_block_num(block):
if not block:
return -1
if type(block) == bytes:
block = block.decode('utf-8')
if type(block) == str:
block = json.loads(block)
return int(block['previous'][:8], base=16)
def run(filename):
b = Blockchain()
# automatically resume from where we left off
start_block = get_previous_block_num(get_last_line(filename)) + 2 # previous + last + 1
with open(filename, 'a+') as file:
for block in b.stream_from(start_block=start_block, full_blocks=True):
file.write(json.dumps(block, sort_keys=True) + '\n')
if __name__ == '__main__':
output_file = '/home/user/Downloads/steem.blockchain.json'
with suppress(KeyboardInterrupt):
run(output_file)
The only thing that needs changing is this line, which indicates a location where the flat file will be stored:
output_file = '/home/user/Downloads/steem.blockchain.json'
You can run this script as many times as you like, and it will continue from the last block it synced.
Useful Utilities
To see how many blocks we currently have, we can simply perform a line count.
wc -l steem.blockchain.json
To inspect a particular block, and pretty-print it:
sed '10000q;d' steem.blockchain.json | python -m json.tool
Replace 10000 with desired block_number + 1.
Great work @furion! Thanks for the dedication and countless hours of hard quality work. Namaste :)
Hope to see more post like this(blockchain dev etc.).
Congratulations @furion!
Your post was mentioned in my hit parade in the following categories:
I am interested in going into the block chain coding business. How do you think is the best way for me to get started? I am going to follow you for more news @furion
good share.
resteem. GREAT INFO!
Thank you for sharing.
cool thanks
How big is the blockchain right now? I am trying to find a tool that says it in GB (or terrabytes lol)
9.5GB as of 20 minutes ago
Nice information.. forward to your post ...
Great post, and it 's obvious that your contribution to the community is highly appreciated. I'm not a programming whiz but I like looking at the code from time to time. I will definitely check this out. Thanks again!
question can you point me what to do if I want only the accounts' name?
I need more and more posts like that.
To clarify, you want to get a list of usernames?
You can use
s = Steem() accounts = s.get_all_usernames()thanks
I would like to do this, but the whole takes up a lot of space.
Hi, @furion,
I've just "stumbled" across your blog today while doing a search on Steemit.
Please pardon me if this is a little off-topic, but I've been getting increasingly desperate to find a solution to the problem of editing my older posts. I've written a number of articles about this (most recently HERE), but can't seem to find "the right people" to talk to about it.
And so, I'm stopping by here to ask if you know of any utility that can "unlock" my posts older than 7 days so that I can edit them? As a frequent content creator, I have a great and pressing need to be able to do so.
I would appreciate any leads, guidance, suggestions, recommendations about who to contact - in other words, anything that you might be able to do to set me on the path to gaining edit access to my old posts would be greatly appreciated.
Thank you in advance for any help or ideas you may be able to offer!
😄😇😄
