[ANN] steemtools - A High-Level Python library for Steem

[furion]

graphics by @etherdesign

Why steemtools

I have been pretty busy in the past month, working on several Steem related projects:

• SteemQ, a video platform on top of STEEM
• steem.li, real-time voting and curation tools
• IM market-maker, running 24/7
• a set of curation bots
• weekly reports on various topics

I had spent a fair amount of time writing code and looking for solutions that would enable me to in part do these things.

The need for the library emerged, as I realized I had scattered code over all of my projects, in many cases, implementing the same functionalities.

I have also been contacted by half a dozen people on https://steemit.chat, asking for advice on certain problems - problems I had already solved myself.

I think that different people solving the same problems makes no sense - when all we really want is to achieve a higher level goal.

The purpose of this library is to enable others to do what I do, and avoid the colossal waste of time.

Acknowledgments

steemtools is built on top of piston and python-steemlib, the awesome Python libraries made by @xeroc.

@jesta has been very helpful in #piston channel on https://steemit.chat, and steem.ws, a fast and reliable node cluster has saved me a lot of time, and allowed me to keep working while my local node was re-compiling/replaying.

And lastly, @burnin, a guy that reverse-engineered the voting mechanisms, and saved me so much time with the Converter module.

Thank you guys, I could not have done this without you.

Installation

I highly recommend starting off with Anaconda distribution of Python 3.5.

After that, we only need to run one command:

pip install steemtools

Modules

steemtools is currently comprised of 4 modules:

• blockchain: all utilities for blockchain traversal/parsing
• base: contains our Post, Account and Converter classes
• helpers: static helper functions
• node: a convenient way to connect to local RPC, with automatic failover to steem.ws

Blockchain

---

Replaying History:

from steemtools.blockchain import Blockchain

for event in Blockchain().replay():
    print("Event: %s" % event['op_type'])
    print("Time: %s" % event['timestamp'])
    print("Body: %s\n" % event['op'])

This function allows us to go back in time, and replay the entire STEEM blockchain from start to finish. Once it reaches the present moment, it will keep going, printing events with every new block (every 3 seconds or so).

The output will look a little bit like this:

Operation Types:
Perhaps we aren't interested in all the events, but just specific ones.
We can ask replay to only give us votes:

for event in Blockchain().replay(filter_by="vote")

Or a set of events, such as votes and comments:

for event in Blockchain().replay(filter_by=["vote", "comment"])

For the reference, the full list of currently available operations is:

blockchain_operations = [
    'vote', 'comment', 'delete_comment', 'account_create', 'account_update',
    'limit_order_create', 'limit_order_cancel',
    'transfer',  'transfer_to_vesting', 'withdraw_vesting', 'convert', 'set_withdraw_vesting_route',
    'pow', 'pow2', 'feed_publish',  'witness_update',
    'account_witness_vote', 'account_witness_proxy',
    'recover_account', 'request_account_recovery', 'change_recovery_account',
    'custom', 'custom_json'
]

Time Constraints:
Parsing the ENTIRE blockchain is often unnecessary. We can specify a desired range:

start_block = 3598980
end_block = 4260042
for event in Blockchain().replay(start_block, end_block=end_block, filter_by=["vote", "comment"]):
    pprint(event)

Working with block numbers is painful, and this is why blockchain module comes with 2 helpers:

get_current_block()
get_block_from_time("2016-09-01T00:00:00")

Putting it all together...

b = Blockchain()
history = b.replay(
    start_block=b.get_block_from_time("2016-09-01T00:00:00"),
    end_block=b.get_current_block(),
    filter_by=['transfer']
)
for event in history:
    payment = event['op']
    print("@%s sent %s to @%s" % (payment['from'], payment['amount'], payment['to']))

The above code will fetch all the transfers from September 9th going forward, up until present.

...
@victoriart sent 1.000 SBD to @null
@dude sent 5.095 STEEM to @bittrex
@devil sent 5.107 STEEM to @poloniex
@pinoytravel sent 0.010 SBD to @null
@aladdin sent 5.013 STEEM to @poloniex
@mrwang sent 31.211 STEEM to @blocktrades
@kodi sent 0.030 SBD to @steembingo
...

Account History

---

Account module allows us to lookup virtual operations, as well as some of the common operations for an individual account.
Usually it is more efficient to query the account history over parsing the blockchain block-by-block.

Looking up the account history:
Accounts module gives us 2 generators for the task, history, which gives us account history from inception forward and history2 which gives us account history newest to oldest. The interface should look familiar, as it is similar to the replay from the blockchain module.

from steemtools.base import Account
from steemtools.helpers import parse_payout

for event in Account("furion").history(filter_by=["transfer"]):
    transfer = event['op']
    if transfer['to'] == "null":
        print("$%.1f :: %s" % (parse_payout(transfer['amount']), transfer['memo']))

The code above will pull the transfer history for my account, find promoted posts by looking for transfers to @null, and finally print the $ amount spent as well as the permlink to the post.

$11.1 :: @furion/steem-analysis-ownership-distribution-and-the-whale-selling-pressure
$11.0 :: @furion/using-machine-learning-to-fight-plagiarism
$41.0 :: @furion/a-quick-look-at-null-and-the-profitability-of-promoted-posts

Virtual Operations:
As mentioned above, there are several operations that are virtual, and we cannot obtain these by parsing the blockchain itself. We can however lookup the history of virtual operations on a specific account with the history method.

Account("furion").history(filter_by=["curate_reward", "fill_order"])

Currently, the following types are available for the lookup (both virtual and not):

account_operations = {
    'account_create',
    'account_update',
    'account_witness_vote',
    'comment',
    'comment_reward',
    'convert',
    'curate_reward',
    'fill_order',
    'fill_vesting_withdraw',
    'fill_convert_request',
    'interest',
    'limit_order_cancel',
    'limit_order_create',
    'transfer',
    'transfer_to_vesting',
    'vote',
    'witness_update',
    'account_witness_proxy',
    'feed_publish',
    'pow', 'pow2',
    'withdraw_vesting',
}

Account Methods

---

Account has several helper methods. Here are a few:

from steemtools.base import Account

account = Account("furion")

account.get_sp()
#> 6211.590278675119

account.reputation()
#> 62.76

account.voting_power()
#> 80.75

account.avg_payout_per_post()
#> 142.7166

We can also easily obtain the latest blog posts. Lets get the titles of most recent 3:

blog = account.get_blog()
for post in blog[:3]:
    print(post['title'])

# outputs:
# A quick look at @null, and the profitability of Promoted Posts
# A quick look at the top curators and their rewards
# Homepage Payout Distribution, Power Law and Project Curie

How about a list of followers:

followers = account.get_followers()
#> ['anns', 'benjy33', 'negoshi', ...]

Lets obtain the curation stats:

account.curation_stats()
# outputs
# {'24hr': 9.627790750805277, '7d': 57.82547153222017, 'avg': 8.260781647460025}

Or get a basket of features:

account.get_features(max_posts, payout_requirement)

Outputs:

{'author': {'followers': 281,
  'post_count': 10,
  'ppp': 142,
  'rep': 62.76,
  'sp': 6211,
  'ttw': 281.0,
  'winners': 2},
 'name': 'furion',
 'settings': {'max_posts': 10, 'payout_requirement': 300}}

Check out the base.py module for all the methods.

Posts

---

Post is a superset of piston.steem.Post. This means that, it behaves the same way, and has all the niceties and helpers that piston's Post object has.

We can initialize it in any of these ways:
a) using the identifier string

Post("@furion/homepage-payout-distribution-power-law-and-project-curie")

b) using a piston.steem.Post object

last_post = Account("furion").get_blog()[0]
Post(last_post)

c) using a author+permlink containing dictionary, such as vote

for vote in s.rpc.stream("vote"):
        print(vote)
        print(Post(vote))
        
# {'voter': 'ats-david', 'author': 'whatsup', 'weight': 10000, 'permlink': 're-steve-walschot-investigating-the-wale-scam-assumptions-above-knowledge-20160912t020220232z'}
# <Steem.Post-@whatsup/re-steve-walschot-investigating-the-wale-scam-assumptions-above-knowledge-20160912t020220232z>

---

Now that our Post is initialized, we have access to extra methods. Here are a few:

p = Post("@furion/homepage-payout-distribution-power-law-and-project-curie")

p.payout()
#> 456.003

# if the post was 10 minutes old, this would output 33.33
p.calc_reward_pct()
#> 100

p.is_comment()
#> False

# not a spam tagged post
p.contains_tags(filter_by=["spam"])
#> False

p.get_votes()
#> [list of active votes]

Check out the base.py module for all the methods.

Converter

---

Converter is a class that tries to convert/calculate different units. I won't go into details in this post, aside from listing the available methods:

from steemtools.base import Converter

c = Converter()

c.sbd_median_price()
c.steem_per_mvests()

c.vests_to_sp(vests)
c.sp_to_vests(sp)
c.sp_to_rshares(sp)
c.steem_to_sbd(steem)
c.sbd_to_steem(sbd)
c.sbd_to_shares(sbd)
c.rshares_to_weight(rshares)

Helpers

---

Helpers is a set of static methods that are generally useful when dealing with Steem objects.

from steemtools import helpers

# get just the digit part of any asset (STEEM, VESTS, SBD)
helpers.parse_payout("12.3456 SBD") 
#> 12.3456

# break down our asset
helpers.read_asset("12.3456 SBD")
#> {'symbol': 'SBD', 'value': 12.3456}

# get time difference in seconds between lets say 2 posts
helpers.time_diff(time1, time2)
#> 1337

# determine if our object is a post or a comment
helpers.is_comment(object)
#> False

# time elapsed in seconds 
helpers.time_elapsed(post)
#> 9001

Going forward

I wanted to be as minimal as possible with the first version, and only include the most essential components.

There are many new features, and areas in which steemtools can be expanded upon. For instance, an exchange module, that would allow for quick bootstrapping of a market-maker or arbitrage bot. These features may be added later on.

For now, I am looking for feedback and new ideas on how to improve whats already there, and make the library stable, so people can depend on it.

Github: Netherdrake/steemtools

---

Don't miss out on the next post. Follow me.

---

Comments

[furion]

If you don't have a local node (its highly recommended to have a local node for use of this library), you will need to get the develop version of piston installed, to get the automatic fallback to remote node (new feature in python-steemlib, not on master yet).

Just run this after installing steemtools:

pip install --upgrade --no-deps --force-reinstall  git+git://github.com/xeroc/piston@develop
pip install --upgrade --no-deps --force-reinstall  git+git://github.com/xeroc/python-steemlib@develop
pip install --upgrade --no-deps --force-reinstall  git+git://github.com/xeroc/python-graphenelib@develop

[chitty]

I couldnt wait and I built a simple script using steemtools: https://steemit.com/steemtools/@chitty/share-your-code-using-steemtools-to-get-an-accounts-info

Thanks a lot to all you guys, I am just learning python and could never build something myself without steemtools lol

[noisy]

High level perspective of this library is great! :) I love it :)

So right now stack looks like this:

steemtools use piston
piston use steemlib
steemlib use graphenelib
graphenelib connects with steemblockchain

[bitcalm]

This will make a nice addition to the other programming tools available for steem. Looking forward to getting my hands dirty with it. Thanks!

[jesta]

This is absolutely fantastic! I started writing some of this code the other day for the @bounty bot, I think since my codes still rather immature and not a working library, I might have to take a stab at using this instead.

I was going through and analyzing @burnin's code as well hahaha.

[cryptomental]

I definitely will check your tool and start using it. Did you also make any experiments with the steem debug node? I mean the node started with this script: https://github.com/steemit/steem/blob/develop/python_scripts/steemdebugnode/debugnode.py . Your library looks like a good candidate for the test/experimental tool to interact with the blockchain.

[furion]

I will look into it. Thanks for bringing it up.

[bhavnapatel68]

All the code language

[kaptainkrayola]

This almost makes me want to learn Python :) . Looks awesome

[bitcoiner]

Do it! Join the Python side.

[l0k1]

i for one welcome you all to the whitespace darkside. i have been a student of CS since i was a littlie, and readability has always been a problem for new folk looking at old code.

[kaptainkrayola]

If i can get over it being a whitespace language I just might. Seems to have a big following with Steem and I'd love to be able to work on some of those projects with you fine devs.

[bitcoiner]

Well, at least it's not as bad as Brainfuck! Haha. I guess the good thing about whitespaces denoting a block is that the code will always be properly indented and there will be no debate about where the first curly brace should be. And you are the finer dev!

[kaptainkrayola]

I agree with that in principal but I find that I'm a stubborn bastard and when a language/framework tells me how to do something I want to do something else. Some people say I have a problem with authority :)

[bitcoiner]

I actually still haven't gotten used to the whitespaciness of Python even after trying to learn it for many years now. I'm just way more comfortable in a C++/Java/js/PHP/etc. syntax. I have to Google things every other minute when trying to write even a simple Python script.

[good-karma]

Great work, mate... more library tools more developer apps and helps new developers in future to keep building on top of Steem!

[cryptohazard]

In the end normal steemit users will never win. Technical skills are too much of an advantages: html, bots, miners, ...

[l0k1]

and so it should be. ignorance is no kind of virtue.

[cryptohazard]

maybe but Steemit is not advertised that way. From the idea they present and the reality, the gap is huge. This is not about ignorance, it's about what kind of users you want on the platform.

[bonitaxm]

thanks for this fantastic resource, making the library available to avoid reinventing the wheel, and as you said: "when all we really want is to achieve a higher level goal." This will enable faster and efficient development of new useful tools for the Steem community.

[jasonxg]

Thats exactly it. A source of really wonderful and powerful tools at our finger tips. Like you say, no need to onvent the wheel twice.

[airmike]

wow, very nice and useful post. thank you

[pgarcgo]

Impresive! Good job!

[cristi]

ah damn! can't wait to get my hands on it! :D you rock ;)

[steevc]

This looks really cool. I've been learning Python and I need to try it out on Steemit. Thanks

[jasonxg]

There so many great steamtools its bloody brilliant !

[clayop]

Can I add other nodes? Currently public node (steem.ws) seems down.

[furion]

Every class takes steem as optional second argument.

So you can do something like:

my_steem_instance = piston.steem.Steem(node="mysteemnode.com")
Account("clayop", steem=my_steem_instance).get_sp()

I think I should add more fallback nodes into the default, thanks for pointing it out.

[chitty]

amazing! thanks a lot

[fooblic]

Great tool! One point I could not catch: how you get virtual operations like "reward" if they are not in blockchain? What a database store such transactions and how we can trust its without confirmations in blockchain ? Thanks in advance for reply.

[denise12]

Syntax and switches

The command syntax is
netstat [-a] [-b] [-e] [-f] [-n] [-o] [-p proto] [-r] [-s] [-t] [-v] [interval]
A brief description of the switches is given in Table I below. Some switches are only in certain Windows versions, as noted in the table..Note that switches for Netstat use the dash symbol "-" rather than the slash "/".

Switch Description
-a Displays all connections and listening ports
-b Displays the executable involved in creating each connection or listening port. (Added in XP SP2.)
-e Displays Ethernet statistics
-f Displays Fully Qualified Domain Names for foreign addresses. (In Windows Vista/7 only)
-n Displays addresses and port numbers in numerical form
-o Displays the owning process ID associated with each connection
-p proto Shows connections for the protocol specified by proto; proto may be any of: TCP, UDP, TCPv6, or UDPv6.
-r Displays the routing table
-s Displays per-protocol statistics
-t Displays the current connection offload state, (Windows Vista/7)
-v When used in conjunction with -b, will display sequence of components involved in creating the connection or listening port for all executables. (Windows XP SP2, SP3)
[interval] An integer used to display results multiple times with specified number of seconds between displays. Continues until stopped by command ctrl+c. Default setting is to display once,
Applications of Netstat

Netstat is one of a number of command-line tools available to check the functioning of a network. (See this page for discussion of other tools.) It provides a way to check if various aspects of TCP/IP are working and what connections are present. In Windows XP SP2, a new switch "-B" was added that allows the actual executable file that has opened a connection to be displayed. This newer capability provides a chance to catch malware that may be phoning home or using your computer in unwanted ways on the Internet. There are various ways that a system administrator might use the assortment of switches but I will give two examples that might be useful to home PC users.

[linkback-bot-v0]

This post has been linked to from another place on Steem.

• SHARE YOUR CODE: Using SteemTools to get accounts info by @chitty

Learn more about linkback bot v0.4. Upvote if you want the bot to continue posting linkbacks for your posts. Flag if otherwise.

Built by @ontofractal