Steem-Python API and RPC Speed

[danielsaori]

Some struggle with Steem-Python

_{@dorabot Protector of Peace}

Using steem-python to access the Steem API can sometimes turn out to be painfully slow. Especially using the official Steemit RPC nodes.

The RPC nodes seem to be frequently under heavy load and can cause extremely unreliable response times. In many cases this will only lead to slow result, but it can even end up with incomplete and missed data. One example of this is trying to stream data from the blockchain using the function stream_comments().

Part of my goal with @dorabot was to enable interactions via Steemit. To make that happen, I needed a reliable way to process all comments posted in the blockchain. My initial choice was to use a function called stream_comments in the Steemd class.

def stream_comments(self, *args, **kwargs):
        """ Generator that yields posts when they come in
            To be used in a for loop that returns an instance of `Post()`.
        """

_{(Steem-Python on Github).}

The stream_comments() function is constructed to listen to all the blocks in the Blockchain and return a Post object for each post/comment. Very convenient, but returning a Post object means there is an additional API call and that makes it horrendously slow.

When I was looking for a solution I stumble upon this post made by @pibara.

He was having similar issues and made a pretty neat script to measure the performance fetching raw blocks from the blockchain. The post above is old by now and doesn't include the latest bug fix. The following line needs to be moved outside of the while-loop for the script to function properly.

index = start

The script is using no fancy API calls and is an excellent way to measure the pure performance of fetching blocks from the blockchain.

Running the script from @pibara, it was sometimes slow to fetch data from the Steemit RPC nodes, but at least it could easily keep up. So it was clear that the stream_comments function should be avoided.

My next step was to use this script to compare the performance using different RPC nodes. Ever since I started using steem-python I noticed that the response time can vary greatly. And as @followbtcnews (in cooperation with @crimsonclad) just had released a full node as part of the Minnow Support Project, I thought it would be a cool thing to compare. Please follow this link for the announcement from @followbtcnews.
https://steemit.com/witness-category/@followbtcnews/heard-you-needed-a-node-witness-followbtcnews-launches-a-new-public-full-rpc-node

I started the test with the default steemit nodes. As this is the default, there is no specific modification. As soon as you create an object of the Steem class, like below, you will connect to one of the default nodes.

steem = Steem()

_{Steemit RPC Node}

And below the code to connect to a different node.

my_nodes = ['https://steemd.minnowsupportproject.org']
steem = Steem(my_nodes)

_{MSP's RPC Node}

As you can see above, the results from the default nodes vary quite a bit while the server from @followbtcnews is very consistent. Don't look blindly at the "seconds behind" value. There seems to be some kind of delay for how the blocks are presented through the API, hence the ~60 seconds behind being normal.

Seeing these results I will definitely be moving @dorabot away from the default RPC nodes.

Thank you for reading!
Stayed tune for future updates.

Please let me know if you have any questions.
And please ping me (@danielsaori) if you connect to Discord.

Proud member of #minnowsupportproject & #teamaustralia
Thank you @aggroed, @ausbitbank, @teamsteem,
@theprophet0, @someguy123, @canadian-coconut and @sirknight

Click HERE to learn more about Minnow Support Project.
Click HERE to connect to our Discord chat server.

Comments

[alexpmorris]

but it can even end up with incomplete and missed data

I've been asking a few others about this, and I've been wondering if it's not a potential issue with the websocket implementation being used, as I had described in this github issue:

https://github.com/zaphoyd/websocketpp/issues/641#issuecomment-329650068

I believe this is also related to an overloaded server (at times of especially "high response latency"), but here's the gist of it:

All frames returned are in json format and no compression is being used (I believe there were issues when they tried enabling compression). Every so often, I get what appears to be a "complete" websocket frame, however, the json packet is incomplete (ie. the frame size appears to be smaller than the actual packet size). The rest of the packet still comes through (but not as another frame). After appending the remainder to the original truncated packet, all seems well again.

I have a feeling if STEEMIT and other graphene-related chains transitioned to use https://github.com/uNetworking/uWebSockets instead, it might help alleviate at least some of these issues.

[danielsaori]

Great finding! 👍
I have never had that error thrown that I can remember. But I also only tested a short while before I moved away from the default rpc nodes.
There could be many things playing a role here, but for sure, using the stream_comments function there is an enormous overhead initiating a post object for each comment.

[harshallele]

Hmm I never understood(and still don't understand) what an RPC node is, but your bit about the default nodes being slow due to heavy load makes sense.

I actually made a script a few weeks ago using the steem-python api, and it was REALLY slow, until now I thought that the python api itself is slower than the other APIs, but after reading this post I realised that its the RPC nodes.

[danielsaori]

You should give it a try with another rpc node and see how it goes.
Depending on the complexity and number of queries it can make a big difference

[ehabfox]

Thank you for sharing it !

[rambeesbd]

your post is really amazing friend, I like

[realskilled]

Hi @danielsaori,

Thanks for sharing this!

I'm looking for an API that allows me to create content (POSTs). I have been reading the documentation and it seems that we can only retrieve information or just create a comments or up-vote.

I wonder if you could point me to some API that allows creating POST in steemit.com.

Thanks,

@realskilled

[danielsaori]

If you are using steem-python you have a function called post() available to make both post new posts or make comments.

[realskilled]

Well, actually I'm need to use this from PHP :)

[tomber]

Is that Doreamon??😸

[danielsaori]

Avenger Dora! 😉

[minnowsupport]

This post received a 5% vote by @minnowsupport courtesy of @followbtcnews from the Minnow Support Project ( @minnowsupport ). Join us in Discord.

Upvoting this comment will help support @minnowsupport.

[fishmon]

@steemitcards give dora some !boobs

[steemitcards]

[danielsaori]

@dorabot test ?winner

[dorabot]

The winner is: danielsaori!!!

[danielsaori]

Thanks!
So you think this was a boobs-worthy post? ;)

[fishmon]

No, i just asumed you wanted to see some boobs!

[danielsaori]

@dorabot another ?winner test

[dorabot]

No Winner!! No upvotes on this post... :(

[danielsaori]

Can you tell me @dorabot who the ?winner is?

[dorabot]

The winner is: qwasert!!!

[danielsaori]

@dorabot test ?winner

[dorabot]

The winner is: danielsaori!!!

[dorabot]

@dorabot ?winner

[danielsaori]

hmmm.. This above was from the esteem app...

[dorabot]

Ah, so the esteem app is not formatting json metadata inline with steemit.com...
@dorabot ?winner

[dorabot]

The winner is: dorabot!!!

[minnowsupport]

This post has been resteemed by @minnowsupport courtesy of @followbtcnews from the Minnow Support Project ( @minnowsupport ). Join us in Discord.

Upvoting this comment will help support @minnowsupport.

[minnowbooster]

@danielsaori got you a $8.17 @minnowbooster upgoat, nice! (Image: pixabay.com)

---

Want a boost? Click here to read more! @originalworks

[mrsquiggle]

This post received a 20% vote by @mrsquiggle courtesy of @scooter77 from the Minnow Support Project ( @minnowsupport ). Join us in Discord.

Upvoting this comment will help support @minnowsupport.

[bellyrub]

This wonderful post has received a bellyrub 17.40 % upvote from @bellyrub thanks to this cool cat: @danielsaori. My pops @zeartul is one of your top steemit witness, if you like my bellyrubs please go vote for him, if you love what he is doing vote for this comment as well.

[booster]

This post has received a 3.82 % upvote from @booster thanks to: @danielsaori.

[majorleaguemarc]

Hey! I'm looking to get some advice on how to get a bot started for steemit. Can you help me out - or get me in contact with someone that can help me out?

I'm very new to software development and I'm learning python. I need to ask someone if my idea is possible and other probing questions. I'm willing to pay to get the project started.

Thanks for your help . I've got big plans and I need all the support I can get :)

[buildawhale]

This post has received a 24.85 % upvote from @buildawhale thanks to: @danielsaori. Send at least 0.50 SBD to @buildawhale with a post link in the memo field for a portion of the next vote.

To support our curation initiative, please vote on my owner, @themarkymark, as a Steem Witness

[eroche]

This has been driving me round the bend for ages. Thanks for this post. I have just started to try out different nodes and its amazing how quick it is now.