Context
This is for beginners.
For when your first Python bot or script for Hive goes to production.
I am not the right guy to post about clean code, but I've learned a few things.
I've made some mistakes and will make more, but maybe I can help you avoid some.
Coroutines
In an earlier post I demonstrated some deployment strategies for Hive bots.
To improve on those bots, you'll need coroutines.
This has nothing to do with Hive.
In the following, I demonstrate how I use them.
Python supports 3 methods to handle coroutines.
multi-processing
This is the most complicated and fastest.
But Python is not fast to begin with.
If computation was the issue, I'd use a different language entirely.
So multi-processing is out the window - I don't want complicated.
asyncio
I am not the right guy to talk about this and it confused me for a while.
async is different in javascript and Python and that made it more complicated.
Normal Python code is already synchronous, while js in browser sometimes just starts coroutines and doesn't wait...
Anyways, normal Python codes executes like this:
import time
def normal(interval):
print("normal for:", interval, "s")
time.sleep(interval)
print('normal finished')
normal(2)
normal(1)
The normal function waits for each part of the procedure and continues line by line.
This means the main thread is blocked when I use time.sleep.
normal for: 2 s
normal finished
normal for: 1 s
normal finished
Async could be used like this:
import asyncio
async def asynced(interval):
print("async for:", interval, "s")
await asyncio.sleep(interval)
print("async finished")
loop = asyncio.new_event_loop()
loop.create_task(asynced(1))
loop.create_task(asynced(3))
loop.run_forever()
While one function is awaiting, another task will get executed.
async for: 1 s
async for: 3 s
async finished
async finished
In javascript in browser, when using await, you kind of want the opposite effect.
Using fetch for example kicks off a coroutine and you get a Promise as result of your fetch.
You need to await or use a callback. That can be confusing.
Anyways, in Python async is also usefull for when you are waiting for an external event.
Because while you are waiting, you can unblock the thread and start a different task.
Also, if one task breaks, the next one will still execute.
threading
In the example I'll use threading and async.
In the script above, the loop.runs_forever. And that still blocks the thread.
import time
import asyncio
import threading
async def job(interval):
print("job for:", interval, "s")
time.sleep(interval)
print("job finished")
loop = asyncio.new_event_loop()
threading.Thread(target=loop.run_forever).start()
asyncio.run_coroutine_threadsafe(job(2), loop)
time.sleep(1)
print("program keeps flowing")
Now the main thread is unblocked, and you can access the async loop in the separate thread with run_coroutine_threadsafe. I use it a a queue, which I add tasks to.
job for: 2 s
program keeps flowing
job finished
Even if the job threw an exception and broke, the main thread would keep rolling.
Complexity
This adds a layer of complexity to Python.
Sequential flow of routines is much easier to handle.
Threadsafety and such things are no joke. Beware of race conditions 😵.
I mean: imagine the async example writing chunks to a file - you'd never be sure in which order they get written. It's a high price to pay...
Performance
The way I understand it, Python gets compiled and executed in such a way, that in the end it only uses one system thread. So what ever you do with threading here, if it is for performance, I'd suggest looking at other soultions.
On the other hand, async libraries for Python such as aiohttp and uvicorn and stuff use async and make the whole thing fast enough to build a websocket server with.
And with flask or other things like it, you can just let gunicorn spin up 100 Python workers in each their own system thread.
Application
In the end it all depends on what you want to build.
I assume the next step from building Python scripts is to get to a service, that others can use, too.
I am not sure what to call an app anymore, but I'll try to build one on this blog.
Conclusion
This post was mainly for @arc7icwolf.
I complained about his while true loop and never showed an alternative.
I'll keep blogging and hope it makes sense...
Great community content! Bravo.
I'm bookmarking all your posts, because once I'll finish the script I'm working on right now, I want to challenge myself and try to build something with the knowledge you have shared, and are still sharing, so far.
You also showed me how to use a signal to interrupt a while True loop without breaking it! The small project I'm working on now use a while loop but with a "flag" (when a condition is met, the flag is changed from True to False), so I hadn't the opportunity to implement what you teached me... but I haven't forgotten it :)
I'm eager to know more about this!
I've got a little demo that I will post next.
I think a webpage would be best.
If you need any help building the thing, find me on Discord, or ask here.
Love it :)))
I believe I'm quite close to finish it: it's not going to be polished, but it's working, and this is what matters the most for me right now ahahah and as always I'd love to hear some feedback from you to better understand where I made a poor work and where I can improve my logic :)
I'd also like to (try to) add at least a coroutine, just to see if I can do it!
So yesterday I worked on an app and realized: that's going to be too large for a blog.
So... I'll post the demo and from then on, I can't post the whole code.
Don't! Unless you have to.
If you keep building with Python and start using frameworks, you will enter the world of asyncio soon enough...