Python venv workflow: absolute paths
Having fun trying out many, locally installed, AI projects ...
Since I last posted on Hive, about various image generators using AI systems, tons of new AI projects came out, many of which you can (or should) install locally on your own personal computer or server. And boy did I have a good time toying around with many of them!
However, as those AI projects mostly use a multitude of Python modules under the hood, "package management" (i.e. which specific module versions a given AI project uses to function) becomes both essential as well as a "pain in the arse" if not taken care of.
Luckily, Python has this concept called "virtual environment" or venv, where you can basically configure separate Python interpreters per individual (AI) project, therewith allowing to manage package dependencies per Python project.
... which led to a bit of a mess with various Python venvs, aliases and different distributions
So far, so good, but as over time I've installed multiple different Python distributions on my same "development Mac" - using for example both Homebrew, Anaconda, etc. - for which I had configured aliases as well (because originally Python 2.7 came pre-installed yet I wanted to use Python3 without having to literally type python3 each time, and the same for pip and pip3), and I have to confess, things turned out to become a bit of a mess on my system.
Also, I thought about using pipenv and pyenv and venv ... if you're anything like me, trying cool stuff out, being a creative coder but no seasoned sysadmin per se, then getting back after - say - 3 months to an environment and project you worked on months ago, can get pretty frustrating.
Solution: workflow using absolute paths instead of source / deacticate !
If you recognize what I'm describing here, and/or if you're thinking about trying out by locally installing these cool shiny new AI tools on your own system (I'm discussing macOS here, which should work on Linux Desktop the same, and on Windows with slight modifications on pathnames but the idea/crux is the same), then you might enjoy this workflow trick I thought of.
When reading this, it might seem trivial to you. But you might also find tremendous value in this "absolute path"-workflow concept, I hope it helps you like it helped me so let's explain what I thought of!
Workflow step 1: create identical ~/venvs/ and ~/pythonprojects/ folder structures
What I mean with this:
Let's assume you keep your self-written Python projects, as subfolders within the folder
~/pythonprojects/, for example if you want to code your own Hive voting bot in Python, create your code folder as~/pythonprojects/hivevoting/and of course just place your self-written code with your.pyfiles in thereNext create an identically named venv in your
~/venvs/folder, so in this example~/venvs/hivevoting/, so use the same name for both project folder and venv and the same structure using absolute paths, via the command:
python3 -m venv ~/venvs/hivevoting
Repeat this for every newly generated project that you write yourself or fork. If you locally work on 30 different projects, also create 30 different venvs using the absolute paths as explained
Workflow step 2: run pip and python as absolute paths without source'ing, nor using the -m flag for installing as module
Normally you'd source a venv prior to running commands like python yourfile.py or pip install -m some_module. And when you're done using that venv you'd run deactivate to switch back to the base interpreter. But you can completely skip this and conversely you don't need to worry about accidentically pip installing globally or into the wrong venv, again by using absolute paths, always.
- So for example if you want to run the file
main.pyof your hivevoting project, use:
~/venvs/hivevoting/bin/python ~/pythonprojects/hivevoting/main.py
At the left you have your absolute path to the correct venv'd python executable/interpreter, and at the right side the absolute path of your project script (of course if you cd into your project script folder already, you can also run)
~/venvs/hivevoting/bin/python main.py
... but why would you? Running the file again while coding you just use the up-arrow key in your terminal using both absolute paths. More verbose does not equate to more typing nonstop but it does give more clarity and understanding.
This way pip installing new modules doesn't have to be specified as either module or globally either, via:
~/venvs/hivevoting/bin/pip install pandas
, or
~/venvs/hivevoting/bin/pip uninstall pandas
Creating (freeze 'ing) a requirements.txt is now done via:
~/venvs/hivevoting/bin/pip freeze > ~/pythonprojects/hivevoting/requirements.txt
And to install your project dependencies to a freshly created venv from an existing/forked requirements.txt file, you do:
~/venvs/hivevoting/bin/pip install -r ~/pythonprojects/hivevoting/requirements.txt
And if you want to know which exact packages and versions are currently installed per venv, just use:
~/venvs/hivevoting/bin/pip list
It's a bit more verbose but personally this makes me understand which exact executable I'm currently using and where it's exactly located on my system! Preciously I didn't "explicitely think" about this and over time that makes you forget leading to confusion (at least in my case it did).
Workflow step 3: Configuring the venv interpreter in your code editor/IDE for autocompletion / code hinting, without project configuration
Both VS Code and PyCharm (Pro and/or Community Edition, CE) let you assign specific Python interpreters - per venv, or globally,- while you're still coding in your IDE/code editor. Because you're using identical venv and project folder names, you can also skip project configurations in the editor project settings! Just select, point-and-click the venv interpreter per project right after opening the project folder in your favorite IDE and you're good to go! :-)
See you in the next post!
I hope you enjoyed this venv workflow and that it may ease your package management workflow like it has for me! See you next time!
Thanks for your contribution to the STEMsocial community. Feel free to join us on discord to get to know the rest of us!
Please consider delegating to the @stemsocial account (85% of the curation rewards are returned).
You may also include @stemsocial as a beneficiary of the rewards of this post to get a stronger support.