Great post to define the problem and the challenge, as well as the finding the solution.
The check about the pandas version is a great touch.
I used cron, which despite its awful interface got the job done.
Can't agree. Crontab's interface is amazing and works very well in comparison to hipster solutions. :)
I would also like to see a installation instruction without conda. (Packacing it with setup.py or at least a requirements.txt) Conda is not a industry standard yet, I code with Python almost ten years, never used it.)
Also, reading the configurational variables outside of the logic may be good for other people. (Ex: environment variables.)
It's a minor thing but looks like you also have a redundant variable call here.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Thanks @emrebeyler for the thorough and helpful review. Much appreciated.
I added support for virtual environment with pip installation of
requirements.txt
in this commit. I usually work on scientific computing projects where there are more heavyweight dependencies, often written in other languages then Python, soconda
makes the most sense. But here the dependencies are all Python and virtual env does a fantastic job.I did consider making the script into more a package... e.g. adding
setup.py
and making it a command line utility withargparse
argument parsing. However, I reckoned that many users will need to modify the source code to suit their custom needs and at this point the added complexity would be more of a barrier to customization.My gripe is that I have trouble remember what the following cron config means:
Also placing minutes before hours seems wrong.
Remnant from when I was writing this code in Jupyter notebook cells!
Thank you for your review, @emrebeyler! Keep up the good work!