Pip-tools for Cross-OS development.
Problem Definition
I am using Prefect.io for a project. Prefect is great but the package requires docker, which requires special packages when running on windows. When I push this project to the Continuous Integration environment, the installation breaks breaks because the Linux environments can't use the windows-specific packages.
The Solution
The solution is to use pip-tools and a requirements.in instead of a requirements.txt. With pip-tools you can generate a unique requirements.txt for each environment.
Components
Pip-tools
The pypi package you will need to make this all work.
https://github.com/jazzband/pip-tools
Requirements.in
List all your top level dependencies here. Anything you would manually install goes here. i.e. pip install pandas scikit-learn
Requirements.txt
The requirements.txt is generated by the pip-tools command pip-compile
.
This will generate your full requirements.txt.
And it will show where each package is required either from the requirements.in or if its a dependency on one of those packages etc.
Steps
1. Write the requirements.in
Put all the packages you need in this file.
This is the only thing you need to include in your versioning.
2 . pip-compile
This step gets all the top level dependencies, anything you would manually install for your project.
In the past, you might have installed all your dependencies from the command line and ran pip freeze > requirements.txt
. But that time is gone.
You can compile this on any environment you need for os-specific installations.
3. pip install
Last updated