Pip-tools for Cross-OS development.

Problem Definition

I am using Prefect.io for a project. Prefect is great but the package requires docker, which requires special packages when running on windows. When I push this project to the Continuous Integration environment, the installation breaks breaks because the Linux environments can't use the windows-specific packages.

The Solution

The solution is to use pip-tools and a requirements.in instead of a requirements.txt. With pip-tools you can generate a unique requirements.txt for each environment.

Components

Pip-tools

The pypi package you will need to make this all work.

https://github.com/jazzband/pip-tools

Requirements.in

List all your top level dependencies here. Anything you would manually install goes here. i.e. pip install pandas scikit-learn

Requirements.txt

The requirements.txt is generated by the pip-tools command pip-compile . This will generate your full requirements.txt. And it will show where each package is required either from the requirements.in or if its a dependency on one of those packages etc.

Steps

1. Write the requirements.in

Put all the packages you need in this file.

# just plain text, one line per package
pandas
scikit-learn

This is the only thing you need to include in your versioning.

2 . pip-compile

pip-compile requirement.in --output-file=requirements.txt

This step gets all the top level dependencies, anything you would manually install for your project.

In the past, you might have installed all your dependencies from the command line and ran pip freeze > requirements.txt . But that time is gone.

You can compile this on any environment you need for os-specific installations.

3. pip install

pip install -r requirements.txt

Last updated