jazzband/pip-tools

> Compile a fully-pinned requirements.txt lockfile from loose Python dependencies, then sync a virtualenv to match it exactly.

GitHub repo · Official website · License: BSD-3-Clause

Overview

pip-tools is two command-line programs — pip-compile and pip-sync — that add lockfile discipline on top of stock pip without replacing it. pip-compile reads your abstract dependencies (from requirements.in, pyproject.toml, setup.cfg, or setup.py), resolves the full transitive tree, and writes a flat requirements.txt where every package is pinned to an exact version and annotated with which requirement pulled it in. pip-sync then makes the active virtualenv byte-for-byte match that file, installing, upgrading, and — crucially — uninstalling whatever is needed. The project started in 2012¹ and has been maintained by the Jazzband collective for most of its life.

Its defining design choice is that the compiled output is a plain requirements.txt that ordinary pip install -r understands. There is no bespoke lockfile format, no vendored resolver database, no wrapper you have to route all installs through. This is why pip-tools remains popular in CI, Docker builds, and teams that want reproducibility without adopting a whole new packaging workflow. The corresponding tradeoff is that pip-tools inherits pip's model wholesale, including its weakest point: locks are resolved per-environment, not universally.

The other tension worth stating up front is that pip-tools is coupled to pip's private internals. It imports from pip._internal, which pip explicitly does not treat as a public API. Every pip release is a potential breakage, and pip-tools ships pin ranges on pip to defend against it — a maintenance tax that shaped the project's release cadence and is the root of most version-compatibility bug reports.

Getting Started

pip-tools installs into each project virtualenv, alongside the project it locks:

$ python -m pip install pip-tools

Declare loose dependencies in requirements.in, compile them, then sync:

$ echo "django" > requirements.in
$ pip-compile requirements.in            # -> requirements.txt, fully pinned
$ pip-sync                               # make the venv match requirements.txt

The generated requirements.txt records provenance and the exact command used:

# This file is autogenerated by pip-compile with Python 3.11
# by the following command:
#
#    pip-compile requirements.in
#
asgiref==3.8.1
    # via django
django==5.0.6
    # via -r requirements.in
sqlparse==0.5.0
    # via django

Architecture / How It Works

pip-compile does not implement its own dependency resolver. It drives pip's resolver machinery — the resolvelib-based backtracking resolver that pip itself adopted in pip 20.3 — by constructing pip's internal objects and asking them to produce a resolved set, then serializing the result. Because it reuses pip's resolution and its finder, pip-compile sees the same indexes, wheels, and environment markers that a real pip install would, which is what makes the lockfile faithful to what pip will actually install.

To do this it reaches into pip._internal (the PackageFinder, resolver, and requirement objects). Those symbols are unstable across pip releases, so pip-tools maintains a compatibility shim layer and declares a supported pip version range in its own dependencies. This is the single most important thing to understand operationally: pip-tools and pip are version-locked partners, and upgrading one can require upgrading the other.

Resolution is per-environment by construction. The output of pip-compile is specific to the interpreter version, OS, and platform it ran on, because environment markers (python_version, sys_platform, etc.) are evaluated against the current environment. A requirements.txt compiled on Linux/CPython 3.11 is not guaranteed valid on macOS or Python 3.9. Teams that ship across platforms are expected to compile — and commit — a separate output file per environment.

pip-sync is the mirror operation. It diffs the installed distributions against the target requirements.txt and converges the environment to it, including removals. It is deliberately narrow: it refuses to touch pip, setuptools, and pip-tools itself, and it is only safe against a file that pip-compile produced — running it against a hand-written requirements file will happily uninstall things you wanted.

--generate-hashes adds --hash=sha256:... lines for every artifact, enabling pip's hash-checking mode so installs fail if an index serves an unexpected file. Layered requirements (-c requirements.txt as a constraint in a second .in) let development locks stay compatible with a production lock without duplicating pins.

Production Notes

• pip version coupling is the top footgun. If you pin pip in CI and let

pip-tools float (or vice versa), you eventually hit "unsupported pip version" or an import error from pip._internal. Pin both, upgrade them together, and read the pip-tools changelog before bumping pip.

• Locks are not portable across environments. Do not compile on your Mac and

ship the result to a Linux container expecting identity. Compile inside an image that matches production (a common pattern is pip-compile inside the same Docker base image used to build). Conditional dependencies and backports-* packages are where this bites.

• pip-sync uninstalls aggressively. That is the point, but it means a shared

or long-lived virtualenv with tools installed out-of-band (linters, debuggers) will get stripped unless those tools are in the synced files. Use dedicated, disposable venvs.

• No changes when the lock already satisfies the input. pip-compile will not

upgrade an existing, still-valid requirements.txt even when newer versions exist. You must pass --upgrade / -U or --upgrade-package NAME (-P) explicitly. Teams are frequently surprised that re-running does nothing.

• Build-time locking needs PIP_CONSTRAINT. Locking a package's own build

dependencies requires cooperation with pip's isolated build environment via the PIP_CONSTRAINT env var; see the upstream discussion in pip#8439². pip-compile --all-build-deps exists to help but is not a turnkey guarantee.

• Speed. Resolution is as fast (or slow) as pip's backtracking resolver.

Pathological dependency conflicts can cause long backtracking with many index round-trips. This is one of the main reasons some teams moved to uv, whose resolver is substantially faster on the same inputs.

When to Use / When Not

Use when:

• You want deterministic, hash-verifiable installs but the output must remain a

vanilla requirements.txt that pip install -r and every existing tool reads.

• You are locking an application (service, Docker image) and control the target

environment.

• You want to add locking to an existing pip-based project with minimal disruption

and no new lockfile format to teach the team.

Avoid when:

• You need one universal, cross-platform lockfile from a single resolve — pip-tools

is per-environment by design; reach for uv or Poetry.

• You want an all-in-one project/dependency/virtualenv/publish manager; pip-tools

is intentionally just compile + sync.

• Resolution speed on a large, conflict-heavy dependency graph is a bottleneck.

Alternatives

• astral-sh/uv — Rust-based; far faster resolution and a uv pip compile command

that is a near drop-in for pip-compile, plus universal cross-platform locks. Use when speed or a single portable lock matters.

• python-poetry/poetry — full project manager with its own poetry.lock and

resolver. Use when you want dependency management, packaging, and publishing in one tool and will adopt its workflow.

• pdm-project/pdm — PEP 582 / standards-focused manager with a lockfile and

plugin system. Use when you want a modern, pyproject.toml-native manager.

• pypa/pipenv — Pipfile + Pipfile.lock with integrated virtualenv handling.

Use when you want the classic app-locking workflow and don't mind its pace.

• Hatch (pypa/hatch) — environment and build manager. Use when packaging and env

matrices matter more than an emitted requirements.txt lock.

History

References

1. ^ pip-tools repository, created 2012-09-10 (GitHub API metadata). https://github.com/jazzband/pip-tools
2. ^ pip issue #8439, "Add a way to pass constraints to the build environment" (PIP_CONSTRAINT). https://github.com/pypa/pip/issues/8439
3. ^ Jazzband collective — project home for community-maintained Python packages. https://jazzband.co/

Tags

python, packaging, dependency-management, lockfile, pip, cli, reproducible-builds, requirements-txt, devops, hashes