Chapter 26: Standing on Giants' Shoulders

Learning Objectives

• Understand what packages and dependency graphs are.
• Learn how to use virtual environments to isolate projects.
• Master package managers like pip, npm, and poetry.
• Understand semantic versioning and the importance of lock files.

Prerequisites

CLI Basics (Ch. 25), File I/O (Ch. 11), Git (Ch. 14)

Why Does This Exist?

Writing everything from scratch is insane. If you need to make an HTTP request, you don't want to manually handle TCP sockets. Every modern app builds on thousands of open-source libraries.

Package managers automate the downloading, versioning, and resolving of these dependencies so you can focus on writing your unique business logic.

History

Perl introduced CPAN in 1995. Python introduced PyPI in 2003. Node.js introduced npm in 2010.

In 2016, the "left-pad" incident occurred: an author deleted a tiny 11-line package from npm, which was a dependency of a dependency of thousands of major projects. Half the internet's build systems broke. This proved how interconnected our dependency graphs are, leading to better lock file and registry architectures.

Mental Model

Think of open-source packages like LEGO sets. PyPI/npm is the massive global LEGO store.

Your package manager (pip/npm) is the delivery service. Your requirements.txt or package.json is your shopping list.

A virtual environment is your dedicated LEGO room. It ensures that the pieces you buy for Project A don't get mixed up with the pieces for Project B.

Internal Working

When you request to install a package, the manager contacts the registry and downloads it.

However, that package might depend on 5 other packages, which depend on 10 more. The package manager uses a dependency resolution algorithm (like a SAT solver) to find a set of versions where every package is happy without conflicting.

Once resolved, it generates a lock file freezing the exact hashes and versions of every downloaded file, ensuring reproducibility.

Syntax

Python (pip & venv)

1python -m venv venv                # Create virtual env
2source venv/bin/activate           # Activate (Mac/Linux)
3pip install requests==2.31.0       # Install specific version
4pip freeze > requirements.txt      # Save dependencies
5pip install -r requirements.txt    # Install from list

Node.js (npm)

1npm init -y                        # Initialize package.json
2npm install express                # Install prod dependency
3npm install --save-dev jest        # Install dev dependency

Semantic Versioning (SemVer)

Versions look like MAJOR.MINOR.PATCH (e.g., 2.31.0).

• MAJOR: Breaking changes.
• MINOR: New features, backwards compatible.
• PATCH: Bug fixes, backwards compatible.

Visual Explanation

Tiny Example

1python -m venv venv
2source venv/bin/activate
3pip install rich

Then in Python:

1from rich import print
2print("[bold red]Hello World![/bold red]")

Walkthrough

We created an isolated bubble (venv). We jumped into it (activate). We downloaded the rich UI library from PyPI. Then we used it in a tiny script to print colored text to the terminal.

Common Mistakes

• Committing node_modules or venv/ to Git: These folders are massive and contain system-specific binaries. Only commit requirements.txt or package.json and let others rebuild the environment themselves.
• Not pinning versions: If you just write requests, it works today. In a year, it might download a breaking update. Always freeze versions.

Debugging

If pip install fails with huge blocks of red text, it's often missing a C compiler because the package requires compiling native extensions. Read the error carefully; it usually tells you what OS package you are missing.

To fix version conflicts, use tools like pipdeptree to see your dependency graph and find out which packages are demanding incompatible versions.

Mini Project

Time: 20 min

Create a virtual environment. Install requests and rich. Write a script that fetches data from the JSONPlaceholder API and prints it using rich. Freeze dependencies to requirements.txt.

# run: pip install requests rich
import requests
from rich import print

response = requests.get("https://jsonplaceholder.typicode.com/users/1")
data = response.json()

print(f"[bold green]User Info:[/bold green]")
print(f"Name: [cyan]{data['name']}[/cyan]")
print(f"Email: {data['email']}")

Bigger Project

Time: 1.5 hr

Set up a Python project with Poetry (poetry new my-project). Create a pyproject.toml with 3 dependencies and 2 dev dependencies (like pytest). Write a utility module and write tests. Use poetry run pytest to run them.

# Poetry workflow reference
# Run each command below in your terminal — this script documents what they do.

STEPS = [
    ("Create project",        "poetry new my-project"),
    ("Add runtime deps",      "poetry add requests pandas rich"),
    ("Add dev-only deps",     "poetry add --group dev pytest black ruff"),
    ("Install everything",    "poetry install"),
    ("Run tests",             "poetry run pytest --cov=my_project"),
    ("Export requirements",   "poetry export -f requirements.txt -o requirements.txt"),
]

print("=== Poetry Project Workflow ===")
for title, cmd in STEPS:
    print(f"  {title:25}  $ {cmd}")

PYPROJECT = '''
[tool.poetry]
name    = "my-project"
version = "0.1.0"
python  = "^3.11"

[tool.poetry.dependencies]
python   = "^3.11"
requests = "^2.31.0"
pandas   = "^2.1.0"
rich     = "^13.6.0"

[tool.poetry.group.dev.dependencies]
pytest = "^7.4.0"
black  = "^23.9.0"
ruff   = "^0.1.0"
'''
print("\n=== pyproject.toml ===")
print(PYPROJECT)

SEM_VER = [
    ("^1.2.3", ">=1.2.3, <2.0.0", "Allow MINOR + PATCH bumps  (most common)"),
    ("~1.2.3", ">=1.2.3, <1.3.0", "Allow PATCH bumps only     (conservative)"),
    ("1.2.3",  "==1.2.3",         "Pin to exact version        (lock it down)"),
]
print("=== Semantic Versioning ===")
for spec, meaning, note in SEM_VER:
    print(f"  {spec:10} -> {meaning:22}  ({note})")

Production Usage

In Docker images, you always copy the requirements.txt, run pip install, and then copy the rest of your code. This caches the dependency installation step.

In Node, CI pipelines use npm ci instead of npm install because it strictly obeys the lock file without updating anything.

Best Practices

• Always use virtual environments.
• Pin production dependencies exactly (using lock files).
• Keep development dependencies (linters, testers) separate from production dependencies.
• Use Dependabot to automatically monitor and update libraries with security vulnerabilities.

Interview Questions

Revision Sheet

Connections

• Connects back to Git (Ch. 14) — you never commit dependencies, only the requirements list.
• Connects to Deployment (Ch. 20) — your host needs to know how to install your packages.