The Package Architecture: Building Importable Code
.jpeg)
Timothy had written dozens of useful Python modules for the library—database utilities, book cataloging functions, report generators. But each script lived in isolation, copied and pasted between projects, versions diverging, improvements lost.
The problem? multiple copies of the same function:
# project_a/utils.py
def format_isbn(isbn):
return isbn.replace("-", "")
# project_b/helpers.py
def format_isbn(isbn): # Same function, different file!
return isbn.replace("-", "")
# project_c/isbn_tools.py
def format_isbn(isbn): # Now we have THREE copies!
return isbn.replace("-", "")
Margaret found him maintaining the same function in seven different locations. "You're managing chaos," she observed. "Come to the Package Architecture—where code becomes reusable, installable, and shareable."
From Script to Package
She showed him the transformation:
The before and after:
# Before: Just a file (a MODULE)
# my_script.py
def process_data():
return "data"
# After: A proper package (a PACKAGE containing modules)
# my_library/
# __init__.py
# core.py
# utils.py
"A package is a directory with an __init__.py file," Margaret explained. "That file—even if empty—tells Python 'this directory is importable.' Everything inside becomes accessible through dot notation."
"Wait," Timothy said. "What's the difference between a module and a package?"
The key distinction:
# MODULE: A single .py file
# isbn.py
def format_isbn(isbn):
return isbn.replace("-", "")
# Import: from isbn import format_isbn
# PACKAGE: A directory with __init__.py containing modules
# library_tools/ # This is the package
# __init__.py # Marker file
# isbn.py # This is a module
# catalog.py # This is a module
# Import: from library_tools.isbn import format_isbn
"Both are importable," Margaret explained. "A module is a single file. A package is a collection of modules organized in a directory. Packages let you group related modules together."
Python's Bytecode Cache
After importing for the first time, Timothy noticed new directories:
The automatically created cache:
# library_tools/
# __init__.py
# isbn.py
# __pycache__/ # Created automatically!
# __init__.cpython-311.pyc # Compiled bytecode
# isbn.cpython-311.pyc # Compiled bytecode
"The __pycache__ directory stores compiled bytecode," Margaret explained. "Python compiles .py files to .pyc files for faster loading. First import compiles, subsequent imports load the cached bytecode—much faster."
What to add to .gitignore:
# .gitignore should include:
__pycache__/
*.pyc
*.pyo
"Always ignore __pycache__ in version control," she advised. "These are machine-generated cache files, not source code."
The Simplest Package
A minimal working example:
# library_tools/
# __init__.py # Makes this a package
# isbn.py # Module inside package
# library_tools/isbn.py
def format_isbn(isbn):
"""Remove hyphens from ISBN"""
return isbn.replace("-", "")
def validate_isbn(isbn):
"""Check if ISBN has correct length"""
clean = format_isbn(isbn)
return len(clean) in (10, 13)
# Now anyone can import it!
from library_tools.isbn import format_isbn
from library_tools.isbn import validate_isbn
# Or import the whole module
from library_tools import isbn
result = isbn.format_isbn("978-0-441-01359-3")
"The package name is the directory name," Margaret noted. "The module name is the file name. The function name is... the function name. Three levels of organization."
What Happens During Import
Before diving deeper, Margaret showed Timothy what Python actually does when you import:
Python's six-step import process:
# When you write:
from library_tools.isbn import format_isbn
# Python performs these steps:
# 1. Check sys.modules - already loaded?
# 2. Search sys.path for library_tools package
# 3. Execute library_tools/__init__.py (first import only!)
# 4. Execute library_tools/isbn.py (if not cached)
# 5. Cache everything in sys.modules
# 6. Bind format_isbn to current namespace
"Imports happen once per session," Margaret explained. "Python caches modules in sys.modules. Second imports are instant—they just retrieve the cached module."
Demonstrating the cache:
import sys
# First import - executes code
from library_tools import isbn
print(isbn) # <module 'library_tools.isbn' from '...'>
# Check the cache
print('library_tools.isbn' in sys.modules) # True
# Second import - uses cache (instant!)
from library_tools import isbn # Doesn't re-execute isbn.py
# You can see all loaded modules
print(list(sys.modules.keys()))
# ['sys', 'builtins', 'library_tools', 'library_tools.isbn', ...]
"This is why __init__.py code runs only once," she noted. "First import executes it, subsequent imports use the cached version. Put initialization logic there, but be careful—it runs at import time, not call time."
The Import Search Path
How Python finds packages:
import sys
# Python searches these locations in order:
print(sys.path)
# [
# '', # Current directory (first!)
# '/usr/lib/python3.11', # Standard library
# '/usr/lib/python3.11/site-packages', # Installed packages
# ...
# ]
# To add custom locations:
sys.path.append('/path/to/my/packages')
# Or set PYTHONPATH environment variable:
# export PYTHONPATH=/path/to/my/packages
"Current directory is first," Margaret warned. "If you have a file named email.py, it shadows the standard library's email module! Always use unique package names."
Import Side Effects and Circular Imports
Import-time execution:
# library_tools/__init__.py
print("Initializing library_tools!") # Runs at import time
# First import anywhere in your program:
import library_tools # Prints: Initializing library_tools!
# Subsequent imports:
import library_tools # Silent - cached
"Be cautious with import-time code," Margaret cautioned. "Keep __init__.py lightweight. Heavy computation at import time slows down program startup."
She also warned about circular dependencies:
A problematic pattern to avoid:
# library_tools/catalog.py
from library_tools.database import save_book # Imports database
# library_tools/database.py
from library_tools.catalog import Book # Imports catalog
# CIRCULAR DEPENDENCY!
# catalog imports database
# database imports catalog
# Neither can finish importing!
"We'll tackle circular import problems in Article 42," she noted. "For now, know that they're a design smell—usually fixable by restructuring code or using import-time tricks."
The init.py File: Package Gateway
Timothy learned that __init__.py is more than a marker:
Three common patterns:
# library_tools/__init__.py
# Option 1: Empty (package exists, but you import from submodules)
# Users must: from library_tools.isbn import format_isbn
# Option 2: Import key functions for convenience
from library_tools.isbn import format_isbn, validate_isbn
from library_tools.database import connect_db
# Now users can: from library_tools import format_isbn
# Option 3: Control what's exported with __all__
from library_tools.isbn import format_isbn, validate_isbn
from library_tools.database import connect_db, _internal_helper
__all__ = ['format_isbn', 'validate_isbn', 'connect_db']
# _internal_helper is available but not in __all__
# When someone does: from library_tools import *
# They only get items in __all__
"Think of __init__.py as the package's front desk," Margaret explained. "It decides what visitors see immediately versus what they must ask for specifically."
Package Initialization
Common initialization patterns:
# library_tools/__init__.py
# Code here runs when the package is first imported!
print("Library Tools v1.0 loaded")
# Common pattern: Set version
__version__ = "1.0.0"
# Common pattern: Lazy imports (import only when needed)
def get_heavy_module():
from library_tools import heavy_processing
return heavy_processing
# Common pattern: Check dependencies
try:
import requests
except ImportError:
raise ImportError("library_tools requires 'requests'. Install with: pip install requests")
# Make key items available at package level
from library_tools.isbn import format_isbn
from library_tools.catalog import Book, Author
# Now users can do:
# from library_tools import format_isbn, Book
"The __init__.py runs once, when the package is first imported," she noted. "Use it for setup, version declaration, and making your API convenient."
Relative vs Absolute Imports
Margaret showed him both import styles:
Package structure for examples:
# Package structure:
# library_tools/
# __init__.py
# isbn.py
# catalog.py
# database/
# __init__.py
# connection.py
# queries.py
# In library_tools/catalog.py - ABSOLUTE IMPORTS
from library_tools.isbn import format_isbn
from library_tools.database.connection import get_db
class Book:
def __init__(self, isbn):
self.isbn = format_isbn(isbn)
def save(self):
db = get_db()
# save to database
# In library_tools/catalog.py - RELATIVE IMPORTS
from .isbn import format_isbn # Same package
from .database.connection import get_db # Subpackage
class Book:
def __init__(self, isbn):
self.isbn = format_isbn(isbn)
"Relative imports use dots," Margaret explained. "One dot means 'current package.' Two dots means 'parent package.'"
Navigating with dots:
# In library_tools/database/queries.py
# Absolute imports - always work
from library_tools.isbn import format_isbn
from library_tools.database.connection import get_db
# Relative imports - package-aware
from ..isbn import format_isbn # Up one level, then into isbn
from .connection import get_db # Same level (database package)
def find_book(isbn):
clean_isbn = format_isbn(isbn) # From parent package
db = get_db() # From same package
return db.query("SELECT * FROM books WHERE isbn = ?", clean_isbn)
When to Use Each:
Absolute imports are preferred because they're always clear about where imports come from, they work from anywhere, and they're easier to refactor. Use the full package path for transparency:
from library_tools.isbn import format_isbn
Relative imports should be used sparingly. They're short and convenient inside large packages, but they can't run the file as a script and they're harder to move code between packages:
from .isbn import format_isbn
"Absolute imports are clearer," Margaret advised. "Use relative imports only within tightly coupled package code. Never in scripts meant to run directly."
The -m Flag: Running Package Code
Timothy tried to run a file with relative imports:
The file with relative imports:
# library_tools/catalog.py
from .isbn import format_isbn # Relative import
class Book:
def __init__(self, isbn):
self.isbn = format_isbn(isbn)
Running it the wrong way vs the right way:
# This FAILS:
$ python library_tools/catalog.py
ValueError: attempted relative import in non-package
# This WORKS:
$ python -m library_tools.catalog
# Runs successfully!
"The -m flag tells Python to run code as part of a package," Margaret explained. "Without it, Python treats the file as a standalone script—relative imports fail. With -m, Python knows the package context—relative imports work."
What Python does in each case:
# When you run: python -m library_tools.catalog
# Python does:
# 1. Finds library_tools package in sys.path
# 2. Treats catalog.py as part of that package
# 3. Resolves relative imports correctly
# When you run: python library_tools/catalog.py
# Python does:
# 1. Runs the file directly (not as package)
# 2. Doesn't know about package structure
# 3. Relative imports fail - no package context!
Subpackages: Nested Organization
Timothy learned to organize complex packages:
A hierarchical structure:
# library_tools/
# __init__.py
# isbn.py
# catalog/
# __init__.py
# book.py
# author.py
# database/
# __init__.py
# connection.py
# queries.py
# reports/
# __init__.py
# pdf.py
# excel.py
# email.py
# Each subdirectory with __init__.py is a subpackage!
# library_tools/catalog/__init__.py
from library_tools.catalog.book import Book
from library_tools.catalog.author import Author
__all__ = ['Book', 'Author']
# Now users can do:
from library_tools.catalog import Book, Author
# Instead of:
from library_tools.catalog.book import Book
from library_tools.catalog.author import Author
"Subpackages group related functionality," Margaret explained. "Each subpackage gets its own __init__.py to control its API."
The all Variable: Controlling Exports
She demonstrated selective visibility:
Public vs private functions:
# library_tools/isbn.py
def format_isbn(isbn):
"""Public API - users should call this"""
return _clean_isbn(isbn)
def validate_isbn(isbn):
"""Public API"""
return _check_length(_clean_isbn(isbn))
def _clean_isbn(isbn):
"""Private helper - internal use only"""
return isbn.replace("-", "").replace(" ", "")
def _check_length(isbn):
"""Private helper"""
return len(isbn) in (10, 13)
# Control what 'from library_tools.isbn import *' imports
__all__ = ['format_isbn', 'validate_isbn']
# _clean_isbn and _check_length are NOT in __all__
"The underscore prefix is a convention for private functions," Margaret noted. "But __all__ explicitly declares your public API. When someone does import *, they only get what's in __all__."
How all filters imports:
# Without __all__:
from library_tools.isbn import *
# Gets: format_isbn, validate_isbn, _clean_isbn, _check_length (everything!)
# With __all__:
from library_tools.isbn import *
# Gets: format_isbn, validate_isbn (only public API)
# But direct imports always work:
from library_tools.isbn import _clean_isbn # This still works!
Making Code Installable: pyproject.toml
Margaret showed him how to make the package installable:
Python comments:
# Project structure:
# my-library-project/
# pyproject.toml # Package metadata
# README.md
# library_tools/ # The package itself
# __init__.py
# isbn.py
# catalog.py
# tests/
# test_isbn.py
Now, the TOML:
# pyproject.toml
[build-system]
requires = ["setuptools>=61.0"]
build-backend = "setuptools.build_meta"
[project]
name = "library-tools"
version = "1.0.0"
description = "Tools for library management"
authors = [{name = "Timothy", email = "timothy@library.com"}]
readme = "README.md"
requires-python = ">=3.8"
dependencies = [
"requests>=2.28.0",
"pandas>=1.5.0"
]
[project.optional-dependencies]
dev = [
"pytest>=7.0",
"black>=22.0"
]
"This tells Python how to install your package," Margaret explained. "Run pip install . in the project directory, and your package becomes available everywhere."
Installation options:
# Regular install - copies files to site-packages
pip install .
# Editable install - creates link to source directory
pip install -e .
# Now from anywhere:
python -c "from library_tools import format_isbn; print(format_isbn('978-0-441-01359-3'))"
# 9780441013593
Editable Installs for Development
Timothy asked about the -e flag. Margaret demonstrated:
What editable mode does:
# Install in editable mode (development)
$ pip install -e .
# What this does:
# 1. Creates link file in site-packages pointing to source
# 2. Changes to source code take effect immediately
# 3. No reinstall needed after editing code!
The workflow comparison:
# Without -e (regular install):
# Edit library_tools/isbn.py
# Must reinstall: pip install .
# Then changes are visible
# With -e (editable install):
# Edit library_tools/isbn.py
# Changes immediately visible!
# Python imports from source directory
"Editable installs are essential during development," Margaret noted. "Edit, test, edit, test—no reinstall loop. When ready to release, use regular install to test the actual distribution."
Distribution
Building and uploading to PyPI:
# Build distribution files
python -m build
# Creates:
# dist/
# library_tools-1.0.0.tar.gz # Source distribution
# library_tools-1.0.0-py3-none-any.whl # Wheel (binary distribution)
# Upload to PyPI (Python Package Index)
python -m twine upload dist/*
# Now anyone can install:
pip install library-tools
Namespace Packages: Shared Namespaces
Timothy learned about packages without __init__.py:
The implicit namespace pattern:
# Modern Python (3.3+) supports implicit namespace packages
# company_tools/
# database/
# connection.py # No __init__.py in company_tools!
# reporting/
# pdf.py
# This allows multiple projects to contribute to the same namespace:
# Project 1: company-tools-database
# company_tools/
# database/
# Project 2: company-tools-reporting
# company_tools/
# reporting/
# Both installed, users can:
from company_tools.database import connection
from company_tools.reporting import pdf
"Namespace packages let multiple distributions share a namespace," Margaret explained. "Useful for plugin architectures and large organizations with many packages."
Entry Points: Command-Line Scripts
She showed him how to make CLI tools:
Declaring console scripts in pyproject.toml:
# pyproject.toml
[project.scripts]
format-isbn = "library_tools.cli:format_isbn_command"
validate-book = "library_tools.cli:validate_command"
The implementation in Python:
# library_tools/cli.py
def format_isbn_command():
"""Entry point for command-line usage"""
import sys
if len(sys.argv) < 2:
print("Usage: format-isbn <isbn>")
sys.exit(1)
from library_tools.isbn import format_isbn
isbn = sys.argv[1]
print(format_isbn(isbn))
def validate_command():
import sys
from library_tools.isbn import validate_isbn
isbn = sys.argv[1] if len(sys.argv) > 1 else ""
if validate_isbn(isbn):
print(f"✓ Valid ISBN: {isbn}")
else:
print(f"✗ Invalid ISBN: {isbn}")
sys.exit(1)
"After installing, these become shell commands," Margaret noted:
Using the installed CLI tools:
$ pip install library-tools
$ format-isbn 978-0-441-01359-3
9780441013593
$ validate-book 978-0-441-01359-3
✓ Valid ISBN: 978-0-441-01359-3
Best Practices: Package Structure
Margaret shared production patterns:
The recommended layout:
# Recommended structure:
# my-project/
# README.md # Documentation
# LICENSE # License file
# pyproject.toml # Package metadata
# .gitignore # Ignore __pycache__, *.pyc, etc.
# src/
# my_package/ # Source code in src/ directory
# __init__.py
# core.py
# utils.py
# tests/ # Tests outside package
# __init__.py
# test_core.py
# test_utils.py
# docs/ # Documentation
# api.md
# guide.md
"Keep tests separate from package code," she advised. "Use src/ layout to prevent import confusion during development."
Why src/ Layout?
Timothy asked why the extra directory. Margaret showed him the problem:
Without src/ layout - the problem:
# WITHOUT src/ layout:
# my-project/
# my_package/
# __init__.py
# core.py
# tests/
# test_core.py
# pyproject.toml
# During development, running tests:
$ python -m pytest tests/
# Python's import search finds:
# 1. Current directory (my-project/)
# 2. Finds my_package/ folder
# 3. Imports from SOURCE directory (not installed package!)
# Problem: Tests run against uninstalled code
# Packaging bugs go undetected until users install!
With src/ layout - the solution:
# WITH src/ layout:
# my-project/
# src/
# my_package/
# __init__.py
# tests/
# pyproject.toml
# During development, running tests:
$ pip install -e . # Must install first
$ python -m pytest tests/
# Python's import search:
# 1. Current directory (my-project/) - doesn't find my_package
# 2. Finds INSTALLED package in site-packages
# 3. Tests run against installed package!
# Benefit: Catches packaging issues immediately
# If import works in tests, it works for users
"The src/ layout forces you to test the installed package," Margaret explained. "Without it, imports might work for you but fail for users. With it, what you test is what ships."
Version Management
Semantic versioning in practice:
# library_tools/__init__.py
__version__ = "1.0.0"
# Use semantic versioning:
# MAJOR.MINOR.PATCH
# 1.0.0 - Initial release
# 1.0.1 - Bug fix (backwards compatible)
# 1.1.0 - New feature (backwards compatible)
# 2.0.0 - Breaking change (not backwards compatible)
API Design
Good vs bad package APIs:
# Good package API - clear and minimal
from library_tools import format_isbn, validate_isbn, Book
# Bad package API - everything exposed
from library_tools import (
format_isbn, validate_isbn, _clean_isbn, _check_length,
_helper1, _helper2, _internal_parser, _debug_function
)
# Use __all__ to define public API
# Use underscore prefix for private functions
# Use __init__.py to expose convenient imports
The Takeaway
Timothy stood in the Package Architecture, where scattered scripts became organized software.
Packages are directories with init.py: The marker file makes directories importable.
Modules vs packages: Modules are single .py files; packages are directories organizing multiple modules.
init.py controls the package interface: Import key items to make them easily accessible.
Imports are cached in sys.modules: First import executes code, subsequent imports use cached version.
sys.path determines import search: Current directory, standard library, site-packages, in that order.
pycache stores compiled bytecode: Speeds up subsequent imports; should be in .gitignore.
all defines the public API: Controls what import * imports, signals intent to users.
Absolute imports are clearer: Use full package paths for transparency.
Relative imports work within packages: Convenient for tightly coupled code, but require -m flag to run.
The -m flag runs code as a module: Enables relative imports by providing package context.
Subpackages organize complexity: Each subdirectory with __init__.py creates a namespace.
src/ layout prevents import confusion: Forces testing against installed package, catches packaging bugs early.
pyproject.toml makes packages installable: Modern standard for package metadata and dependencies.
pip install -e . for development: Editable install links to source—changes visible immediately without reinstall.
Entry points create CLI commands: Turn Python functions into shell commands.
version tracks releases: Use semantic versioning for clear compatibility signals.
Private functions use underscores: Convention signals internal use, but doesn't prevent import.
Circular imports are design smells: When modules import each other, restructure or use import-time tricks.
Tests live outside the package: Keep test code separate from production code.
README.md documents usage: First thing users see, should show install and basic examples.
Namespace packages share names: Allow multiple distributions to contribute to one namespace.
The Python Package Architecture
Timothy had discovered how to transform his collection of useful functions into professional, reusable packages.
The Package Architecture revealed that good code isn't just correct—it's organized, discoverable, and shareable.
He learned that __init__.py serves as both a marker and a gateway, that import paths reflect structure, that Python caches imports for performance, that __all__ declares intent, and that pyproject.toml bridges the gap between code and distribution.
Also, he understood the mechanics of Python's import system—how sys.path determines search order, how sys.modules caches loaded code, and how the -m flag provides package context for relative imports.
Most importantly, he understood that packaging isn't an afterthought but a fundamental skill—the difference between writing scripts for yourself and creating software for the world.
Aaron Rose is a software engineer and technology writer at tech-reader.blog and the author of Think Like a Genius.
Comments
Post a Comment