Python Package Builder for AWS Lambda

Build AWS Lambda deployment packages and layers for multiple Python versions (3.10+) and architectures (x86_64/arm64).

Features

🐍 Multiple Python versions: Support for Python 3.10, 3.11, 3.12, 3.13, and 3.14 (default)
🏗️ Multi-architecture: Build for both x86_64 (amd64) and arm64 (aarch64) by default
📦 Bulk packaging: Uses requirements.txt by default to package multiple dependencies
🔄 Two output formats: Single combined archive (default) or individual archives per package
⚡ Fast builds: Based on Amazon Linux 2023 for compatibility with Lambda runtime
🚀 Zero-config: Run without arguments for sensible defaults

Quick Start

Default Usage (Recommended)

Simply run the build script with no arguments:

./build-multiarch.sh

Default behavior:

Uses Python 3.14 (latest)
Reads from requirements.txt in the current directory
Builds for both x86_64 and arm64 architectures
Creates single combined archive per architecture

Result: combined-python3.14-x86_64.zip and combined-python3.14-aarch64.zip in ./output/

Customizing the Build

# Use a specific Python version
./build-multiarch.sh --python 3.13

# Build only for x86_64
./build-multiarch.sh --skip-arm64

# Create individual archives per package
./build-multiarch.sh --individual

# Build a single package instead of requirements.txt
./build-multiarch.sh --package numpy

# Combine options
./build-multiarch.sh --python 3.12 --skip-arm64 --individual

Command Reference

build-multiarch.sh Options

Option	Description	Default
`--python VERSION`	Python version (3.10-3.14)	3.14
`-r, --requirements FILE`	Path to requirements.txt	requirements.txt
`--package NAME`	Single package name (overrides requirements.txt)	-
`--skip-x86`, `--skip-amd64`	Skip x86_64 architecture build	Build both
`--skip-arm64`	Skip arm64 architecture build	Build both
`--individual`	Create individual archives per package	Single combined
`--image NAME`	Custom Docker image name	lambda-zipper
`--help`	Show help message	-

Usage Examples

# Default: Python 3.14, requirements.txt, multi-arch, combined archive
./build-multiarch.sh

# Use different Python version
./build-multiarch.sh --python 3.13

# Build only for x86_64 architecture
./build-multiarch.sh --skip-arm64

# Create individual archives for each package
./build-multiarch.sh --individual

# Use custom requirements file
./build-multiarch.sh -r ./my-requirements.txt

# Build a single package (ignores requirements.txt)
./build-multiarch.sh --package numpy

# Combine multiple options
./build-multiarch.sh --python 3.12 --skip-arm64 --individual -r prod-requirements.txt

Advanced Usage

Manual Docker Build (Lower-Level)

For direct Docker control, you can build and run containers manually:

Single Package Mode

Build a Docker image:

docker build --build-arg PYTHON_VERSION=3.14 -t lambda-zipper .

Package a single library:

docker run --rm -v $(pwd)/output:/package lambda-zipper numpy

Requirements.txt with Individual Archives

docker run --rm \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper

Requirements.txt with Combined Archive

docker run --rm \
  -e SINGLE_FILE=true \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper

Supported Python Versions

Python Version	Amazon Linux 2023	Installation Method	Status
3.10	✅ Available	dnf package	Supported
3.11	✅ Available	dnf package	Supported
3.12	✅ Available	dnf package	Supported
3.13	✅ Available	dnf package	Supported
3.14	⚙️ Built from source	Compiled with optimizations	Supported (Default)

Note: Python 3.14 is automatically compiled from source when building the Docker image, which takes longer but ensures compatibility with the latest Python version.

Output Formats

Lambda Layer Format

python/
  lib/
    pythonX.Y/
      site-packages/
        [your packages here]

Upload as a Lambda Layer in the AWS Console.

Flat Deployment Format

[your packages at root level]

Extract into your Lambda deployment package alongside your code.

Architecture Support

x86_64 (amd64): Standard Lambda runtime
arm64 (aarch64): AWS Graviton2 Lambda runtime (cost-effective)

Choose the architecture that matches your Lambda function configuration.

Real-World Examples

Data Science Stack

Create requirements.txt:

numpy==1.26.4
pandas==2.1.4
scipy==1.11.4
scikit-learn==1.3.2

Build with defaults (Python 3.14, multi-arch, combined archive):

./build-multiarch.sh

Result: combined-python3.14-x86_64.zip and combined-python3.14-aarch64.zip ready for Lambda Layer

Web Scraping Stack

Create requirements.txt:

requests==2.31.0
beautifulsoup4==4.12.2
lxml==4.9.3

Build for Python 3.13, x86_64 only:

./build-multiarch.sh --python 3.13 --skip-arm64

Computer Vision with OpenCV

Create requirements.txt:

Pillow==10.2.0
opencv-python-headless==4.9.0.80
numpy==1.26.4

Build individual archives for selective deployment:

./build-multiarch.sh --individual

Result: Separate archives for each package, allowing you to pick and choose

AWS SDK Only (Single Package)

Build only boto3 for both architectures:

./build-multiarch.sh --package boto3

Result: boto3-python3.14-x86_64.zip and boto3-python3.14-aarch64.zip

Legacy Python Version Support

Build for Python 3.10 with specific requirements:

./build-multiarch.sh --python 3.10 -r legacy-requirements.txt

Project Structure

.
├── Dockerfile              # Multi-version, multi-arch capable Docker image
├── package.sh             # Packaging script with requirements.txt support
├── build-multiarch.sh     # Helper script for multi-arch builds
├── requirements.example.txt
└── readme.md

How It Works

Base Image: Uses Amazon Linux 2023 for Lambda runtime compatibility
Python Installation: Installs specified Python version (3.10-3.14) with development headers
Package Installation: Uses pip with --target flag to install packages
Packaging: Creates zip archives with proper Lambda Layer structure
Multi-arch: Leverages Docker buildx for platform-specific builds

Troubleshooting

Docker buildx not available

Enable buildx in Docker Desktop settings or install manually:

docker buildx version

Requirements file not found

By default, the script looks for requirements.txt in the current directory. Specify a different file:

./build-multiarch.sh -r path/to/my-requirements.txt

Package has binary dependencies

The build includes gcc, g++, and cmake for compiling binary extensions. Most packages should work, but some may require additional system libraries. Modify the Dockerfile to add specific dependencies.

Lambda Layer size limits

Lambda has a 250MB (unzipped) limit for layers. For large packages:

Use --individual flag to split packages into separate archives
Deploy multiple smaller layers
Use Lambda deployment package directly
Consider AWS Lambda Extensions or Container Images

Build takes too long

Python 3.14 is compiled from source and takes longer. For faster builds:

Use Python 3.13 or earlier: ./build-multiarch.sh --python 3.13
Build only one architecture: ./build-multiarch.sh --skip-arm64
Docker caches layers, so subsequent builds are faster

Advanced Docker Usage

Custom Dockerfile Modifications

Add system dependencies:

# Add after the base install in Dockerfile
RUN dnf -y install postgresql-devel libxml2-devel

Building Without the Helper Script

# Build for specific platform
docker buildx build \
  --platform linux/amd64 \
  --build-arg PYTHON_VERSION=3.14 \
  -t lambda-zipper:custom \
  --load .

# Run with requirements (combined archive)
docker run --rm \
  -e SINGLE_FILE=true \
  -v $(pwd)/requirements.txt:/input/requirements.txt \
  -v $(pwd)/output:/package \
  lambda-zipper:custom

# Run with single package
docker run --rm \
  -v $(pwd)/output:/package \
  lambda-zipper:custom numpy

Output File Naming

Archives are named with the following pattern:

Combined archives (default):

combined-python{VERSION}-{ARCH}.zip
Example: combined-python3.14-x86_64.zip, combined-python3.14-aarch64.zip

Individual package archives (with --individual):

{PACKAGE}-python{VERSION}-{ARCH}.zip
Example: numpy-python3.14-x86_64.zip, requests-python3.13-aarch64.zip

Contributing

Feel free to submit issues and enhancement requests!

License

This project is open source and available for use and modification.

Name		Name	Last commit message	Last commit date
.github		.github
Dockerfile		Dockerfile
LICENSE		LICENSE
build-multiarch.sh		build-multiarch.sh
package.sh		package.sh
readme.md		readme.md
requirements.example.txt		requirements.example.txt

FilesExpand file tree

Directory actions

More options

Directory actions

More options

Latest commit

History

Folders and files

readme.md

Python Package Builder for AWS Lambda

Features

Quick Start

Default Usage (Recommended)

Customizing the Build

Command Reference

build-multiarch.sh Options

Usage Examples

Advanced Usage

Manual Docker Build (Lower-Level)

Single Package Mode

Requirements.txt with Individual Archives

Requirements.txt with Combined Archive

Supported Python Versions

Output Formats

Lambda Layer Format

Flat Deployment Format

Architecture Support

Real-World Examples

Data Science Stack

Web Scraping Stack

Computer Vision with OpenCV

AWS SDK Only (Single Package)

Legacy Python Version Support

Project Structure

How It Works

Troubleshooting

Docker buildx not available

Requirements file not found

Package has binary dependencies

Lambda Layer size limits

Build takes too long

Advanced Docker Usage

Custom Dockerfile Modifications

Building Without the Helper Script

Output File Naming

Contributing

License