S3 Python Packages

s3transfer

Amazon S3 Transfer Manager for Python

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

moto

A library that allows you to easily mock out tests based on AWS infrastructure.

cloudpathlib

Python pathlib-style classes for cloud storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.

pathy

simple, flexible, offline capable, cloud storage with a Python path-like interface

fs-s3fs

Amazon S3 filesystem for PyFilesystem2

airbyte-source-declarative-manifest

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

yolov5

Packaged version of ultralytics/yolov5 + many extra features

opendal

Apache OpenDAL: One Layer, All Storage.

synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3

tiledb

Python interface to the TileDB storage engine

s3-concat

Concat multiple files in s3

remotezip

Python module to access single members of a zip archive without downloading the full content from a remote web server.

aioaws

Asyncio compatible SDK for aws services.

astro-sdk-python

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

django-s3direct

Directly upload files to S3 compatible services with Django.

cloudstorage

Unified cloud storage API for storage services.

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

databricks-utils

`databricks-utils` is a python package that provide several utility classes/func that improve ease-of-use in databricks notebook.

cloud-volume

Read and write Neuroglancer datasets programmatically.

cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.

ministack

Ministack: Free, open-source local AWS emulator - 55+ services, Terraform compatible, real databases. Free forever. MIT licensed.

datap-rs

A config-driven Rust server that publishes Parquet and Delta datasets as fast, typed HTTP APIs from local disk or object storage, with interchangeable DuckDB or Arrow+DataFusion backends, JSON and Arrow IPC output, and production-ready features like auth, metrics, and hot reloads.

object-storage-proxy

Fast in-process object storage reverse proxy with python interface.