S3 Python Packages

s3transfer

Amazon S3 Transfer Manager for Python

smart-open

Utils for streaming large files (S3, HDFS, gzip, bz2...)

moto

A library that allows you to easily mock out tests based on AWS infrastructure.

cloudpathlib

Python pathlib-style classes for cloud storage services such as Amazon S3, Azure Blob Storage, and Google Cloud Storage.

pathy

simple, flexible, offline capable, cloud storage with a Python path-like interface

airbyte-source-declarative-manifest

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

fs-s3fs

Amazon S3 filesystem for PyFilesystem2

yolov5

Packaged version of ultralytics/yolov5 + many extra features

opendal

Apache OpenDAL: One Layer, All Storage.

synapse-s3-storage-provider

Synapse storage provider to fetch and store media in Amazon S3

tiledb

Python interface to the TileDB storage engine

s3-concat

Concat multiple files in s3

cloudstorage

Unified cloud storage API for storage services.

remotezip

Python module to access single members of a zip archive without downloading the full content from a remote web server.

astro-sdk-python

Astro SDK allows rapid and clean development of {Extract, Load, Transform} workflows using Python and SQL, powered by Apache Airflow.

django-s3direct

Directly upload files to S3 compatible services with Django.

cloud-volume

Read and write Neuroglancer datasets programmatically.

databricks-utils

`databricks-utils` is a python package that provide several utility classes/func that improve ease-of-use in databricks notebook.

object-storage-proxy

Fast in-process object storage reverse proxy with python interface.

adk-extra-services

A Python package providing additional service implementations for the Google ADK framework (S3, Redis, MongoDB, Azure, etc)

aioaws

Asyncio compatible SDK for aws services.

cloud-files

Threaded Python and CLI client library for AWS S3, Google Cloud Storage (GCS), in-memory, and the local filesystem.

megfile

Megvii FILE Library - Working with Files in Python same as the standard library

airbyte-source-facebook-marketing

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.