Data Pipeline Python Packages

flupy

Fluent data pipelines for python and your shell

elementary-data

The dbt-native data observability solution for data & analytics engineers. Monitor your data pipelines in minutes. Available as self-hosted or cloud service with premium features.

doit

CLI task management & automation tool

airbyte-source-declarative-manifest

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

covalent

Pythonic tool for orchestrating machine-learning/high performance/quantum-computing workflows in heterogeneous compute environments.

gusty

Making DAG construction easier

ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

airbyte-source-s3

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

hydra-etl

Declarative ETL pipelines in YAML, validated before they run. CLI, REST API, visual Studio, MCP server.

datris-mcp-server

The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.

augraphy

Augmentation pipeline for rendering synthetic paper printing, faxing, scanning and copy machine processes

airbyte-source-facebook-marketing

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

cereyan

Orchestration for Python data pipelines, without the platform. A fast Rust core drives flows, schedules, backfills, retries, and a live UI. Agentic-ready: MCP tools, agent-attributed runs, human-in-the-loop approvals. One wheel, no runtime dependencies.

airbyte-source-gcs

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

piperider-nightly

Code review for data in dbt

airbyte-source-azure-blob-storage

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

batchit

Batch any Python iterator by count, elapsed time, or both. Zero dependencies.

datarax

Differentiable data pipelines for JAX/Flax NNX: HuggingFace, TFDS and ArrayRecord sources, augmentation stages, scan-based epochs, exact mid-epoch resume.

airbyte-source-google-ads

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

airbyte-source-github

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

mloda

mloda.ai - Open Data Access for AI and ML. Plugin-based. Traceable. Framework-agnostic.

dagster-authkit

Community Auth System for self-hosted Dagster OSS - simple RBAC, Audit-log, and Session Management

airbyte-source-sftp-bulk

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.

airbyte-source-shopify

Open-source data movement for ELT pipelines and AI agents — from APIs, databases & files to warehouses, lakes, and AI applications. Both self-hosted and Cloud.