Pipelines Python Packages

meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

pypyr

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

pipestat

Pipeline results reporting package

pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

torchx-nightly

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

pytreeclass

Visualize, create, and operate on pytrees in the most intuitive way possible.

polyaxon

AI Infra / AI Orchestration / AI Control Plane

vents

Open source connections catalog, integrations, tools, alerting, and notifications library.

kfp-notebook

Elyra extends JupyterLab with an AI centric approach.

calkit-python

Integrated, automated, accelerated research projects.

taipy

Turns Data and AI algorithms into production-ready web applications in no time.

lsst-pex-config

Configuration interface and history-tracking for LSST Data Management.

mabel

😊 mabel is a platform for authoring data processing systems.

mcp-gitlab

MCP server for GitLab API — 76 tools for AI assistants (Claude, Cursor, VS Code). Projects, MRs, pipelines, CI/CD, approvals, issues, code review.

lsst-pipe-base

LSST Data Management: base classes for data processing tasks

ploomber

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

lsst-ctrl-mpexec

Execution framework for PipelineTask

data-expectations

Are your data meeting your expectations?

lsst-ctrl-bps

A PipelineTask execution framework for multi-node processing for the LSST Batch Production Service (BPS).

taipy-gui

Turns Data and AI algorithms into production-ready web applications in no time.

glide

Easy ETL