Pipelines Python Packages

torchx

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

meltano

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

pipestat

Pipeline results reporting package

torchx-nightly

TorchX is a universal job launcher for PyTorch applications. TorchX is designed to have fast iteration time for training/research and support for E2E production ML pipelines when you're ready.

zenml

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

mage-ai

🧙 Build, run, and manage data pipelines for integrating and transforming data.

polyaxon

MLOps Tools For Managing & Orchestrating The Machine Learning LifeCycle

pipefunc

Lightweight fast function pipeline (DAG) creation in pure Python for scientific (HPC) workflows 🕸️🧪

kfp-notebook

Elyra extends JupyterLab with an AI centric approach.

pypyr

pypyr task-runner cli & api for automation pipelines. Automate anything by combining commands, different scripts in different languages & applications into one pipeline process.

vents

Open source connections catalog, integrations, tools, alerting, and notifications library.

taipy

Turns Data and AI algorithms into production-ready web applications in no time.

metaxy

Pluggable sample-level metadata versioning for incremental multimodal pipelines.

pytreeclass

Visualize, create, and operate on pytrees in the most intuitive way possible.

data-expectations

Are your data meeting your expectations?

mlprimitives

Primitives for machine learning and data science.

zenml-nightly

ZenML 🙏: One AI Platform from Pipelines to Agents. https://zenml.io.

calkit-python

That's a nice result you got there, but how did you calculate it?

ploomber

The fastest ⚡️ way to build data pipelines. Develop iteratively, deploy anywhere. ☁️

mabel

😊 mabel is a platform for authoring data processing systems.

tenzir

Tenzir CLI bindings and binary launchers

taipy-gui

Turns Data and AI algorithms into production-ready web applications in no time.

tibanna

Tibanna helps you run your genomic pipelines on Amazon cloud (AWS). It is used by the 4DN DCIC (4D Nucleome Data Coordination and Integration Center) to process data. Tibanna supports CWL/WDL (w/ docker), Snakemake (w/ conda) and custom Docker/shell command.

lsst-pipe-base

LSST Data Management: base classes for data processing tasks