Data Ingestion Python Packages

oneagent-sdk

Enables custom tracing of Python applications in Dynatrace

ingestr

ingestr is a CLI tool to copy data between any databases with a single command seamlessly.

datris-mcp-server

The Data Control Plane for AI Agents — acquire, validate, and land data over MCP with scoped keys, approval gates, and an audit log. Self-hosted, open source.

rypipe

Format- and source-agnostic ingestion framework: a common execution runtime for turning record-oriented data sources into typed columnar data. Rust core, Python bindings; adapters live in separate packages.

aioscraper

Async Python framework for large-scale API data collection: JSON/REST fan-out with adaptive rate limiting, retries, priority queues and pipelines.

omniload

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

caland

Small tools for the ugly gaps on Databricks: stevin, lely, leeghwater, caland

llm-markdownify

Convert documents, images to high-quality Markdown using Vision LLMs. Built for RAG ingestion pipelines.

lely

Small tools for the ugly gaps on Databricks: stevin, lely, leeghwater, caland

dlt-filesystem

omniload is a polyglot data loader based on dlt. Copy data between any source and any destination.

leeghwater

Small tools for the ugly gaps on Databricks: stevin, lely, leeghwater, caland

stevin

Small tools for the ugly gaps on Databricks: stevin, lely, leeghwater, caland

druid-to-pinot-migrator

CLI and library to translate Apache Druid ingestion specs into Apache Pinot schemas, table configs, and ingestion jobs.

sequor

Build complete API integrations with YAML and SQL. Rapid development without vendor lock-in and per-row costs.

bors

A highly flexible and versatile service integration framework.

enterprise-crawler-framework

Production-oriented infrastructure for reliable Python crawlers and data-collection bots.

feedunify

A high-performance, unifying library for data ingestion pipelines from multiple sources.

nanostream

Modules for ETL Pipelines

metalpipe

Modules for ETL Pipelines

skeem

Infer SQL DDL statements from tabular data.

comet-hunter

An image ingestion and processing system, designed to assist in the discovery of comets.

dataforge-dfg

Data Forge is a high-performance, CLI-first data integration tool designed to streamline the lifecycle of data from ingestion to transformation. Built with Python, it provides a robust framework for handling both ETL and ELT workflows with a focus on automation, reliability, and developer experience.

openauc

Open-source Python toolkit for ingesting, validating, plotting, generating and archiving analytical ultracentrifugation data.

sdmxflow

SDMX ingestion for statistical data warehouses: reproducible datasets, metadata history, and exported codelists in an append-only structure.