Data Engineering Pipeline Python Packages

quickstart-vdk

One framework to develop, deploy and operate data workflows with Python and SQL.

7K 485 72

wxdata

A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.

2K 25 1

vdk-core

Versatile Data Kit SDK Core

2K 485 72

marshmallow-pyspark

Marshmallow serializer integration with pyspark

1K 12 4

vdk-jupyterlab-extension

A Jupyterlab extension for using VDK

1K 485 72

vdk-lineage-model

One framework to develop, deploy and operate data workflows with Python and SQL.

1K 485 72

xontrib-pipeliner

Easily process the lines using pipes in xonsh.

838 63 4

vdk-control-cli

VDK Control CLI allows user to create, delete, manage and their Data Jobs in Kubernetes runtime.

554 485 72

vdk-impala

One framework to develop, deploy and operate data workflows with Python and SQL.

522 485 72

airflow-provider-vdk

Airflow provider for Versatile Data Kit.

484 485 72

vdk-huggingface

Integrate VDK with Huggingface as both data source and target

473 485 72

plotsim

Plotsim generates multi-table synthetic datasets with behavioral trajectories, correlations, and causal lags. Config-driven, deterministic, no real data required.

458 0 0

vdk-data-sources

One framework to develop, deploy and operate data workflows with Python and SQL.

379 485 72

vdk-heartbeat

One framework to develop, deploy and operate data workflows with Python and SQL.

367 485 72

vdk-trino

Versatile Data Kit SDK plugin provides support for trino database and trino transformation templates.

367 485 72

vdk-plugin-control-cli

One framework to develop, deploy and operate data workflows with Python and SQL.

357 485 72

vdk-control-api-auth

Versatile Data Kit plugin library provides support for authentication.

345 485 72

perustats

Tools to download and process public datasets from Peru (INEI & BCRP).

343 5 1

vdk-test-utils

Provides utilities for testing Versatile Data Kit SDK plugins.

330 485 72

pybro-cli

🔥 PyBro CLI - A Collection of Python Dev Tools for Debian 24.04

314 0 0

vdk-server

One framework to develop, deploy and operate data workflows with Python and SQL.

313 485 72

vdk-ingest-http

One framework to develop, deploy and operate data workflows with Python and SQL.

302 485 72

tiny-blocks

Tiny Blocks to build large and complex data pipelines!

281 3 0

vdk-notebook

One framework to develop, deploy and operate data workflows with Python and SQL.

255 485 72