Data Engineering Pipeline Python Packages

quickstart-vdk

One framework to develop, deploy and operate data workflows with Python and SQL.

10K 481 67

vdk-core

Versatile Data Kit SDK Core

3K 481 67

wxdata

A Python package of end-to-end weather data clients & raw data clients with VPN/PROXY support, data processors that decode variable keys from GRIB format into a plain-language format & various tools for assisting Python automated workflows, querying meteorological datasets and filling gaps in meteorological data.

2K 24 1

plotsim

Plotsim generates multi-table synthetic datasets with behavioral trajectories, correlations, and causal lags. Config-driven, deterministic, no real data required.

2K 0 0

vdk-jupyterlab-extension

One framework to develop, deploy and operate data workflows with Python and SQL.

2K 481 67

marshmallow-pyspark

Marshmallow serializer integration with pyspark

2K 12 4

xontrib-pipeliner

Let your pipe lines flow thru the Python code in xonsh.

1K 62 4

vdk-control-cli

One framework to develop, deploy and operate data workflows with Python and SQL.

937 481 67

vdk-lineage-model

VDK Lineage Model plugin defines common lineage model and classes used for managing lineageinformation in other VDK plugins.

879 481 67

vdk-trino

One framework to develop, deploy and operate data workflows with Python and SQL.

727 481 67

vdk-impala

One framework to develop, deploy and operate data workflows with Python and SQL.

650 481 67

airflow-provider-vdk

One framework to develop, deploy and operate data workflows with Python and SQL.

529 481 67

vdk-test-utils

One framework to develop, deploy and operate data workflows with Python and SQL.

498 481 67

vdk-server

Versatile Data Kit SDK plugin that facilitates the installation of a local Control Service.

487 481 67

vdk-heartbeat

Versatile Data Kit Heartbeat and Health Test

466 481 67

vdk-huggingface

One framework to develop, deploy and operate data workflows with Python and SQL.

416 481 67

vdk-plugin-control-cli

One framework to develop, deploy and operate data workflows with Python and SQL.

416 481 67

tiny-blocks

Tiny Blocks to build large and complex data pipelines!

396 3 0

vdk-control-api-auth

One framework to develop, deploy and operate data workflows with Python and SQL.

371 481 67

vdk-ingest-http

One framework to develop, deploy and operate data workflows with Python and SQL.

363 481 67

vdk-kerberos-auth

One framework to develop, deploy and operate data workflows with Python and SQL.

336 481 67

perustats

PeruStats provides a unified, developer-friendly interface to Peru's main public data portals — scraping, caching, and organizing microdata so you can focus on analysis instead of plumbing.

333 5 1

vdk-postgres

One framework to develop, deploy and operate data workflows with Python and SQL.

302 481 67

vdk-csv

One framework to develop, deploy and operate data workflows with Python and SQL.

277 481 67