Data Engineering Python Packages

awswrangler

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

84.8M 4K 750

great-expectations

Always know what to expect from your data.

19.1M 12K 2K

apache-airflow-providers-common-sql

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

12.9M 47K 18K

snowflake-snowpark-python

Snowflake Snowpark Python API

11.8M 341 158

dagster-pipes

An orchestration platform for the development, production, and observation of data assets.

8.3M 16K 2K

apache-airflow-providers-snowflake

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

8.2M 47K 18K

dagster

An orchestration platform for the development, production, and observation of data assets.

8.1M 16K 2K

dagster-shared

An orchestration platform for the development, production, and observation of data assets.

7.7M 16K 2K

prefect

Prefect is a workflow orchestration framework for building resilient data pipelines in Python.

7.3M 24K 3K

apache-airflow-providers-mysql

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

7.2M 47K 18K

apache-airflow-providers-ssh

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

7.2M 47K 18K

apache-airflow

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

7.1M 47K 18K

dagster-graphql

An orchestration platform for the development, production, and observation of data assets.

6.5M 16K 2K

dagster-webserver

An orchestration platform for the development, production, and observation of data assets.

6.3M 16K 2K

apache-airflow-providers-common-compat

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

6.2M 47K 18K

apache-airflow-providers-google

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

6.2M 47K 18K

apache-airflow-providers-databricks

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

6.1M 47K 18K

dagster-postgres

An orchestration platform for the development, production, and observation of data assets.

5.2M 16K 2K

apache-airflow-providers-smtp

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

5M 47K 18K

dlt

data load tool (dlt) is an open source Python library that makes data loading easy 🛠️

5M 6K 624

apache-airflow-providers-http

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

4.6M 47K 18K

apache-airflow-providers-slack

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

4.5M 47K 18K

apache-airflow-providers-cncf-kubernetes

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

4.4M 47K 18K

apache-airflow-providers-fab

Apache Airflow - A platform to programmatically author, schedule, and monitor workflows

4.4M 47K 18K