data-engineering

Clean APIs for data cleaning. Python implementation of R package Janitor

Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

A comprehensive Python package template to kickstart and standardize your MLOps initiatives and data pipelines.

A curated collection of 300+ engineering blog articles from top tech companies. Learn how the best engineering teams solve real-world problems at scale.

The most comprehensive SQL guide from a real-world expert! Learn everything from basics to advanced queries, optimizations, and real-world SQL

Quilt is a Scientific Data Management Platform on AWS that helps teams and AI find, trust, and reuse data through deeply versioned, context-rich data packages.

Open-source ETL/ELT you deploy on your own servers or cloud. Built on DuckDB: no-code/low-code visual pipelines or SQL, 385 components, dbt, CDC, data quality, reverse ETL, lineage, MCP for AI agents. No vendor cloud, no per-row billing.

A curated, but incomplete, list of data-centric AI resources.

A Data Engineering & Machine Learning Knowledge Hub

Home of the Open Data Contract Standard (ODCS).

📙 Awesome Data Catalogs and Observability Platforms.

Enforce Data Contracts

🌊 Continuously synchronize the systems where your data lives, to the systems where you _want_ it to live, by managing your data flows with Estuary. 🌊

FlyFish is a data visualization coding platform. We can create a data model quickly in a simple way, and quickly generate a set of data visualization solutions by dragging.

A comprehensive guide to building a modern data warehouse with SQL Server, including ETL processes, data modeling, and analytics.

Egeria core

Un repositorio más con conceptos básicos, desafíos técnicos y recursos sobre ingeniería de datos en español 🧙✨

Community-driven, simple, yet powerful framework for fast, cost-effective distributed Compute over Data.

Neum AI is a best-in-class framework to manage the creation and synchronization of vector embeddings at large scale.

Supplementary Materials for the The Complete dbt (Data Build Tool) Bootcamp Udemy course

Practical Data Engineering: A Hands-On Real-Estate Project Guide

Python framework for building efficient data pipelines. It promotes modularity and collaboration, enabling the creation of complex pipelines from simple, reusable components.

Open-source agentic data engineering harness for dbt, SQL, and cloud warehouses. 100+ tools, 10 warehouses, AI-powered.

Know your data better！Datavines is Next-gen Data Observability Platform, support metadata manage and data quality.

Supercharge BigQuery with BigFunctions

Compilation of high-profile real-world examples of failed machine learning projects

VectorFlow is a high volume vector embedding pipeline that ingests raw data, transforms it into vectors and writes it to a vector DB of your choice.

数据流引擎是一款面向数据集成、数据同步、数据交换、数据共享、任务配置、任务调度的底层数据驱动引擎。数据流引擎采用管执分离、多流层、插件库等体系应对大规模数据任务、数据高频上报、数据高频采集、异构数据兼容的实际数据问题。

Synmetrix – production-ready open source semantic layer on Cube