data-engineering

SQL Translator is a tool for converting natural language queries into SQL code using artificial intelligence. This project is 100% free and open source.

pandas on AWS - Easy integration with Athena, Glue, Redshift, Timestream, Neptune, OpenSearch, QuickSight, Chime, CloudWatchLogs, DynamoDB, EMR, SecretManager, PostgreSQL, MySQL, SQLServer and S3 (Parquet, CSV, JSON and EXCEL).

A list of useful resources to learn Data Engineering from scratch

Maestro: Netflix’s Workflow Orchestrator

Blazing-fast Data-Wrangling toolkit

Memphis.dev is a highly scalable and effortless data streaming platform

Learn how to design, develop, deploy and iterate on production-grade ML applications.

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

An Awesome List of Open-Source Data Engineering Projects

Apache DevLake is an open-source dev data platform to ingest, analyze, and visualize the fragmented data from DevOps tools, extracting insights for engineering excellence, developer experience, and community growth.

Learn to build your Second Brain AI assistant with LLMs, agents, RAG, fine-tuning, LLMOps and AI systems techniques.

Meltano: the declarative code-first data integration engine that powers your wildest data and ML-powered product ideas. Say goodbye to writing, maintaining, and scaling your own API integrations.

Apache Hamilton helps data scientists and engineers define testable, modular, self-documenting dataflows, that encode lineage/tracing and metadata. Runs and scales everywhere python does.

A new SOTA for RAG — an original retrieval architecture and an open-source knowledge base for humans and agents.

A low code Machine Learning personalized ranking service for articles, listings, search results, recommendations that boosts user engagement. A friendly Learn-to-Rank engine

Data Contracts engine for the modern data stack. https://www.soda.io

Datart is a next generation Data Visualization Open Platform

Python Stream Processing

The best place to learn data engineering. Built and maintained by the data engineering community.

Feathr – A scalable, unified data and AI engineering platform for enterprise

A curated list of awesome dbt resources

More than 2000+ Data engineer interview questions.

MLRun is an open source MLOps platform for quickly building and managing continuous ML applications across their lifecycle. MLRun integrates into your development and CI/CD environment and automates the delivery of production data, ML pipelines, and online applications.

🔥🔥🔥 Open source Reverse ETL - alternative to hightouch and census.

A comprehensive list of 180+ YouTube Channels for Data Science, Data Engineering, Machine Learning, Deep learning, Computer Science, programming, software engineering, etc.

👾 nao is an open source analytics agent. (1) Create context with nao-core cli, (2) deploy nao chat interface for everyone

:bar_chart: :clipboard: Dashboards using YAML or JSON files

ktx is an executable context layer for data and analytics agents 🐙 Allow Claude Code, Codex, or other AI agents to query analytical databases accurately and with full context of your company

Python Streaming DataFrames for Kafka

Concurrent Python made simple