bigdata

This is a repo with links to everything you'd ever want to learn about data engineering

🚀2.3x faster than MinIO for 4KB object payloads. RustFS is an open-source, S3-compatible high-performance object storage system supporting migration and coexistence with other S3-compatible platforms such as MinIO and Ceph.

High-performance, scalable time-series database designed for Industrial IoT (IIoT) scenarios

Empowering Data Intelligence with Distributed SQL for Sharding, Scalability, and Security Across All Databases.

大数据入门指南 :star:

A curated list of awesome big data frameworks, ressources and other awesomeness.

JuiceFS is a distributed POSIX file system built on top of Redis and S3.

Data Agent Ready Warehouse : One for Analytics, Search, AI, Python Sandbox. — rebuilt from scratch. Unified architecture on your S3.

Out-of-Core hybrid Apache Arrow/NumPy DataFrame for Python, ML, visualization and exploration of big tabular data at a billion rows per second 🚀

Upserts, Deletes And Incremental Processing on Big Data.

A Cloud Native Batch System (Project under CNCF)

100+套大数据可视化炫酷大屏Html5模板；包含行业：社区、物业、政务、交通、金融银行等，全网最新、最多，最全、最酷、最炫大数据可视化模板。陆续更新中

A data integration framework

🔨 用 JSON 来生成结构化的 SQL 语句，基于 Vue3 + TypeScript + Vite + Ant Design + MonacoEditor 实现，项目简单（重逻辑轻页面）、适合练手~

Apache Avro is a data serialization system.

大数据学习，从零开始学习大数据，包含大数据学习各阶段学习视频、面试资料

GridDB is a next-generation open source database that makes time series IoT and big data fast,and easy.

Aix-DB 基于 LangChain/LangGraph 框架，结合 MCP Skills 多智能体协作架构，实现自然语言到数据洞察的端到端转换。

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

基于开源的flink，对其实时sql进行扩展；主要实现了流与维表的join，支持原生flink SQL所有的语法

Byzer (former MLSQL): A low-code open-source programming language for data pipeline, analytics and AI.

大数据知识仓库涉及到数据仓库建模、实时计算、大数据、数据中台、系统设计、Java、算法等。

Distributed Big Data Orchestration Service

Google, Naver multiprocess image web crawler (Selenium)

Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks

:truck: Agile Data Preparation Workflows made easy with Pandas, Dask, cuDF, Dask-cuDF, Vaex and PySpark

First open-source data discovery and observability platform. We make a life for data practitioners easy so you can focus on your business.

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

Data syncing in golang for ClickHouse.

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.