apache-spark

The open source AI engineering platform for agents, LLMs, and ML models. MLflow enables teams of all sizes to debug, evaluate, monitor, and optimize production-quality AI applications while controlling costs and managing access to models and data.

lakeFS - Data version control for your data lake | Git for data

Simple and Distributed Machine Learning Python Library porting ML algorithms for Spark

Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.

Kubernetes operator for managing the lifecycle of Apache Spark applications on Kubernetes.

.NET for Apache® Spark™ makes Apache Spark™ easily accessible to .NET developers.

Apache Spark docker image

Feathr – A scalable, unified data and AI engineering platform for enterprise

A curated list of awesome Apache Spark packages and resources.

The Internals of Apache Spark

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

MapReduce, Spark, Java, and Scala for Data Algorithms Book

R interface for Apache Spark

C# and F# language binding and extensions to Apache Spark

Fully managed Apache Parquet implementation

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

pyspark methods to enhance developer productivity 📣 👯 🎉

A command-line tool for launching Apache Spark clusters.

A list about Apache Kafka

REST web service for the true real-time scoring (<1 ms) of Scikit-Learn, R and Apache Spark models