spark

High performance data store solution

This is the github repo for Learning Spark: Lightning-Fast Data Analytics [2nd Edition]

50+ DockerHub public images for Docker & Kubernetes - DevOps, CI/CD, GitHub Actions, CircleCI, Jenkins, TeamCity, Alpine, CentOS, Debian, Fedora, Ubuntu, Hadoop, Kafka, ZooKeeper, HBase, Cassandra, Solr, SolrCloud, Presto, Apache Drill, Nifi, Spark, Consul, Riak

Jupyter magics and kernels for working with remote Spark clusters

Taier is a big data development platform for submission, scheduling, operation and maintenance, and indicator information display

PySpark-Tutorial provides basic algorithms using PySpark

Apache DataFusion Comet Spark Accelerator

Scalable master data management, identity resolution, entity resolution, and deduplication using ML

GraphFrames is a package for Apache Spark which provides DataFrame-based Graphs

Apache Amoro(incubating) is a Lakehouse management system built on open data lake formats.

A Data Engineering & Machine Learning Knowledge Hub

Apache Celeborn is an elastic and high-performance service for shuffle and spilled data.

ADAM is a genomics analysis platform with specialized file formats built using Apache Avro, Apache Spark, and Apache Parquet. Apache 2 licensed.

Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.

NVIDIA cuDF for Apache Spark plugin - accelerate Apache Spark with GPUs

A free tutorial for Apache Spark.

Sparkling Water provides H2O functionality inside Spark cluster

R interface for Apache Spark

Apache Livy is an open source REST interface for interacting with Apache Spark from anywhere.

Open source project for data preparation for GenAI applications

An open protocol for secure data sharing

C# and F# language binding and extensions to Apache Spark

A connector for Spark that allows reading and writing to/from Redis cluster

Expressive types for Spark.

TiSpark is built for running Apache Spark on top of TiDB/TiKV

A curated list of awesome resources related to the Ada and SPARK programming language

DoEKS is a tool to build, deploy and scale Data Platforms on Amazon EKS

This repository contains the development code for sparkMeasure, an Apache Spark performance analysis and troubleshooting library. It simplifies collecting, aggregating, and exporting Spark task/stage metrics, and is designed for practical use by developers and data engineers in interactive analysis, testing, and production monitoring workflows.

80+ DevOps & Data CLI Tools - AWS, GCP, GCF Python Cloud Functions, Log Anonymizer, Spark, Hadoop, HBase, Hive, Impala, Linux, Docker, Spark Data Converters & Validators (Avro/Parquet/JSON/CSV/INI/XML/YAML), Travis CI, AWS CloudFormation, Elasticsearch, Solr etc.

Scriptis is for interactive data analysis with script development(SQL, Pyspark, HiveQL), task submission(Spark, Hive), UDF, function, resource management and intelligent diagnosis.