#Big-data
Showing 60 of 224 repositories tagged #big-data, ranked by stars
The Patterns of Scalable, Reliable, and Performant Large-Scale Systems
ClickHouseยฎ is a real-time analytics database management system
Apache Spark - A unified analytics engine for large-scale data processing
Data science Python notebooks: Deep learning (TensorFlow, Theano, Caffe, Keras), scikit-learn, Kaggle, big data (Spark, Hadoop MapReduce, HDFS), matplotlib, pandas, NumPy, SciPy, Python essentials, AWS, and various command lines.
Apache Flink
Open-source IoT Platform - Device management, data collection, processing and visualization.
An open source cybersecurity protocol for syncing decentralized graph data.
The Data Engineering Cookbook
Official repository of Trino, the distributed SQL query engine for big data, formerly known as PrestoSQL (https://trino.io)
Cloud-native OSS search engine for observability
The most widely used Python to C compiler
A fast, scalable, high performance Gradient Boosting on Decision Trees library, used for ranking, classification, regression and other machine learning tasks for Python, R, Java, C++. Supports computation on CPU and GPU.
Apache DataFusion SQL Query Engine
Apache Beam is a unified programming model for Batch and Streaming data processing.
H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
Arkime is an open source, large scale, full packet capturing, indexing, and database system.
The Open Source Feature Store for AI/ML
The AI search platform
Data-Centric Pipelines and Data Versioning
High-performance data engine for AI and multimodal workloads. Process images, audio, video, and structured data at any scale
Simple and Distributed Machine Learning
Easily turn large sets of image urls to an image dataset. Can download, resize and package 100M urls in 20h on one machine.
๐ FASTJSON2 is a Java JSON library with excellent performance.
Data Science Roadmap from A to Z
๐จ ๐ ๐ป ๐ GraphScope: A One-Stop Large-Scale Graph Computing System from Alibaba | ไธ็ซๅผๅพ่ฎก็ฎ็ณป็ป
Extensible SQL Lexer and Parser for Rust
Koalas: pandas API on Apache Spark
Drop-in Apache Spark replacement written in Rust, unifying batch processing, stream processing, and compute-intensive AI workloads.
A crazy fast analytical database, built on bitmaps. Perfect for ML applications. Learn more at: http://docs.featurebase.com/. Start a Docker instance: https://hub.docker.com/r/featurebasedb/featurebase
ArcticDB is a high performance, serverless DataFrame database built for the Python Data Science ecosystem.
Open-source BI for engineers
Apache DataFusion Ballista Distributed Query Engine
A search engine which can hold 100 trillion lines of log data.
Fluid, elastic data abstraction and acceleration for BigData/AI applications in cloud. (Project under CNCF)
The Auron accelerator for distributed computing framework (e.g., Spark) leverages native vectorized execution to accelerate query processing
Open source security data lake for threat hunting, detection & response, and cybersecurity analytics at petabyte scale on AWS
Apache Spark & Python (pySpark) tutorials for Big Data Analysis and Machine Learning as IPython / Jupyter notebooks
๐ 500+ curated resources for Data Analysis & Data Science: Python, SQL, Statistics, ML, AI, Visualization, Cheatsheets, Roadmaps, Interview Prep. For beginners and experts.
:bar_chart: :clipboard: Dashboards using YAML or JSON files
Tonbo is an embedded database for serverless and edge runtimes.
Example end to end data engineering project.
Extension for Scikit-learn is a seamless way to speed up your Scikit-learn application
One advanced and mature open-source MPP (Massively Parallel Processing) database. Open source alternative to Greenplum Database.
Scalable, reliable, distributed storage system optimized for data analytics and object store workloads.
Automated Deep Learning without ANY human intervention. 1'st Solution for AutoDL challenge@NeurIPS.
Datumbox is an open-source Machine Learning framework written in Java which allows the rapid development of Machine Learning and Statistical applications.
๐ Awesome Data Catalogs and Observability Platforms.
A @ClickHouse fork that supports high-performance vector search and full-text search.
Career Resources for Data Science, Machine Learning, Big Data and Business Analytics Career Repository
Sparkling Water provides H2O functionality inside Spark cluster
Server for the ListenBrainz project, including the front-end (javascript/react) code that it serves and all of the data processing components that LB uses.
An open protocol for secure data sharing
Un repositorio mรกs con conceptos bรกsicos, desafรญos tรฉcnicos y recursos sobre ingenierรญa de datos en espaรฑol ๐งโจ
๐ State-of-the-art learned data structure that enables fast lookup, predecessor, range searches and updates in arrays of billions of items using orders of magnitude less space than traditional indexes
AI projects
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Workflows and interfaces for neuroimaging packages
An on-line movie recommender using Spark, Python Flask, and the MovieLens dataset
Source code for the CERN Open Data portal
A Powerful Data Visualization Tool. Uses TypeScript And Vue3. Scenario-specific templates. User-friendly interfaces. ไธๆฌพๆฐๆฎๅฏ่งๅๅบ็จๆญๅปบๅทฅๅ ท