#Data-processing
Showing 60 of 74 repositories tagged #data-processing, ranked by stars
Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
A collection of handy Bash One-Liners and terminal tricks for data processing and Linux system maintenance.
Incremental engine for long horizon agents 🌟 Star if you like it!
Miller is like awk, sed, cut, join, and sort for name-indexed data such as CSV, TSV, and tabular JSON
Unified querying, transformation, and modification of JSON, TOML, YAML, XML, INI, HCL, KDL and CSV.
Data processing for and with foundation models! 🍎 🍋 🌽 ➡️ ➡️🍸 🍹 🍷
Easy Data Preparation with latest LLMs-based Operators and Pipelines.
A GPU-accelerated library containing highly optimized building blocks and an execution engine for data processing to accelerate deep learning training and inference applications.
High-performance AI pipeline engine with a C++ core and 50+ Python-extensible nodes. Build, debug, and scale LLM workflows with 13+ model providers, 8+ vector databases, and agent orchestration, all from your IDE. Includes VS Code extension, TypeScript/Python SDKs, and Docker deployment.
A light-weight, flexible, and expressive statistical data testing library
Kubernetes-native platform to run massively parallel data/streaming jobs
The Context Layer for unstructured data: typed, versioned datasets over S3, GCS, Azure
Large-scale pretraining for dialogue
Toolkit for Machine Learning, Natural Language Processing, and Text Generation, in TensorFlow. This is part of the CASL project: http://casl-project.ai/
Python Stream Processing
Scalable data pre processing and curation toolkit for LLMs
Extract Transform Load for Python 3.5+
Data and tools for generating and inspecting OLMo pre-training data.
Concurrent Python made simple
Source code accompanying book: Data Science on the Google Cloud Platform, Valliappa Lakshmanan, O'Reilly 2017
Distribute and run AI workloads on Kubernetes magically in Python, like PyTorch for ML infra.
Large-scale pretrained models for goal-directed dialog
Google Cloud Dataflow provides a simple, powerful model for building both batch and streaming parallel data processing pipelines.
Command line tool to download and extract data from HTML/XML pages or JSON-APIs, using CSS, XPath 3.0, XQuery 3.0, JSONiq or pattern matching. It can also create new or transformed XML/HTML/JSON documents.
Integrating the Best of TF into PyTorch, for Machine Learning, Natural Language Processing, and Text Generation. This is part of the CASL project: http://casl-project.ai/
Advanced and Fast Data Transformation in R
Machine Learning notebooks for refreshing concepts.
Harmonious distributed data analysis in Rust.
📈 PatternPy: A Python package revolutionizing trading analysis with high-speed pattern recognition, leveraging Pandas & Numpy. Effortlessly spot Head & Shoulders, Tops & Bottoms, Supports & Resistances. For experts & beginners. #TradingMadeEasy 🔥
Lineage metadata API, artifacts streams, sandbox, API, and spaces for Polyaxon
A multi-cloud framework for big data analytics and embarrassingly parallel jobs, that provides an universal API for building parallel applications in the cloud ☁️🚀
An end-to-end data engineering pipeline that orchestrates data ingestion, processing, and storage using Apache Airflow, Python, Apache Kafka, Apache Zookeeper, Apache Spark, and Cassandra. All components are containerized with Docker for easy deployment and scalability.
Python Adaptive Signal Processing
Predict the Power Production of a solar panel farm from Weather Measurements using Machine Learning
Super fast list of dicts to pre-formatted tables conversion library for Python 2/3
A pure Python implementation of Apache Spark's RDD and DStream interfaces.
Tangle is a web app that allows the users to build and run Machine Learning pipelines without having to set up development environment.
MCP Server and CLI for Apache Spark History Server. Debug Spark applications from AI agents, scripts, or the terminal.
Remote Sensing and GIS Software Library; python module tools for processing spatial data.
High performance data processing employs high performance computing (HPC) to process data, which is then translated into information and knowledge. The advent of high-performance computing and data analytics enabled real-time interrogation of extremely large data sets.
Desktop GUI for SNAP based on NetBeans Platform
A full data warehouse infrastructure with ETL pipelines running inside docker on Apache Airflow for data orchestration, AWS Redshift for cloud data warehouse and Metabase to serve the needs of data visualizations such as analytical dashboards.
A simple package to abstract away the process of creating usable DataFrames for data analytics. This package is heavily inspired by the amazing Python library, Pandas.
🎯 Visual Excel data processing tool - Build complex workflows without coding. Drag & drop interface with real-time preview. Cross-platform desktop app.
DocWire SDK: Award-winning modern data processing in C++20. SourceForge Community Choice & Microsoft support. AI-driven processing. Supports nearly 100 data formats, including email boxes and OCR. Boost efficiency in text extraction, web data extraction, data mining, document analysis. Offline processing is possible for security and confidentiality
Deep learning tools for predicting oil well data
Prosto is a data processing toolkit radically changing how data is processed by heavily relying on functions and operations with functions - an alternative to map-reduce and join-groupby
DeepStream Libraries offer CVCUDA, NvImageCodec, and PyNvVideoCodec modules as Python APIs for seamless integration into custom frameworks.
Data Visualization Tutorial | Matplotlib | Seaborn | Pandas
A blazing fast exporter for your Elasticsearch data.
End to end data engineering project
⇔ IterTable is a Pythonic API for iterating through tabular data formats, including CSV, XLSX, XML, and JSON.
Fast Generic Execution Graph/Network
Visual AI development framework for training and inference of ML models, scaling pipelines, and automating workflows with Python
“Data science” is just about as broad of a term as they come. It may be easiest to describe what it is by listing its more concrete components: Data exploration & analysis. Included here: Pandas; NumPy; SciPy; a helping hand from Python's Standard Library.
Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.
Slipstream provides a data-flow model to simplify development of stateful streaming applications.
Little utility to decode Metastock files and write them in text format
Framework for processing and filtering datasets
This document forms the basis of several workshops/talks that get into everyday programming with R, but also includes mirrored code in Python as Jupyter notebooks.