#Chunking
Showing 26 of 26 repositories tagged #chunking, ranked by stars
NCRF++, a Neural Sequence Labeling Toolkit. Easy use to any sequence labeling tasks (e.g. NER, POS, Segmentation). It includes character LSTM/CNN, word LSTM/CNN and softmax/CRF components.
Fully neural approach for text chunking
Adaptive Chunking: automatically select the best chunking method per document for RAG. Accepted at LREC 2026.
The RAG Experiment Accelerator is a versatile tool designed to expedite and facilitate the process of conducting experiments and evaluations using Azure Cognitive Search and RAG pattern.
A package for parsing PDFs and analyzing their content using LLMs.
Pure Rust PDF library for AI/RAG: structure-aware chunking, no ML, no C deps.
PDFStract - Extract, Chunking and Embedding Layer in Your RAG Pipeline - Available as CLI - WEBUI - API
Open-source toolkit for reliable RAG pipelines: convert PDFs to Markdown, clean documents, inspect chunks, compare chunking strategies, and enrich metadata for LLM applications.
A Python CLI to test, benchmark, and find the best RAG chunking strategy for your Markdown documents.
Deprecated/unmaintained Postgres extensions to support end-to-end Retrieval-Augmented Generation (RAG) pipelines
Live TS segmenter and HLS manifest creation in Go
An LLM GUI application; enables you to interact with your files, offering dynamic parameters that can modify response behavior during runtime.
An Overview of the Latest Document Chunking Research
One library to split them all: Sentence, Code, Docs. Chunk smarter, not harder — built for LLMs, RAG pipelines, and beyond.
Грамматический Словарь Русского Языка (+ английский, японский, etc)
smart-llm-loader is a lightweight yet powerful Python package that transforms any document into LLM-ready chunks. Spend less time on preprocessing headaches and more time building what matters. From RAG systems to chatbots to document Q&A, SmartLLMLoader handles the heavy lifting so you can focus on creating exceptional AI applications.
Labelling Sequential Data in Natural Language Processing with R - using CRFsuite
Build document-native LLM applications
Rust CLI implementing the Recursive Language Model (RLM) pattern for Claude Code. Process documents 100x larger than context windows through intelligent chunking, SQLite persistence, and recursive sub-LLM orchestration.
LLM Chatbot w/ Retrieval Augmented Generation using Llamaindex. It demonstrates how to impl. chunking, indexing, and source citation.
Retrieval-augmented generation (RAG) for remote & local LLM use
🧠✂️ SemanticSlicer — A smart text chunker for LLM-ready documents.
Embedding-driven, context-aware text chunking for Semantic Kernel and RAG workflows in .NET
A very fast, feature-packed, AI-first Markdown (CommonMark/GFM) gem for Ruby, based on pulldown-cmark (Rust).
Privacy-first document intelligence engine — parse PDFs, DOCX, PPTX, XLSX & CSV into AI-ready chunks for RAG pipelines. Includes HITL review, 3-layer memory chat, and a production FastAPI server.
Easily create a RAG app based on your Discord messages