#Tokenization
Showing 48 of 48 repositories tagged #tokenization, ranked by stars
💫 Industrial-strength Natural Language Processing (NLP) in Python
🎒 Token-Oriented Object Notation (TOON) – Compact, human-readable, schema-aware JSON for LLM prompts. Spec, benchmarks, TypeScript SDK.
Easy token price estimates for 400+ LLMs. TokenOps.
Secure Vault for Customer PII/PHI/PCI/KYC Records
LunaSec - Dependency Security Scanner that automatically notifies you about vulnerabilities like Log4Shell or node-ipc in your Pull Requests and Builds. Protect yourself in 30 seconds with the LunaTrace GitHub App: https://github.com/marketplace/lunatrace-by-lunasec/
Ravencoin Core integration/staging tree
All the slides, accompanying code and exercises all stored in this repo. 🎈
👑 spaCy building blocks and visualizers for Streamlit apps
Trankit is a Light-Weight Transformer-based Python Toolkit for Multilingual Natural Language Processing
Solidity based "BIKE RENTAL SHOP" on Ethereum network.
Build a ChatGPT like LLM from scratch in PyTorch, explained step by step.
CodeChain's official implementation in Rust.
TokenScript schema, specs and paper
Implementation of the LongRoPE: Extending LLM Context Window Beyond 2 Million Tokens Paper
Language AI Engineering Lab, a place where you can deeply understand and build modern Language AI systems, from fundamentals to production.
The Libre Multilingual Analyzer, a Natural Language Processing (NLP) C++ toolkit.
This repository comprises the theoretical and technical aspects of tokenisation of real world assets.
Transform and optimize your markdown documentation for Large Language Models (LLMs) and RAG systems. Generate llms.txt automatically.
klmbr - a prompt pre-processing technique to break through the barrier of entropy while generating text with LLMs
Reference Solidity implementation of the CMTAT security token framework developed by CMTA to tokenize financial instruments.
Implementation of the dynamic chunking mechanism in H-net by Hwang et al. of Carnegie Mellon
On-chain RWA Tokenization Framework
A model which uses your social media posting predict your MBTI personality type.
Ready-to-run examples showing NoPII PII protection with OpenAI, Anthropic, LangChain, LlamaIndex, and more
a unix-like du command line tool to count token usage per files and directories
Tokenize, encrypt/decrypt, mask your data on the fly with Vaulty proxy
Solana RWA (Real World Assets) backend, RWA tokenization - token, NFT
Use Python and NLTK to build out your own text classifiers and solve common NLP problems
Get identifiers, paths, URLs and words from the previous command output and use them for the next command in @xonsh.
The ultimate open source betting protocol. PolyCash is a P2P blockchain platform for wallets, asset issuance, bonds & gaming.
Solana RWA (Real World Assets) frontend, RWA tokenization - token, NFT
Solana RWA (Real World Assets) smart contract, RWA tokenization - token, NFT
ERC-3643 - Raptor Version is a simple, educational look at the T-REX standard. Using Solidity and Web3, this project demystifies tokenized securities. Remember, Raptor is for learning, not production. Dive in for an accessible peek into blockchain finance!
An open-source DeFi protocol with smart contracts and a web dashboard for tokenized funds, automated liquidity provisioning, and on-chain yield optimization.
Context Limiter & Output Vetter for context bloat. It is a highly specialized, structure-aware JSON built specifically to intercept and compress MCP responses before they annihilate your LLM's context window.
The RWA Token Program is a wrapper and extension program for Solana Token Extensions that creates a uniform approach to permissions tokens on SVM blockchains.
The Implementation of The Ledger of Things Node. Layer 1 decentralized blockchain platform for the tokenization of objects. Proof of Scan protocol. Useful smart-contracts and dApps.
Pecu Novus is a Layer-1 blockchain network designed for financial applications, focusing on hybrid architecture, scalability, and user-friendliness to bridge traditional finance and blockchain technology.
The project is a simple sentiment analysis using NLP. The project in written in python with Jupyter notebook. It shows how to do text preprocessing (removing of bad words, stop words, lemmatization, tokenization). It further shows how to save a trained model, and use the model in a real life suitation. The machine learning model used here is k-Nearest Neighbor which is used to build the model. Various performance evaluation techniques are used, and they include confusion matrix, and Scikit-learn libraries classification report which give the accuracy, precision, recall and f1- score preformance of the model. The target values been classified are positive and negative review.
Private version of CMTAT security token in Noir (Aztec network DSL)
Institutional-grade protocol for tokenizing Real-World Assets (RWAs) on Solana. Features: on-chain compliance, KYC/AML verification, and DeFi integrations for global markets.
Smart contract for AFRX Security Token (ERC-3643) — a regulated digital asset representing tokenized equity in Afrail Inc., a U.S.-registered smart infrastructure company. Built on Ethereum (ERC-20 base) and compliant with SEC Regulation D Rule 506(c) and Regulation S. Backed by 57.7M Class A and 20M Preferred shares. Fully KYC/AML-compli
Provides a comprehensive solution for detecting plagiarism and finding similarities between text documents
Real estate tokenization platform built on multiple blockchains enabling fractional ownership of premium properties through tokenization.
Smart Contract and Tokenization Framework for Africa Fast Ferry Ltd Compliant with SEC Regulation D Rule 506(c) and Regulation S — Combining Regulated Equity (AFFEQ) and Revenue-Sharing (AFFR) Tokens based on ERC-3643 Standards.
🍶 llm-distillery ⇢ use LLMs to run map-reduce summarization tasks on large documents until a target token size is met.
Privacy-first institutional asset tokenization designed to automate regulatory compliance and auditing while preserving strong client confidentiality.
Collection of high-quality resources on blockchain, tokenization, and DLT-based capital markets (EU-Focus)