Intel® Extension for Transformers ===========================

An Innovative Transformer-based Toolkit to Accelerate GenAI/LLM Everywhere

🏭Architecture   |   💬NeuralChat   |   😃Inference on CPU   |   😃Inference on GPU   |   💻Examples   |   📖Documentations

🏃Installation

Quick Install from Pypi

pip install intel-extension-for-transformers

For system requirements and other installation tips, please refer to Installation Guide

🌟Introduction

Intel® Extension for Transformers is an innovative toolkit designed to accelerate GenAI/LLM everywhere with the optimal performance of Transformer-based models on various Intel platforms, including Intel Gaudi2, Intel CPU, and Intel GPU. The toolkit provides the below key features and examples:

• Seamless user experience of model compressions on Transformer-based models by extending Hugging Face transformers APIs and leveraging Intel® Neural Compressor

• Advanced software optimizations and unique compression-aware runtime (released with NeurIPS 2022's paper Fast Distilbert on CPUs and QuaLA-MiniLM: a Quantized Length Adaptive MiniLM, and NeurIPS 2021's paper Prune Once for All: Sparse Pre-Trained Language Models)

• Optimized Transformer-based model packages such as Stable Diffusion, GPT-J-6B, GPT-NEOX, BLOOM-176B, T5, Flan-T5, and end-to-end workflows such as SetFit-based text classification and document level sentiment analysis (DLSA)

• NeuralChat, a customizable chatbot framework to create your own chatbot within minutes by leveraging a rich set of plugins such as Knowledge Retrieval, Speech Interaction, Query Caching, and Security Guardrail. This framework supports Intel Gaudi2/CPU/GPU.

• Inference of Large Language Model (LLM) in pure C/C++ with weight-only quantization kernels for Intel CPU and Intel GPU (TBD), supporting GPT-NEOX, LLAMA, MPT, FALCON, BLOOM-7B, OPT, ChatGLM2-6B, GPT-J-6B, and Dolly-v2-3B. Support AMX, VNNI, AVX512F and AVX2 instruction set. We've boosted the performance of Intel CPUs, with a particular focus on the 4th generation Intel Xeon Scalable processor, codenamed Sapphire Rapids.

🔓Validated Hardware

Hardware	Fine-Tuning		Inference
	Full	PEFT	8-bit	4-bit
Intel Gaudi2	✔	✔	WIP (FP8)	-
Intel Xeon Scalable Processors	✔	✔	✔ (INT8, FP8)	✔ (INT4, FP4, NF4)
Intel Xeon CPU Max Series	✔	✔	✔ (INT8, FP8)	✔ (INT4, FP4, NF4)
Intel Data Center GPU Max Series	WIP	WIP	WIP (INT8)	✔ (INT4)
Intel Arc A-Series	-	-	WIP (INT8)	✔ (INT4)
Intel Core Processors	-	✔	✔ (INT8, FP8)	✔ (INT4, FP4, NF4)

In the table above, "-" means not applicable or not started yet.

🔓Validated Software

Software	Fine-Tuning		Inference
	Full	PEFT	8-bit	4-bit
PyTorch	2.0.1+cpu, 2.0.1a0 (gpu)	2.0.1+cpu, 2.0.1a0 (gpu)	2.1.0+cpu, 2.0.1a0 (gpu)	2.1.0+cpu, 2.0.1a0 (gpu)
Intel® Extension for PyTorch	2.1.0+cpu, 2.0.110+xpu	2.1.0+cpu, 2.0.110+xpu	2.1.0+cpu, 2.0.110+xpu	2.1.0+cpu, 2.0.110+xpu
Transformers	4.35.2(CPU), 4.31.0 (Intel GPU)	4.35.2(CPU), 4.31.0 (Intel GPU)	4.35.2(CPU), 4.31.0 (Intel GPU)	4.35.2(CPU), 4.31.0 (Intel GPU)
Synapse AI	1.13.0	1.13.0	1.13.0	1.13.0
Gaudi2 driver	1.13.0-ee32e42	1.13.0-ee32e42	1.13.0-ee32e42	1.13.0-ee32e42
intel-level-zero-gpu	1.3.26918.50-736~22.04	1.3.26918.50-736~22.04	1.3.26918.50-736~22.04	1.3.26918.50-736~22.04

Please refer to the detailed requirements in CPU, Gaudi2, Intel GPU.

🔓Validated OS

Ubuntu 20.04/22.04, Centos 8.

🌱Getting Started

Chatbot

Below is the sample code to create your chatbot. See more examples.

Serving (OpenAI-compatible RESTful APIs)

NeuralChat provides OpenAI-compatible RESTful APIs for chat, so you can use NeuralChat as a drop-in replacement for OpenAI APIs. You can start NeuralChat server either using the Shell command or Python code.

# Shell Command
neuralchatserver start --configfile ./server/config/neuralchat.yaml

# Python Code
from intelextensionfortransformers.neuralchat import NeuralChatServerExecutor
server_executor = NeuralChatServerExecutor()
serverexecutor(c, logfile="./neuralchat.log")

NeuralChat service can be accessible through OpenAI client library, curl commands, and requests library. See more in NeuralChat.

Offline

from intelextensionfortransformers.neuralchat import build_chatbot
chatbot = build_chatbot()
response = chatbot.predict("Tell me about Intel Xeon Scalable Processors.")

Transformers-based extension APIs

Below is the sample code to use the extended Transformers APIs. See more examples.

INT4 Inference (CPU)

We encourage you to install NeuralSpeed to get the latest features (e.g., GGUF support) of LLM low-bit inference on CPUs. You may also want to use v1.3 without NeuralSpeed by following the document

from transformers import AutoTokenizer
from intelextensionfor_transformers.transformers import AutoModelForCausalLM
model_name = "Intel/neural-chat-7b-v3-1"     
prompt = "Once upon a time, there existed a little girl,"
tokenizer = AutoTokenizer.frompretrained(modelname, trustremotecode=True) inputs = tokenizer(prompt, returntensors="pt").inputids
model = AutoModelForCausalLM.frompretrained(modelname, loadin4bit=True) outputs = model.generate(inputs)

You can also load GGUF format model from Huggingface, we only support Q40/Q50/Q8_0 gguf format for now.

from transformers import AutoTokenizer from intelextensionfor_transformers.transformers import AutoModelForCausalLM

Specify the GGUF repo on the Hugginface
model_name = "TheBloke/Llama-2-7B-Chat-GGUF"
Download the the specific gguf model file from the above repo
gguffile = "llama-2-7b-chat.Q40.gguf"
make sure you are granted to access this model on the Huggingface.
tokenizer_name = "meta-llama/Llama-2-7b-chat-hf"
prompt = "Once upon a time, there existed a little girl,"
tokenizer = AutoTokenizer.frompretrained(tokenizername, trustremotecode=True)
inputs = tokenizer(prompt, returntensors="pt").inputids
model = AutoModelForCausalLM.frompretrained(modelname, gguffile = gguffile) outputs = model.generate(inputs)

You can also load PyTorch Model from Modelscope >Note:require modelscope

from transformers import TextStreamer from modelscope import AutoTokenizer from intelextensionfor_transformers.transformers import AutoModelForCausalLM modelname = "qwen/Qwen-7B"     # Modelscope modelid or local model prompt = "Once upon a time, there existed a little girl,"

model = AutoModelForCausalLM.frompretrained(modelname, loadin4bit=True, model_hub="modelscope") tokenizer = AutoTokenizer.frompretrained(modelname, trustremotecode=True) inputs = tokenizer(prompt, returntensors="pt").inputids streamer = TextStreamer(tokenizer) outputs = model.generate(inputs, streamer=streamer, maxnewtokens=300)

You can also load the low-bit model quantized by GPTQ/AWQ/RTN/AutoRound algorithm.

from transformers import AutoTokenizer from intelextensionfor_transformers.transformers import AutoModelForCausalLM, GPTQConfig

Hugging Face GPTQ/AWQ model or use local quantize model
modelname = "MODELNAMEORPATH"
prompt = "Once upon a time, a little girl"
tokenizer = AutoTokenizer.frompretrained(modelname, trustremotecode=True) inputs = tokenizer(prompt, returntensors="pt").inputids model = AutoModelForCausalLM.frompretrained(modelname, trustremotecode=True) outputs = model.generate(inputs)

INT4 Inference (GPU)

import intelextensionfor_pytorch as ipex
from intelextensionfor_transformers.transformers.modeling import AutoModelForCausalLM
from transformers import AutoTokenizer
import torch
device_map = "xpu" model_name ="Qwen/Qwen-7B" tokenizer = AutoTokenizer.frompretrained(modelname, trustremotecode=True) prompt = "Once upon a time, there existed a little girl," inputs = tokenizer(prompt, returntensors="pt").inputids.to(device_map)
model = AutoModelForCausalLM.frompretrained(modelname, trustremotecode=True,                                               devicemap=devicemap, loadin4bit=True)
model = ipex.optimizetransformers(model, inplace=True, dtype=torch.float16, quantizationconfig=True, device=device_map)
output = model.generate(inputs)

Note: Please refer to the example and script for more details.

Langchain-based extension APIs

Below is the sample code to use the extended Langchain APIs. See more examples.

from langchaincommunity.llms.huggingfacepipeline import HuggingFacePipeline
from langchain.chains import RetrievalQA
from langchain_core.vectorstores import VectorStoreRetriever
from intelextensionfor_transformers.langchain.vectorstores import Chroma
retriever = VectorStoreRetriever(vectorstore=Chroma(...))
retrievalQA = RetrievalQA.from_llm(llm=HuggingFacePipeline(...), retriever=retriever)

🎯Validated Models

You can access the validated models, accuracy and performance from Release data or Medium blog.

📖Documentation

OVERVIEW
NeuralChat				Neural Speed
NEURALCHAT
Chatbot on Intel CPU		Chatbot on Intel GPU			Chatbot on Gaudi
Chatbot on Client				More Notebooks
NEURAL SPEED
Neural Speed		Streaming LLM		Low Precision Kernels		Tensor Parallelism
LLM COMPRESSION
SmoothQuant (INT8)		Weight-only Quantization (INT4/FP4/NF4/INT8)			QLoRA on CPU
GENERAL COMPRESSION
Quantization		Pruning		Distillation		Orchestration
Data Augmentation		Export		Metrics		Objectives
Pipeline		Length Adaptive			Early Exit
TUTORIALS & RESULTS
Tutorials		LLM List		General Model List		Model Performance

🙌Demo

• LLM Infinite Inference (up to 4M tokens)

https://github.com/intel/intel-extension-for-transformers/assets/109187816/1698dcda-c9ec-4f44-b159-f4e9d67ab15b

• LLM QLoRA on Client CPU

https://github.com/intel/intel-extension-for-transformers/assets/88082706/9d9bdb7e-65db-47bb-bbed-d23b151e8b31

📃Selected Publications/Events

• Blog published on Huggingface: Building Cost-Efficient Enterprise RAG applications with Intel Gaudi 2 and Intel Xeon (May 2024)
• Blog published on Intel Developer News: Efficient Natural Language Embedding Models with Intel® Extension for Transformers (May 2024)
• Blog published on Techcrunch: Intel and others commit to building open generative AI tools for the enterprise (Apr 2024)
• Video on YouTube: Intel Vision Keynotes 2024 (Apr 2024)
• Blog published on Vectara: Do Smaller Models Hallucinate More? (Apr 2024)
• Blog of Intel Developer News: Use the neural-chat-7b Model for Advanced Fraud Detection: An AI-Driven Approach in Cybersecurity (March 2024)
• CES 2024: CES 2024 Great Minds Keynote: Bringing the Limitless Potential of AI Everywhere: Intel Hybrid Copilot demo (Jan 2024)
• Blog published on Medium: Connect an AI agent with your API: Intel Neural-Chat 7b LLM can replace Open AI Function Calling (Dec 2023)
• NeurIPS'2023 on Efficient Natural Language and Speech Processing: Efficient LLM Inference on CPUs (Nov 2023)
• Blog published on Hugging Face: Intel Neural-Chat 7b: Fine-Tuning on Gaudi2 for Top LLM Performance (Nov 2023)
• Blog published on VMware: AI without GPUs: A Technical Brief for VMware Private AI with Intel (Nov 2023)

View Full Publication List

Additional Content

• Release Information
• Contribution Guidelines
• Legal Information
• Security Policy
• Apache License

Acknowledgements

• Excellent open-source projects: bitsandbytes, FastChat, fastRAG, ggml, gptq, llama.cpp, lm-evauation-harness, peft, trl, streamingllm and many others.

• Thanks to all the contributors.

💁Collaborations

Welcome to raise any interesting ideas on model compression techniques and LLM-based chatbot development! Feel free to reach us, and we look forward to our collaborations on Intel Extension for Transformers!