Vision Language Models (VLMs) for Optical Character Recognition (OCR).

| Feature | Support | | :---------------------- | :---------------------------------------------------------------------- | | File Types | :whitecheckmark: PDF, TIFF, PNG, JPG/JPEG, BMP, GIF, WEBP | | VLM Engines | :whitecheckmark: Ollama, OpenAI Compatible (vLLM, SGLang, OpenRouter), OpenAI, Azure OpenAI | | Output Modes | :whitecheckmark: Markdown, HTML, plain text, JSON, BBox | | Batch OCR | :whitecheckmark: Processes many files concurrently with Python, CLI, and web app | | Pipelines | :whitecheckmark: Per-page routing for heterogeneous documents (IndependentPagePipeline) |

🆕Recent Updates

• v0.2.0 (Jun 1, 2025):

• v0.3.0 (Jul 5, 2025):

• v0.3.1 (Oct 11, 2025):

• v0.4.0 (Dec 15, 2025):

• v0.4.3:

• v0.4.4:

• v0.5.0 (May 4, 2026):

• v0.6.0 (Jul 10, 2026):

Table of Contents

• Overview
• Supported Models
• Prerequisites
• Web Application
• Python package
• CLI

✨Overview

Stream OCR resuts in real-time

img/webapp/tablemarkdowndemo.PNG width=1000 >

Batch processing many files

img/webapp/batch_processing.PNG width=1000 >

⭐Supported Models

Open-weights (ALL Supported!!)

• Qwen3-VL
• Qwen2.5-VL
• Llama-3.2
• LLaVa-1.5

Proprietary

🚦Prerequisites

• Python 3.x
• For PDF processing: poppler library.
• At least one VLM inference engine setup (Ollama, OpenAI/Azure API keys, or an OpenAI-compatible API endpoint).

pip install ollama # For Ollama
pip install openai # For OpenAI (compatible) and Azure OpenAI

🌎Web Application

https://github.com/user-attachments/assets/b196453c-fd2c-491a-ba1e-0a77cf7f5941

Installation

:whale:Running with Docker

docker pull daviden1013/vlm4ocr-app:latest
docker run -p 5000:5000 daviden1013/vlm4ocr-app:latest

If port 5000 is already in use on your machine, you can map it to a different local port. For example, to map it to local port 8080:

docker run -p 8080:5000 daviden1013/vlm4ocr-app:latest

Using Ollama with the Dockerized App

Docker Desktop (Windows/Mac): In the VLM4OCR web UI, set the Ollama Host to http://host.docker.internal:11434. Linux: Run the Docker container with host networking:

docker run --network="host" daviden1013/vlm4ocr-app:latest

Install from source

# Install python package
pip install vlm4ocr
Clone source code
git clone https://github.com/daviden1013/vlm4ocr.git
Run Web App
cd vlm4ocr/services/web_app
python run.py

🐍Python package

Installation

Python package is available on PyPi

pip install vlm4ocr

Quick start

from vlm4ocr import VLLMVLMEngine
vlm_engine = VLLMVLMEngine(model="Qwen/Qwen3-VL-30B-A3B-Instruct")

To use other VLM inference engines:

from vlm4ocr import OpenAICompatibleVLMEngine
vlmengine = OpenAICompatibleVLMEngine(model="<modename>", baseurl="<baseurl>", apikey="<apikey>")

Ollama

from vlm4ocr import OllamaVLMEngine
vlmengine = OllamaVLMEngine(modelname="llama3.2-vision:11b-instruct-fp16")

img/openai-logomarkwhite.png width=16 /> OpenAI API

Follow the Best Practices for API Key Safety to set up API key.

export OPENAIAPIKEY=<yourAPIkey>

from vlm4ocr import OpenAIVLMEngine
vlm_engine = OpenAIVLMEngine(model="gpt-4o-mini")

img/Azureicon.png width=32 /> Azure OpenAI API

Follow the Azure AI Services Quickstart to set up Endpoint and API key.

export AZUREOPENAIAPIKEY="<yourAPI_key>"
export AZUREOPENAIENDPOINT="<your_endpoint>"

from llm_ie.engines import AzureOpenAIVLMEngine
vlm_engine = AzureOpenAIVLMEngine(model="gpt-4o-mini",                                    api_version="<your api version>")

We define OCR engine and specify output formats.

from vlm4ocr import OCREngine
Image/PDF paths
imagepath = "/examples/synthesizeddata/GPT-4osynthesizednote1page_1.jpg"
pdfpath = "/examples/synthesizeddata/GPT-4osynthesizednote_1.pdf"
Define OCR engine
ocr = OCREngine(vlmengine, outputmode="markdown")

Run OCR sequentially (process one image at a time) for single or multiple files:

# OCR for a single image ocrresults = ocr.sequentialocr(image_path, verbose=True)

OCR for a single pdf (multiple pages)
ocrresults = ocr.sequentialocr(pdf_path, verbose=True)
OCR for multiple image and pdf files
ocrresults = ocr.sequentialocr([pdfpath, imagepath], verbose=True)
Auto-correct page rotation before OCR — "tesseract" uses Tesseract OSD, "vlm" reuses the OCR engine's VLM
ocrresults = ocr.sequentialocr(imagepath, rotatecorrection="vlm", verbose=True)
Inspect OCR results
len(ocr_results) # 2 files
ocrresults[0].inputdir
ocr_results[0].filename
len(ocr_results[0]) # PDF file number of pages
ocrtext = ocrresults[0].to_string() # OCR text (all pages concatenated)

Run OCR concurrently and write to file:

import asyncio

async def run_ocr():     response = ocr.concurrentocr([imagepath1, imagepath2], concurrentbatch_size=4)     async for result in response:         if result.status == "success":             filename = result.filename             ocrtext = result.tostring()             with open(f"{filename}.md", "w", encoding="utf-8") as f:                 f.write(ocr_text)
asyncio.run(run_ocr())

Run OCR with bounding boxes (outputmode="bbox"). Leave userprompt empty for full-text bbox OCR, or set it to a free-text instruction for targeted extraction:

from vlm4ocr import VLLMVLMEngine, OCREngine
vlm_engine = VLLMVLMEngine(model="Qwen/Qwen3.5-35B-A3B")
Full-text OCR with bounding boxes
ocr = OCREngine(vlmengine=vlmengine, output_mode="bbox")
ocrresults = ocr.sequentialocr(image_path)
Targeted extraction with bounding boxes
ocr = OCREngine(
    vlmengine=vlmengine,
    output_mode="bbox",
    user_prompt="Extract patient name, date of birth, Platelets, CMP glucose, BUN, and RBCs values.",
)
ocrresults = ocr.sequentialocr(image_path)
Inspect bbox results
for pagenum, page in enumerate(ocrresults[0].pages):
    for item in page.bboxes:
        print(item.label, item.bbox, item.text)
# Visualize on the source page (targeted extraction: color by label, show both)     annotated = page.plotbboxes(showlabel=True, show_text=True, color="label")     annotated.save(f"annotatedpage{pagenum}.png")

Note that this is a synthesized data with NO real PHI!!

[ 	{"bbox_2d": [508, 143, 620, 158], "label": "patient name", "text": "Mary Johnson"}, 	{"bbox_2d": [508, 160, 610, 176], "label": "date of birth", "text": "Jan 29, 1990"}, 	{"bbox_2d": [484, 405, 569, 421], "label": "Platelets", "text": "280 x10⁹/L"}, 	{"bbox_2d": [484, 455, 560, 470], "label": "CMP glucose", "text": "90 mg/dL"}, 	{"bbox_2d": [484, 540, 562, 555], "label": "BUN", "text": "12 mg/dL"}, 	{"bbox_2d": [484, 778, 690, 794], "label": "RBCs", "text": "0 - 1 per high power field"} ]

Supply few-shot examples to improve OCR accuracy:

from PIL import Image import asyncio from vlm4ocr import FewShotExample, VLLMVLMEngine, OCREngine

Load few-shot examples
Note that the example text should be the expected OCR output for the example image. Do not include any extra instructions.
example1image = Image.open("example_1.JPG")
with open("example_1.txt", "r") as f:
    example1text = f.read()
example2image = Image.open("example_2.JPG") with open("example_2.txt", "r") as f:     example2text = f.read()
fewshotexamples = [     FewShotExample(image=example1image, text=example1text, maxdimensionpixels=512),     FewShotExample(image=example2image, text=example2text, maxdimensionpixels=512) ]
async def run_ocr():     response = ocr.concurrentocr([imagepath1, imagepath2], concurrentbatchsize=4, fewshotexamples=fewshot_examples)     async for result in response:         if result.status == "success":             filename = result.filename             ocrtext = result.tostring()             with open(f"{filename}.md", "w", encoding="utf-8") as f:                 f.write(ocr_text)
asyncio.run(run_ocr())

💻CLI

Installation

Install the Python package on PyPi and CLI tool will be automatically installed.

pip install vlm4ocr

Usage

# OpenAI compatible API
vlm4ocr --inputpath /examples/synthesizeddata/ \
        --outputpath /examples/ocroutput/ \
        --output_mode markdown \
        --log \
        --vlmengine openaicompatible \
        --model Qwen/Qwen2.5-VL-7B-Instruct \
        --api_key EMPTY \
        --base_url http://localhost:8000/v1 \
        --concurrentbatchsize 4

# OpenAI API export OPENAIAPIKEY=<api key> vlm4ocr --inputpath /examples/synthesizeddata/GPT-4osynthesizednote_1.pdf \         --output_mode HTML \         --log \         --vlm_engine openai \         --model gpt-4.1-mini \         --concurrentbatchsize 4