parallel-corpora-tools

PHP

Tools for filtering and cleaning parallel and monolingual corpora for machine translation and other natural language processing tasks.

Last updated Apr 21, 2026

Stars

Forks

Issues

Stars/day

Attention Score

View on GitHub ↗ Open in github.dev

Topics

#cleaning #corpora #corpus-tools #data-processing #data-science #filtering #language #language-processing #machine #machine-translation #natural-language #natural-language-processing #neural #neural-machine-translation #nlp #nmt #translation

Language breakdown

PHP 74.9%

Shell 25.1%

▸ Files click to expand

README

Corpora Cleaning Tools

Tools for filtering and cleaning parallel and monolingual corpora in order to train better (neural) machine translation systems.

Inspired by the Data Filtering and Data Pre-processing sections of Tilde's WMT17 paper. This repository includes some of the more basic scripts that can help to get rid of the majority of junk from parallel corpora.

Tools included

parallel - tools for parallel corpora
mono - tools for monolingual corpora

Requirements

pip install subword-nmt pip install langid

Publications

If you use this tool, please cite the following paper:

Matīss Rikters (2018). "Impact of Corpora Quality on Neural Machine Translation." In Proceedings of the 8th Conference Human Language Technologies - The Baltic Perspective (Baltic HLT 2018) (2018).

@inproceedings{Rikters2018BalticHLT,
	author = {Rikters, Matīss},
	booktitle={In Proceedings of the 8th Conference Human Language Technologies - The Baltic Perspective (Baltic HLT 2018)},
	title = {{Impact of Corpora Quality on Neural Machine Translation}},
	address={Tartu, Estonia},
	year = {2018}
}

🔗 More in this category