Context Vector Data Description (CVDD): An unsupervised anomaly detection method for text

This repository provides a PyTorch implementation of Context Vector Data Description (CVDD), a self-attentive, multi-context one-class classification method for unsupervised anomaly detection on text as presented in our ACL 2019 paper.

Citation and Contact

You find the ACL 2019 paper at https://www.aclweb.org/anthology/P19-1398.

If you find our work useful, please also cite the paper:

@inproceedings{ruff2019,   title     = {Self-Attentive, Multi-Context One-Class Classification for Unsupervised Anomaly Detection on Text},   author    = {Ruff, Lukas and Zemlyanskiy, Yury and Vandermeulen, Robert and Schnake, Thomas and Kloft, Marius},   booktitle = {Proceedings of the 57th Conference of the Association for Computational Linguistics},   month     = {jul},   year      = {2019},   pages     = {4061--4071} }

If you would like to get in touch, just drop an email to contact@lukasruff.com.

Abstract

> There exist few text-specific methods for unsupervised anomaly detection, and for those that do exist, none utilize pre-trained models for distributed vector representations of words. In this paper we introduce a new anomaly detection method---Context Vector Data Description (CVDD)---which builds upon word embedding models to learn multiple sentence representations that capture multiple semantic contexts via the self-attention mechanism. Modeling multiple contexts enables us to perform contextual anomaly detection of sentences and phrases with respect to the multiple themes and concepts present in an unlabeled text corpus. These contexts in combination with the self-attention weights make our method highly interpretable. We demonstrate the effectiveness of CVDD quantitatively as well as qualitatively on the well-known Reuters, 20 Newsgroups, and IMDB Movie Reviews datasets.

Installation

This code is written in Python 3.7 and requires the packages listed in requirements.txt.

Clone the repository to your machine and directory of choice:

git clone https://github.com/lukasruff/CVDD-PyTorch.git

To run the code, we recommend setting up a virtual environment, e.g. using virtualenv or conda:

virtualenv

# pip install virtualenv
cd <path-to-CVDD-PyTorch-directory>
virtualenv myenv
source myenv/bin/activate
pip install -r requirements.txt

conda

cd <path-to-CVDD-PyTorch-directory>
conda create --name myenv
source activate myenv
while read requirement; do conda install -n myenv --yes $requirement; done < requirements.txt

After installing the packages, run python -m spacy download en to download the spaCy en library.

Running experiments

You can run CVDD experiments using the main.py script.

The following are examples on how to run experiments on Reuters-21578, 20 Newsgroups, and IMDB Movie Reviews as reported in the paper.

Reuters-21578

Here's an example on reuters with 'ship' (--normalclass 6) considered to be the normal class using GloVe6B word embeddings for a CVDD model with --nattentionheads 3 and --attention_size 150.

cd <path-to-CVDD-PyTorch-directory>
activate virtual environment
source myenv/bin/activate  # or 'source activate myenv' for conda
change to source directory
cd src
create folder for experimental output
mkdir ../log/test_reuters
run experiment
python main.py reuters cvddNet ../log/testreuters ../data --device cpu --seed 1 --cleantxt --embeddingsize 300 --pretrainedmodel GloVe6B --adscore contextdistmean --nattentionheads 3 --attentionsize 150 --lambdap 1.0 --alphascheduler logarithmic --nepochs 100 --lr 0.01 --lrmilestone 40  --normal_class 6;

The indexation of classes is [0, 1, 2, 3, 4, 5, 6] for ['earn', 'acq', 'crude', 'trade', 'money-fx', 'interest', 'ship'].

20 Newsgroups

Here's an example on newsgroups20 with 'comp' (--normal_class 0) considered to be the normal class using FastTexten word embeddings for a CVDD model with --nattentionheads 3 and --attentionsize 150.

cd <path-to-CVDD-PyTorch-directory>
activate virtual environment
source myenv/bin/activate  # or 'source activate myenv' for conda
change to source directory
cd src
create folder for experimental output
mkdir ../log/test_newsgroups20
run experiment
python main.py newsgroups20 cvddNet ../log/testnewsgroups20 ../data --device cpu --seed 1 --cleantxt --embeddingsize 300 --pretrainedmodel FastTexten --adscore contextdistmean --nattentionheads 3 --attentionsize 150 --lambdap 1.0 --alphascheduler logarithmic --nepochs 100 --lr 0.01 --lrmilestone 40 --normal_class 0;

The indexation of classes is [0, 1, 2, 3, 4, 5] for ['comp', 'rec', 'sci', 'misc', 'pol', 'rel'].

IMDB Movie Reviews

Here's an example on training a CVDD model with --nattentionheads 10 and --attention_size 150 on the full imdb training set (selected via --normalclass -1) using GloVe42B word embeddings.

cd <path-to-CVDD-PyTorch-directory>
activate virtual environment
source myenv/bin/activate  # or 'source activate myenv' for conda
change to source directory
cd src
create folder for experimental output
mkdir ../log/test_imdb
run experiment
python main.py imdb cvddNet ../log/testimdb ../data --device cpu --seed 1 --cleantxt --embeddingsize 300 --pretrainedmodel GloVe42B --adscore contextdistmean --nattentionheads 10 --attentionsize 150 --lambdap 10.0 --alphascheduler soft --nepochs 100 --lr 0.01 --lrmilestone 40 --normal_class -1;

Have a look into main.py for all the possible arguments and options.

License

MIT