CONLL-U to Pandas DataFrame

Turn CONLL-U documents into Pandas DataFrames for easy NLP!

Install

pip install conll-df

Usage

curl -O https://raw.githubusercontent.com/UniversalDependencies/UD_English/master/en-ud-train.conllu

import pandas as pd
from conlldf import conlldf
path = 'en-ud-train.conllu'
df = conlldf(path, fileindex=False)
df.head(40).to_html()

Output (truncated):

Function arguments

| Name | Type | Description | |---|---|---| | path | str | Path to CONLL-U file | | add_gov | bool | Create extra columns for governor word, lemma, POS and function | | skip_morph | bool | Enable if you'd like to skip the parsing of morphological and extra fields | | v2 | bool/'auto' | CONLL-U version of file. By default, detect from data | | drop | list | list of column names you don't need | | add_meta | bool | add columns for sentence-level metadata | | categories | bool | Convert columns to categorical format where possible | | file_index | bool | Include filename in index levels | | extra_fields | list/'auto' | Names of extra fields in the last column. By default, detect from data | | kwargs | dict | additional arguments to pass to pandas.read_csv()` |

Configuring these arguments can increase speed a lot, so if speed is important to you, turn off the features you don't need.

Where to from here?

If you're working with Python and CONLL-U, you might want to take a look at [tücan, which provides a command-line and web-app interface for exploring CONLL-U datasets.

Alternatively, there's plenty of cool stuff you can do with Pandas by itself. Here are some toy examples:

Pivot table

piv = df.pivot_table(columns='f', index=['x'], aggfunc=len)
piv.fillna(0).astype(int).to_html()

Create a chainable search method

def searcher(df, column, query, inverse=False):
    """Search column for regex query"""
    bool_ix = df[column].str.contains(query)
    return df[boolix] if not inverse else df[~boolix]
pd.DataFrame.search = searcher
get nominal subjects starting with a, b or c
df.search('f', 'nsubj').search('w', '^[abc]').head().to_html()

Create a concordancer

def _conclines(match, df=False, column=False):
    """Apply this to each sentence"""
    s, i = match.name
    sent = df['w'].loc[s]
    match['left'] = sent.loc[:i-1].str.cat(sep=' ')
    match['right'] = sent.loc[i+1:].str.cat(sep=' ')
    formatted = match['w']
    if column != 'w':
        formatted += '/' + match[column]
    match['match'] = formatted
    return match
def conc(df, column, query):     """Build simple concordancer"""     # get query matches     matches = df[df[column].str.contains(query)]     # add left and right columns     lines = matches.apply(_conclines, df=df, column=column, axis=1)     return lines[['left', 'match', 'right']]
pd.DataFrame.conc = conc lines = df.head(1000).conc('l', 'be') lines.head(10).to_html()