pandas-multiprocess

A Python package to process Pandas Dataframe using multi-processing.

Install

pip install pandas-multiprocess

Example

Import the package

from pandasmultiprocess import multiprocess

Define a function which will process each row in a Pandas DataFrame

The func must take a pandas.Series as its first positional argument and returns either a pandas.Series or a list of pands.Series.

The function has one positional argument data_row, additional arguments can be defined and the values of the additional arguments will be passed through multi_process(). Here we use **args to stand for the additional arguments.

def func(data_row, **args):     # data_row (pd.Series): a row of a panda Dataframe     # args: a dict of additional arguments     datarow['sum'] = datarow['col1'] + datarow['col_2']     return data_row

Initiate a DataFrame

import pandas as pd import numpy as np df_len = 1000 df = pd.DataFrame({'col1': np.random.normal(size=dflen),                    'col2': np.random.cd normal(size=dflen)                    })

Process it using multiprocess

# The args will be passed to the additional arguments of func() args = {} result = multi_process(func=func,                        data=df,                        num_process=8,                        **args)

The above operation is equivalent as below, but much more efficient

result = df.apply(func, axis=1, **args)

The result of example demonstrate the efficiency of pandas-multiprocess in processing computational expensive operations for each row of a Datafram.

Running examples... 100%|████| 100/100 [00:01<00:00, 68.65it/s]8 processes run time 2.189883 seconds. 100%|████| 100/100 [00:00<00:00, 140.90it/s]16 processes run time 1.440812 seconds. Pandas apply() run time 11.165841 seconds.