es_pandas

Read, write and update large scale pandas DataFrame with ElasticSearch.

Requirements

Installation The package is hosted on PyPi and can be installed with pip:

pip install es_pandas

Deprecation Notice

Supporting of ElasticSearch 5.x will by deprecated in future version.

Usage

import time
import pandas as pd
from espandas import espandas
Information of es cluseter
es_host = 'localhost:9200'
index = 'demo'
crete es_pandas instance
ep = espandas(eshost)
Example data frame
df = pd.DataFrame({'Num': [x for x in range(100000)]})
df['Alpha'] = 'Hello'
df['Date'] = pd.datetime.now()
init template if you want
doc_type = 'demo'
ep.initestmpl(df, doc_type)
Example of write data to es, use the template you create
ep.toes(df, index, doctype=doctype, threadcount=2, chunk_size=10000)
set useindex=True if you want to use DataFrame index as records' id
ep.toes(df, index, doctype=doctype, useindex=True, threadcount=2, chunksize=10000)
delete records from es
ep.toes(df.iloc[5000:], index, doctype=doctype, optype='delete', threadcount=2, chunk_size=10000)
Update doc by doc _id
df.iloc[:1000, 1] = 'Bye'
df.iloc[:1000, 2] = pd.datetime.now()
ep.toes(df.iloc[:1000, 1:], index, doctype=doctype, op_type='update')
Example of read data from es
df = ep.to_pandas(index)
print(df.head())
return certain fields in es
heads = ['Num', 'Date']
df = ep.to_pandas(index, heads=heads)
print(df.head())
set certain columns dtype
dtype = {'Num': 'float', 'Alpha': object}
df = ep.to_pandas(index, dtype=dtype)
print(df.dtypes)
infer dtype from es template
df = ep.topandas(index, inferdtype=True)
print(df.dtypes)
use query_sql parameter if you want to do query in sql
Example of write data to es with pandas.io.json
ep.toes(df, index, doctype=doctype, usepandasjson=True, threadcount=2, chunk_size=10000)
print('write es doc with pandas.io.json finished')