IndexedTables.jl

IndexedTables provides tabular data structures where some of the columns form a sorted index. It provides the backend to JuliaDB, but can be used on its own for efficient in-memory data processing and analytics.

Data Structures

IndexedTables offers two data structures: IndexedTable and NDSparse.

Both types store data in columns.
IndexedTable and NDSparse differ mainly in how data is accessed.
Both types have equal performance for Table operations (select, filter, etc.).

Quickstart

using Pkg
Pkg.add("IndexedTables")
using IndexedTables
t = table((x = 1:100, y = randn(100)))
select(t, :x)
filter(row -> row.y > 0, t)

`IndexedTable` vs. `NDSparse`

First let's create some data to work with.

using Dates
city = vcat(fill("New York", 3), fill("Boston", 3))
dates = repeat(Date(2016,7,6):Day(1):Date(2016,7,8), 2)
vals = [91, 89, 91, 95, 83, 76]

IndexedTable

(Optionally) Sorted by primary key(s), pkey.
Data is accessed as a Vector of NamedTuples.

using IndexedTables
julia> t1 = table((city = city, dates = dates, values = vals); pkey = [:city, :dates]) Table with 6 rows, 3 columns: city        dates       values ────────────────────────────── "Boston"    2016-07-06  95 "Boston"    2016-07-07  83 "Boston"    2016-07-08  76 "New York"  2016-07-06  91 "New York"  2016-07-07  89 "New York"  2016-07-08  91
julia> t1[1] (city = "Boston", dates = 2016-07-06, values = 95)

NDSparse

Sorted by index variables (first argument).
Data is accessed as an N-dimensional sparse array with arbitrary indexes.

julia> t2 = ndsparse((city=city, dates=dates), (value=vals,))
2-d NDSparse with 6 values (1 field named tuples):
city        dates      │ value
───────────────────────┼──────
"Boston"    2016-07-06 │ 95
"Boston"    2016-07-07 │ 83
"Boston"    2016-07-08 │ 76
"New York"  2016-07-06 │ 91
"New York"  2016-07-07 │ 89
"New York"  2016-07-08 │ 91
julia> t2["Boston", Date(2016, 7, 6)] (value = 95)