GPandas

GPandas is a high-performance data manipulation and analysis library written in Go, drawing inspiration from Python's popular pandas library. It provides efficient and easy-to-use data structures, primarily the DataFrame, to handle structured data in Go applications.

Project Structure

• benchmark/: Contains benchmark scripts for performance evaluation against Python's pandas:
• dataframe/: Houses the core DataFrame implementation:
• plot/: Provides interactive chart generation capabilities using go-echarts v2:

• gpandas.go: Serves as the primary entry point for the GPandas library. It provides high-level API functions for DataFrame creation and data loading.
• gpandas_sql.go: Extends GPandas to interact with SQL databases and Google BigQuery:
• tests/: Contains unit tests to ensure the correctness and robustness of GPandas. It follows the exact same dir structure as the project for easy navigation.
• examples/: Contains example programs demonstrating GPandas features:

• utils/collection/: Contains generic collection utilities:

Code Functionality

GPandas is designed to provide a familiar and efficient way to work with tabular data in Go. Key functionalities include:

Core DataFrame Operations

• DataFrame Creation: Construct columnar DataFrames from in-memory data using gpandas.DataFrame(), or load from external sources like CSV files using gpandas.Read_csv(). Each DataFrame uses a map[string]*Series structure for efficient columnar access.
• Column Manipulation:

• Data Merging: Combine DataFrames based on common columns with DataFrame.Merge(), supporting:

• Data Export:

• Data Display:

Indexing and Selection

GPandas provides pandas-like indexing capabilities for intuitive data access:

• Column Selection
• Label-based Indexing (Loc)
• Position-based Indexing (iLoc)
• Index Management

Filtering and Selection by Condition

GPandas supports pandas-like boolean filtering for row subsetting. Filter and Where return a chainable, error-deferred FilterChain; terminate the chain with .Result() (or .MustResult()):

• Filter(): Keep rows where a column satisfies a comparison, e.g. df.Filter("Age", dataframe.GreaterThan, int64(25)).Result(). Supported operators: Equals, NotEquals, GreaterThan, GreaterThanOrEqual, LessThan, LessThanOrEqual. Numeric comparisons work across int/int64/float64, and null values never match a comparison.
• Chaining: Conditions can be combined fluently; the first error is carried through and surfaced by .Result():

result, err := df.
      Filter("Department", dataframe.Equals, "Engineering").
      Filter("Salary", dataframe.GreaterThan, 90000.0).
      Result()

• Where(): Keep rows for which a predicate returns true. The predicate receives a map[string]any of the row (nulls as nil), enabling arbitrary multi-column conditions:

result, err := df.Where(func(row map[string]any) bool {
      age, _ := row["Age"].(int64)
      return age > 25 && row["City"] == "NYC"
  }).Result()

Summary Statistics

GPandas provides exploratory data analysis helpers over numeric columns:

• Describe(): Returns a DataFrame of count, mean, std (sample, ddof=1), min, 25%, 50%, 75%, max per numeric column. Quantiles use linear interpolation and nulls are ignored.
• Column aggregations: Mean(), Sum(), Std(), Median(), Min(), Max() each return a map[string]float64 keyed by numeric column name.
• NullCount(): Returns a map[string]int of null counts per column.
• ValueCounts(column): Returns a DataFrame of unique values and their frequencies (descending), excluding nulls.

Transforming Columns

GPandas supports element-wise and row-wise transformations:

• Apply(column, fn): Transform each value of a column with fn func(any) any (nulls passed as nil). The result column type is inferred from the returned values; mixed integer and floating-point results are promoted to float64 (pandas-like).
• Map(column, mapping): Replace values in a column according to a map[any]any; unmapped values are kept unchanged.
• ApplyRow(fn): Transform whole rows with fn func(map[string]any) map[string]any, useful for deriving new columns. New keys are appended (sorted) after the existing columns.

Handling Missing Data

GPandas provides null-aware cleaning operations:

• FillNA(value): Replace nulls across all compatible columns with a constant. Incompatible columns are left unchanged.
• FillNAColumn(column, value): Fill nulls in a single column.
• FillNAMethod(method): Forward fill ("ffill") or backward fill ("bfill") nulls by propagation.
• DropNA(how, subset): Drop rows containing nulls. how is "any" (default) or "all"; subset limits the columns considered.
• IsNA() / NotNA(): Return boolean DataFrames indicating null / non-null cells.

Adding Columns

GPandas supports adding and inserting columns in place:

• Assign(name, series): Add a new column (or replace an existing one) from a Series.
• AssignFunc(name, fn): Add a column computed from each row with fn func(map[string]any) any; the type is inferred.
• Insert(loc, name, series): Insert a column at a specific position.

Unique Values and Deduplication

• Unique(column): Distinct values in first-appearance order (includes a single nil if the column has nulls).
• NUnique(column): Count of distinct non-null values.
• Duplicated(subset, keep): Boolean slice marking duplicate rows. keep is "first" (default), "last", or "none".
• DropDuplicates(subset, keep): Return a new DataFrame with duplicate rows removed.

Type Casting and Introspection

• AsType(column, targetType): Convert a column to FloatCol{}, IntCol{}, StringCol{}, or BoolCol{} (string aliases like "float64" also accepted). Nulls are preserved.
• DTypes(): Map of column name to data type name.
• Info(): Human-readable summary of rows, columns, non-null counts, and dtypes.

Aggregation and Window Functions

GPandas supports flexible aggregation and time-series style window operations:

• GroupBy(...).Agg(spec): Apply multiple aggregation functions per column at once, e.g. gb.Agg(map[string][]dataframe.AggFunc{"Salary": {dataframe.AggSum, dataframe.AggMean}}). Supported functions: AggSum, AggMean, AggCount, AggMin, AggMax, AggStd, AggMedian, AggFirst, AggLast. Result columns are named <column>_<func>.
• Rolling(window): Moving-window aggregations — .Mean(), .Sum(), .Min(), .Max(), .Std(). Positions without a full window of non-null values are null.
• Shift(periods): Shift values down (positive) or up (negative), filling vacated cells with null.
• CumSum() / CumMax() / CumMin() / CumProd(): Cumulative operations over numeric columns; nulls are skipped and preserved.

Reshaping with Stack, Unstack, and MultiIndex

• Stack(): Reshape wide → long, producing index/variable/value columns (null cells dropped).
• Unstack(): Inverse of Stack, reshaping the long format back to wide.
• SetMultiIndex(columns): Build a composite (flattened) index by joining the given columns' values.

String Methods

String columns expose a vectorized accessor via df.Str(column) (or series.Str() on a *StringSeries):

• Lower(), Upper(), Strip(), Title(), Replace(old, new) → *StringSeries
• Contains(substr), StartsWith(prefix), EndsWith(suffix) → *BoolSeries
• Len() → *Int64Series; Split(sep) → [][]string

See examples/advanced/ for a complete working example of aggregation, window functions, reshaping, and string methods.

Statistics, Sampling, and Chaining

• Corr() / Cov(): Pairwise Pearson correlation and sample covariance matrices over numeric columns (returned as a square DataFrame indexed by column name).
• Sample(n, seed...): Randomly select n rows without replacement; an optional seed makes the selection deterministic.
• Pipe(fn): Apply a custom func(DataFrame) (DataFrame, error) for fluent method chaining.

DateTime and Categorical Types

• ToDatetime(column, layout): Parse a string column into a datetime column (auto-detects common layouts when layout is empty). Then df.Dt(column) exposes Year(), Month(), Day(), Hour(), Weekday(), Date(), and more.
• AsCategorical(column): Convert a column to a memory-efficient categorical type backed by integer codes; Categories(column) lists the distinct categories.

Multi-key Merge

• MergeOn(other, on, how): Join two DataFrames on multiple key columns (inner, left, right, full), generalizing Merge.

Additional Visualizations

• PlotScatter(xCol, yCol, opts): Scatter chart from two numeric columns.
• PlotHistogram(column, bins, opts): Histogram of a numeric column.
• PlotHeatmap(opts): Heatmap of numeric columns (pairs well with Corr()).

Data Loading from External Sources

• CSV Reading: Efficiently read CSV files into DataFrames with gpandas.Read_csv(), leveraging concurrent processing for performance.
• JSON I/O: Read records-oriented JSON with gpandas.Read_json() and export with DataFrame.ToJSON().
• Excel I/O: Read .xlsx files with gpandas.Readexcel() and export with DataFrame.ToExcel() (powered by excelize).
• Parquet I/O: Read .parquet files with gpandas.Readparquet() and export with DataFrame.ToParquet() (powered by parquet-go). Note: columns are written as non-nullable, so nulls are stored as zero values.
• SQL Database Integration:

import (     "github.com/apoplexi24/gpandas/dataframe"     "github.com/apoplexi24/gpandas/plot"     "github.com/apoplexi24/gpandas/utils/collection" )

// Create DataFrame categories, _ := collection.NewStringSeriesFromData([]string{"A", "B", "C"}, nil) values, _ := collection.NewFloat64SeriesFromData([]float64{10.0, 20.0, 30.0}, nil)
df := &dataframe.DataFrame{     Columns: map[string]collection.Series{         "category": categories,         "value":    values,     },     ColumnOrder: []string{"category", "value"},     Index:       []string{"0", "1", "2"}, }
// Generate bar chart opts := &plot.ChartOptions{     Title:      "Sample Bar Chart",     Width:      900,     Height:     500,     OutputPath: "output/chart.html", } df.PlotBar("category", "value", opts)

go get github.com/apoplexi24/gpandas

git clone https://github.com/apoplexi24/gpandas.git
   cd gpandas

go mod download

opts := &plot.ChartOptions{       OutputPath: "output/chart.html",  // Required   }

os.MkdirAll("output", 0755)