hipDF - GPU DataFrames on AMD GPUs

[!NOTE]

This README is derived from the original RAPIDSAI project's README. More care is necessary to remove/modify parts that are only applicable to the original version.

[!NOTE]

This ROCm™ port is derived work based on the NVIDIA RAPIDS® cuDF project (version 25.02). It aims to

follow the latter's directory structure and API naming as closely as possible to minimize porting friction for users that are interested in using both projects.

Resources

RAPIDS resources:

cudf (Python) documentation
libcudf (C++/CUDA) documentation
RAPIDS Community: Get help, contribute, and collaborate.

Overview

Built based on the Apache Arrow columnar memory format, hipDF is a GPU DataFrame library for loading, joining, aggregating, filtering, and otherwise manipulating data.

hipDF provides a pandas-like API that will be familiar to data engineers & data scientists, so they can use it to easily accelerate their workflows without going into the details of HIP programming.

For example, the following snippet downloads a CSV, then uses the GPU to parse it into rows and columns and run calculations:

import cudf
tipsdf = cudf.readcsv("https://github.com/plotly/datasets/raw/master/tips.csv") tipsdf["tippercentage"] = tipsdf["tip"] / tipsdf["total_bill"] * 100
display average tip by dining party size
print(tipsdf.groupby("size").tippercentage.mean())

Or, you can use hipDF as a no-code-change accelerator for pandas, using cudf.pandas. cudf.pandas supports 100% of the pandas API, utilizing hipDF for supported operations and falling back to pandas when needed:

%load_ext cudf.pandas  # pandas operations now use the GPU!
import pandas as pd
tipsdf = pd.readcsv("https://github.com/plotly/datasets/raw/master/tips.csv") tipsdf["tippercentage"] = tipsdf["tip"] / tipsdf["total_bill"] * 100
display average tip by dining party size
print(tipsdf.groupby("size").tippercentage.mean())

Quick Start

[!NOTE]

Currently, a docker image is not available for AMD GPUs.

Installation

[!CAUTION]

Incompatibility notice: Mixing RAPIDS and ROCmDS packages/installations is not supported. To avoid conflicts, strictly separate and isolated environments must be maintained if it is required to install both RAPIDS and ROCm-DS packages on the same system.

[!NOTE]

We support only AMD GPUs. Use the RAPIDS package for NVIDIA GPUs.

ROCm/GPU Requirements

ROCm HIP SDK compilers version 7.0.2
Build requirements: rocthrust-dev, rocm-llvm-dev, hipcub (Ubuntu)
Runtime requirements: rocm-llvm-dev (Ubuntu)
Officially supported architecture (gfx90a, gfx942).
Ubuntu 22.04+
Python 3.10 or later

Install from AMD PyPI

See install instructions.

Build/Install from Source

See build instructions.

Open GPU Data Science

The ROCm-DS suite of open source software libraries aims to enable execution of end-to-end data science and analytics pipelines entirely on AMD GPUs. It relies on ROCm HIP primitives for low-level compute optimization, but exposing that GPU parallelism and high-bandwidth memory speed through user-friendly Python interfaces.

Apache Arrow on GPU

The GPU version of Apache Arrow is a common API that enables efficient interchange of tabular data between processes running on the GPU. End-to-end computation on the GPU avoids unnecessary copying and converting of data off the GPU, reducing compute time and cost for high-performance analytics common in artificial intelligence workloads. As the name implies, hipDF uses the Apache Arrow columnar data format on the GPU. Currently, a subset of the features in Apache Arrow are supported.

🔗 More in this category