Solana Conformance Suite

This tool allows for validation of targets (e.g. Firedancer) against Solana Agave by running it against a series of predefined tests. It takes binary or human-readable Protobuf messages, as well as FlatBuffers fixtures, as inputs and runs them through the specified targets. It also includes functionality to validate targets for other issues, such as memory corruption.

Requirements

This tool works on RHEL or Ubuntu.

Build dependencies (for compiling vendored tools from source):

• cmake (3.x+)
• make
• g++ (or clang++)
• git (with submodule support)

Installation

Clone this repository and install dependencies:

# Clone with submodules
git clone --recurse-submodules <repo-url>
cd solana-conformance
Or if already cloned, initialize submodules:
git submodule update --init --recursive
For RHEL, run:
source install.sh
For Ubuntu, run:
source install_ubuntu.sh

The install scripts will:

• Install system packages (Python, cmake, etc.)
• Install vendored dependencies to opt/bin/ (flatc, buf)
• Create a Python virtual environment
• Install Python packages and generate bindings

For reproducibility, solana-conformance vendors all dependencies.

The deps.sh script manages vendored dependencies:

./deps.sh           # Install all dependencies to opt/bin/ ./deps.sh flatc     # Build only flatc ./deps.sh buf       # Download only buf ./deps.sh --status  # Check what's installed (including submodules) ./deps.sh --clean   # Remove built artifacts

Building flatc requires cmake, make, and g++. The buf binary is downloaded pre-built.

All built content lives in opt/ (gitignored). Source is in shlr/ (submodules).

Install auto-completion

solana-conformance --install-completion

You will need to reload your shell + the testsuiteenv venv to see the changes.

Currently Supported Harness Types

list-harness-types will provide the most updated list.

$ solana-conformance list-harness-types
Available harness types: 
ElfLoaderHarness
 
InstrHarness
 
SyscallHarness
 
VmInterpHarness
 
VmValidateHarness
 
TxnHarness
 
BlockHarness
 
TypeHarness

Protobuf

Each target must contain a function entrypoint that takes in a Context input message and outputs a Effects message (see proto/invoke.proto as an example).

Function Entrypoints:
ElfLoaderHarness: solcompatelfloaderv1
InstrHarness: solcompatinstrexecutev1
SyscallHarness: solcompatvmsyscallexecute_v1
VmInterpHarness: solcompatvminterpv1
VmValidateHarness: solcompatvmvalidatev1
TxnHarness: solcompattxnexecutev1
BlockHarness: solcompatblockexecutev1
TypeHarness: solcompattypeexecutev1

Updating definitions

All message definitions are defined in protosol. Anytime, protofuf definitions are updated in protosol, you will need to run the following command:

./fetchandgenerate.sh

Protosol Version

The protosol schemas are vendored as a git submodule in shlr/protosol. The version is pinned by the submodule commit.

To update to a different version:

cd shlr/protosol git checkout <tag-or-branch> cd ../.. ./fetchandgenerate.sh

FlatBuffers

In addition to Protobuf, this tool supports FlatBuffers fixtures (.fix files). FlatBuffers fixtures are automatically detected and converted to Protobuf format for processing.

Supported Formats

• Protobuf (.fix, .elfctx, .instrctx, etc.) - Standard format
• FlatBuffers (.fix) - Auto-detected and converted, used by honggfuzz/solfuzz fuzzing

Updating FlatBuffers Definitions

FlatBuffers schemas are defined in protosol. The fetchand_generate.sh script generates both Protobuf and FlatBuffers Python bindings:

./fetchandgenerate.sh

This generates both Protobuf and FlatBuffers bindings from the shlr/protosol submodule.

Using FlatBuffers Fixtures

FlatBuffers fixtures work transparently with all commands:

# Run tests on FlatBuffers fixtures
solana-conformance run-tests -i fixtures/ -s agave.so -t firedancer.so
Download fixtures (may be FlatBuffers or Protobuf)
solana-conformance download-fixtures -n solelfloader_diff
Validate fixtures and check their format
solana-conformance validate-fixtures -i fixtures/
solana-conformance validate-fixtures -i fixtures/ -v  # verbose

Output Format Control

When creating or regenerating fixtures, you can control the output format:

# Auto mode (default): upgrades to FlatBuffers when supported
ELF fixtures -> FlatBuffers, other fixtures -> Protobuf
solana-conformance create-fixtures -i inputs/ -s $SOLFUZZ_TARGET -o output/
Force Protobuf output
solana-conformance create-fixtures -i inputs/ -s $SOLFUZZ_TARGET -o output/ -F protobuf
Force FlatBuffers output (currently only for ELFLoaderHarness)
solana-conformance create-fixtures -i inputs/ -s $SOLFUZZ_TARGET -o output/ -F flatbuffers -h ElfLoaderHarness

Output format options (-F / --output-format):

• auto (default) - Upgrades to FlatBuffers when supported, otherwise Protobuf
• protobuf - Always output Protobuf format
• flatbuffers - Output FlatBuffers format (ELFLoaderHarness only)

Note: FlatBuffers output is currently only supported for ELFLoaderHarness (solcompatelfloaderv1/_v2). Other harness types (InstrHarness, SyscallHarness, TxnHarness, etc.) will automatically fall back to Protobuf output because FlatBuffers schemas don't exist for those fixture types in protosol.

Example output:

[OK] bug_3ec6cbcd.fix      Format: flatbuffers      Entrypoint: solcompatelfloaderv1      ELF size: 1702 bytes

Summary: 10 files checked   Valid:   10   Invalid: 0   Formats: 3 Protobuf, 7 FlatBuffers

Python API

from testsuite.flatbuffersutils import (
    FixtureLoader,
    detect_format,
    isflatbuffersoutput_supported,
)
from pathlib import Path
Unified loading (auto-detects format)
loader = FixtureLoader(Path('fixture.fix'))
print(f"Format: {loader.format_type}")  # 'flatbuffers' or 'protobuf'
print(f"Entrypoint: {loader.fn_entrypoint}")
print(f"ELF size: {len(loader.elf_data)} bytes")
Format detection only
with open('fixture.fix', 'rb') as f:
    data = f.read()
    
Quick heuristic detection
fmt = detect_format(data)  # Returns 'flatbuffers', 'protobuf', or 'unknown'
Validated detection (actually tries to parse)
fmt = detect_format(data, validate=True)  # More accurate but slower
Check if FlatBuffers output is supported for a harness
isflatbuffersoutputsupported("solcompatelfloader_v1")  # True
isflatbuffersoutputsupported("solcompatinstrexecute_v1")  # False

Downloading Fixtures and Crashes

Fixtures and crash inputs produced by the fuzzing infrastructure can be downloaded directly.

Setup

• Install dependencies:

pip install -e ".[octane]"

• GCS credentials (for downloading artifacts from Google Cloud Storage):

Credentials are auto-detected in this order: 1. GOOGLEAPPLICATIONCREDENTIALS environment variable 2. gcloud legacy credentials (~/.config/gcloud/legacy_credentials/<account>/adc.json) 3. GCE metadata service (when running on Google Compute Engine) For manual setup:

# Option 1: Use gcloud CLI (recommended)
   gcloud auth application-default login
   
   # Option 2: Set credentials path explicitly
   export GOOGLEAPPLICATIONCREDENTIALS=/path/to/credentials.json

Debugging workflow

# List available lineages and their repro counts
solana-conformance list-repros
Download fixtures (prefers .fix files)
solana-conformance download-fixtures -n solelfloader_diff -o fixtures/
Download crash inputs (prefers .fuzz files)
solana-conformance download-crashes -n solelfloader_diff -o crashes/
Download a single repro by hash
solana-conformance download-fixture <hash> -l solelfloader_diff -o output/
Debug a mismatch (downloads, creates fixtures, runs tests)
solana-conformance debug-mismatch <hash> -l solelfloader_diff \
    -s $SOLFUZZTARGET -t $FIREDANCERTARGET -o debug_output/
Debug all mismatches in a lineage
solana-conformance debug-mismatches -n solelfloader_diff -l 5 \
    -s $SOLFUZZTARGET -t $FIREDANCERTARGET -o debug_output/

GCS Authentication Troubleshooting

GCS credentials are auto-detected from gcloud legacy credentials, so most users won't need manual setup. If you still see authentication errors:

# Check if gcloud is authenticated
gcloud auth list
Check for legacy credentials (auto-detected)
ls ~/.config/gcloud/legacy_credentials/*/adc.json
Re-authenticate if needed
gcloud auth application-default login
Or set credentials explicitly
export GOOGLEAPPLICATIONCREDENTIALS=~/.config/gcloud/legacy_credentials/<account>/adc.json

Note: On GCE VMs without a service account attached, the metadata service won't work. The tool will automatically fall back to gcloud legacy credentials if available.

Setting up Environment

To setup the solana-conformance environment, run the following command and you will be all set:

source testsuiteenv/bin/activate

Environment Variables

Set these environment variables to avoid passing target paths on every command:

# Path to Agave/Solana reference target (.so file)
export SOLFUZZTARGET=/path/to/libsolfuzzagave.so
Path to Firedancer target (.so file)
export FIREDANCERTARGET=/path/to/libfdexecsolcompat.so

With these set, you can run commands without -s and -t:

# Instead of: solana-conformance run-tests -i fixtures/ -s /path/to/agave.so -t /path/to/fd.so

Just use:
solana-conformance run-tests -i fixtures/

Other useful environment variables:

• OCTANEAPIURL - Override the default API endpoint for downloading fixtures/crashes
• GOOGLEAPPLICATIONCREDENTIALS - Path to GCS credentials JSON file

Usage

Run the following to view all supported commands or refer to commands.md:

solana-conformance --help

Preferred Debugging

Use the following command instead if you want the ability to debug in GDB:

<gdb / rust-gdb> --args python3.11 -m testsuite.testsuite exec-instr --input <inputfile> --target <sharedlib>

Refer to exec-instr command for more information.

Recommended usage is opening two terminals side by side, and running the above command on the same output while changing the target parameter to the two desired targets and stepping through the debugger for each corresponding test case.

Uninstalling

source clean.sh

Running Integration Tests

To run the integration test suite:

# Ensure dependencies are installed
./deps.sh
Activate the virtual environment
source testsuiteenv/bin/activate
Run tests (requires target .so files)
./runintegrationtests.sh

Troubleshooting

Check Dependencies

Run the dependency checker to diagnose issues:

solana-conformance check-deps

Example output:

=== solana-conformance FlatBuffers Status ===

[OK] flatbuffers package: v24.3.25 [OK] numpy package: v1.26.4 (faster parsing) [OK] FlatBuffers bindings: generated
[OK] FlatBuffers support is ready!

Common Issues

"undefined symbol: _sanitizercov8bitcounters_init"

Cause: The shared library was built with sanitizer coverage instrumentation (-fsanitize-coverage) but is being loaded outside a fuzzer harness that would normally provide these symbols.

Fix: This is now handled automatically! The tool will:

• Build a sancov stub library (/tmp/solanaconformancesancovstub/libsancovstub.so)
• Set up LD_PRELOAD with the stub and ASAN libraries
• Configure ASAN_OPTIONS appropriately

If you still see issues, you can manually set up the environment:

# Locate ASAN library ldconfig -p | grep libasan

Set LD_PRELOAD manually
export LDPRELOAD=/lib64/libasan.so.8:/tmp/solanaconformancesancovstub/libsancov_stub.so
export ASANOPTIONS=detectleaks=0:verifyasanlink_order=0

"cannot allocate memory in static TLS block"

Cause: The shared library uses thread-local storage (TLS) and was loaded via dlopen() after program start, when the TLS block is already allocated.

Fix: The tool automatically adds target libraries to LD_PRELOAD to ensure TLS is allocated at startup. If you see this error, ensure you're using the latest version.

"FlatBuffers support not available"

Cause: The flatbuffers Python package is not installed.

Fix:

pip install flatbuffers>=24.0.0 
Or re-run the install script:
 source install.sh

"FlatBuffers bindings not generated"

Cause: The Python bindings generated from .fbs schemas don't exist.

Fix:

# First ensure flatc is installed ./deps.sh

Then generate bindings
./fetchandgenerate.sh

"flatc not found"

Cause: The vendored FlatBuffers compiler is not built. For reproducibility, solana-conformance only uses the vendored flatc built from shlr/flatbuffers/ and does not use system-installed versions.

Fix:

# Initialize submodules (if not done during clone) git submodule update --init --recursive

Build flatc from source
./deps.sh
Check status
./deps.sh --status

Note: Requires cmake, make, and g++ to build from source.

"buf not found"

Cause: The vendored buf tool is not installed. For reproducibility, solana-conformance only uses the vendored buf in opt/bin/ and does not use system-installed versions.

Fix:

# Install buf (downloads pre-built binary) ./deps.sh buf

Or install all dependencies
./deps.sh
Check status
./deps.sh --status

"Failed to parse fixture" with FlatBuffers file

Cause: The fixture file may be corrupted, empty, or in an unexpected format.

Debug:

from testsuite.flatbuffersutils import FixtureLoader from pathlib import Path

loader = FixtureLoader(Path('problematic.fix')) print(f"Valid: {loader.is_valid}") print(f"Format: {loader.format_type}") print(f"Error: {loader.error_message}")

Example error output:

Valid: False Format: unknown Error: Failed to convert FlatBuffers to Protobuf: problematic.fix   FlatBuffers conversion error: error: unpack_from requires a buffer of at least 24 bytes...

"Unable to parse fixture: Not recognized as Protobuf or FlatBuffers"

Cause: The file is neither valid Protobuf nor FlatBuffers. It may be:

• A raw .fuzz input (not a fixture)
• A corrupted download
• A different file format

Check file type:

file problematic.fix hexdump -C problematic.fix | head

Slow FlatBuffers parsing

Cause: Numpy is not installed.

Fix:

pip install numpy>=1.24.0

With numpy, large ELF data is parsed significantly faster.

After git pull, FlatBuffers fixtures don't work

Cause: Schema updates require regenerating bindings.

Fix:

./fetchandgenerate.sh

This fetches the latest protosol schemas and regenerates both Protobuf and FlatBuffers bindings.

"Cowardly refusing to install hooks with core.hooksPath set"

Cause: You have custom git hooks configured via core.hooksPath. This is just a warning during installation and doesn't affect functionality.

What it means: The pre-commit install step was skipped. This is fine - the pre-commit hooks are only used for development (formatting, linting) and are not required to use solana-conformance.

Fix (optional): If you want pre-commit hooks for development:

# Temporarily unset core.hooksPath git config --unset core.hooksPath pre-commit install 
Re-set your hooks if needed