Solana RPC

[!NOTE]

This repo contains steps to configure a slightly more performant RPC than a regular one

🤖 Server

Requirements

Solana Hardware Compatibility List can be found here

• CPU

• RAM

• Disk

Consider a larger ledger disk if longer transaction history is required

[!CAUTION]

Accounts and ledger should not be stored on the same disk

Sol User

Create a new Ubuntu user, named sol, for running the validator:

sudo adduser sol

It is a best practice to always run your validator as a non-root user, like the sol user we just created.

Ports Opening

Note: RPC port remains closed, only SSH and gossip ports are opened.

For new machines with UFW disabled:

• Add OpenSSH first to prevent lockout if you don't have password access
• Open required ports
• Close RPC port

sudo ufw enable
sudo ufw allow 22/tcp sudo ufw allow 20000/udp sudo ufw allow 8000:8100/tcp sudo ufw allow 8000:8100/udp
sudo ufw deny 8899/tcp sudo ufw deny 8899/udp
sudo ufw reload

Hard Drive Setup

On your Ubuntu computer make sure that you have at least 2TB of disk space mounted. You can check disk space using the df command:

df -h

To see the hard disk devices that you have available, use the list block devices command:

lsblk -f

Drive Formatting: Ledger

Assuming you have an nvme drive that is not formatted, you will have to format the drive and then mount it.

For example, if your computer has a device located at /dev/nvme0n1, then you can format the drive with the command:

sudo mkfs -t xfs /dev/nvme0n1

Next, check that you now have a UUID for that device:

lsblk -f

In the fourth column, next to your device name, you should see a string of letters and numbers that look like this: 6abd1aa5-8422-4b18-8058-11f821fd3967. That is the UUID for the device

Mounting Your Drive: Ledger

So far we have created a formatted drive, but you do not have access to it until you mount it. Make a directory for mounting your drive:

sudo mkdir -p /mnt/ledger

Next, change the ownership of the directory to your sol user:

sudo chown -R sol:sol /mnt/ledger

Now you can mount the drive:

sudo mount -o noatime -o logbufs=0 -o nofail  /dev/nvme0n1 /mnt/ledger

Formatting And Mounting Drive: AccountsDB

You will also want to mount the accounts db on a separate hard drive. The process will be similar to the ledger example above.

Assuming you have device at /dev/nvme1n1, format the device and verify it exists:

sudo mkfs -t xfs /dev/nvme1n1

Then verify the UUID for the device exists:

lsblk -f

Create a directory for mounting:

sudo mkdir -p /mnt/accounts

Change the ownership of that directory:

sudo chown -R sol:sol /mnt/accounts

And lastly, mount the drive:

sudo mount -o noatime -o logbufs=0 -o nofail   /dev/nvme1n1 /mnt/accounts

Modify fstab

sudo vi /etc/fstab

should contains something similar

# /etc/fstab: static file system information.
#
Use 'blkid' to print the universally unique identifier for a
device; this may be used with UUID= as a more robust way to name devices
that works even if disks are added and removed. See fstab(5).
#
<file system> <mount point>   <type>  <options>       <dump>  <pass>
...
UUID="03477038-6fa7-4de3-9ca6-4b0aef52bf42" /mnt/ledger xfs defaults,noatime,logbufs=8,largeio,inode64,allocsize=64m,nofail 0 0 UUID="68ff3738-f9f7-4423-a24c-68d989a2e496" /mnt/accounts xfs defaults,noatime,logbufs=8,largeio,inode64,allocsize=64m,nofail 0 0

To verify the mounts are active with correct options:

mount | grep -E 'ledger|accounts'
or
findmnt /mnt/ledger /mnt/accounts

🌵 Agave Client

Install rustc, cargo and rustfmt

curl https://sh.rustup.rs -sSf | sh
source $HOME/.cargo/env
rustup component add rustfmt

When building the master branch, please make sure you are using the latest stable rust version by running:

rustup update
rustup default nightly
rustup override set nightly

you may need to install libssl-dev, pkg-config, zlib1g-dev, protobuf etc.

sudo apt-get update
sudo apt-get install libssl-dev libudev-dev pkg-config zlib1g-dev llvm clang libclang-dev cmake make libprotobuf-dev protobuf-compiler lld

Download the source code

git clone https://github.com/anza-xyz/agave.git
cd agave
let's asume we would like to build v3.1.11 of validator
export TAG="v3.1.11" 
git switch tags/$TAG --detach

Build

export RUSTFLAGS="-Clink-arg=-fuse-ld=lld -Ctarget-cpu=native"
./scripts/cargo-install-all.sh  --release-with-lto --validator-only .
export PATH=$PWD/bin:$PATH

🦾 System Tuning

Set CPU Governor to Performance

echo "performance" | tee /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor

Enable CPU Boost

With the governor set to performance, ensure CPU boost (Turbo Boost / Precision Boost) is also enabled — on some systems it is a separate knob:

cat /sys/devices/system/cpu/cpufreq/boost
echo 1 | sudo tee /sys/devices/system/cpu/cpufreq/boost

[!NOTE]

The GRUB parameter amdpstate=passive allows boost to function on AMD CPUs. On Intel, boost is controlled via the intelpstate driver and is enabled by default in performance mode.

Disable THP (Transparent Huge Pages)

Linux transparent huge page (THP) support allows the kernel to automatically promote regular memory pages into huge pages. Huge pages reduces TLB pressure, but THP support introduces latency spikes when pages are promoted into huge pages and when memory compaction is triggered.

echo never > /sys/kernel/mm/transparent_hugepage/enabled

Disable KSM (Kernel Samepage Merging)

Linux kernel samepage merging (KSM) is a feature that de-duplicates memory pages that contains identical data. The merging process needs to lock the page tables and issue TLB shootdowns, leading to unpredictable memory access latencies. KSM only operates on memory pages that has been opted in to samepage merging using madvise(...MADV_MERGEABLE). If needed KSM can be disabled system wide by running the following command:

echo 0 > /sys/kernel/mm/ksm/run

Disable automatic NUMA memory balancing

Linux supports automatic page fault based NUMA memory balancing and manual page migration of memory between NUMA nodes. Migration of memory pages between NUMA nodes will cause TLB shootdowns and page faults for applications using the affected memory

echo 0 > /proc/sys/kernel/numa_balancing

NVMe I/O Scheduler

For NVMe devices the kernel may default to mq-deadline or bfq, both of which add queueing latency. Set the scheduler to none (passthrough) so I/O goes directly to the NVMe queue:

echo none | sudo tee /sys/block/nvme*/queue/scheduler

Disable read-ahead — RocksDB and accounts access patterns are largely random, so prefetching wastes bandwidth:

echo 0 | sudo tee /sys/block/nvme*/queue/readaheadkb

Persist both across reboots with a udev rule:

sudo bash -c 'cat > /etc/udev/rules.d/60-nvme-scheduler.rules <<EOF
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/scheduler}="none"
ACTION=="add|change", KERNEL=="nvme[0-9]*", ATTR{queue/readaheadkb}="0"
EOF'
sudo udevadm control --reload-rules && sudo udevadm trigger

Huge and Gigantic Pages

Agave's accounts-db uses memory-mapped files extensively. Huge pages (2 MB) and gigantic pages (1 GB) reduce TLB (Translation Lookaside Buffer) pressure, which improves performance for random access patterns across large memory regions like the accounts index.

sudo mkdir -p /mnt/hugepages
sudo mount -t hugetlbfs -o pagesize=2097152,min_size=228589568 none /mnt/hugepages
sudo mkdir -p /mnt/gigantic sudo mount -t hugetlbfs -o pagesize=1073741824,min_size=27917287424 none /mnt/gigantic
cat /proc/mounts | grep hugetlbfs 
none /mnt/hugepages hugetlbfs rw,seclabel,relatime,pagesize=2M,min_size=95124124 0 0
 
none /mnt/gigantic hugetlbfs rw,seclabel,relatime,pagesize=1024M,min_size=540092137472 0 0

To persist across reboots, add to /etc/fstab:

none /mnt/hugepages hugetlbfs pagesize=2M,min_size=218M 0 0
none /mnt/gigantic hugetlbfs pagesize=1G,min_size=26G 0 0

[!NOTE]

The kernel must have enough free contiguous memory to allocate huge pages. Allocate them early after boot for best results. Monitor availability with cat /proc/meminfo | grep -i huge.

Sysctl Tuning

sudo bash -c "cat >/etc/sysctl.d/21-agave-validator.conf <<EOF
TCP Buffer Sizes (10k min, 87.38k default, 12M max)
net.ipv4.tcp_rmem=10240 87380 12582912
net.ipv4.tcp_wmem=10240 87380 12582912
Increase UDP buffer sizes
net.core.rmem_default = 134217728
net.core.rmem_max = 134217728
net.core.wmem_default = 134217728
net.core.wmem_max = 134217728
Connection backlog (critical for RPC node)
net.core.somaxconn = 65535
net.ipv4.tcpmaxsyn_backlog = 65535
NIC receive queue (critical for QUIC/UDP)
net.core.netdevmaxbacklog = 250000
XDP ancillary buffer
net.core.optmem_max = 65536
TCP Optimization
net.ipv4.tcpcongestioncontrol=bbr
net.core.default_qdisc=fq
net.ipv4.tcp_fastopen=3
net.ipv4.tcp_timestamps=0
net.ipv4.tcp_sack=1
net.ipv4.tcptwreuse=1
net.ipv4.tcpnometrics_save=1
net.ipv4.tcpmoderatercvbuf=1
Kernel Optimization
kernel.timer_migration=0
kernel.hungtasktimeout_secs=30
kernel.pid_max=49152
kernel.nmi_watchdog=0
Virtual Memory Tuning
vm.swappiness=0
NOTE: With swappiness=0 and no swap partition, the OOM killer triggers early.
This is intentional — swapping accounts-db to disk would be worse than restarting.
vm.maxmapcount = 2000000
vm.stat_interval=10
vm.dirty_ratio=40
vm.dirtybackgroundratio=10
vm.minfreekbytes=3000000
vm.dirtyexpirecentisecs=36000
vm.dirtywritebackcentisecs=3000
vm.dirtytimeexpireseconds=43200
Increase number of allowed open file descriptors
fs.nr_open = 2000000
Increase Sync interval (default is 3000)
fs.xfs.xfssyncd_centisecs = 10000
EOF"

sudo sysctl -p /etc/sysctl.d/21-agave-validator.conf

Increase systemd and session file limits

Add

LimitNOFILE=2000000
LimitMEMLOCK=2000000000

DefaultLimitNOFILE=2000000
DefaultLimitMEMLOCK=2000000000

to the [Manager] section of /etc/systemd/system.conf.

sudo systemctl daemon-reload

sudo bash -c "cat >/etc/security/limits.d/90-solana-nofiles.conf <<EOF
Increase process file descriptor count limit
- nofile 2000000
Increase memory locked limit (kB)
- memlock 2000000
EOF"

NIC Tuning

Disable irqbalance

The GRUB irqaffinity= parameter pins hardware interrupts away from the PoH core, but irqbalance runs as a service and periodically overrides manual IRQ assignments. Disable it:

sudo systemctl disable --now irqbalance

Increase NIC ring buffers and reduce interrupt coalescing

Default NIC ring buffers are small and interrupt coalescing adds latency. Tune with ethtool (replace eth0 with your interface name):

ip link show

# For just the name of the primary interface (the one with the default route):
ip route get 1.1.1.1 | awk '{print $5; exit}'

# Check current and maximum ring buffer sizes
ethtool -g eth0
Increase RX/TX ring buffers to reduce packet drops under QUIC burst traffic
sudo ethtool -G eth0 rx 4096 tx 4096
Disable adaptive coalescing and minimize interrupt delay for lower latency
sudo ethtool -C eth0 adaptive-rx off adaptive-tx off rx-usecs 0 tx-usecs 0

To persist across reboots, add these commands to a systemd ExecStartPost or a /etc/networkd-dispatcher script.

NUMA Pinning

AMD Zen 3/4 CPUs have multiple CCDs (Core Complex Dies), each with its own L3 cache. Cross-CCD memory access adds latency. NUMA configuration depends on your topology:

# Check your NUMA topology
numactl --hardware

Single NUMA node (e.g., Ryzen 9 7950X or single-CCD EPYC): no NUMA pinning needed — skip this step.

2 NUMA nodes with non-uniform distances: pin to a single node to keep memory accesses local:

ExecStart=/usr/bin/numactl --cpunodebind=0 --membind=0 /home/sol/bin/validator.sh

4+ NUMA nodes with uniform distances (e.g., multi-socket or NPS4 EPYC where cross-node distance is equal): use interleaved memory to distribute evenly across all nodes. Single-node binding is too restrictive — the validator's thread count exceeds one node's core count, and --membind to one node limits available RAM.

ExecStart=/usr/bin/numactl --interleave=all /home/sol/bin/validator.sh

[!IMPORTANT]

If you isolate a PoH core (e.g., --experimental-poh-pinned-cpu-core 10), make sure it is reachable by your NUMA config. For example, if core 10 is on NUMA node 1, using --cpunodebind=0 would prevent the PoH thread from reaching it. Use --interleave=all or include the correct node in --cpunodebind.

Isolate one core for PoH

[!NOTE]

Many thanks for that guide to ax | 1000x.sh

find out the nearest available core. in most cases, it's core 2 (cores 0 and 1 are often used by the kernel). if you have more cores, you can choose another available nearest core.

Know your topology

lstopo

check your cores and hyperthreads look at the "cores" table to find your core and its hyperthread. for example, if you choose core 10, its hyperthread might be 34 (in my case)

lscpu --all -e

the easiest way to find the hyperthread for eg core 10:

cat /sys/devices/system/cpu/cpu10/topology/threadsiblingslist

Isolate the core and its hyperthread

in my case the hyperthread for core 10 is 34 /etc/default/grub (don't forget to run update-grub and reboot afterwards)

GRUBCMDLINELINUXDEFAULT="quiet amdpstate=passive nvmecore.defaultpsmaxlatencyus=0 nohzfull=10,34 isolcpus=domain,managed_irq,10,34 irqaffinity=0-9,11-33,35-47"

update-grub

[!NOTE]

- nohz_full=10,34 enables full dynamic ticks for core 10 and its hyperthread 34 to reducing overhead and latency.

- isolcpus=domain,managed_irq,10,34 isolates core 10 and hyperthread 34 from the general scheduler

- irqaffinity=0-9,11-33,35-47 directs interrupts away from core 10 and hyperthread 34

Set the poh thread to core 10

add the cli to your validator

...
--experimental-poh-pinned-cpu-core 10 \ 
...

there is a bug with coreaffinity if you isolate your cores: link

#!/bin/bash
main pid of agave-validator
agave_pid=$(pgrep -f "^agave-validator --identity")
if [ -z "$agave_pid" ]; then
    logger "setaffinity: agavevalidator_404"
    exit 1
fi
find thread id
threadpid=$(ps -T -p $agavepid -o spid,comm | grep 'solPohTickProd' | awk '{print $1}')
if [ -z "$thread_pid" ]; then
    logger "setaffinity: solPohTickProd404"
    exit 1
fi
currentaffinity=$(taskset -cp $threadpid 2>&1 | awk '{print $NF}') if [ "$current_affinity" == "10" ]; then     logger "setaffinity: solPohTickProdalready_set"     exit 1 else     # set poh to cpu 10     sudo taskset -cp 10 $thread_pid     logger "setaffinity: setdone"      # $thread_pid fi

sudo setcap capnetraw,capbpf,capnet_admin=p /home/sol/agave/bin/agave-validator

mkdir -p /home/sol/bin

vi /home/sol/bin/validator.sh

chmod +x /home/sol/bin/validator.sh

[Unit]
Description=Solana Validator
After=network.target
StartLimitIntervalSec=0
[Service] Type=simple Restart=always RestartSec=1 LimitNOFILE=2000000 LimitMEMLOCK=2000000000 LogRateLimitIntervalSec=0 User=sol Envir Environment=SOLANAMETRICSCONFIG=host=https://metrics.solana.com:8086,db=mainnet-beta,u=mainnet-beta_write,p=password ExecStart=/home/sol/bin/validator.sh ExecStop=/home/sol/agave/bin/agave-validator --ledger /mnt/ledger exit
[Install] WantedBy=multi-user.target

cat > logrotate.sol <<EOF
/home/sol/agave-validator.log {
  rotate 7
  daily
  missingok
  postrotate
    systemctl kill -s USR1 sol.service
  endscript
}
EOF
sudo cp logrotate.sol /etc/logrotate.d/sol systemctl restart logrotate.service

sudo systemctl enable --now sol

sudo systemctl status sol.service

agave-validator -l /mnt/ledger wait-for-restart-window --min-idle-time 2

ls -lh /mnt/ledger/snapshot-*.tar.zst

sudo systemctl restart sol

git clone https://github.com/jito-labs/shredstream-proxy.git --recurse-submodules
cd shredstream-proxy

export RUSTFLAGS="-Clink-arg=-fuse-ld=lld -Ctarget-cpu=native"
cargo build --release --bin jito-shredstream-proxy
mkdir ./bin cp target/release/jito-shredstream-proxy ./bin/jito-shredstream-proxy export PATH=$PWD/bin:$PATH

export LEDGERDIR=/mnt/ledger; bash -c "$(curl -fsSL https://raw.githubusercontent.com/jito-labs/shredstream-proxy/master/scripts/gettvu_port.sh)"

vi /home/sol/bin/shredstream.sh

chmod +x /home/sol/bin/shredstream.sh

[Unit]
Description=Shredstream Proxy
After=network.target
StartLimitIntervalSec=0
[Service] Type=simple Restart=always RestartSec=1 LogRateLimitIntervalSec=0 User=sol Envir Envir ExecStart=/home/sol/bin/shredstream.sh
[Install] WantedBy=multi-user.target

sudo systemctl enable --now shredstream

sudo systemctl status shredstream.service