Node exporter

[][quay] [][hub] [][goreportcard]

Prometheus exporter for hardware and OS metrics exposed by \*NIX kernels, written in Go with pluggable metric collectors.

The Windows exporter is recommended for Windows users. To expose NVIDIA GPU metrics, prometheus-dcgm can be used.

Installation and Usage

If you are new to Prometheus and nodeexporter there is a simple step-by-step guide.

docker run -d \
  --net="host" \
  --pid="host" \
  -v "/:/host:ro,rslave" \
  quay.io/prometheus/node-exporter:latest \
  --path.rootfs=/host

---
version: '3.8'
services:   node_exporter:     image: quay.io/prometheus/node-exporter:latest     containername: nodeexporter     command:       - '--path.rootfs=/host'     network_mode: host     pid: host     restart: unless-stopped     volumes:       - '/:/host:ro,rslave'

--collector.filesystem.mount-points-exclude=^/(dev|proc|sys|var/lib/docker/.+|var/lib/kubelet/.+)($|/)

Name | Description | OS ---------|-------------|---- arp | Exposes ARP statistics from /proc/net/arp. | Linux bcache | Exposes bcache statistics from /sys/fs/bcache/. | Linux bonding | Exposes the number of configured and active slaves of Linux bonding interfaces. | Linux btrfs | Exposes btrfs statistics | Linux boottime | Exposes system boot time derived from the kern.boottime sysctl. | Darwin, Dragonfly, FreeBSD, NetBSD, OpenBSD, Solaris conntrack | Shows conntrack statistics (does nothing if no /proc/sys/net/netfilter/ present). | Linux cpu | Exposes CPU statistics | Darwin, Dragonfly, FreeBSD, Linux, Solaris, OpenBSD cpufreq | Exposes CPU frequency statistics | Linux, Solaris diskstats | Exposes disk I/O statistics. | Darwin, Linux, OpenBSD dmi | Expose Desktop Management Interface (DMI) info from /sys/class/dmi/id/ | Linux dmmultipath | Exposes DM-multipath device and path metrics from /sys/block/dm-*. | Linux edac | Exposes error detection and correction statistics. | Linux entropy | Exposes available entropy. | Linux exec | Exposes execution statistics. | Dragonfly, FreeBSD fibrechannel | Exposes fibre channel information and statistics from /sys/class/fc_host/. | Linux filefd | Exposes file descriptor statistics from /proc/sys/fs/file-nr. | Linux filesystem | Exposes filesystem statistics, such as disk space used. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD hwmon | Expose hardware monitoring and sensor data from /sys/class/hwmon/. | Linux infiniband | Exposes network statistics specific to InfiniBand and Intel OmniPath configurations. | Linux ipvs | Exposes IPVS status from /proc/net/ipvs and stats from /proc/net/ipvs_stats. | Linux kernelhung | Exposes number of tasks that have been detected as hung from /proc/sys/kernel/hungtaskdetectcount. | Linux loadavg | Exposes load average. | Darwin, Dragonfly, FreeBSD, Linux, NetBSD, OpenBSD, Solaris mdadm | Exposes statistics about devices in /proc/mdstat (does nothing if no /proc/mdstat present). | Linux meminfo | Exposes memory statistics. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD netclass | Exposes network interface info from /sys/class/net/ | Linux netdev | Exposes network interface statistics such as bytes transferred. | Darwin, Dragonfly, FreeBSD, Linux, OpenBSD netisr | Exposes netisr statistics | FreeBSD netstat | Exposes network statistics from /proc/net/netstat. This is the same information as netstat -s. | Linux nfs | Exposes NFS client statistics from /proc/net/rpc/nfs. This is the same information as nfsstat -c. | Linux nfsd | Exposes NFS kernel server statistics from /proc/net/rpc/nfsd. This is the same information as nfsstat -s. | Linux nvme | Exposes NVMe info from /sys/class/nvme/ | Linux os | Expose OS release info from /etc/os-release or /usr/lib/os-release | any powersupplyclass | Exposes Power Supply statistics from /sys/class/power_supply | Linux pressure | Exposes pressure stall statistics from /proc/pressure/. | Linux (kernel 4.20+ and/or CONFIG\PSI) rapl | Exposes various statistics from /sys/class/powercap. | Linux schedstat | Exposes task scheduler statistics from /proc/schedstat. | Linux selinux | Exposes SELinux statistics. | Linux sockstat | Exposes various statistics from /proc/net/sockstat. | Linux softnet | Exposes statistics from /proc/net/softnet_stat. | Linux stat | Exposes various statistics from /proc/stat. This includes boot time, forks and interrupts. | Linux tapestats | Exposes statistics from /sys/class/scsi_tape. | Linux textfile | Exposes statistics read from local disk. The --collector.textfile.directory flag must be set. | any thermal | Exposes thermal statistics like pmset -g therm. | Darwin thermal\_zone | Exposes thermal zone & cooling device statistics from /sys/class/thermal. | Linux time | Exposes the current system time. | any timex | Exposes selected adjtimex(2) system call stats. | Linux udpqueues | Exposes UDP total lengths of the rxqueue and tx_queue from /proc/net/udp and /proc/net/udp6. | Linux uname | Exposes system information as provided by the uname system call. | Darwin, FreeBSD, Linux, OpenBSD vmstat | Exposes statistics from /proc/vmstat. | Linux watchdog | Exposes statistics from /sys/class/watchdog | Linux xfs | Exposes XFS runtime statistics. | Linux (kernel 4.4+) zfs | Exposes ZFS performance statistics. | FreeBSD, Linux, Solaris

Disabled by default

node_exporter also implements a number of collectors that are disabled by default. Reasons for this vary by collector, and may include:

• High cardinality
• Prolonged runtime that exceeds the Prometheus scrapeinterval or scrapetimeout
• Significant resource demands on the host

Name | Description | OS ---------|-------------|---- buddyinfo | Exposes statistics of memory fragments as reported by /proc/buddyinfo. | Linux cgroups | A summary of the number of active and enabled cgroups | Linux cpu\_vulnerabilities | Exposes CPU vulnerability information from sysfs. | Linux devstat | Exposes device statistics | Dragonfly, FreeBSD drm | Expose GPU metrics using sysfs / DRM, amdgpu is the only driver which exposes this information through DRM | Linux drbd | Exposes Distributed Replicated Block Device statistics (to version 8.4) | Linux ethtool | Exposes network interface information and network driver statistics equivalent to ethtool, ethtool -S, and ethtool -i. | Linux interrupts | Exposes detailed interrupts statistics. | Linux, OpenBSD ksmd | Exposes kernel and system statistics from /sys/kernel/mm/ksm. | Linux lnstat | Exposes stats from /proc/net/stat/. | Linux logind | Exposes session counts from logind. | Linux meminfo\_numa | Exposes memory statistics from /sys/devices/system/node/node[0-9]/meminfo, /sys/devices/system/node/node[0-9]/numastat. | Linux mountstats | Exposes filesystem statistics from /proc/self/mountstats. Exposes detailed NFS client statistics. | Linux network_route | Exposes the routing table as metrics | Linux nvmesubsystem | Exposes NVMe over Fabrics subsystem path health metrics from /sys/class/nvme-subsystem/. | Linux pcidevice | Exposes pci devices' information including their link status and parent devices. | Linux perf | Exposes perf based metrics (Warning: Metrics are dependent on kernel configuration and settings). | Linux processes | Exposes aggregate process statistics from /proc. | Linux qdisc | Exposes queuing discipline statistics | Linux slabinfo | Exposes slab statistics from /proc/slabinfo. Note that permission of /proc/slabinfo is usually 0400, so set it appropriately. | Linux softirqs | Exposes detailed softirq statistics from /proc/softirqs. | Linux sysctl | Expose sysctl values from /proc/sys. Use --collector.sysctl.include(-info) to configure. | Linux swap | Expose swap information from /proc/swaps. | Linux systemd | Exposes service and system status from systemd. | Linux tcpstat | Exposes TCP connection status information from /proc/net/tcp and /proc/net/tcp6. (Warning: the current version has potential performance issues in high load situations.) | Linux wifi | Exposes WiFi device and station statistics. | Linux xfrm | Exposes statistics from /proc/net/xfrm_stat | Linux zoneinfo | Exposes NUMA memory zone metrics. | Linux

Deprecated

These collectors are deprecated and will be removed in the next major release.

Name | Description | OS ---------|-------------|---- ntp | Exposes local NTP daemon health to check time | any runit | Exposes service status from runit. | any_ supervisord | Exposes service status from supervisord. | any_

Perf Collector

The perf collector may not work out of the box on some Linux systems due to kernel configuration and security settings. To allow access, set the following sysctl parameter:

sysctl -w kernel.perfeventparanoid=X

• 2 allow only user-space measurements (default since Linux 4.6).
• 1 allow both kernel and user measurements (default before Linux 4.6).
• 0 allow access to CPU-specific data but not raw tracepoint samples.
• -1 no restrictions.

By default, the perf collector will only collect metrics of the CPUs that node_exporter is running on (ie runtime.NumCPU. If this is insufficient (e.g. if you run node_exporter with its CPU affinity set to specific CPUs), you can specify a list of alternate CPUs by using the --collector.perf.cpus flag. For example, to collect metrics on CPUs 2-6, you would specify: --collector.perf --collector.perf.cpus=2-6. The CPU configuration is zero indexed and can also take a stride value; e.g. --collector.perf --collector.perf.cpus=1-10:5 would collect on CPUs 1, 5, and 10.

The perf collector is also able to collect tracepoint counts when using the --collector.perf.tracepoint flag. Tracepoints can be found using perf list or from debugfs. And example usage of this would be --collector.perf.tracepoint="sched:schedprocessexec".

Sysctl Collector

The sysctl collector can be enabled with --collector.sysctl. It supports exposing numeric sysctl values as metrics using the --collector.sysctl.include flag and string values as info metrics by using the --collector.sysctl.include-info flag. The flags can be repeated. For sysctl with multiple numeric values, an optional mapping can be given to expose each value as its own metric. Otherwise an index label is used to identify the different fields.

Examples

Numeric values

Single values

Multiple values

net.ipv4.tcp_rmem = 4096	131072	6291456

nodesysctlnetipv4tcp_rmem{index="0"} 4096
nodesysctlnetipv4tcp_rmem{index="1"} 131072
nodesysctlnetipv4tcp_rmem{index="2"} 6291456

nodesysctlnetipv4tcprmemmin 4096
nodesysctlnetipv4tcprmemdefault 131072
nodesysctlnetipv4tcprmemmax 6291456

String values

Single values

Multiple values

kernel.seccomp.actionsavail = killprocess kill_thread trap errno trace log allow

nodesysctlinfo{key="kernel.seccomp.actionsavail", index="0", value="killprocess"} 1
nodesysctlinfo{key="kernel.seccomp.actionsavail", index="1", value="killthread"} 1
...

Textfile Collector

The textfile collector is similar to the Pushgateway, in that it allows exporting of statistics from batch jobs. It can also be used to export static metrics, such as what role a machine has. The Pushgateway should be used for service-level metrics. The textfile module is for metrics that are tied to a machine.

To use it, set the --collector.textfile.directory flag on the node_exporter commandline. The collector will parse all files in that directory matching the glob *.prom using the text format. Note: Timestamps are not supported.

To atomically push completion time for a cron job:

echo mybatchjobcompletiontime $(date +%s) > /path/to/directory/mybatchjob.prom.$$ mv /path/to/directory/mybatchjob.prom.$$ /path/to/directory/mybatchjob.prom

To statically set roles for a machine using labels:

echo 'role{role="application_server"} 1' > /path/to/directory/role.prom.$$ mv /path/to/directory/role.prom.$$ /path/to/directory/role.prom

Filtering enabled collectors

The node_exporter will expose all metrics from enabled collectors by default. This is the recommended way to collect metrics to avoid errors when comparing metrics of different families.

For advanced use the nodeexporter can be passed an optional list of collectors to filter metrics. The parameters collect[] and exclude[] can be used multiple times (but cannot be combined). In Prometheus configuration you can use this syntax under the scrape config.

params:     collect[]:       - cpu       - meminfo

params:     exclude[]:       - netdev

./node_exporter --web.config.file=web-config.yml