Summary

🔸 Unbound dashboard in Grafana 🔸 Prometheus time series database 🔸 Unbound metrics exporter in Go 🔸 Log aggregation with Loki 🔸 Unbound setup is available at unbound-redis 🔸 Refer to info.md for dashboard details and release.md for changes

Specs:

|Grafana |Prometheus|Loki |Go |OS |HW |

|:-------|:---------|:------|:-------|:----------------------------|:-----------------------|

|11.1.0|2.42.0 |3.1.0|1.21.5|raspios-bookworm-arm64-lite|Raspberry Pi 4 Model B|

#

Steps

  🔸 Grafana ➜ Prometheus ➜ Unbound Exporter ➜ Loki ➜ Import Dashboard

❯ Grafana

• Download:

There are 2 versions OSS and Enterprise. OSS version is more than enough. Enterprise version installs too many extra packages (like unattended-upgrades and more). Below cmd downloads Grafana OSS for arm64. > wget https://dl.grafana.com/oss/release/grafana11.1.0arm64.deb

• Install:

> sudo apt install musl > ℹ️ Note: > musl package is required since Grafana 10.2.3

> sudo dpkg -i grafana11.1.0arm64.deb

> ℹ️ Note: > A tweaked grafana.ini is available in the release. It reduces memory footprint, removes usage collection, stops calls to grafana server/repo and has few more optimizations. You can use grafana.ini either from the release or the default one. Default config is located at /etc/grafana/grafana.ini

• Start:

> sudo systemctl daemon-reload > sudo systemctl enable grafana-server > sudo systemctl start grafana-server

• UI:

> http://<RP-IP>:3000/ > Default user/pass ➟ admin/admin

#

❯ Prometheus

• Install:

> sudo apt install prometheus

• Config:

Enable unbound-exporter scraping in prometheus. A trimmed down prometheus config, prometheus.yml is available in the release. Take a backup of existing prometheus.yml, if you are interested in the default config. Copy prometheus.yml from the release to /etc/prometheus/ dir.

> ℹ️ Note: > Provided prometheus.yml has only unbound-exporter metric collection enabled. Default metric collection for node and prometheus exporters are removed. Scraping interval is set to 5m.

• Remove Node Exporter:

Node exporter exports machine metrics. It is installed as part of prometheus pkg and runs as systemd service. It is not needed for unbound-dashboard. Unless you are already using it, remove node exporter. Below cmd will remove 8 node-exporter related pkgs. > Remove: > sudo apt --purge autoremove prometheus-node-exporter > Disable scrape config: > Provided prometheus.yml has node exporter scrapping config removed.

• UI:

> http://<RP-IP>:9090/

#

❯ Unbound Exporter

• I wrote my own exporter in Go. It is more efficient and tailored for this dashboard. A prebuilt binary (for arm64) is available in the release. Source code is available at unbound-exporter.

• Config:

Modify Unbound config. Edit /etc/unbound/unbound.conf * Enable extended stats. Add option under server: tag > extended-statistics: yes

* Enable Unix domain socket for collecting stats. It is faster than default TCP. Add below options under remote-control: tag > control-interface: "/var/run/unbound.sock" > control-use-cert: no

• Install:

Copy unbound-exporter binary from the release to /usr/local/bin/ dir. Make sure it is under ownership of root and executable. > Change ownership [If needed]: > sudo chown root:root /usr/local/bin/unbound-exporter > Make it executable [If needed]: > sudo chmod +x /usr/local/bin/unbound-exporter

• Service:

Create service to automatically run unbound-exporter at startup. Copy prometheus-unbound-exporter.service from the release to /etc/systemd/system/ dir. Make sure it is under the ownership of root. Enable and start the service. > Change ownership [If needed]: > sudo chown root:root /etc/systemd/system/prometheus-unbound-exporter.service > Enable: > sudo systemctl enable prometheus-unbound-exporter > Start: > sudo systemctl start prometheus-unbound-exporter > Status: > sudo systemctl status prometheus-unbound-exporter

> ℹ️ Note: > Provided prometheus-unbound-exporter.service passes 2 paramaters. Blocklist file path and Unbound unix domain socket URI. Change them accordingly if you are using different path and name.

• Usage:

> unbound-exporter -h > >

#

❯ Loki

• Download:

Download Loki and Promtail > curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/loki3.1.0arm64.deb" > curl -O -L "https://github.com/grafana/loki/releases/download/v3.1.0/promtail3.1.0arm64.deb"

• Install:

> sudo dpkg -i loki3.1.0arm64.deb > sudo dpkg -i promtail3.1.0arm64.deb

• Logging:

Enable Unbound logging. * Edit /etc/unbound/unbound.conf. Add/Modify below options under server: tag > log-replies: yes > log-tag-queryreply: yes > log-local-actions: yes > logfile: /var/log/unbound/unbound.log

> ℹ️ Note: > Make sure verbosity: is set to 0.

* Create log dir > sudo mkdir /var/log/unbound > sudo chown unbound:unbound /var/log/unbound

* Enable log rotation > Copy file unbound from the release under logrotate dir to /etc/logrotate.d/ dir. Make sure it is under the ownership of root.

• Config:

Replace /etc/loki/config.yml and /etc/promtail/config.yml with the config.yml files in the release under loki and promtail dirs respectively. Both should be under the ownership of root.

> ℹ️ Note: > Provided loki config.yml is optimized to process large data set and metrics calculation for Unbound logs. It is tweaked after some thorough testing in Unbound logs specific dashboard with 6 panels. Default loki config can hardly handle 2 panels with small data set, it throws errors and timeouts with large data set and multiple requests. > Provided promtail config.yml enables Unbound logs scrapping.

• Restart:

> sudo systemctl restart loki > sudo systemctl restart promtail

#

❯ Import Dashboard

• Open Grafana UI ➟ http://<RP-IP>:3000/

* Click Data Sources under Configuration. Click Add data source select Prometheus. Add below options: > Name ➟ Prometheus > Default ➟ On > Add URL ➟ http://localhost:9090 > Add Scrape interval ➟ 5m > Hit ➟ Save & test

* Click Data Sources under Configuration. Click Add data source select Loki. Add below options: > Name ➟ Loki > Add URL ➟ http://localhost:3100 > Add Maximum lines ➟ 100000 > Hit ➟ Save & test

* Dashboard, unbound-dashboard.json is available in the release. Click Import under Dashboards. Click Upload JSON file. Select unbound-dashboard.json. Add below options: > Folder ➟ Dashboards > Select Prometheus ➟ Data Source > Select Loki ➟ Data Source > Hit ➟ Import

#

❯ ℹ️ Tips & Notes

• Grafana:

How to ➟ Change grafana landing page to unbound dashboard & Switch between Dark (default) and Light theme.

> Open Grafana UI ➟ http://<RP-IP>:3000/ > Click Profile under top right icon > Under Preferences select General/Unbound in Home Dashboard drop down > Change theme in Interface theme drop down > Hit Save

There is an additional panel in the dashboard at the top right, not visible in the preview. It shows unbound-exporter status and may be beneficial. If you are not interested in that simply remove it. Screenshot below: >

• Prometheus:

How to ➟ Remove time series (metrics) collected by prometheus instantly for fresh start & Reduce prometheus journal logging. > Enable admin API: > sudo nano /etc/default/prometheus > Add at the top: > ARGS="--web.enable-admin-api --log.level=warn" > Save & Exit

> To delete all metrics of specific exporter add job_name as last argument in delete cmd: > Delete: > curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/delete_series?match[]={job="node"}' > Purge: > curl -X POST -g 'http://localhost:9090/api/v1/admin/tsdb/clean_tombstones' > Restart: > sudo systemctl restart prometheus