# basic-benchmark A self-contained kit for benchmarking a **freshly installed Fedora 44** machine (and comparing it against another build or an earlier baseline), using the *same* tests on both sides, with continuous power/thermal logging so you get efficiency comparisons — not just scores. The kit makes **no assumptions about the host**: no baked-in CPU/GPU/user/host names. Specs are *captured* on whatever machine runs it (`./capture-specs.sh`). It targets a stock Fedora 44 with internet access, and can provision missing tools itself via `dnf`. The suite is split into two parts so it can be driven from another device: | Part | Needs a display? | Where to run it | What it covers | |------|------------------|-----------------|----------------| | **headless** | no | over SSH from another device | CPU, memory, storage, thermals, power logging | | **headed** | yes | on the desktop, in the KDE session | GPU (glmark2, vkmark) | Both parts append their part name to the result tag, so they never overwrite each other's CSVs. > **PassMark PerformanceTest** is the primary cross-platform score. The suite is > self-contained: no cloud uploads, no license keys. --- ## 1. Workflow ``` capture specs headless (from any device) headed (at the machine) ───────────── ────────────────────────── ─────────────────────── ./capture-specs.sh before ssh -t \ 'cd basic-benchmark && \ ./run-benchmarks.sh before headless' -> results/power-before-headless.csv results/phases-before-headless.csv ./run-benchmarks.sh before headed -> results/power-before-headed.csv ───────────────── change hardware / reinstall ───────────────── repeat all with "after", then: power-report.py --tags \ before-headless before-headed \ after-headless after-headed ``` You can run the parts in any order; run both on the same machine back to back. --- ## 2. Files | File | What it is | |------|------------| | `README.md` | This guide. | | `specs.txt` | Pointer to the spec capture (no hardcoded host). | | `specs-detailed.txt` | Pointer to the spec capture (no hardcoded host). | | `capture-specs.sh` | Captures a portable spec sheet for the *current* host. | | `benchmarks.txt` | The suite definition: every test, commands, ground rules, results table. | | `run-benchmarks.sh` | Runs the suite (`headless`/`headed`/`all`), starts/stops power logging, drops phase markers. | | `powerlog.py` | Background sampler (~1 Hz) for power, temps, clocks, GPU, fans. | | `power-report.py` | Turns the CSVs into PNG graphs, per-phase tables and efficiency numbers. | | `compare-report.py` | Cross-device / before-after results table + one bar chart per benchmark (`results/graphs/compare-*.png`). | | `results/` | Created at runtime: CSVs, logs, `results/graphs/*.png`. | | `tools/` | Bundled binaries: PassMark (`tools/pt/`). Ships with the kit — no download needed. | | `tools/corpus/` | Fixed sample inputs for the app workloads (LibreOffice docs, a GEGL/GIMP photo, Inkscape SVGs), fetched once and checksum-pinned. | --- ## 3. Prerequisites Target: a **stock Fedora 44 with internet access** (x86-64 for the bundled PassMark binary). Nothing host-specific is required — no fixed user, hostname, or hardware. - The **headless** part runs from any terminal (local or SSH); the **headed** part must run inside a graphical session (Wayland or X11) because GPU tests need a live display. - `sudo` access for your user (no passwordless sudo assumed). Power logging, the CPU governor and `turbostat` need root. Over SSH, allocate a TTY (`ssh -t ...`) so `sudo` can prompt; the script keeps the sudo timestamp alive for the whole run so you type the password once. - Python 3 (present on Fedora). `power-report.py` needs `matplotlib`. - **Missing tools are installed automatically** from dnf when possible: ```bash INSTALL_DEPS=1 ./run-benchmarks.sh before headless # auto-install, no prompt ./run-benchmarks.sh doctor # show what's missing ``` Interactive runs default to `INSTALL_DEPS=ask` (a y/N prompt). The mapping is `7z→7zip`, `sysbench→sysbench`, `fio→fio`, `stress-ng→stress-ng`, `sensors→lm_sensors`, `turbostat→kernel-tools`, `cpupower→cpupowerutils`, `glmark2`, `vkmark`, `glxinfo→mesa-demos`, `vulkaninfo→vulkan-tools`, `matplotlib→python3-matplotlib`, plus `ncurses-libs`/`ncurses-compat-libs` for PassMark. Set `INSTALL_DEPS=0` to never install. - The LLM test needs **no** install: `llama-bench` and its small model are bundled under `tools/llama/` (see §4). - The **app workloads** (LibreOffice, GEGL, Inkscape, GIMP; see §4) are optional: each step runs only if its command is installed, otherwise it is skipped and listed by `./run-benchmarks.sh doctor`. --- ## 4. One-time tool setup **Almost nothing to download.** PassMark, `llama-bench`, and the LLM model ship inside `tools/`. (The optional app-workload corpora in §4 are fetched once, on first run.) Just make sure the ncurses compat libs are present for PassMark: ```bash sudo dnf install -y ncurses-libs ncurses-compat-libs # if it asks for libncurses.so.5: sudo ln -sf /usr/lib64/libncurses.so.6 /usr/lib64/libncurses.so.5 ./tools/pt/PerformanceTest/PerformanceTest_Linux_x86-64 -h # help / flags ``` If the bundled binary is missing (e.g. non-x86-64 host), re-fetch it: 1. Download `https://www.passmark.com/downloads/pt_linux_x64.zip`. 2. Unpack into `tools/pt/`: ```bash mkdir -p tools/pt && cd tools/pt unzip ~/Downloads/pt_linux_x64.zip # zips unpack to PerformanceTest/PerformanceTest_Linux_x86-64 (older: pt_linux_x64) chmod +x PerformanceTest/PerformanceTest_Linux_x86-64 ``` PassMark invokes `sudo dmidecode -t 17` for RAM SPD details; it's optional, and it's silent once the script's sudo keepalive is active. ### LLM inference — llama.cpp (bundled) **Already bundled** under `tools/llama/`: a `llama-bench` runtime (`bin/`, with its shared libs) and a small model (`models/MiniCPM5-2B-Q8_0.gguf`, ~2.5 GB). No download or install needed. The test runs **token generation on CPU/RAM** (`-ngl 0`), which is memory-bandwidth-bound — so its `tg` tok/s is a direct indication of how well the machine's RAM serves LLM inference. ```bash tools/llama/bin/llama-bench \ -m tools/llama/models/MiniCPM5-2B-Q8_0.gguf -ngl 0 -p 0 -n 128 -r 2 ``` Override with `LLAMA_MODEL=` (any GGUF), or `LLAMA_NGL=` to offload to GPU. ### App workloads — LibreOffice / GEGL / Inkscape / GIMP (optional) Stock distro applications are driven over a fixed sample corpus. **No Phoronix Test Suite is involved** — each is the app's own CLI, timed, named after the benchmark. Install the ones you want to measure: ```bash sudo dnf install -y libreoffice-writer libreoffice-calc # documents -> PDF sudo dnf install -y gegl04-tools # GEGL operations sudo dnf install -y inkscape # SVG -> PNG sudo dnf install -y gimp # GIMP batch operations (GIMP 3) ``` On first run the corpora are **fetched once** into `tools/corpus/` and checksum-verified (`lo-sample-documents-1.zip`, `sample-photo-6000x4000-1.zip`, `svg-test-files-1.zip`; ~12 MB total), then reused so both sides process identical bytes. Missing apps are skipped; skip the whole group with `RUN_APPS=0`. GIMP 3 reshaped the Script-Fu API, so the GIMP step picks a version-appropriate script and is best-effort — check `results/gimp-.txt` if a number looks wrong. ### Phoronix Test Suite (optional, short core only) Already installed (`phoronix-test-suite` 10.8.6). To use it: ```bash sudo phoronix-test-suite make-download-cache # once, needs internet ``` Only the short core (`build-linux-kernel`, `c-ray`, `stream`) is used, behind `RUN_PTS=1`. The long suite is intentionally **not** included. > `run-benchmarks.sh` finds PassMark at `tools/pt/pt_linux_x64` **or** > `tools/pt/PerformanceTest/PerformanceTest_Linux_x86-64`. Override the location > with `TOOLS=/path` (or set `PT=` directly). --- ## 5. Quick start On the target machine (fresh Fedora 44, internet): ```bash cd basic-benchmark ./run-benchmarks.sh doctor # shows what's present/missing (runs nothing) INSTALL_DEPS=1 ./run-benchmarks.sh doctor # provision missing tools via dnf ./capture-specs.sh before # record this machine's spec sheet ``` ### Part 1 — headless (run from another device over SSH) ```bash ssh -t 'cd basic-benchmark && ./run-benchmarks.sh before headless' ``` The `-t` matters: it gives `sudo` a TTY so power logging and the governor work. If tools are missing, this prompts to install them (or set `INSTALL_DEPS=1`). ### Part 2 — headed (run at the machine, in a graphical terminal) ```bash ./run-benchmarks.sh before headed ``` This errors out (exit 2) if no display is attached, rather than silently producing useless results. Run it when a graphical session is active. ### Compare ```bash python3 power-report.py --tags before-headless before-headed ``` Outputs: `results/graphs/power-before-headless.png`, `power-before-headed.png`, plus per-phase power/thermal tables and efficiency. (Comparison graph accepts any number of tags.) **Legacy single-shot:** `./run-benchmarks.sh before` runs `all` (headless, then headed if a display is present) under the plain `before` tag. **Median of 3 (recommended):** tag each repetition, then report all: ```bash for i in 1 2 3; do ssh -t "cd basic-benchmark && ./run-benchmarks.sh before-hl$i headless" ./run-benchmarks.sh "before-hd$i" headed # at the target done python3 power-report.py --tags before-hl1 before-hl2 before-hl3 \ before-hd1 before-hd2 before-hd3 ``` --- ## 6. What the run script does (and rough times) Headless part (`~25-30 min` per pass): | # | Step | ~Time | Notes | |---|------|-------|-------| | 0 | Env capture → `results/env-.txt` | <1 min | CPU model, tools, hashes, GPU pinning | | 0b | Idle baseline (≥15 s) | ~18 s | true idle power/thermal window | | - | Start power sampler | 0 | runs the whole part | | 1 | Pin CPU governor → `performance` | <1 min | sudo; **restored on exit** | | 2 | PassMark PT Linux `-r 1` then `-r 2` | 8-12 min | CPU suite + Memory suite (separate windows) | | 3 | `7z b` (all threads + `-mmt1`) | ~2 min | integer/cache proxy; isolated single-core | | 4 | `openssl speed` aes-256-gcm / sha256 | 2-3 min | crypto | | 5 | `sysbench` cpu 1-thread / all-thread / memory | 1-2 min | isolated single-core window | | 6 | llama.cpp generation (`2B`, CPU/RAM) | ~15 s | bundled; token-generation tok/s | | 6b | LibreOffice docs→PDF (20 docs) | ~0.5-1 min | fixed corpus | | 6c | GEGL image ops (9 on one 6000×4000 photo) | ~0.5-1 min | fixed corpus | | 6d | Inkscape SVG→PNG (fixed set) | ~1-2 min | fixed corpus | | 6e | GIMP batch ops (4) | ~1-2 min | GIMP 3, best-effort | | 7 | `fio` 4×30 s | 3-4 min | same SSD, control; runs as your user | | 8 | `stress-ng` 5 min + `turbostat` | ~8 min | thermals/power | | 9 | `systemd-analyze` | <1 min | informational | | + | PTS short core (`RUN_PTS=1`) | +30-45 min | kernel build, c-ray, stream | Headed part (`~4-5 min` per pass): | # | Step | ~Time | Notes | |---|------|-------|-------| | 1 | `glmark2` | ~1-2 min | pinned to the dGPU | | 2 | `vkmark` | ~2-3 min | pinned to the dGPU | Every step writes its own `results/-.txt/json`. Options (env): ```bash RUN_PTS=1 ./run-benchmarks.sh before headless # add the 3 short Phoronix tests RUN_APPS=0 ./run-benchmarks.sh before headless # skip the app workloads CORPUS=/path ./run-benchmarks.sh before headless # app corpus cache location GOVERNOR=powersave ./run-benchmarks.sh after headless # pin a governor (see §7) RUN_GPU=0 ./run-benchmarks.sh before # skip GPU tests in 'all' mode TOOLS=/path ./run-benchmarks.sh before headless # custom tools dir GPU_PCI=0000:03:00.0 ./run-benchmarks.sh before headed # force a GPU FIOFILE=/data/fio.bin ./run-benchmarks.sh before headless # fio scratch file ``` --- ## 7. Ground rules for a fair BEFORE/AFTER 1. Same OS image, kernel, mesa, and **tool versions** on both runs (recorded in `env-.txt`; tool binaries are hashed). 2. Same storage drive, same filesystem/options. 3. Same GPU, driver and displays; note which GPU drives the displays. The script auto-detects the headless dGPU and pins GPU tests to it (see §8). 4. Machine idle: close browsers, stop docker/VMs (`sudo systemctl stop docker docker.socket`). 5. Governor pinned to `performance` for timed runs and **restored afterwards** (the script does both). To measure another profile, use `GOVERNOR=`: ```bash GOVERNOR=powersave ./run-benchmarks.sh after headless # -> after-powersave-headless GOVERNOR=performance ./run-benchmarks.sh after headless # -> after-performance-headless ./run-benchmarks.sh after headless # -> after-headless (performance) ``` An explicit `GOVERNOR` is appended to the tag, so both profiles can be captured on the same run set and compared side by side with `compare-report.py` (the default `performance` keeps the plain tag, so existing before/after pairs stay comparable). 6. Warm up once, then take the **median of 3** runs. Run both parts back to back on the same side. 7. Note ambient/room temperature next to the results. 8. **Back up** `results/` (and `~/.phoronix-test-suite` if used) before reinstalling or re-imaging — copy them somewhere off the machine. --- ## 8. Power & thermal logging `powerlog.py` samples about once per second for the whole part: - **package W** and **core W** — from RAPL energy counters (`/sys/class/powercap/intel-rapl*`; root-only). - **CPU temp** — `k10temp` (or `coretemp` on other platforms). - **CPU MHz** — average `scaling_cur_freq`. - **GPU W / GPU temp / GPU busy** — amdgpu hwmon, **tied to the dGPU** (the first `amdgpu` hwmon exposing `power1_average`, so a hybrid iGPU+dGPU box logs the discrete card, not the iGPU). - **fan1, fan2** — `nct6798`. Files: - `results/power-.csv` — `epoch,iso,pkg_w,core_w,cpu_temp_c,cpu_mhz,gpu_w,gpu_temp_c,gpu_busy,fan1,fan2` - `results/phases-.csv` — `epoch,phase`, one line per benchmark, so the log can be split per benchmark. Run it standalone for a single test (Ctrl-C to stop): ```bash sudo python3 powerlog.py --tag before ``` ### GPU pinning The suite pins GPU tests to the GPU that owns **no connected display** (the RX 6700 XT here, while both monitors are on the Cezanne iGPU). It exports: - `DRI_PRIME=pci-` for OpenGL (`glmark2`) - `MESA_VK_DEVICE_SELECT` / `VK_LOADER_DEVICE_SELECT=` for Vulkan (`vkmark`) `glxinfo -B` and `vulkaninfo --summary` are captured so you can confirm which device actually ran. ### Graphs and efficiency ```bash python3 power-report.py --tags before-headless python3 power-report.py --tags before-headless before-headed python3 power-report.py --tags before-headless before-headed \ after-headless after-headed ``` Reports per phase: **avg / p95 / peak W**, **avg / peak °C**, **energy (kJ)**, and **score-per-watt**. Efficiency needs a scores file `results/scores-.csv`; the name should match a phase marker (exact or substring). Phase names: ``` env idle PassMark-cpu PassMark-mem 7zip 7zip-1t openssl sysbench-1t sysbench-nt sysbench-mem llama fio-seqread fio-seqwrite fio-randread fio-randwrite thermal-load systemd libreoffice gegl inkscape gimp glmark2 vkmark pts-core ``` Example `results/scores-before-headless.csv`: ``` PassMark-cpu,20707 PassMark-cpu-single,3342 PassMark-mem,2450 7zip,58935 openssl-aes,30.17 sysbench-1t,4810 llama-tg,18.2 libreoffice,3.9 gegl,1.8 inkscape,12.4 gimp,0.33 fio-seqread,2010.4 glmark2,5989 ``` ### True wall power (optional) The numbers above are **package** watts. For whole-system wall power, add a smart plug (Tasmota/Shelly) or a UPS that reports load, poll it on a second background loop, and merge on the `epoch` column. --- ## 9. Reading the results - PassMark **CPU Mark** scales with core count; PassMark **CPU Single Threaded** isolates IPC/clock. Read them together. - Compare **single-thread vs all-thread** separately — thread counts differ between platforms. - Compare **bandwidth and latency** independently; PassMark Memory Mark also rewards capacity. - Components carried over unchanged (same SSD, same dGPU, same displays) should land near-flat; a big move there means a setup fault, not a win. - For efficiency, **score/W higher is better**, **energy kJ lower is better**. ### Comparing devices / before-after ```bash python3 compare-report.py # every tag found in results/ python3 compare-report.py --tags t14-1 server-1-headless desktop-before-headless \ --label t14-1="ThinkPad T14" --label server-1-headless=server ``` Merges every `results/scores-.csv` into one table (also written to `results/compare-scores.csv`) and draws **one bar chart per benchmark** to `results/graphs/compare-.png`. Metrics missing from an older score file are back-filled from the raw results (`fio` JSON, `openssl`, `7z`, `sysbench-mem`, `glmark2`/`vkmark`), so existing runs compare without re-running. Needs `matplotlib`; use `--no-graphs` for the table alone. Fill the results table in `benchmarks.txt` with the medians after each side. --- ## 10. Backing up / reinstalling Before any reinstall or re-image, copy the kit and its results somewhere safe: ```bash cp -a ~/basic-benchmark /path/to/backup/basic-benchmark-$(date +%F) cp -a ~/.phoronix-test-suite /path/to/backup/pts-backup # if PTS was used ``` After the new build: put the kit back, run `INSTALL_DEPS=1 ./run-benchmarks.sh doctor` to provision tools, then run both parts (`headless` over SSH, `headed` at the machine) with `after`, and `python3 power-report.py --tags before-headless before-headed`. --- ## 11. Troubleshooting | Symptom | Cause / fix | |---|---| | `powerlog: no RAPL counters found` | Not running as root. Use `sudo`; otherwise power columns stay blank. | | Power columns blank in the CSV | Same as above, or RAPL not exposed. `turbostat` output still gives PkgWatt. | | `WARNING: no sudo` / sampler skipped over SSH | No TTY for the password prompt. Re-run as `ssh -t ...`. | | `ERROR: no display` from `headed` | The headed part must run in the KDE session (Konsole), not over SSH. | | `GPU: skipped` | No `DISPLAY`/`WAYLAND_DISPLAY` (e.g. over SSH) or `RUN_GPU=0`. | | GPU tests run on the wrong card | Override with `GPU_PCI=`/`GPU_VD=`; check the renderer in `glxinfo-.txt`. | | PassMark: `ncurses: cannot initialize terminal type` | `TERM` unset / no terminal. The script sets `TERM=xterm`; if it persists, run over `ssh -t`. | | PassMark warns `sudo: a terminal is required` | Its internal `sudo dmidecode` (RAM SPD) is optional; it's silent once the sudo keepalive is active. | | `skip (not found: …/pt_linux_x64)` | PassMark not installed at either expected path — see §4. | | `report failed` / no matplotlib | `sudo dnf install -y python3-matplotlib`. | | Scores show `no phase match` | Fix names in `scores-.csv` to match a phase name (§8). | | Governor left on `performance` | Shouldn't happen (restored on exit); set it back with `sudo cpupower frequency-set -g powersave`. | | App step says `skip (… missing)` | That app isn't installed; `sudo dnf install -y `, or `RUN_APPS=0` to silence. | | App step says `skip (corpus unavailable)` | Corpus download failed (no curl/wget or no network); re-run with network, or pre-seed `tools/corpus/`. | | GIMP step errors / odd number | GIMP 3 reshaped Script-Fu; inspect `results/gimp-.txt` (see §4). | --- ## 12. Scope notes - Portable by design: run it on any Fedora 44 host; specs are captured with `./capture-specs.sh`, never hardcoded. - The suite is intentionally split: **headless** for SSH-driven runs, **headed** for the display-bound GPU tests. - **PassMark** is the primary cross-platform score. The suite is self-contained and offline: no uploads and no license keys. - Phoronix is optional and **short only** — no long suite. - `powerlog.py`/`power-report.py` are plain Python 3; the only Python dep is `matplotlib` (installed on demand; see §3).