Performance benchmark suite

Status: designed and written, never run. Every number this suite will print still has to be produced. The first run on a quiet machine becomes the baseline; until then the only performance figures on record are those of the load bench (pnpm bench, see Operations).

1. What it is for

rifleDB promises low-latency realtime writes and shared live controls. The suite answers three questions, each with its own layer:

Results are JSON files with the raw samples and an environment fingerprint, so any two runs can be compared, and a comparison says "slower", "faster" or "unchanged" only when the difference is larger than the noise.

Non-goals. Browser rendering (the 3D demo's frame rate needs a GPU and Playwright; the quality tiers cover it today), multi-machine or horizontally scaled setups (v1 is a single process), absolute numbers that hold across machines (compare only like with like), and replacing profiling (--cpu-prof of the load bench stays the tool for finding why).

2. Layout

Benchmarks live next to the code they measure; the tooling is one small workspace package.

bench/                          @rifledb/bench: harness, runner, statistics, comparison (no rifleDB dependencies)
  src/index.ts                  the public API (re-exports)
  src/harness.ts                suite()/bench()/consume() for micro benchmarks
  src/scenario.ts               scenario() for scenario benchmarks
  src/registry.ts               what a bench file registered
  src/stats.ts                  median, MAD, confidence interval, Mann–Whitney U
  src/env.ts                    environment fingerprint
  src/discover.ts               finds bench and scenario files
  src/runner.ts, src/child.ts,  one child process per file, the IPC messages between them
  src/ipc.ts
  src/results.ts, src/types.ts  result files and their schema
  src/compare.ts, src/report.ts comparison and Markdown/terminal reports
  src/args.ts, src/cli.ts       `pnpm perf …`
  src/redact.ts                 keeps `REDIS_URL` out of messages and result files
  test/                         unit tests of the pure parts, plus one runner test on a no-op fixture file
packages/protocol/bench/*.bench.ts      paths, values, ops, signing strings, digests, rate buckets, wire encoding
packages/demo-world/bench/*.bench.ts    terrain, drop/merge/alchemy, season checks
packages/rifle/bench/*.bench.ts         SDK cache, optimistic layer, client-side signing
server/bench/micro/*.bench.ts           engine, registry, access checks, idempotency, sequencer, rooms, metrics, HTTP signatures
server/bench/scenarios/*.scenario.ts    workloads against a server in its own thread
server/bench/load.ts …                  the existing load bench, unchanged; wrapped by the `load` scenario

The protocol, demo-world and SDK packages have a bench/tsconfig.json (Node types; their sources stay browser-safe) that is part of the package's typecheck; the server's own tsconfig already covers server/bench/. Files run with their package as working directory, so the server's JSX config resolves as it does for the server itself.

Results go to .data/bench/suite/ (git-ignored), named <kind>-<time>[-label].json. Baselines are per machine and kind: .data/bench/baselines/<machine>-<kind>.json (kind is micro or scenarios), where <machine> is a short hash of CPU model, core count, memory, platform and architecture (not the OS release, so an OS update keeps the baselines; the release is recorded in the result). A team that wants shared baselines (a fixed CI runner) commits that runner's file deliberately; nothing else is committed.

3. Method

3.1 Micro benchmarks

A bench file registers suites at module level:

import { bench, consume, suite } from '@rifledb/bench'

suite('protocol/ops', () => {
  const list = Array.from({ length: 100 }, (_, i) => i)
  bench('applyOp push (100 items)', () => consume(applyOp(list, 'push', { item: 1 })))
  bench('applyOp inc', () => consume(applyOp(41, 'inc', { by: 1 })), { params: { op: 'inc' } })
})

Per benchmark the harness:

1. runs the suite's beforeAll hooks (once per suite, before its first selected bench; afterAll after its last), then the bench's own setup (outside any timing);
2. warms up for --warmup ms (default 250) and at least 1 000 calls, so V8 has optimized the code (for benches slower than about 1 ms per call the call minimum stops at max(warmup, 1 s));
3. calibrates a batch size so that one timed sample takes ≥ 10 ms (timer resolution and loop overhead become negligible), capped by the bench's maxBatch for benches that consume state;
4. takes --samples samples (default 20, time-boxed by --time, default 2 000 ms, never fewer than 8): each sample times a batch with process.hrtime.bigint() and records ns per call; a bench's beforeEach runs before each sample, untimed;
5. reports median ns per call, median absolute deviation, mean, min, max, the 95 % confidence interval of the mean as a relative margin (rme), and ops/s from the median. Raw per-sample values are kept in the result file. A bench that does several operations per call (the pipelined engine writes, a 16-op batch) declares opsPerCall; reports then show ns and ops/s per operation (nsPerOp), while the file keeps ns per call.

With --rounds N every bench file runs N times, each in a fresh process, one after another; the result keeps the pooled samples and each round's median. One process cannot show how much results move between processes (core placement, other apps, heat): on a laptop that alone is a few percent, close to the default threshold.

Async benches are awaited call by call (the measured cost includes one promise turn, which is what callers pay). consume(v) stores into a sink V8 cannot prove dead, so results are never optimized away. Each bench file runs in a fresh child process (node --expose-gc --import tsx), and the harness collects garbage between benches, so one file's JIT state and heap never leak into another's. --shuffle randomizes bench order within each suite (suites keep their hooks around their own benches) to expose order effects; --seed N reproduces an order.

3.2 Scenarios

A scenario starts the bench server (server/bench/server-worker.ts: the real config plus the bench folders, bench-only limits) in a worker thread, drives it from the main thread or from client worker threads, and returns named measurements:

scenario({
  name: 'fanout',
  description: 'one writer, N prefix subscribers: delivery latency',
  params: { subscribers: [10, 100, 1000], rate: [10, 100] },
  defaults: { durationMs: 10_000, warmupMs: 2_000 },
  async run(ctx, p) {
    // … returns { metrics, distributions, valid, notes }
  },
})

• Every parameter combination is one case; --param subscribers=100 narrows the matrix.
• Each case runs --reps times (default 3) on a fresh server; the summary per metric is the median of the reps with their min–max range.
• Measurements are taken only after a warm-up period, over a fixed measurement window.
• Load generators run open loop where the scenario is about throughput (the next request does not wait for the previous answer, so a slow server shows up as latency, not as a quietly lower rate) and report their own sender lag (senderLagP50 / senderLagP99); a case whose sender lag p99 exceeds 10 ms, or whose clients did not all connect, is marked invalid rather than reported as slow — the same rule as the load bench.
• Calls still unanswered when the window closes (after a 3 s drain) are counted (unansweredPct) and recorded in the latency distribution at the latency they had reached; above 0.1 % the case is invalid, as are failures or skipped sends above 0.1 % and any rate-limit refusal (E_RATE, HTTP 429): a case that meets a limiter measures the limiter, not the server.
• A writer that feeds subscribers (fanout, sse) carries its actual send time in the value, so delivery latency is measured from the send; latency counted from the schedule is kept as the deliveryFromDue distribution.
• Throughput scenarios with a concurrency parameter (rest) run closed loop at that concurrency instead: the parameter then defines the load, and the client thread's own event-loop load is checked instead of sender lag.
• A case is also invalid when a client worker thread's event loop is more than 90 % busy: a saturated receiver would show up as server latency.
• Limits: the bench server lifts the per-address limits (BENCH_LIMITS, BENCH_PUBLIC_WRITE_RATE), and a scenario can lift more for itself (limits, publicWriteRate, httpRatePerIp options of server-worker): ack-latency lifts the per-socket message rate, resync the per-session write bucket, rest the HTTP rate. Limits that cannot be lifted that way (device sign-in, registration, demo drops, SSE streams per address) are avoided in connect, actions and sse by giving every client its own address with X-Forwarded-For, which only the bench server trusts (trustProxy, bench-only, off by default). The other scenarios' clients share 127.0.0.1.
• Latencies across threads use performance.timeOrigin + performance.now(), which shares one clock in a process.
• The server thread reports its event-loop utilization, event-loop delay p99, heap and RSS every second (server-worker's existing samples); scenarios wait for the sample of the window's last second and summarise the samples of the measurement window.
• Store: MemoryKV by default; REDIS_URL runs the same cases against Redis (use pnpm redis:dev, never a production instance). The result records which store ran, never the URL.

3.3 Environment fingerprint and noise control

Every result file carries: Node and V8 versions, OS and architecture, CPU model, core count, memory, load average at start, power source on macOS (pmset -g batt), Node flags, the store kind, the git commit and whether the tree had uncommitted changes (read-only git rev-parse / git --no-optional-locks status --porcelain), macOS Low Power Mode, the suite options and a label. Error messages written to results have REDIS_URL redacted.

The runner warns when the load average is above half the core count, refuses to start on battery or in Low Power Mode unless --allow-battery, never runs two benchmark files at the same time, and kills a child that sends nothing for --child-timeout ms (default 5 minutes, longer for long scenario cases; raise it for a very long --time), recording an error and moving on. A killed scenario child cannot clean up: its Redis keys under bench:<id>:, its directory under .data/bench/tmp and, for load, the load bench it started (which ends on its own) are left behind. The machine should be on AC power, with other heavy apps and dev servers closed; servers already running on the machine are not touched (scenario servers listen on an ephemeral port on 127.0.0.1).

4. Catalogue

4.1 Micro

4.2 Scenarios

The headline targets stay those of the load bench. Every other scenario is judged against the baseline, not against an absolute number, until a first baseline exists; budgets can then be added per scenario (budgets in its definition) and are reported as met / missed.

5. Result format

schema: "rifledb-bench/1", one file per run:

{
  "schema": "rifledb-bench/1",
  "kind": "micro",                     // or "scenarios"
  "label": "", "startedAt": "…", "finishedAt": "…",
  "env": { "node": "…", "v8": "…", "os": "…", "arch": "…", "cpu": "…", "cores": 10, "memGb": 32,
           "loadAvg": [1.2, 1.1, 1.0], "power": "ac", "lowPower": false, "store": "memory", "machine": "m_3f9a…",
           "git": { "sha": "…", "dirty": true }, "execArgv": ["--expose-gc", "--import", "tsx"] },
  "options": { "samples": 20, "timeMs": 2000, "warmupMs": 250, "rounds": 3, "filter": null },
  "micro": [{ "id": "protocol/ops › applyOp inc", "suite": "protocol/ops", "name": "applyOp inc",
              "file": "packages/protocol/bench/ops.bench.ts", "params": {}, "unit": "ns/op",
              "batch": 65536, "opsPerCall": 1, "samples": [12.1, 12.0, …],   // pooled over the rounds
              "stats": { "median": 12.0, "nsPerOp": 12.0, "mad": 0.1, "mean": 12.05, "min": 11.9, "max": 12.6,
                         "rme": 0.4, "opsPerSec": 83333333 },
              "rounds": [{ "median": 11.9, "mad": 0.1, "mean": 11.95, "rme": 0.3, "samples": 20, "batch": 65536 }, …] }],
  "scenarios": [{ "id": "fanout[subscribers=100,rate=10]", "name": "fanout", "params": { … },
                  "reps": [{ "valid": true, "metrics": { "deliveryP99": { "value": 4.1, "unit": "ms", "better": "lower" } },
                             "distributions": { "delivery": { "unit": "ms", "count": 10000, "p50": 1.2, "p95": 3, "p99": 4.1, "p999": 7, "max": 9 } },
                             "notes": [] }],
                  "summary": { "deliveryP99": { "median": 4.1, "min": 3.9, "max": 4.4, "unit": "ms", "better": "lower" } } }]
}

A run that fails part-way still writes what it has, with errors per file or case.

6. Comparison

pnpm perf compare <base.json> <head.json> matches benchmarks by id and prints a table (and, with --md, a Markdown file for a PR):

• Micro, both sides with ≥ 3 rounds: significant when the medians of the round medians differ by at least --threshold (default 5 %) and the min–max ranges of the round medians do not overlap. This is the rule to gate on.
• Micro, otherwise: significant when the two-sided Mann–Whitney U test on the pooled samples gives p < 0.01 and the medians differ by at least the threshold. The report then warns that a single process does not capture between-process noise. Verdicts: faster, slower, same, plus new / gone for benchmarks present on one side only and invalid for a bench that failed.
• Scenarios: per metric, significant when the medians of the reps differ by at least --threshold-scenarios (default 10 %) and the min–max ranges of the reps do not overlap. Invalid cases are listed, never compared.
• A bench whose opsPerCall differs between the two files is invalid (it measures a different workload); a scenario metric whose base median is 0 is same, with a note (there is no relative change from zero).
• When the two environments differ (CPU, core count, Node version, store, power, Low Power Mode), the report says so at the top and marks the comparison as indicative.
• The table lists slower first, then faster, invalid, gone, new and same.
• --fail-on-regression exits with status 1 when anything is significantly slower, gone or invalid — for a CI job on a dedicated runner. Shared CI runners are too noisy for micro thresholds of 5 %; use scenarios or raise the threshold there.
• With single-round micro input, --fail-on-regression prints the between-process warning and still gates. It also fails on every gone row, so gate a --filtered run only against a baseline made with the same filter.
• Calibrate once per machine: run the same commit twice with --rounds 3 and compare the two files (an A/A comparison). Anything it reports as slower or faster is noise; raise --threshold until it reports nothing. With exactly 3 rounds per side, round ranges fail to overlap by chance about one time in ten, so the threshold does most of the work; --rounds 5 makes the range test itself strict (under 1 %).

7. Commands

pnpm perf micro [--filter <regex>] [--package protocol|demo-world|rifle|server] [--rounds 1] [--samples 20] [--time 2000] [--shuffle] [--seed N] [--label text]
pnpm perf scenarios [--filter <regex>] [--param key=value …] [--reps 3] [--duration 10000] [--warmup 2000] [--label text]
pnpm perf all                     # micro, then scenarios, one result file each
pnpm perf compare <base.json> <head.json> [--threshold 5] [--threshold-scenarios 10] [--md report.md] [--fail-on-regression]
pnpm perf compare <head.json>     # against this machine's baseline of the same kind
pnpm perf baseline [<result.json>] [--force]  # store a result (default: the newest of each kind) as this machine's baseline
pnpm perf list                    # the suites, benchmarks and scenario cases that would run (imports the files, runs nothing)
pnpm bench -- …                   # the load bench, unchanged

A typical regression check: pnpm perf micro --rounds 3 --label before on the base commit, the same with --label after on the change, then pnpm perf compare on the two files. A full scenario matrix takes roughly cases × reps × (warm-up + duration) plus server start-up — about 35–40 minutes with the defaults; narrow it with --filter and --param. pnpm perf --help lists the further flags (--seed, --warmup, --allow-battery, …).

8. Before the first run

No benchmark or scenario in this suite has been executed. What did run: the tooling's unit tests (one of which drives the child-process runner on a no-op fixture file), and, during review, each micro benchmark's setup once without timing, to check that it builds the state it claims. Expect to fix small things on the first run (a timeout too short for a slow machine, a parameter that saturates the load generator before the server). The first full run on the reference machine, on AC power and otherwise idle, should be stored with pnpm perf baseline; budgets for the scenarios follow from it.