Measure capture, concurrency and output delivery

The capture benchmark compares serial blocking and Eio capture on one owned graph: one session, four windows and sixteen panes. Each sample captures the graph 100 times by default and selects active panes using both a compiled expression and a native predicate over each same snapshot.

The harness checks exact ordered IDs across captures and modes, one observed tmux client and one child process per mode, and zero attached clients afterward. Owned fixture teardown verifies pane and daemon exit. Both modes reuse one bound connection for their captures; they perform no per-object acquisition.

Reproduce¶

Prepare the exact release fixture, then build the benchmark outside normal test loops:

$ opam exec -- dune build bench/capture.exe

Run it with Python 3.11 or newer. Choose a new output file each time:

$ opam exec -- python3 bench/capture.py scratch/tmux/tmux-3.8/tmux \
    --iterations 100 \
    --repeat 5 \
    --output scratch/capture-baseline.json

The output records compiler and tmux versions, binary and behavior-source hashes, OS/architecture, CPU availability, starting load average, whole-command time, per-sample process time, client and daemon CPU usage, and OCaml heap allocation during the capture loop. Samples include program startup, binding, the capture loop and client closure. Fixture creation and compilation are outside the per-sample interval. Mode order alternates across repetitions.

Capture and graph size¶

These observations use OCaml 5.5.2 on one shared Linux host. Each table entry is median milliseconds for 50 captures, followed by its min–max across three samples. The raw files retain the source commit, behavior hash and host load.

Change --windows and --panes-per-window to measure larger graphs. The same tmux 3.8 workload, with 50 captures and three samples per mode, produced:

The largest graph has 256 panes. Each observation still uses batched graph acquisition. Allocation counters cover the OCaml heap during the capture loop; they do not measure process RSS or tmux’s allocations.

Heap retained across captures¶

Keeping one pane keeps its captured graph reachable. The retention benchmark measures this with a blocking client and one session containing one pane per window. Each sample runs in a fresh process, warms the connection, then captures the same graph 100 times. It measures live OCaml heap after full major collection before capture, while retained values remain reachable, and after releasing them.

On OCaml 5.5.2 and tmux 3.8, six samples per mode produced these live-heap increases in KiB. Every sample had the same retained-byte count:

Retaining 100 detached (id, command, path) summaries used 13.2 KiB in each graph shape; 100 Pane.target values used 6.2 KiB. With 64 windows and 25 captures, retaining one pane per capture used 940.0 KiB, while retaining only the latest pane still used 37.6 KiB. Targets remain connection-bound; retaining one does not authorize a future connection.

These are total live-heap differences, including list cells and projected values, not exact arena sizes. The discard control and every release phase returned 40 bytes below the warm baseline, a small baseline-overhead offset. This measurement does not cover process RSS, tmux memory, relationship-array allocation or a general leak guarantee. Mode order rotates across samples. The harness verifies topology, retained value counts, detached clients and owned daemon/pane teardown. Sample durations include the client-exit check; whole-command duration includes fixture setup and teardown.

Build the benchmark:

$ opam exec -- dune build bench/retention.exe

Run the 64-window case with a new output path:

$ opam exec -- python3 bench/retention.py scratch/tmux/tmux-3.8/tmux \
    --windows 64 \
    --captures 100 \
    --repeat 6 \
    --output scratch/retention.json

Completed control-lane mutations¶

The control benchmark measures typed fLibtmux.Operations.set_option mutations through one persistent Eio control client. Each sample starts a fresh one-session, one-window, one-pane daemon, warms the connection, then completes 128 distinct option writes with 1, 8 or 32 concurrent callers. Six samples per cell rotate the order of those caller counts.

Every handle must finish successfully. After the measured phase, an independent read checks every option name and value. Launch logs and process/client counts verify one persistent helper, with no subprocess per mutation. Owned teardown checks helper, client, pane and daemon exit.

Build the executable outside normal test loops:

$ opam exec -- dune build bench/control.exe

Run the complete matrix on one endpoint fixture:

$ opam exec -- python3 bench/control.py scratch/tmux/tmux-3.8/tmux \
    --requests 128 \
    --concurrency 1 8 32 \
    --repeat 6 \
    --output scratch/control-baseline.json

OCaml 5.5.2 results on the two supported endpoints:

Raw 3.3a and 3.8 records retain all 768 operation observations per cell, six sample durations, source/executable/binary hashes and platform details. One-minute host load started at 2.12 for 3.3a and 9.14 for 3.8, with 12 available CPUs. Other host work was active. These samples do not establish a release-speed comparison or a recommended caller count. Increasing callers does not add lanes: the engine still serializes this client.

The native phase includes command construction, completion checks and result-row allocation. It excludes connection, warmup, observer probes, independent verification and teardown. Per-operation time starts immediately before fLibtmux_eio.Client.submit, after a benchmark worker becomes available; it includes admitted lane queueing and fLibtmux_eio.Client.await, but excludes waiting for that worker slot. The pooled p95 uses nearest rank over the 768 observations. Allocation includes the benchmark’s own command and result values. The “sample_seconds” field includes setup and teardown; “consumer_and_topology_cpu_seconds” covers reaped children through consumer exit, including the topology probe, rather than daemon or whole-sample CPU.

The raw “whole_command_seconds” field times the collector’s entry point. Timing the outer command separately also includes Python and opam startup.

These results qualify this fixed mutation workload. They do not qualify the stream-gap or mixed-load failures in compatibility.

Concurrent reads, streaming and cancellation¶

The Eio benchmark runs bounded concurrent option reads with distinct expected values. It checks response attribution and input order, then measures future output from an owned cat process with terminal echo disabled. Output cursors must advance. Finally it cancels active byte waits and checks subscription cleanup. One primary client becomes two while a stream lane is active; both observed client and child-process counts are checked. No clients remain after closure, and owned pane/daemon exit is verified.

Each option read starts a read-only tmux process inside the client’s pump. The request rates below include process launch and are separate from the control-lane mutation measurements above.

Build the executable outside normal test loops:

$ opam exec -- dune build bench/concurrency.exe

Run the three phases against an owned release fixture:

$ opam exec -- python3 bench/concurrency.py scratch/tmux/tmux-3.8/tmux \
    --requests 128 \
    --concurrency 8 \
    --streams 10 \
    --cancels 10 \
    --repeat 3 \
    --output scratch/concurrency-baseline.json

Request time includes queueing until the attributed response. Stream time runs from sending literal input through observing the process’s output. Cancellation time runs from the cancellation request through the observer receiving it, including subscription cleanup. The p95 uses nearest rank across all samples.

On tmux 3.8, each row contains 384 requests, 30 stream samples and 30 cancellations:

The eight-request workload also passes on 3.3, 3.3a and 3.4. Higher concurrency increases queueing on the retained connection. These measurements come from a shared host running other work; compare raw sample ranges and load before choosing a workload or drawing performance conclusions.

Mixed commands, stream pressure and membership changes¶

The mixed-load benchmark overlaps retained command submissions, seven fast stream readers, seven deliberately stalled readers, membership changes and cancellation. Its default runs three waves with eight command workers and four attributed output frames per stream lane. It adds no work to the unit-test loops.

Build the executable separately:

$ opam exec -- dune build bench/mixed.exe

Run the bounded workload against one owned release fixture:

$ opam exec -- python3 bench/mixed.py scratch/tmux/tmux-3.8/tmux \
    --output scratch/mixed-baseline.json

Each wave submits 64 commands without yielding and requires the 65th to reject with the declared admission limit. Eight queued operations are cancelled before writing; their handles retain Not_started. Sixteen subscribers occupy seven shared session lanes. Together with the load connection’s primary lane this reaches its eight-lane limit; a separate controller connection sends output and changes membership. The seventeenth subscriber and an additional session lane must reject with their specific capacity errors. These expected rejections are recorded separately from correctness failures.

Commands set unique user-option names and values. The receipt retains operation IDs, worker IDs, submission indexes and the actual order in which awaiters observe results. It verifies every resulting option value, including absence for cancelled operations. Ordinary mutations use the retained control lane; read_option verification runs afterward through its temporary subprocess route and has a separate phase duration. Each verified option emits its operation ID, completed count and read duration before the observer handshake. The validation phase includes those handshakes; individual read durations exclude them. The observer acknowledgements and resource queries add measured overhead, so these are instrumented workload timings.

Fast streams must deliver exact ordered bytes with continuous generation and byte offsets. Tiny unconsumed queues must report Gap Overflow, then kEnd. Splitting and removing a temporary pane changes a watched window’s membership; affected readers must report Gap Membership_changed, then End. Explicit readmission uses a fresh observation and generation. Unchanged lanes continue independently; the receipt requires observed stream progress while commands remain pending in each wave. Each wave also cancels an active observer after its retained wait reaches Acknowledged, releases the wait through the controller, and checks the eventual result before and after client closure.

Native event callbacks synchronously sample owned Linux process trees before yielding to telemetry, retaining their occurrence-time pending-command counts. Progress handshakes separately sample process trees and attached tmux clients, including while the churn pane is alive. Emission timestamps and counts stay separate from event occurrence; queued telemetry cannot manufacture overlap. Raw records distinguish the Python harness, OCaml process, daemon, command-client children and pane children. RSS/FD/process/client maxima are maxima of these observations; per-process RSS high-water marks come from the kernel. They do not establish continuous aggregate resource peaks. The controller contributes a primary client and, during its retained wait, an auxiliary client in addition to the load connection’s eight clients. The sampler’s own retained pidfds affect its FD count.

Per-worker receipts report completed quotas, cancellation counts, latency samples and first/last observation times. They characterize this bounded workload rather than proving scheduler fairness, a service-level objective or general capacity. The 1 MiB queued-byte and 32 MiB payload limits are recorded source constraints; this workload pressures admission counts, subscriber counts and lane counts, not those byte maxima. No automatic retries or timeout increases hide a failure.

Exercise teardown after interruption with a new output file:

$ opam exec -- python3 bench/mixed.py scratch/tmux/tmux-3.8/tmux \
    --rounds 1 \
    --frames 2 \
    --inject-failure \
    --output scratch/mixed-interrupted.json

EXPECTED_FAILURE means the harness intentionally interrupted the native process while a temporary pane was alive. It still requires owned daemon, pane and command-client exit, reaping, zero remaining children and removal of the private fixture directory. Unexpected errors or failed cleanup produce FAIL; an interruption is not a successful workload measurement.

The summary records each outcome and source identity. The linked gzip archives contain the complete JSON receipts; the summary includes their compressed and uncompressed SHA-256 digests.

Raw results retain exact source and executable hashes, fixture-helper hash, versions, command arguments, workload bounds, host load, resource observations and cleanup receipts. Keep failed receipts beside passing measurements when assessing a release. Current collector qualification is PARTIAL across the supported releases:

Earlier records predate collector corrections; use their recorded source hashes. A later passing sample does not resolve an unexplained failure. These outcomes qualify this mixed benchmark separately from the existing native runtime tests.

The three 3.4 cores identify the same upstream write path. A later tmux correction guards null control state; that guard is present in the exact 3.8 source. The precise triggering sequence remains unproved. The supported range is 3.3a–3.8; the 3.3 measurements above are historical evidence. The 3.4 mixed workload remains a failed qualification.

Benchmarks are a separate tier. Correctness gates and test-loop timings are listed in testing, not used as throughput measurements. Keep a single benchmark run under ten minutes and a complete sweep under one hour.

Compatibility · Control · Guide