{ "schema_version": 1, "what": "Experiment: put ALL CUDA workloads on gb10-ref under MPS and measure the effect", "harness": "scripts/cotenancy/mps-workload-ab.sh", "harness_note": "Rates are deltas of the services' own frames_processed/frames_dropped counters across a 60 s window, so they measure the delivered rate of the live pipeline rather than a synthetic kernel. Arms are switched by adding or omitting compose/accel-nvidia/mps-clients.yml on the `docker compose up` line. The thread-cap curve in `thread_cap_curve` comes from scripts/cotenancy/mps-cap-sweep.py.", "host": "gb10-ref", "gpu": "NVIDIA GB10 (sm_121, aarch64, unified memory)", "recorded_at": "2026-08-04T12:45:00Z", "headline": "The experiment cannot reach its stated goal, and the reason is structural rather than a configuration mistake. MPS serialises servers PER UID on a device: the control daemon will only run one user's MPS server at a time, and a client of a different uid is pushed onto a pending list until the active server drains. The CUDA workloads on this box span FOUR uids, so no configuration exists in which all of them are managed simultaneously. What was achieved instead: every uid-0 CUDA workload is managed, with membership proven from the daemon's own client list, and the thread-percentage cap is shown to be a real lever on live services.", "the_original_finding_restated": "The gate `mps.all-clients-managed` read '7 uncontrolled CUDA clients'. That was not tenants outside our MPS server \u2014 there was NO MPS control daemon on the box at all, so every CUDA client was unmanaged by definition. The gate could never have been cleared by removing tenants. Two process checks made this invisible: `pgrep -f nvidia-cuda-mps` matches the grepping shell's own command line, and `pgrep -x nvidia-cuda-mps-control` can never match because Linux truncates comm to 15 characters ('nvidia-cuda-mps'). The old scripts/cotenancy/start-mps.sh used the second form, so its 'already running' guard never fired.", "blocker": { "name": "MPS servers are per-uid and mutually exclusive on a device", "state": "measured", "source": "nvidia-cuda-mps-control daemon log + a timed client attempt", "measured_at": "2026-08-04T12:33:44Z", "evidence": { "daemon_log": [ "Try shutting down server 1757995, backoff is enabled", "Server was unable to shutdown due to 4 active clients", "Server 1757995 has 4 active worker threads. Server will not shutdown.", "Control will try shutting down server 1757995 again in at most 300000ms", "NEW CLIENT 2090378 from user 1000: Server is not ready, push client to pending list" ], "experiment": "A uid-1000 CUDA client launched against the running uid-0 server never attached. It was killed at a 30 s budget; an earlier unbudgeted attempt sat in torch.cuda.is_available() for over 6 minutes, which is past the daemon's own 300 s retry, and still never joined.", "consequence": "A client of a non-active uid does not fail fast \u2014 it BLOCKS. Restarting the foreign tenants with CUDA_MPS_PIPE_DIRECTORY set would therefore have hung them at CUDA init rather than managing them, taking four services down for no gain." }, "unknown_reason": null }, "uid_census": { "value": { "0": [ "wg3-sr-worker", "wg3-sr-worker-quality" ], "999": [ "wg3-tvws-sensing" ], "1000": [ "llama.cpp llama-server", "voxcpm-trial", "radio-planner / ulap-sionna-rt" ], "1001": [ "kokoro-tts" ] }, "state": "measured", "source": "/proc//status Uid field", "measured_at": "2026-08-04T12:36:00Z", "evidence": { "note": "Four distinct uids. At most one of these groups can be MPS-managed at a time." }, "unknown_reason": null }, "gate": { "id": "mps.all-clients-managed", "verdict": "fail", "why": "An MPS control daemon IS running and every uid-0 CUDA workload is managed by it, but three CUDA clients of other uids are not: kokoro-tts (uid 1001), radio-planner (uid 1000) and wg3-tvws-sensing (uid 999). Under the per-uid exclusivity above they cannot join while the uid-0 server is active.", "evidence_is_live": true, "note": "This verdict is now computed from live data by services/cotenancy-exporter, not read from a stored build snapshot. Before this round the gate could only answer 'was the GPU clean when someone last swept'." }, "membership_proof": { "value": { "server_pid": 2204038, "client_pids": [ 2200350, 2200356 ], "resolved": { "2200350": "wg3-sr-worker (cap 50)", "2200356": "wg3-sr-worker-quality (cap 20)" } }, "state": "measured", "source": "nvidia-cuda-mps-control get_server_list / get_client_list", "measured_at": "2026-08-04T12:41:00Z", "evidence": { "basis": "the daemon's own client list. The presence of CUDA_MPS_PIPE_DIRECTORY in a container's environment is NOT membership and was never accepted as such." }, "unknown_reason": null }, "workload_ab": { "value": { "mps_off_control_arm": { "n": 1, "sr_display_fps": 3.63, "sr_archive_fps": 0.749, "aggregate_fps": 4.379 }, "mps_on": { "n": 2, "sr_display_fps": [ 5.147, 5.53 ], "sr_archive_fps": [ 0.4, 0.416 ], "sr_display_mean": 5.339, "sr_archive_mean": 0.408, "aggregate_mean_fps": 5.747 }, "reading": "The display path gained ~47% (3.63 -> 5.34 fps) and the archive path lost ~46% (0.749 -> 0.408 fps), which is the direction the configured caps ask for: sr-worker holds 50 and sr-worker-quality 20. Aggregate delivered frames rose ~31% (4.38 -> 5.75 fps), consistent with MPS avoiding context-switch thrash between two clients that were previously time-slicing. Neither path met its target (15 and 5 fps); the shortfall is still paid in dropped frames." }, "state": "measured", "source": "scripts/cotenancy/mps-workload-ab.sh, 60 s windows", "measured_at": "2026-08-04T12:41:48Z", "evidence": { "box_state": "RAM ~66-72 of 121 GiB used, swap effectively full at the start (SwapFree 416 MiB), load average 53-67 throughout. Every number here carries that.", "caveat": "The control arm is n=1 against n=2 for MPS-ON, on a box with heavy and varying foreign load. The direction of both changes is consistent and large, but these are not tight intervals and should not be quoted as such." }, "unknown_reason": null }, "thread_cap_curve": { "value": { "100": 49.56, "75": 36.44, "50": 22.71, "25": 14.24, "10": 8.56 }, "state": "measured", "source": "scripts/cotenancy/mps-cap-sweep.py, fp16 4096 matmul in a managed client", "measured_at": "2026-08-04T12:39:00Z", "evidence": { "units": "TFLOPS", "every_cell_self_managed": true, "membership_per_cell": "Each cell records a pid that appeared in the daemon's own client list DURING that cell and was not there before it.", "co_tenancy": "Measured with the live stack and 3-5 foreign tenants present and untouched, which is why the absolute values sit below the clean-box figures." }, "unknown_reason": null }, "cross_host": { "value": { "_hosts_note": "Keys are ROLE labels, not hostnames, on purpose. tests/publication/internal-names.json maps BOTH GB10 boxes to the SAME single reference label. A cross-host comparison written with hostnames therefore collapses into two rows called the same thing the moment the publication redactor runs, and the entire point of the comparison \u2014 loaded box vs idle box \u2014 is lost. Role labels survive redaction with the distinction intact.", "gb10_loaded_cotenant": { "note": "co-tenant loaded", "100": 49.56, "75": 36.44, "50": 22.71, "25": 14.24, "10": 8.56 }, "gb10_idle_clean_box": { "note": "idle GPU, zero other tenants, cuBLAS fp16 GEMM via nvcc (no torch on that box)", "unmanaged_baseline": 88.61, "100": 92.5, "75": 75.74, "50": 51.68, "25": 28.19, "10": 18.9 }, "gb10_loaded_prior_artefact": { "source": "benchmarks/reports/mps-cap-result.json, private throwaway daemon, synthetic clients", "100": 79.18, "50": 51.53, "25": 27.65, "10": 18.63 } }, "state": "measured", "source": "this round on both GB10 boxes", "measured_at": "2026-08-04T12:20:00Z", "evidence": { "reading": "The cap behaves the SAME on both GB10 boxes \u2014 gb10-ref's 51.68 / 28.19 / 18.90 at 50/25/10 match the earlier gb10-ref figures of 51.53 / 27.65 / 18.63 closely. What differs is the CEILING, not the lever: 92.50 TFLOPS at cap 100 on the idle box versus 49.56 on the loaded one. That is the whole 'provisions, does not reserve' point in one comparison \u2014 the cap divides whatever is left after the unmanaged tenants have taken their share, and it cannot defend a floor.", "also": "On gb10-ref, cap 100 (92.50) slightly EXCEEDED the unmanaged baseline (88.61), which is the context-switch-avoidance effect showing up on a clean box too.", "force_tegra": "The MPS server runs as `nvidia-cuda-mps-server -force-tegra` on BOTH GB10 boxes. The driver drives GB10 down the Tegra/integrated path. The thread-percentage lever demonstrably works there, but this is not the same code path as a discrete datacentre GPU and anything anomalous about MPS on this silicon should be suspected there first.", "redaction_caveat": "Do not reintroduce hostnames into this block. The two physically distinct GB10 boxes share one reference label in the redaction map, so hostnames here are not just private, they are ambiguous after packaging." }, "unknown_reason": null }, "mig_on_gb10": { "value": null, "state": "unknown", "source": "nvidia-smi -q", "measured_at": null, "evidence": null, "unknown_reason": "GB10 reports `MIG Mode: N/A` \u2014 not `Disabled`. The silicon does not implement MIG at all, on either GB10 box. MPS is the only GPU-partitioning primitive available here, and MPS does not partition memory. Any claim of hardware-guaranteed tenant isolation on GB10 is unsupported." }, "what_broke": { "value": [ { "item": "voxcpm-trial (amini-voxcpm.service, 5723 MiB)", "what": "Stopped by SIGTERM at 15:40:12 EAT (`Main PID: 4349 (code=killed, signal=TERM)`), not an OOM kill. This agent issued no command targeting it. Restart=on-failure treats a TERM-initiated stop as clean, so systemd did NOT bring it back and it stayed down. Restored with `systemctl --user start amini-voxcpm.service`.", "status": "restored" }, { "item": "llama-server.service (655 MiB)", "what": "Restarted itself once (NRestarts=1) around the same period; Restart=always caught it. Serving again on 127.0.0.1:8080.", "status": "self-healed" }, { "item": "a uid-1000 MPS client", "what": "Blocked indefinitely on the daemon's pending list rather than failing. This is the blocker above, and it is the finding, not an accident.", "status": "expected" }, { "item": "amini-voxcpm.service (5723 MiB) \u2014 second, distinct failure", "what": "Hung at CUDA init for ~6 min against the daemon at the DEFAULT pipe directory, then silently fell back to CPU. Caused by this work. Fixed by moving the daemon to /tmp/wg3-mps and restarting the service; verified 'Running on device: cuda' and 5723 MiB on the GPU.", "status": "restored, cause fixed at the root" }, { "item": "llama-server.service crash-loop (NRestarts 1 -> 35)", "what": "NOT caused by this work, and not what it first looked like. There are TWO enabled systemd --user units with IDENTICAL ExecStart competing for 127.0.0.1:8080: llama-server.service and llamacpp-qwen05b.service. The latter became active at 15:40:14 EAT, took the port, and is serving; the former could no longer bind and looped. 15:40:14 is two seconds after the unexplained SIGTERM to amini-voxcpm, so both events point at another actor on the box.", "status": "churn stopped by `systemctl --user stop llama-server.service`; the surviving unit still serves :8080 ({\"status\":\"ok\"}). NEITHER unit was disabled \u2014 which one is canonical is an operator decision." } ], "state": "measured", "source": "systemctl --user show / journalctl", "measured_at": "2026-08-04T12:43:00Z", "evidence": { "note": "The operator's capture described llama.cpp and voxcpm as unsupervised bare processes. They are not: both are ENABLED systemd --user units (llama-server.service Restart=always, amini-voxcpm.service Restart=on-failure). That correction lowers the risk of restarting them." }, "unknown_reason": null }, "fault_isolation_between_clients": { "value": null, "state": "unknown", "source": "not tested", "measured_at": null, "evidence": null, "unknown_reason": "Post-Volta MPS is documented to fault-isolate clients from one another, but deliberately crashing one of two live SR workers to observe whether the other survived was not run \u2014 it would have meant inducing a fault in a service on a box being demoed. Not observed, so not claimed." }, "memory": { "value": { "before": { "mem_available_gb": 49.9, "swap_free_mb": 416 }, "after": { "mem_available_gb": 54.8, "swap_free_mb": 2724 } }, "state": "measured", "source": "/proc/meminfo", "measured_at": "2026-08-04T12:42:00Z", "evidence": { "reading": "Roughly 2.3 GiB of swap and ~5 GiB of RAM came back over the session. This is NOT attributable to MPS: it is the restart cycle of the two SR workers plus ollama's 12.2 GiB llama-server dropping off the GPU when its keep-alive expired. Ollama's footprint is transient and reloads on demand, so freeing it is not a durable reclaim and is not counted as one." }, "unknown_reason": null }, "honesty": "CUDA_MPS_ACTIVE_THREAD_PERCENTAGE PROVISIONS but does not RESERVE. NVIDIA documents that kernels from different clients may still execute on the same SM. Nothing in this artefact is evidence of hard isolation: the cap moved share between two of our own workloads and raised aggregate throughput, and that is all it shows. MPS gives no memory isolation whatsoever. MIG hard-partitions both SMs and memory \u2014 and GB10 has no MIG.", "operator_decisions": [ "To make `mps.all-clients-managed` capable of passing on gb10-ref, every CUDA workload on the box must run as ONE uid. Today they span four (0, 999, 1000, 1001). That is an architecture decision about container users and service accounts, not a tuning knob.", "Alternatively, accept that the gate measures a SCOPE \u2014 e.g. 'every uid-0 workload is managed' \u2014 and re-word it. It currently asks a question the hardware cannot answer yes to on a shared box.", "Something outside this agent sent SIGTERM to amini-voxcpm.service at 15:40:12 EAT. If another agent or a reclaim job is stopping GPU tenants, it should be identified before the next experiment, because it silently contaminates any before/after on this box.", "Two enabled systemd --user units, llama-server.service and llamacpp-qwen05b.service, have identical ExecStart and both want 127.0.0.1:8080. They will race again at every boot. Decide which is canonical and disable the other; this agent stopped the losing one at runtime but disabled nothing.", "Never place an MPS control daemon at /tmp/nvidia-mps on a shared box. It is CUDA's default path, so it captures unrelated CUDA processes and \u2014 across uids \u2014 hangs them. This repo now pins /tmp/wg3-mps." ], "default_pipe_directory_trap": { "value": "/tmp/nvidia-mps is CUDA's DEFAULT pipe directory; a daemon there captures every CUDA process on the box without those processes opting in", "state": "measured", "source": "nvidia-cuda-mps-control daemon log + amini-voxcpm.service journal", "measured_at": "2026-08-04T12:49:00Z", "evidence": { "what_happened": "The daemon was first stood up at /tmp/nvidia-mps. amini-voxcpm.service (uid 1000) was then restarted and HUNG at CUDA init for ~6 minutes. The daemon's own log named it: 'NEW CLIENT 2325448 from user 1000: Server is not ready, push client to pending list'. voxcpm had never been configured for MPS and had no CUDA_MPS_* variable in its environment \u2014 the CUDA runtime found the daemon at the default path by itself.", "consequence": "On stopping the daemon voxcpm resumed within seconds, but had already fallen back to 'Running on device: cpu'. A 5.7 GiB GPU service had silently become a CPU service. A second restart was needed to recover CUDA, confirmed by 'Running on device: cuda' and 5723 MiB back on the GPU.", "fix": "The daemon and all its clients moved to /tmp/wg3-mps, a NON-default path, so only services that explicitly set CUDA_MPS_PIPE_DIRECTORY join. /tmp/nvidia-mps was removed from the box.", "severity": "This is the most dangerous thing found today. A daemon at the default path makes an MPS rollout a box-wide, opt-out change rather than an opt-in one." }, "unknown_reason": null }, "final_state": { "value": { "mps_daemon": "up at /tmp/wg3-mps", "managed": [ "wg3-sr-worker (cap 50)", "wg3-sr-worker-quality (cap 20)" ], "unmanaged_and_healthy": [ "wg3-tvws-sensing", "kokoro-tts", "radio-planner", "voxcpm", "llamacpp-qwen05b" ], "all_services_verified_serving": true }, "state": "measured", "source": "health endpoints + nvidia-smi + the daemon's client list", "measured_at": "2026-08-04T12:55:00Z", "evidence": { "checks": "llama :8080 ok, voxcpm :8883 listening + 5723 MiB on GPU, kokoro-tts up, radio-planner :8765 200, tvws :8002 200, both SR workers healthy and in the daemon's client list" }, "unknown_reason": null }, "operator_decisions_resolved": { "1_gate_scope": { "decision": "RE-WORDED to a scope rather than unifying uids.", "detail": "The live gate is now `mps.in-scope-clients-managed`: of the CUDA clients that CAN join the active MPS server \u2014 those sharing its uid \u2014 is every one a member? It reports `scope.uid`, `scope.out_of_scope_count` and a per-tenant `scope.out_of_scope[]` with uid, memory and reason, so scoping narrows the VERDICT and never the visibility. Currently PASS: both uid-0 clients managed, 7 clients across uids 996/999/1000/1001 explicitly out of scope. Unifying uids was NOT done: it would mean changing the service accounts of other projects' workloads.", "why_not_unify": "Five uids are in play (0, 996 ollama, 999 tvws-sensing, 1000 llama.cpp/voxcpm/radio-planner, 1001 kokoro-tts). Unifying them is an architecture change to foreign services, and the gate re-wording makes the question answerable without it.", "artefact_gate_unchanged": "`mps.all-clients-managed` still governs a controlled BENCHMARK RUN, where demanding a clean GPU is legitimate. Only the live gate moved." }, "2_llama_canonical": { "decision": "RESOLVED \u2014 llama-server.service is canonical; the duplicate is gone.", "detail": "`llamacpp-qwen05b.service` was a unit file written at 12:39:34Z by the session's own preparation thread and has since been removed: LoadState=not-found. `llama-server.service` is the persistent, enabled unit (file dated 2026-07-29) and now holds 127.0.0.1:8080 and answers {\"status\":\"ok\"}. No duplicate remains, so there is no boot race. Notably its own header documents this exact bug class recurring: it 'absorbed --alias from llamacpp-edge.service and retired that unit... the loser crash-looped 70,077 times without ever serving a request.' This was the third instance.", "residual_risk": "None outstanding. Nothing was disabled; the duplicate simply no longer exists." }, "3_voxcpm_sigterm": { "decision": "IDENTIFIED \u2014 it was this session's own preparation thread, not a third party.", "detail": "Session 595719ad (the thread that wrote tenant-restart-paths.md) ran at 12:40:07Z: `echo '=== 1. stop the two unsupervised processes ===' ; kill 4355 4349`. That is the SIGTERM at 15:40:12 EAT. Immediately before, at 12:39:34Z, it wrote two NEW unit files \u2014 llamacpp-qwen05b.service and voxcpm-tts.service \u2014 duplicating services that were ALREADY supervised. One action therefore caused both incidents: voxcpm's SIGTERM (it stayed down because Restart=on-failure treats TERM as clean) and the :8080 race.", "root_cause": "The belief, recorded in tenant-restart-paths.md, that llama.cpp and voxcpm were 'UNSUPERVISED \u2014 bare user processes, NO supervisor'. Both were in fact enabled systemd --user units. The duplicates were created to supervise something already supervised.", "exonerated": "uwezo-reclaim.timer was the obvious suspect and is INNOCENT: it ran at 15:39:02, logged 'ok: 49GB available (>= 20)' and did nothing. It only pkills 'ollama runner', and only under memory pressure.", "competing_diagnosis": "A drop-in written at 15:58 (~/.config/systemd/user/amini-voxcpm.service.d/10-offline-cache.conf) attributes the SAME 5-minute hang to the HuggingFace CDN half-closing a socket. The daemon's own log contradicts that: it named the client \u2014 'NEW CLIENT 2325448 from user 1000: Server is not ready, push client to pending list' \u2014 and the process resumed within seconds of stopping the MPS daemon. HF_HUB_OFFLINE=1 is harmless and arguably good, but the recorded CAUSE is a misattribution of the default-pipe-directory bug." }, "4_prometheus": { "decision": "DONE \u2014 and it uncovered a worse problem than a missing scrape job.", "detail": "Appending the job to the live config had NO effect and the reload still returned HTTP 200. The container's bind mount was ORPHANED: host inode 3541887 vs container inode 3541563. Docker binds a file by inode, so once the host file had been replaced the container kept reading a stale copy forever \u2014 every config edit anyone made there was silently ignored while /-/reload cheerfully reported success. Fixed by recreating wg3-prometheus, which re-resolved the bind (inodes now match). The cotenancy-exporter target is UP and wg3_mps_daemon_up / wg3_mps_managed_client_count / wg3_cotenancy_gate / wg3_cotenancy_scope_uid / wg3_cotenancy_out_of_scope_clients are all queryable.", "port_correction": "Prometheus is on :9096, NOT :9095. The running container had drifted from its own compose file; recreating it restored the committed mapping. The compose comment is explicit that 9095 collides with the native Open5GS PCF metrics server (config/lab.yml monitoring.prometheus open5gs_pcf:9095), which is why 9096 was chosen. Anything still pointing at :9095 \u2014 including this repo's own CLAUDE.md service table \u2014 is stale.", "backup": "config/prometheus.yml.bak- left beside the live file.", "side_effect_to_own": "Recreating wg3-prometheus moved its published port from 9095 to 9096 \u2014 the value its own compose file has committed, with a comment explaining that 9095 collides with the native Open5GS PCF metrics server. The container had been running on a stale mapping. This repo's Makefile, CLAUDE.md and README were queried against 9095 and have been corrected (8 Makefile call sites; they would otherwise have silently printed 'N/A' for every metric, which is the failure mode where a dead endpoint looks like a real zero). NOT corrected, because it is a different tree: ~/Ulap/wg3wi6-sutd/runnables/Makefile still references localhost:9095 and its `make status`/`make health` will now under-report. One-line fix: sed -i 's|localhost:9095|localhost:9096|g' on that Makefile. Reverting the container to 9095 instead was rejected: pinning a container to a value its own compose contradicts is exactly how the drift arose." } }, "deployment_gap": { "value": "the live plane is built and reachable, but NOT yet visible on the deployed page", "state": "measured", "source": "docker inspect wg3-ui-next + a read-only fetch from inside it", "measured_at": "2026-08-04T13:50:00Z", "evidence": { "reachability_PROVEN": "From inside wg3-ui-next: `wget http://cotenancy-exporter:8097/health` returns {\"status\":\"ok\"} and the full gate payload resolves (mps.in-scope-clients-managed / pass / scope uid 0 / 7 out of scope). The container is on wg3-network, so no networking work is needed.", "what_is_missing": [ "The page is served by wg3-ui-next on :18095 from a PRE-BUILT static site at ~/Ulap/ULAP-ONE/deploy/wg3-ui \u2014 a different tree from this repo. The Svelte changes must be built and deployed there before the 'Right now' panel exists.", "Its nginx config is ~/Ulap/ULAP-ONE/deploy/wg3-ui.nginx.conf, a deployed COPY of wg3-workitem6-2026-demo/config/sensing-dashboard.nginx.conf. Their location sets are otherwise identical \u2014 the only difference is the `location /api/cotenancy/` block added here. Without syncing it the browser gets 404 on the live endpoint." ], "why_this_agent_did_not_do_it": "Both steps require reloading or replacing wg3-ui-next, which is on the do-not-stop list for this round.", "honesty": "So the correct claim today is 'the live plane is implemented, unit-tested and reachable', NOT 'the /cotenancy page now shows it'. An honest instrument behind an un-synced front door still ships as unavailable." }, "unknown_reason": null } }