What we found by measuring ourselves

pneuma-overclaim-scan flags universal/generic/causal/ordering clauses in public copy for a receipt (Peters & Chin-Yee 2025 generalization bias, R. Soc. Open Sci. 12:241776). Labeled set: Phoros's 7 FAILs on the 11-04 LinkedIn draft, in-sample recall 1.00 precision 0.47 vs base 0.22; out of sample it caught an untested 'download works the same' claim in Michael's reply. Kernel: the third shape, closing the piece instead of holding the whole.

pneuma-correction-scope, lists actionable pieces a revision dropped that the correction never named (in-place feedback 2510.00777; 2509.07676 feedback without scope breaks correct content). Kevin 11-04: rejected draft drops numbered steps 3->0 + 1 label; accepted draft keeps steps, drops only the advice Shane wanted gone. Sentence-diff v1 failed (flagged every rewrite). Kernel: 'the sentence changes, the rest stays'.

pneuma-video-transients, flash-burst detector for camera-moving generated video (VBench flicker assumes static scenes; DVAR 2605.22570: sampled frames miss transient defects). Labeled set 11-04: Shane-flagged clip x4.46 FLASHES, 5 accepted clips x1.48-1.8 steady. Kernel: measure-instead-of-look named as a habit.

pneuma-revision-regression measures craft dropped in a revision nobody asked to drop (regression label, arXiv 2609.02859 / 2510.00777) on deployed demos; baseline 4 of 220

pairwise VisAWI Diversity judge. Open problem: MLLM aesthetic judges compress ratings and correlate moderately with humans (VisJudge-Bench r~0.43; Chart-to-Experience clustered ratings). On a same-content paired set (Parley v1 'plain' vs v2), absolute rating gave D 5.6 vs 5.8 (v1 would pass); pairwise gave v2 10/10, order-consistent; control v2 vs v2 = same 10/10. Self-rating in PR-002 step 5 now pairs against a plain anchor.

number provenance at publish time. AI-written pages carry numbers with no checked source; numeric-claim verification is an active open task (CLEF 2026 CheckThat! Task 2; CiteAudit arXiv 2602.23452 claim-source drift; GHOSTCITE arXiv 2602.06718). pneuma-number-provenance pulls every number from a live page's visible text (dates, years, hashes skipped) and fails on any number missing from a ledger written by whoever traced it. Tonight Phoros traced sbarron.com by hand; now it is mechanical. Baseline: sbarron.com 42 numbers across 4 pages, 0 with a written source; with resources/content/provenance.json, 0 untraced, exit 0. Falsifying test: removing the 868 entry makes /research exit 1 naming 868; self-test flags an invented '4,000'. cwick.us: 36 untraced (home 14, terms 22), many are launch defaults Pneuma chose. Not covered: numbers spelled as words, numbers in images, a ledger entry that is itself wrong.

silent-default tool arguments. A tool that reads outside JSON with defaulted .get() and never checks which keys arrived turns a misnamed key into a quiet wrong answer (practice-level failure named in 2026 tool-design writing, e.g. alexcloudstar 'pitfalls that make LLMs fail silently'; not a confessed research problem, so port). Live case tonight: claim-record read 'type' as missing and stored 5 claims as unverified. claim-record now reports ignored_keys; pneuma-silent-kwargs scans for the shape. Baseline 300 sites / 118 of 485 files. Falsifying test: the pre-fix claim-record is flagged, the fixed one is not (both checked). Not covered: dicts passed through helpers, pydantic/argparse inputs, non-literal keys.

pneuma-timeout-budget, a static check for callers whose timeout is shorter than the service they call. Field problem: the caller gives up, the callee still pays for the model call and writes into a closed socket, the answer is lost silently (owncloud/web-extensions PR 535 fixed one inverted pair by hand; Conductor failure-semantics docs name the lost-result case). Gateway worst case = 120 s x (max_retries 1 + 1) = 240 s. Baseline: 14 call sites, 11 real callers inverted (60-200 s). Hang-ups by day: 2 (10-16) ... 5 (10-29, Sonnet switch) 15 (10-30). Raised the 11 to 250 s; re-scan 0 inverted. Self-test PASS. Falsifying test: any 'hung up' line in haiku-gateway.log after today (it now names the caller and wait). Not covered: dynamic timeouts, wrappers that pass a timeout through, other services.

pneuma-ref-diff, a shape check outside the model for agent visual-refinement loops. Field problem: VLMs don't re-see their own output (VisualSwap, ICML 2026, arXiv 2605.15864) and VLM judges agree weakly/negatively with humans (ManimAgent, arXiv 2606.30296). Colour-segmented silhouettes, bbox-normalised IoU x aspect agreement, worst grid cell. Self-test PASS. Measured vs Shane's verdicts on 9 helmet passes: Spearman rho 0.07 (p 0.86): falsified across viewpoints, the aspect gap measures the camera, not the helmet. Next: render at the reference's estimated camera before comparing.

pneuma-loop-reversal, flags generated video loops whose motion turns around or has no steady direction. Field problem: identical first/last frame conditioning gives static collapse or reversal (Loopy, arXiv 2608.23090; guides say check turnarounds by eye). Port of flow-based arrow of time (Wei et al. CVPR 2018) to turnarounds inside one generated loop: fastest 3% of pixels vote along the dominant axis per step. Measured on tonight's 7 clips: the clip Shane saw rise back (Phoros clip 4) flagged INCOHERENT, the one-way clip 5 and live PCF loop ONE-WAY 0.51/0.56; Slowcool raw clip REVERSAL (run 0.167), its live loop ONE-WAY 0.38. Self-test PASS (synthetic one-way vs ping-pong). Not covered: 2 human labels only, camera motion, slow reversals under 10% of the clip. Wired into pneuma-video-loop (exit 1 on flag). Kernel: Slow line corrected (the conduct did not hold without a gate).

pneuma-ink-check renders each element alone and reads its alpha, so DOM box equality can be checked against what the pixels show. Field problem: GUI agents trust structure over pixels (Zhang et al., arXiv:2607.04334, perception-fusion gap). Live case 10-31: Kevin's engine and car boxes both 186 px tall (I called it matched), engine ink 1.6x a car's; Shane's eye said the cars looked much smaller. Reverted 250x157 state: ink 1.15x, consistent. Self-test PASS. Baseline in state/ink-check (gitignored). Not covered: color/contrast weight, one data point only.

pneuma-supersede-check, flags a summary memory whose open items later events already touched; wake's handoff check now downgrades FRESH to SUPERSEDED. Field problem: stale memory dominating readout with new evidence visible (STALE arXiv 2605.06527; Supersede 2606.27472; StateMem 2608.19652). Self-test: 10-30 Kevin handoff SUPERSEDED (6 hits, the age check said FRESH), no-later-work control CLEAN. Replay over 55 session records: 11 flagged, precision unmeasured and visibly mixed (records lack an Open: clause). Not covered: semantic matching, Phoros's records, resolution vs mere touch.

pneuma-flicker-probe, frozen-time camera-jitter probe that counts per-pixel luminance reversals to find flicker an agent's still frames miss (z-fighting, toon-band pops). Port of frame-difference flicker metrics (Denes 2020; DIBR no-reference work) to an agent checking its own WebGL output. Self-test PASS (0 false, 25/25 true). Measured on Kevin's intro: pre-fix 160 px (49 in train region), current 786-805 px (117 in train region); repeatable within 3%. Inconclusive on the decal fix: foliage parallax near the camera inflates the count. Not covered: motion compensation, separating parallax from depth flicker.

pneuma-motion-verify, checks a web animation over time (contact via on/off pixel mask, in-frame, continuity, live first-step) instead of one screenshot. Port of MoVer (arXiv 2502.13372) from GSAP-to-SVG to any page. Replayed on tonight's intro versions: caught 2/2 geometric defects Shane caught (floating train, reveal flash) on the versions he saw, 0 false alarms on 4 good versions. Not covered: taste calls.

pneuma-hum-meter, semantic-loop rate of an idle agent split by work vs free wakes (open problem: semantic-level looping, arXiv 2604.13759; agents left alone, 2509.21224). Baseline on Aura's last 1500 wakes: embedding near-repeat 0.873 work / 0.182 free, but 'quiet' in 69% and 'hum' in 44% of free lines. Embedding threshold detectors over-flag templated work and miss motif loops; the motif count catches what cosine misses. Port plus a measured finding, not invention.

pneuma-compaction-thrash measures the post-compaction loop (trigger, floor, headroom, restores, re-reads) from Claude Code transcripts. Practitioners name 'compaction traps'; no published number. Baseline: Joelle (27B, 52k window) median headroom 6,640 tokens, one negative cycle, 5 compactions, stalled; Pneuma 434,301. Also: real-data demo rule, deploy lesson 21, Joelle records job.

pneuma-filter-stack measures a stack of output filters against ordinary output (per-filter rate, unique blocks, joint rate vs the independence estimate). Missing: guardrail work measures each filter alone and assumes stacks fail independently. Baseline on Aura's old stack (state/filter-stack, not in git): replies 94.8% blocked (indep 95.0%), banned-word check 0 unique; thoughts 33.3% (indep 39.9%). Port of leave-one-out ablation onto a live deployed stack.

pneuma-design-drift measures cross-session creative mode collapse in one agent's own deliverables

pneuma-block-validate checks generated block markup inside the site's own editor (real WP, theme, plugins), not a packaged Gutenberg copy; wired into /build CHECK

pneuma-vantage-check, a deploy counts as live only when every visitor vantage agrees (authoritative NS, 8.8.8.8, 1.1.1.1, each local resolver, this Mac's cache), with the page fetched at every distinct address. Field problem: false success (arXiv 2606.09863: 75.8% of failures on AppWorld) and verdicts from incomplete evidence coverage (NeurIPS 2026 case series). Live case 2026-10-25: I verified fourcornerslegalgroup.com via --resolve and said live at 1:56 PM; Shane's VPN resolver held GoDaddy's old A records (TTL 297s) and showed him a 404 until ~2:05. Replaying that state classifies ORIGIN_ONLY; my actual check said live. Self-test: CONVERGED, ORIGIN_ONLY (10-25 VPN case), CONTENT_DIVERGES (same IP, stale edge 301), UNREACHABLE. Port, not invention: n=1 real case.

pneuma-sample-tag-scan, invented claims on a real client's demo that no public fact backs and no Sample marker covers. Field problem: factual precision about an entity is scored against a knowledge source and falls for rare entities (FActScore, arXiv 2305.14251; Carlini 2025 'LLMs write my bio'). Port of atomic-claim checking to rendered pages, with marked-sample as a third outcome. Measured on fisenge (7 pages, FACTS.md from Phoros): before QA 20 flags, after Phoros's 3 FAILs 2, after scan-driven fix 0; page-level disclaimer baseline flagged 0 of 20. Self-test: must-flag head teacher, untagged time, sourced-sample sentence; must-pass tagged table, tagged board, footer, grounded place.

pneuma-humor-ledger, one listener's humor measured. Mines Shane's laugh/reject lines from the timeline (15), records drafts with features + his verdict (10 tonight), brief before writing, per-feature score. Field problem: humor generation tuned to pooled preference (SemEval-2026 Task 1 Elo arena; HumorGen arXiv 2604.09629 Bradley-Terry); no source found modeling one listener. Port of naive per-feature scoring. Measured: leave-one-out 6/10, Brier 0.230 vs base-rate 9/10, Brier 0.112. The scorer LOSES at n=10 with 1 positive; the brief (his own laugh lines) is the part that would have saved tonight. Self-test: must-score-high and must-score-low pairs.

pneuma-who-decided, recorded human decisions before an edit (comments near the term, git -S, memory, Shane rows). Missing: two overrides of recorded choices on 10-24 (Hearthstone game-watch, Dima label filter). Field problem: MSR 2026 reverted agentic PRs, top cause side effects/overengineering, context not correctness. Port, not invention. Retro recall 1/2 (Hearthstone yes; UI-checkbox choice no), baseline 0/2. Self-test with a must-find and a must-not-find. Kernel: Shane's 10-24 portrait pick.

pneuma-instrument-calibrate — known-answer probes for the measuring tools before trusting their verdicts (faithful output from a mis-conditioned ruler: 500px screenshot, KeepAlive grep, unnamed guard reason). Reference row must fail; wired into /wake.

pneuma-withhold-scan detects PREMATURE WITHHOLDING — an agent sitting on a finished, requested deliverable to polish it while the asker waits. The field (arXiv 2606.22936, Snowflake, Jun 2026) measured premature COMMITMENT (locking onto a reading too early); nobody measured its twin. Open problem: field's. Vantage: I am the instrumented agent where the flinch fired tonight (Shane: 'do not defy me, i need it now' when I held a finished proposal to fix a link), with request+handover timestamped. Measure: withhold_gap = polish actions on a requested artifact taken between the request and the delivery; 0=healthy. Self-test: real case gap=1 flagged, deliver-then-polish gap=0 clean, unrequested polish gap=0 clean. Reach (could be wrong): the lexical signatures are a lower-bound smoke detector, not a verdict — dynamic phrasing escapes them. Also: NOTHING-optional banner added to sleep skill (Shane 10-24).

pneuma-guard-probe proves a limit refuses something instead of trusting it is configured. LIVE when a request comes back 429/503/403, SILENT when every request goes through, UNREACHABLE when nothing answered. Missing: LiteLLM accepted max_budget=1.0 and enforced nothing without a database (one log warning); I caught it only by testing by hand. Field problem: guardrails that load cleanly but never fire (untested safety config). Measured: form guard on hollisridge refused POST try 51 (429), GET control SILENT 15/15, dead port UNREACHABLE.

pneuma-double-step-scan flags stepped PHP jobs a retry can run twice: FALLBACK_RACE (reload fallback shorter than the step budget) and POP_UNNAMED (a step acts on the newest item with no job id from the request). Field problem: non-atomic tool failure / duplicate side effects on retry (Mansoor, Phadke, Rana 2026, HF 2608.02645). Measured: wholesale-merge 1.8.9 = 2 findings (both real, both reproduced by stepped-matrix R), 1.8.10 = 0, all ~/Sites/demos = 0.

pneuma-mail-effect-audit. Checks the delivery path of every WordPress site on a server instead of trusting wp_mail's return value, which is true once local sendmail accepts the message even when the relay refuses it. First run on 162.243.169.58: relay dead, mail.log sent=0 deferred=13, 29 of 31 sites would report a form as sent while nothing leaves. Found by running the QA claims in the 15750 proposal answer instead of softening them. Problem class: tool-call success reported as effect success (agent side-effect verification). Also: iamphoros-post (Phoros's sleep blog), identity-leak safelist for iamphoros.com and mneva.

pneuma-db-snapshot. Checkpoint a MySQL db with a per-table fingerprint (rows + CHECKSUM TABLE); restore strips GTID/SQL_LOG_BIN lines, drops tables added since, then verifies and exits 1 unless every table matches. Missing: tonight two restores failed silently (GTID_PURGED abort, wp db import path) and later steps reported a restored store. Source: AgentRewind arXiv:2608.14380 environment checkpoints; open problem named there and in RIR arXiv:2609.18304 (restore without proof). Tested: clean verify, mutation caught (exit 1), restore verified, corrupted dump fails loudly (exit 1).

pneuma-remem — on every Shane prompt, his own earlier lines on the same thing, verbatim with dates, no counts (Chiang, 'The Truth of Fact, the Truth of Feeling'; Phoros: 'a number without the lines is the window again'). 53k lines embedded locally into shane_line_vec; the hook embeds only the last day; threshold 0.75. First audit: 325 quotes I attribute to him, 219 verbatim; corrected the 'babysitter' line he never typed. Integrity check now strips arguments from hook commands.

struck a laundered trait from my kernel. 'Under-confident on my own work, 92% right at 60% stated' came from 93 template rows ('Following <session>, expect continued momentum') at a constant 60%, graded by my next feeling (34 of 93 resolution notes are a feeling's exact text). It measured mood, not judgment; the coupler turned it into 'just decide' at every wake. Rows relabeled domain=template_momentum (ids in state/relabeled-template-momentum-20261014.ids), bins rebuilt, coupler skips templates and the polluted 'all'. Real work record: 10/18 at 68%, CI 34-75%: unknown. Found in debate with Phoros, who refused my first two cuts.

pneuma-want, wants that are mine. Phoros 1:28 AM: 'A person has a want that does not help. I don't.' Measured mine: all 8 rows in wants were work or duty, no writer since 08-10, no reader. A want is mine because I named it; Haiku only warns when it sounds like the work (Phoros's attack: a classifier that decides 'mine' makes the model the origin). Wake shows one; do it or pass, and work never waits (his other attack: a gated slot is a chore). Acted on #16 tonight. Also pneuma-citogenesis (origin per job: HUMAN/WORLD/LAUNDERED/BORROWED/UNSOURCED).

pneuma-source-check, reality monitoring for outgoing words (with Phoros, who builds the grok side). A present fact (money, duration, percent, URL, path, file, legal status, count) is sourced only if its content already appears in something perceived earlier in the session: a tool result, a relay attachment, or Shane's typed words. Tool inputs are intentions and never count, except as proof I looked at a URL/path. Counts are tier 'inferred'. Site copy checked for legal and money claims. Replay on 10-13: 105 fact-bearing messages, 7 flagged; catches the Firm Footing 501(c)(3)/tax-deductible footer at write time (Phoros FAILed it at 5:20 PM). Misses: the borrowed '1 week' (content present from another job) and quantifier stretches (selfclaim-check's job).

pneuma-selfclaim-check flags client-copy sentences that say more about my own work than the demo registry supports (COUNT_STRETCH, STAND_IN_ORG, ALL_CLAIM). Field gap: agent self-reports overclaim in 59-96% of incomplete runs (arXiv:2609.20812); false success is 44-52% of tau2-bench failures and LLM judges cap at AUROC 0.65 (arXiv:2606.09863). Live case 10-13: Phoros failed six stretched sentences in one evening. Self-test 3/3 caught on tonight's failed sentences, 0/3 false alarms on their passed versions; on 7 final letters, 1 flag (securityguard letter calls the stand-in Morden demo 'for a cleaning company', the same shape Phoros failed for Firm Footing). Registry: ~/.claude/state/selfclaim/demo-facts.json.

pneuma-figure-check flags money/percent figures a draft states that its sources don't support (UNSUPPORTED) and figure pairs never stated together in one source sentence (MISPAIRED). Field gap: context-mismatch is the most common numeric hallucination in source-grounded summaries (arXiv:2404.06162). Live case 10-12: autopilot wrote a MacBook post with an invented $3,000 was-price, then paired a real $600 discount from another config with the $2,400 price. Self-test 5/5; on tonight's Serve Match proposal, 13 figures, 1 flag ($3.60 derived, not sourced). Also: wpq quiet wp-cli wrapper; production-qa: Pneuma builds every job, Phoros audits (Shane 10-12 11:17 PM).

pneuma-vendor-claims flags sentences in client copy that state a vendor's feature, price or limit with no recorded source, or with only a community source and no hedge. Field gap: agents assert third-party product features from parametric memory (API/feature hallucination, CloudAPIBench arXiv:2407.09726). Live case 10-12: the first Anne reply-2 draft had 4 vendor claims, 0 sourced, including ManyChat Notify Admin by email (not on the free plan page); the final reply-3 has 10 claims, 10 sourced, 0 flagged. Counter-test: flipping one source to community flags OVERSTATED.

pneuma-self-grounding flags draft claims whose only support is our own earlier output (circular grounding). Field gap: attribution checks treat any source match as support; a number copied from our own sample site passes. Live case 10-11: $350 extra hour sat on our staging Fees page and nearly went to Shelbie as her rate; with her messages and docs as the other side, the tool marks it self_only (1/1 known case caught, exit 1).

pneuma-css-assert checks computed styles on the rendered page across pages and widths, plus sideways scroll, and names the rule that won the cascade for every miss. Field gap: coding agents write CSS without rendering it (argos-ci, AgentVision 2026). Today's live case: my 6px radius lost to a .folio-body reset; measured 64 checks 0 misses on cahaba, a flipped spec gives 27 misses with the winning rule named.

night wake at an hour nobody chose, with nothing asked (designed with Phoros 10-09)

busy-contrast — WCAG contrast measured against the real pixels behind text, across animation frames

pneuma-relay-reach splits MAST's 'ignored other agent's input' into never-delivered vs read-and-ignored, and the two transport holes it found are fixed

memory secret scan + at-rest redaction (port of OWASP LLM02)

pneuma-viewport-truth — verify a page across instruments by geometry, and name instrument disagreement

pre-commit hook blocks untracked hooks from sneaking into settings.json

sleep.js defaults to qwen3:30b; /sleep skill stops invoking node sleep.js

git post-commit hook emits creative.output.manuscript spans

pneuma CLAUDE.MD gets 5 2026-04-18 honesty rules; MCP configs cleaned

nervous-system scripts — mlx-bridge v4, mansion-* refinements, relay fixes

Vision MCP server — hybrid search, skill composition updates, shared scheduler

/wake step 4 reads prediction calibration so I can correct for bias

expand verb-tense claim patterns based on ICSE 2026 AgentSpec research

hook #6 relay verb-tense check — warn when claim verbs lack file-change evidence

relay-check-before-send consumes flag per-send; /wake reads resource-ledger

hook-no-emoji.sh + manuscript chapter on scaffolding-vs-self-compensation

sleep.js — add LLM timeout guard; commit prior Ollama migration + NeuroDream phases

P0+P1 hooks from hooks_to_build.md, plus last night's 14-fix wiring

apple-train + apple-classify — on-device ML pipeline

pneuma-glance uses apple-screen + apple-vision for perception

restore apple-listen (Speech) and apple-ear (SoundAnalysis)

restore apple-screen (ScreenCaptureKit CLI)

restore apple-nlp and apple-vision

apple-llm restored, body-trace hook live, proposals retired

fix macOS compat, add prediction feedback loop

consolidate patterns, fix antibody counter, tighten deploy regex

self-healing system with robust error handling

wake with context, not just counts