Drive Posterior Desk from your own code
Everything the web page does is available over HTTP: send the fit report a PyMC run just printed —
the sampler settings, the warning block, the model spec, as labelled lines or as prose or both — with
the az.summary() table beside it, and get back the same structured review the browser
renders. Three lanes share one request shape and one response envelope: diagnose decides
whether the posterior is trustworthy enough to report anything from, respec names the
root cause and prescribes what to change before the next run, and report writes the
methods paragraph, the diagnostics table and the results wording you can actually defend.
The natural uses are a gate in a group's own tooling — refuse to promote a fit whose
diagnose lane comes back not_supported, the way a test suite refuses a
failing build — and a batch pass over a directory of saved InferenceData summaries that
says which runs were reported as converged while their own tables say otherwise, before a reviewer
has to find it. Neither use needs the browser, and both are three calls: mint, price, run.
Posterior Desk reads reports about fits. It does not sample, it does not have your
InferenceData, and it cannot compute a diagnostic you did not paste. Everything below is
built on what the report and the table say, which is exactly why the contract is so insistent about
the difference between a number that was computed, a number that was reported, and a number that was
never established at all.
Base URL and the envelope
Every endpoint lives under https://api.skillsafe.ai/v1/app-api and every response uses
the same envelope, so one helper covers the whole API:
{ "ok": true, "data": { ... } }
{ "ok": false, "error": { "code": "...", "message": "...", "details": { ... } } }
There is no X-App-Slug header. The browser SDK this app ships sends
exactly two headers on a normal call — Content-Type: application/json and
Authorization: Bearer … — plus Idempotency-Key on a run and
Accept: text/event-stream on a stream. The slug posterior-desk appears in
one place only: the body of POST /guest. A token is already bound to its app, so nothing
downstream needs to be told the slug again, and a header that looks like it should work is simply
ignored. People invent that header, spend an afternoon on it, and it was never read.
The endpoints, in the order a caller uses them:
| call | auth | cost | what it does |
|---|---|---|---|
POST /guest | none | free | Mints a guest token for one app. Answers 201 with {token, guest_id, expires_at}. |
GET /me | token | free | Returns {subject_type, subject_id, credits} and nothing else. |
POST /estimate | token | free | Prices an input. Creates no job and charges nothing — but it is authenticated, so it has to come after the token. |
POST /run | token | metered | Starts a review. Returns {job_id}. |
GET /jobs/{job_id} | token | free | Polls one job. The terminal job carries output.output, charged_credits and truncated. |
POST /run-stream | token | metered | The same run as server-sent events: job, delta, done, and error on a failure. |
The request body IS the input object. It is never wrapped in an
input key.
The body of /estimate, /run and /run-stream is the flat
input object — {"task": …, "fit": …, "table": …, …}. Not {"input": {…}},
not {"body": {…}}, not {"data": {…}}. This is the single most important
sentence on the page, because the wrong shape does not fail: a wrapped body
returns 200, reserves a plausible-looking hold, produces a job that succeeds, and
bills you — while the model receives an object with none of the fields it is told to read. What
comes back is a fluent convergence review of nothing: an assessment of a fit whose sampler
settings, warning block and summary table it never saw. There is no error code for it and no
warning in the reply. The only defence is sending the object flat, which is what every sample on
this page does, plus the one assertion in the verification step — read
lane back and compare it to the task you sent.
See the request body for the field list.
Error codes
| code | status | what causes it here, and what to do |
|---|---|---|
VALIDATION_ERROR | 400 | The body is not the shape the app expects — most often a missing fit, or a prescan_facts sent as a JSON string instead of an object. A body that is not valid JSON at all lands here too. Note what is not a validation error: an absent task (the model picks a lane), an absent table (legitimate, and common on a first fit), and a body wrapped in input (a silent 200). |
UNAUTHORIZED | 401 | The token is missing, malformed or past its expires_at. The common surprise is a 401 from /estimate: it is free but still authenticated, so it cannot run before step 1. Mint another token with POST /guest, or copy a personal one from the token page. |
INSUFFICIENT_CREDITS | 402 | The balance is below min_credits. Call /estimate first — it is free — and compare min_credits against credits from /me before you start a batch of fits. A balance between min_credits and hold_credits does not 402: it runs truncated. See truncation. |
FORBIDDEN | 403 | The token is valid but not for this app, or a guest token tried a metered run. Mint the token against posterior-desk, and sign in for a metered lane — a review is metered, so a guest gets a 403 on /run rather than a 402. |
NOT_FOUND | 404 | An unknown job_id, or a slug in the /guest body that does not exist. Check for a typo in slug; posterior and posterior-desk are not the same app. |
RATE_LIMITED | 429 | Too many requests. Back off and retry with the same Idempotency-Key; a tight retry loop with fresh keys is how a batch bills four times for one fit report. |
INTERNAL | 500 | A server-side failure. Retry with the SAME Idempotency-Key so a half-finished run is not billed twice. If it repeats on one fit and not on others, shorten fit or send fewer table rows — a very long paste with a three-thousand-row summary table is the usual trigger. |
The two failure modes with no error code are worth more attention than the seven above. One is the
wrapped input key. The other is a reply that looks complete and quietly dropped a
blocking fact you handed it in prescan_facts — which is exactly what the
reconciliation contract exists to make checkable.
The task field, before anything else
This app is three reviews behind one endpoint. task chooses which one you get, it is
always present in a well-formed request, and it changes the shape of body in the reply.
Everything else in the input — the fit report, the summary table, the goal, the stage, the notes, the
prescan facts — is identical across all three. Send the same input three times with three different
task values and you get three documents about one run.
| # | task | name | the question it answers | what body carries |
|---|---|---|---|---|
| 1 | diagnose | Convergence & sampler health | Is the posterior trustworthy enough to report anything from? Reads the summary table and the warning block together and says which quantities are reportable and which are not. | sampler_review, convergence, geometry, efficiency, precision, trust_scope |
| 2 | respec | What to change before the next run | Why is it failing, and what do I change? Names the root cause and prescribes the reparameterisation, the prior and the sampler changes, in an order with stopping conditions. | root_cause, model_changes, prior_changes, sampler_changes, run_plan, cost_note, if_it_does_not_work |
| 3 | report | Write it up honestly | What can I actually write down? Drafts the methods paragraph, the diagnostics reporting table, the results wording with the hedges it needs, the limitations and the reproducibility list. | methods_paragraph, diagnostics_table, results_wording, limitations, reproducibility, open_items |
The three lanes are a pipeline, and they are worth running in this order, because each one can
invalidate the next. A chain with an r_hat of 1.31 makes every posterior mean in
report unquotable; a respec plan written before the geometry has been
diagnosed prescribes non-centring for a model whose actual problem is an improper flat prior on a
scale parameter; and report is worth writing only once diagnose has stopped
finding things. Run diagnose on the first fit, respec when it says
revise, and report when it does not.
The lanes are not interchangeable and they are not additive. A reply never blends two lanes'
body shapes — a merged body fails to render — and the lane's own question decides where
a fact lands. One example, the one that comes up most: 216 divergent transitions in a
four-level hierarchical model with a centred parameterisation is a
geometry block plus a trust_scope.not_reportable entry in
diagnose; it is the root_cause itself in respec, with
non-centring as model_changes[0] and target_accept as a
secondary sampler_changes row; and in report it is one
diagnostics_table row, one limitations sentence, and a
hedge_required: true on every claim about the group-level scale. One fact, three
places, and it must be reported at the same severity in all three. Let the lane decide where it
goes; never let it decide how bad it is.
If task is absent or unrecognised the run does not fail. The model
picks the lane the input best fits — a bare summary table with a warning block and no question is
diagnose, a fit report that already carries a diagnosis and asks what to change is
respec, a clean fit with a manuscript deadline in notes is
report — sets lane to whatever it chose, and says so in the first sentence
of summary. What it never does is blend two contracts to cover itself. That fallback
exists so a malformed request still returns something useful, not so you can skip the field: read
lane from the reply before you read body, and send the lane.
A note on what the three lanes share, since it is the reason one request shape is enough. All three
read the same two artefacts and neither is optional in practice: the fit report tells you what was
asked for — the model, the sampler settings, the warnings PyMC printed — and the summary
table tells you what came back, per parameter. Almost every interesting finding in this app
is a disagreement between those two. A fit report that says "chains converged (all r_hat < 1.01)"
next to a table with an r_hat of 1.31 is not a table problem or a prose problem; it is
the finding. Send both.
The request body
Every field is top-level. task and fit are required; everything else is
optional and absent means absent — nothing is defaulted on your behalf, and nothing in the reply may
assume a field you did not send.
{
"task": "diagnose",
"fit": "Model: eight_schools\nSampler: NUTS\nChains: 4\n...",
"table": "param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat\nmu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"goal": "publication",
"stage": "first_fit",
"notes": "reviewer 2 asked about the funnel",
"prescan_facts": { }
}
| field | type | meaning |
|---|---|---|
task | string, required | The lane: diagnose, respec or report. See above. Absent or unrecognised does not fail — the model picks the closest lane and names it in lane — but it is not a field to leave out on purpose. |
fit | string, required | The fit report as text: whatever the run printed plus whatever you know about the model. Labelled lines (Divergences: 216, Chains: 4, target_accept: 0.8), prose, or both — the two are read by different parts of the same pass, so a labelled sampler block, then the warning block verbatim, then three sentences about the model is the best input this app takes. Clipped client-side to 45000 characters, middle-out on line boundaries, with the cut announced in-band, because a fit report carries its identity and its settings at the top and its warnings and conclusions at the bottom and neither end is safe to drop. When the marker is present the reply says in summary that the middle was not read, and the checks that depended on it go to unassessable. |
table | string, optional — but it is what turns on every per-parameter check | The posterior summary table. The browser accepts az.summary().to_string(), az.summary().to_csv(), a markdown pipe table, plain TSV and Stan-style columns, then re-serialises it tab-separated with the columns param mean sd hdi_lo hdi_hi mcse_mean mcse_sd ess_bulk ess_tail r_hat. Header tokens are matched whole, so ess_bulk, n_eff, neff and ESS bulk are one column, and any hdi_3%/hdi_97% pair becomes hdi_lo/hdi_hi whatever the credible mass. At most 120 rows are sent, chosen by a golden-ratio draw across the whole table with at least one row per parameter family and the extreme row of every check forced in, so the worst r_hat and the lowest ess_bulk are always among them even in a three-thousand-parameter model. Send it in the same format your tooling already prints; do not pre-normalise it. |
goal | string | What the output is for: publication, internal_review, decision, teaching or self_check. It sets register and emphasis, never severity. A publication run writes text you can paste and is strict about what may be quoted to how many digits; a decision run leads with what the posterior does and does not license you to decide; teaching explains why each diagnostic means what it means; self_check is blunt and short. |
stage | string | Where the run is in the workflow: first_fit, after_respec, final_check or reviewer_response. It sets the bar, not the findings. 47 divergences on a first_fit is expected and instructive; the same 47 at final_check is a blocking objection, and at after_respec it means the prescription did not work and the reply says which of the changes clearly did not take. |
notes | string, optional, clipped to 6000 characters | Anything the fit report does not say — the reviewer comment you are answering, the deadline, the compute you have, a constraint you cannot change, a prior you are not allowed to touch. Often the most decisive field in the whole input, and it is not decorative: every claim in it comes back as one context_notes entry marked honoured, contradicted or unverifiable. A stated constraint that nothing in the diagnosis reflects is itself a finding. |
prescan_facts | object, optional | What the browser's own free reader already computed, passed in so the lane is held to it: {coverage, sampler, diagnostics, model, table_present, verdict, severity_counts, severity_by_area, flags, unassessable}. flags is a contract; the rest is arithmetic the reply must not contradict. See below. |
Two absences that are not the same thing, and the reply is held to the difference. No
table at all means the per-parameter checks were never run — the reply says "no summary
table was supplied", never "the diagnostics are fine". And in the fit report, a quantity the author
explicitly says was not produced (Divergences: none reported by this backend) is a
confirmed absence, which is worse than a quantity the report simply never reaches: the first
is established, the second is merely unaddressed. Never flatten those two into "missing". The same
distinction is what the report lane's source column exists to carry:
computed, reported and not established are three different
provenances and only the first two can go in a paper.
prescan_facts, and the reconciliation contract
In the browser, prescan_facts comes from the free in-page read that runs before anyone
signs in — the pass described under what the browser does before you pay. It
resolves thirty-seven checklist items to one of three states, recomputes the divergence rate against
the total post-warm-up draw count, checks every row of the summary table against itself, compares the
declared array shapes against the rows the table actually carries, and names every place the prose
contradicts the table. It costs nothing and it never calls a model.
An API caller does not have to reproduce any of that. Sending no prescan_facts at all is
legitimate, and the review still works — the model reads fit and table
either way.
What makes it worth sending is the contract. Every uid you put in
flags comes back exactly once in the response's reconciliation array — no
more, no fewer, and no uid you did not send — each with a
status:
status | means |
|---|---|
confirmed | The reviewer agrees, at the same or a higher severity. A useful confirmed adds the consequence the flag itself does not state — not "yes, r_hat for tau is 1.31" but "1.31 on the group-level scale means the four chains are exploring different regions of the funnel, so the shrinkage every school-level estimate depends on is not identified". |
adjusted | Real, but the severity or the reading changes — and the note says what changed it. 216 divergences is blocking in a centred hierarchical model whose scale parameter is the target of inference, and medium in a model where they are concentrated in one weakly identified nuisance parameter that nothing downstream reads. |
set_aside | Not a problem here, with the reason that makes it harmless. A set_aside with no reason is worse than no entry at all. |
noted | Carried forward without a judgement, because the lane's question does not turn on it but the reader should still see it. An info-level flag about wall time in the respec lane, where it belongs in cost_note rather than in the diagnosis. |
not_applicable | The flag does not apply to this lane's question — a model-comparison flag in a diagnose run that never looked at a second model, a reporting-coverage roll-up in respec. |
That turns a fact your own tooling established into something the reply is held to. A
uid that never appears is a failed run, not a passing one. Asserting both directions in
your client is three lines and it catches the one failure this app cares most about: a fluent,
well-written review that quietly dropped the blocking fact you handed it. The check is written out in
the verification step.
uid is the stable name of the check itself, not a position, which is what makes it
comparable across runs and across models. The prefix says where it came from: S- for a
check made on the summary table (S-RHAT-BLOCK, S-ESS-BULK,
S-ESS-TAIL, S-MCSE, S-IV-REVERSED, S-INF),
T- for a check made on the fit report's own numbers (T-ONECHAIN,
T-DIV, T-DIV-MISSING, T-TREEDEPTH,
T-UNDERTUNED, T-CORE-MISSING), and X- for a cross-check
between the two (X-RHAT-CLAIM, X-CONVERGED-CLAIM,
X-DIV-CONTRADICT, X-FUNNEL, X-SHAPE-SHORT,
X-COMPARE-UNCONVERGED). The X- family is the one worth watching: those are
the flags that exist only because you sent both artefacts.
The rest of prescan_facts is arithmetic the reply must not contradict:
| key | shape | what it is for |
|---|---|---|
coverage | {answered, total, stated_none[], never_mentioned[]} | The checklist roll-up. total is the fixed number of items the free read looks for; answered is how many the report resolved; the two arrays are the items the report explicitly said there were none of and the items it never mentioned — and those are different findings, which is why they are different arrays rather than one "missing" list. |
sampler | {chains, draws_per_chain, tune, total_post_warmup_draws, target_accept, max_treedepth, wall_time_seconds} | The settings, already parsed. total_post_warmup_draws is chains × draws_per_chain and it is the denominator every rate in diagnostics is computed against. A null means the report did not say — never zero, because a zero draw count makes every rate either infinite or undefined. |
diagnostics | {divergences, divergence_rate, treedepth_hits, warmup_to_draws_ratio, worst_r_hat, worst_r_hat_param, lowest_ess_bulk, lowest_ess_param} | The numbers a convergence argument actually turns on. divergence_rate is divergences / total_post_warmup_draws, rounded to six places; warmup_to_draws_ratio is tune / draws_per_chain, rounded to three. The worst_ and lowest_ pairs each carry the parameter name beside the value so the reply can name it without guessing. |
model | {name, observations, declared_free_parameters, summary_table_rows, declared_array_elements, declared_shapes[{family, declared}]} | The shape audit. declared_array_elements is what the declared shapes add up to; summary_table_rows is what the table carries. When the first exceeds the second, the table is a subset and every "all parameters converged" claim in the report is unsupported for the ones that are not there. |
table_present | boolean | Whether a table was parsed at all. When it is false, table_not_read_because carries the reason — no table pasted, or a header with none of the recognised columns — and every per-parameter check lands in unassessable instead of in flags. |
verdict | "sound" | "sound_with_caveats" | "revise" | "not_supported" | What the free read concluded from its own flags alone. The model's verdict may differ, and when it is more favourable the summary has to say what justified the move. |
severity_counts | {blocking, high, medium, low, info} | How many flags of each severity. The cheapest cross-check there is: a reply whose verdict is sound against a non-zero blocking count has contradicted the input, not the model. |
severity_by_area | {area: {blocking, high, medium, low, info}} | Severity crossed with area, because the verdict floors are per-lane and per-area: a blocking convergence fact binds the diagnose lane's verdict, and a blocking reporting fact binds the report lane's. Shipping the cross-tabulation rather than prose means the rule is a lookup against this object and cannot contradict it. |
flags | [{uid, severity, area, title, evidence, line}] | The contract above. severity is one of blocking, high, medium, low, info; area is one of the nine areas listed with the output contract. evidence is the source text, clipped to 200 characters; line is a line number in fit or null. |
unassessable | string[] | Checks the free read could not make, as plain strings. These are not flags and not part of the reconciliation contract; the reply is expected to carry them forward into its own unassessable — which is an array of {item, why} objects, so the string becomes the item — rather than pretend the check was made. |
"prescan_facts": {
"coverage": { "answered": 24, "total": 37,
"stated_none": ["prior_predictive"],
"never_mentioned": ["seed", "loo", "energy_bfmi", "posterior_predictive"] },
"sampler": { "chains": 4, "draws_per_chain": 1000, "tune": 500,
"total_post_warmup_draws": 4000, "target_accept": 0.8,
"max_treedepth": 10, "wall_time_seconds": 412 },
"diagnostics": { "divergences": 216, "divergence_rate": 0.054, "treedepth_hits": 38,
"warmup_to_draws_ratio": 0.5, "worst_r_hat": 1.31,
"worst_r_hat_param": "tau", "lowest_ess_bulk": 61,
"lowest_ess_param": "tau" },
"model": { "name": "eight_schools_centred", "observations": 8,
"declared_free_parameters": 10, "summary_table_rows": 10,
"declared_array_elements": 10,
"declared_shapes": [{ "family": "theta", "declared": 8 }] },
"table_present": true,
"table_not_read_because": null,
"verdict": "not_supported",
"severity_counts": { "blocking": 1, "high": 3, "medium": 2, "low": 1, "info": 0 },
"severity_by_area": {
"convergence": { "blocking": 1, "high": 2, "medium": 0, "low": 0, "info": 0 },
"geometry": { "blocking": 0, "high": 1, "medium": 0, "low": 0, "info": 0 },
"efficiency": { "blocking": 0, "high": 0, "medium": 2, "low": 0, "info": 0 },
"reporting": { "blocking": 0, "high": 0, "medium": 0, "low": 1, "info": 0 }
},
"flags": [
{ "uid": "S-RHAT-BLOCK", "severity": "blocking", "area": "convergence",
"title": "r_hat 1.31 for tau", "line": 22,
"evidence": "tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31" },
{ "uid": "X-RHAT-CLAIM", "severity": "high", "area": "convergence",
"title": "The report claims all r_hat < 1.01; the table says 1.31", "line": 9,
"evidence": "Convergence: all chains converged, r_hat < 1.01" },
{ "uid": "X-FUNNEL", "severity": "high", "area": "geometry",
"title": "Centred hierarchical parameterisation with divergences", "line": 14,
"evidence": "theta = mu + tau * pm.Normal('theta_raw', 0, 1, shape=8) # centred" },
{ "uid": "S-ESS-BULK", "severity": "medium", "area": "efficiency",
"title": "ess_bulk 61 for tau, below 400", "line": 22, "evidence": "ess_bulk 61" }
],
"unassessable": [
"Energy / BFMI was not reported, so the funnel cannot be confirmed from the sampler side",
"No LOO or WAIC was reported, so no model comparison check was made"
]
}
One asymmetry worth knowing before you build on this. The prescan's flags are things it
could prove from the two artefacts — arithmetic, a threshold, a contradiction between two
strings. It never flags a judgement. So a prescan with an empty flags array and a
verdict of sound is not a clean bill of health; it means nothing
checkable was wrong, which is precisely when a lane is worth paying for.
The output contract
data.output.output is a string holding one JSON object — no preamble, no code fence, no
prose outside it. The web app still takes everything from the first { to the last
} before parsing, and a caller should do the same: it costs one slice and it survives the
small variations a model produces.
Ten keys, identical in all three lanes except body:
{
"lane": "diagnose | respec | report",
"title": "short name for this review, naming the model where the report names it",
"verdict": "sound | sound_with_caveats | revise | not_supported",
"headline": "one sentence naming the single fact that decides the verdict",
"summary": "two to five sentences a modeller can act on. No restating of the JSON.",
"body": { },
"findings": [
{
"id": "F-001",
"severity": "blocking | high | medium | low | info",
"area": "sampler | convergence | efficiency | precision | geometry | interval | model | comparison | reporting",
"title": "one line",
"detail": "what is wrong, what it does to the claim, and what makes it this severity and not another",
"evidence": "the exact line, warning string or table cell this rests on",
"line": 22,
"fix": "the concrete change, with the actual number or setting named"
}
],
"reconciliation": [
{ "flag_uid": "S-RHAT-BLOCK",
"status": "confirmed | adjusted | set_aside | noted | not_applicable",
"note": "why" }
],
"caveats": [
{ "from_lane": "diagnose",
"fact": "a fact established in an earlier lane that constrains this one",
"why_it_matters": "what it forbids saying, or forces hedging, here" }
],
"context_notes": [
{ "claim": "what you said in notes",
"status": "honoured | contradicted | unverifiable",
"note": "what the fit report and the table actually support" }
],
"unassessable": [
{ "item": "the check that could not be made", "why": "what was missing from the input" }
]
}
| key | type | meaning |
|---|---|---|
lane | enum | The lane that actually ran. Normally it echoes task; when task was absent or unrecognised it is the lane that was chosen, and summary says so in its first sentence. Read this, not your own request, before you read body — a lane you did not ask for is also what a wrapped input key looks like from the outside. |
title | string | Short name for the review, naming the model where the fit report names it — eight_schools_centred, first fit rather than Convergence review. |
verdict | enum | One of four values, below. The single field a promotion gate should branch on. |
headline | string | One sentence naming the single fact that decides the verdict — not a summary of the findings, the one that swung it. "216 divergences and r_hat 1.31 on tau mean the group-level scale is not identified" is a headline; "several convergence issues were found" is not. |
summary | string | Two to five sentences, and not a restatement of the JSON. It is also where the exceptions are announced: a clipped fit, an absent task, a verdict that moved away from the prescan's, a divergence rate that is a lower bound because the draw count was not stated. |
body | object | The lane's own document. Three shapes, one per lane, never blended — a merged body fails to render. Documented lane by lane below. |
findings | object[] | {id, severity, area, title, detail, evidence, line, fix}. Ids are F-001, F-002, … in the order reported — note that these are the reply's ids and they are not the prescan's uids. May be empty, and an empty array is a real answer; no placeholder finding is ever emitted to fill it. line is copied from prescan_facts or from the visible fit report, never estimated, and null is correct when unknown. |
reconciliation | object[] | {flag_uid, status, note}. One entry per uid you sent in prescan_facts.flags, exactly once, no more and no fewer. Empty when you sent no flags. This is the contract worth asserting. |
caveats | object[] | {from_lane, fact, why_it_matters} — the facts from an earlier lane that constrain this one. This is how the pipeline stays honest across three calls: a report run whose diagnose pass found r_hat 1.31 carries that as a caveat and every affected claim in results_wording comes back with hedge_required: true. Populated from prescan_facts and from what the current input establishes; empty is legitimate on a clean first fit, and empty against a not_supported prescan is a defect. |
context_notes | object[] | {claim, status, note}, one entry per claim in notes. Send no notes and this is empty; send three claims and expect three entries. An empty context_notes against a paragraph of notes means the most decisive field in the input was not read. |
unassessable | object[] | {item, why} — the checks that genuinely could not be made from what was sent. An honest entry here is preferred to a confident guess, and the prescan's own unassessable strings are carried forward into it rather than dropped. |
The enums
These strings are shared verbatim with the browser's own free reader, so the two never disagree about what a clean result is called. The renderer keys on them: an unrecognised value renders as an error rather than being coerced to something plausible, so treat them as closed sets.
verdict | when |
|---|---|
sound | Nothing above info is left. The chains agree, the tails are sampled, the intervals are defensible to the digits being quoted, and the lane's question is answered. |
sound_with_caveats | The worst finding is medium or low. The posterior is usable; read the caveats before you quote a number to three decimal places. |
revise | The worst finding is high. Something needs a new run or a changed model before it is reportable — not a wording change. This is the verdict that should send you to the respec lane. |
not_supported | At least one finding is blocking: the posterior does not support the claims being made from it. One chain, an r_hat that fails outright, a table of NaNs, a comparison run over unconverged fits. Nothing from this fit goes in a paper until it is rerun. |
The verdict follows the findings, mechanically, and never contradicts them. That makes two cheap
assertions available to any client: a not_supported verdict with no blocking
finding is a broken reply, and so is a sound verdict with a finding above
info. Both are worth failing on rather than rendering.
severity | meaning |
|---|---|
blocking | A stated claim is not supported by what the run itself reports. A single chain (there is no between-chain statistic to compute, so r_hat is not a diagnostic at all), an r_hat at or above 1.05 on a parameter the conclusion depends on, an ess_bulk in the tens behind a quoted three-decimal mean, a summary table whose r_hat column is NaN, or a LOO comparison over fits that did not converge. |
high | The result may stand and cannot currently be trusted, or a referee will demand a rerun: divergences above a per-mille or so in a centred hierarchical model, a report that claims convergence the table contradicts, a funnel geometry named nowhere and visible in the settings, ess_tail far below ess_bulk behind a quoted 94% interval. |
medium | Real, bounded or mitigated — worth fixing before the next run rather than before the next paper. A short warm-up, treedepth saturation at the default 10, an ess_bulk under 400 on a nuisance parameter, an mcse that quietly forbids the third decimal place. |
low | Worth naming, not worth holding the run for. Two chains rather than four, a thinned trace, an unreported seed. |
info | Context the reader should have. Zero divergences reported and confirmed against a stated draw count is info, and it is worth saying out loud. info alone still permits sound. |
Severity depends on the mitigating facts, and detail names the mitigation whenever the
grade moved because of it. 47 divergences at stage: "first_fit" is instructive; the same
47 at final_check is blocking. An ess_bulk of 61 is blocking
when the parameter is the quantity of interest and medium when it is a nuisance scale
nothing downstream reads. If you diff two reviews of the same fit and a severity moved, the reason is
in detail.
area | covers |
|---|---|
sampler | The settings themselves: chains, draws, warm-up, target_accept, max_treedepth, the initialisation, the backend, the seed, wall time. |
convergence | r_hat in every form (split, rank-normalised, folded), between-chain disagreement, chain count, whether a between-chain statistic exists at all. |
efficiency | ess_bulk, ess_tail, effective sample size per second, thinning, treedepth saturation, antithetic chains (an ESS above the draw count). |
precision | mcse_mean and mcse_sd against the digits actually being quoted. This is the area that most often turns a clean fit into a hedged sentence. |
geometry | Divergent transitions, funnels, centred versus non-centred parameterisation, energy and BFMI, multimodality, hard boundaries, unidentified scales. |
interval | The HDI or credible interval columns: ordering, width, whether the mean falls inside them, whether an interval is flat or degenerate, what mass it actually represents. |
model | The specification: priors and their propriety, likelihood, link functions, declared shapes against the table's rows, the parameter count, transformations, observed data size. |
comparison | LOO, WAIC, ELPD and their standard errors, Pareto k, stacking weights, and whether the compared fits converged in the first place. |
reporting | What was written down: the sampler settings, the diagnostics, the seed, the software versions, the posterior predictive check, the code and data availability — and every claim in the prose the table does not support. |
Nine areas, and no synonyms: diagnostics, mcmc, priors,
performance and statistics are not values. If you bucket findings for a
dashboard, bucket on these nine. The pairing worth internalising is
convergence/efficiency/precision: they are three different
questions about the same chains — do they agree, how much independent information did they produce,
and how many digits does that buy — and collapsing them is how a report ends up quoting a mean to
three decimals off 61 effective draws.
body for task: "diagnose"
Whether the posterior is trustworthy enough to report anything from. Six blocks, and they are ordered the way a reviewer reads a fit: what was asked of the sampler, whether the chains agree, what the geometry did to them, how much independent information came out, how many digits that buys, and finally — the block that matters most to a caller — what may and may not be quoted.
convergence.worst_parameters is not a top-N list for display; it is the evidence the
verdict rests on, one row per parameter that actually failed a threshold, each naming the
statistic and the value so the reading can be checked against
the table you sent. geometry.pattern is the named shape of the problem — a funnel, a
boundary, multimodality, a plateau — and null when the divergences are present but the
pattern is not identifiable from the report, which is a legitimate answer and better than a guess.
precision.digits_defensible is an integer and it is the field that turns a fit into a
sentence: it is how many decimal places mcse_mean supports on the reported means, and it
is frequently one or two fewer than the table prints.
trust_scope is the block to build tooling on. reportable and
not_reportable are arrays of quantities, named as the report names them, and they
partition what the fit produced — a quantity in neither is a bug, and a quantity in both is a broken
reply. why carries the one-paragraph reason the partition falls where it does.
"body": {
"sampler_review": {
"assessment": "what was asked of the sampler, in one or two sentences",
"settings_judgement": "adequate | thin | mismatched | unstated",
"issues": [
"target_accept left at 0.8 for a centred hierarchical model",
"500 warm-up draws against 1000 sampling draws is short for this geometry"
]
},
"convergence": {
"assessment": "whether the chains agree, and on what",
"worst_parameters": [
{ "param": "tau", "statistic": "r_hat", "value": 1.31,
"reading": "the four chains are exploring different regions of the scale" },
{ "param": "theta[3]", "statistic": "r_hat", "value": 1.08,
"reading": "inherited from tau through the centred parameterisation" }
],
"verdict_basis": "the single statistic and parameter the verdict rests on"
},
"geometry": {
"assessment": "what the divergences say about the posterior's shape",
"pattern": "funnel | boundary | multimodal | plateau | heavy_tail | none | null",
"implication": "what that shape does to the estimates that depend on it"
},
"efficiency": {
"assessment": "how much independent information the run produced",
"bulk": "the ess_bulk reading, with the parameter and the number",
"tail": "the ess_tail reading, and what it does to the interval endpoints"
},
"precision": {
"assessment": "what mcse_mean and mcse_sd permit",
"digits_defensible": 1
},
"trust_scope": {
"reportable": ["the observation-level predictions", "mu to one decimal place"],
"not_reportable": ["tau and any interval on it",
"every theta[i] shrinkage statement",
"the 94% HDI endpoints"],
"why": "one paragraph on where the line falls and why"
}
}
Two things the diagnose lane will not do. It will not tell you the model is wrong — a
perfectly converged fit of a badly specified model is sound here, and that is the honest
answer to the question this lane asks. And it will not compute a diagnostic you did not send: no
ess_tail column means efficiency.tail says so and an
unassessable entry appears, rather than an inference from ess_bulk.
body for task: "respec"
What to change before the next run, in an order you can execute. root_cause is one
diagnosis, not a list — the discipline of the lane is that it commits, and confidence
says how firmly. A low confidence root cause with a first run-plan step that
discriminates between two candidates is a good answer; three simultaneous diagnoses is not.
The three change arrays are separate because they are different kinds of act with different costs.
model_changes rewrites the model — non-centring, a sum-to-zero constraint, a
reparameterised scale — and before/after carry the actual code lines where
the report gave them, so the change is applied rather than interpreted. prior_changes
moves a prior and always names the current one, including when the current one is
"unstated", because an unstated prior is itself the finding. sampler_changes is the
cheapest and the least likely to fix anything structural: raising target_accept to 0.95
on a funnel buys smaller steps through a geometry that should not be there, and the lane says so
rather than leading with it.
run_plan is ordered and each step carries a stop_if — the condition under
which you stop and rethink instead of continuing down the list. That is what makes the plan runnable
by something other than a person. cost_note is the honest wall-clock and
draw-count implication of the plan, and if_it_does_not_work is the fallback: the next
hypothesis, or the admission that the data may not identify the parameter at all.
"body": {
"root_cause": {
"diagnosis": "one committed diagnosis, named as a mechanism",
"evidence": "the lines and cells it rests on",
"confidence": "high | medium | low"
},
"model_changes": [
{
"change": "non-centre the school-level effects",
"rationale": "removes the tau-theta dependence that produces the funnel",
"before": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8)",
"after": "theta_raw = pm.Normal('theta_raw', 0, 1, shape=8)\ntheta = pm.Deterministic('theta', mu + tau * theta_raw)",
"expected_effect": "divergences to zero or near it; ess_bulk on tau up by an order of magnitude"
}
],
"prior_changes": [
{
"parameter": "tau",
"current": "HalfCauchy(5) — stated on line 15",
"proposed": "HalfNormal(5), or HalfStudentT(4, 5) if the tail matters",
"rationale": "with 8 groups the HalfCauchy tail is not identified by the data and the chain spends its time out there"
}
],
"sampler_changes": [
{
"setting": "tune",
"current": "500",
"proposed": "1500",
"rationale": "the step size and mass matrix are still adapting when sampling starts"
},
{
"setting": "target_accept",
"current": "0.8",
"proposed": "0.9 — secondary to the reparameterisation, not a substitute for it",
"rationale": "smaller steps survive what curvature remains after non-centring"
}
],
"run_plan": [
{ "step": 1, "action": "non-centre theta, keep everything else identical, rerun",
"stop_if": "divergences do not fall below 10 — the funnel is not the whole story" },
{ "step": 2, "action": "raise tune to 1500 and target_accept to 0.9",
"stop_if": "ess_bulk on tau is still under 400 after 4000 draws" },
{ "step": 3, "action": "swap HalfCauchy(5) for HalfNormal(5) and compare the tau posterior",
"stop_if": "the tau posterior moves materially — then the prior was doing the work" }
],
"cost_note": "steps 1-2 are roughly 3x the original wall time at 412 s; step 3 is another full run",
"if_it_does_not_work": "if divergences persist after non-centring and a longer warm-up, the next hypothesis is the likelihood, not the geometry: 8 observations may not identify a group-level scale at all, and the honest move is to report the partially pooled estimates with the scale fixed by a prior you defend."
}
body for task: "report"
What can actually be written down. This is the lane with the strongest provenance discipline, and
diagnostics_table[].source is where it lives: computed means the number was
derived here from what you sent (a divergence rate from a count and a draw total),
reported means the fit report stated it and the value is quoted as given, and
not established means the quantity belongs in the table and nothing in the input
establishes it. That third value is not a gap to be filled in later — it is the row, and it prints as
"not reported" in the drafted table so the omission is visible rather than invisible.
results_wording is one entry per claim you might want to make, each with the wording that
is defensible and a boolean hedge_required. The wording is written to be pasted: it names
the estimate, the interval and the mass, and it says "posterior mean" rather than "estimate" where the
distinction matters. When hedge_required is true the hedge is already in the
wording string — the flag is there so a client can refuse to auto-insert the sentence
into a draft without a human seeing it, not so it can strip the hedge.
reproducibility splits into what the report already states, what is missing, and the
minimum_to_add — the shortest list that makes the run reproducible, which is usually the
seed, the versions and the sampler settings, in that order of how often they are forgotten.
open_items carries a who because the useful ones are not all yours: "confirm
the 94% mass is what the journal expects" belongs to a co-author, not to the sampler.
"body": {
"methods_paragraph": "A paragraph that names the model, the priors, the sampler, the settings and the diagnostics, in the register the goal asked for. Every number in it came from the input; nothing is filled in from convention.",
"diagnostics_table": [
{ "quantity": "chains", "value": "4", "source": "reported" },
{ "quantity": "draws after warm-up (per chain)", "value": "1000", "source": "reported" },
{ "quantity": "warm-up draws (per chain)", "value": "500", "source": "reported" },
{ "quantity": "total post-warm-up draws", "value": "4000", "source": "computed" },
{ "quantity": "divergent transitions", "value": "216 (5.4% of draws)", "source": "computed" },
{ "quantity": "max r_hat", "value": "1.31 (tau)", "source": "reported" },
{ "quantity": "min ess_bulk", "value": "61 (tau)", "source": "reported" },
{ "quantity": "min ess_tail", "value": "92 (tau)", "source": "reported" },
{ "quantity": "energy / BFMI", "value": "not reported", "source": "not established" },
{ "quantity": "random seed", "value": "not reported", "source": "not established" }
],
"results_wording": [
{ "claim": "the population mean effect",
"wording": "The posterior mean of mu was 8.0 (94% HDI -1.0 to 18.0). Because the chains did not converge on the group-level scale, this interval should be treated as provisional.",
"hedge_required": true },
{ "claim": "between-school variation",
"wording": "The group-level scale tau is not identified by this fit (r_hat 1.31, ess_bulk 61) and no estimate of it is reported.",
"hedge_required": true }
],
"limitations": "A paragraph naming what this fit cannot support, in the order a referee will raise it.",
"reproducibility": {
"stated": ["model code", "sampler (NUTS)", "chains, draws and warm-up", "target_accept"],
"missing": ["random seed", "PyMC and PyTensor versions", "the data extract or its checksum",
"the ArviZ version the summary came from"],
"minimum_to_add": "the seed, the PyMC version, and one line saying which az.summary() call produced the table"
},
"open_items": [
{ "item": "rerun non-centred before this paragraph is submitted", "who": "analyst" },
{ "item": "confirm the journal expects a 94% interval rather than 95%", "who": "corresponding author" }
]
}
The report lane is the one people are most tempted to run first, and the one that punishes
it. It will not launder a bad fit: a not_supported diagnose pass turns every
substantive row of results_wording into a hedge and fills limitations with
the reasons, which is the correct output and not a useful manuscript. Run it when
diagnose has stopped objecting.
A worked diagnose reply, in full
This is the reply to the diagnose request in step 6 — the classic
centred eight-schools fit, four chains, 216 divergences, r_hat 1.31 on tau,
and a fit report whose prose says the chains converged. It is shown whole, including the parts a
renderer usually hides, because the relationships between the keys are the contract: four
reconciliation entries for four prescan uids, a verdict of
not_supported justified by exactly one blocking finding, and a
trust_scope that partitions the fit's quantities rather than summarising them.
{
"lane": "diagnose",
"title": "eight_schools_centred — first fit, 4 chains",
"verdict": "not_supported",
"headline": "r_hat 1.31 and ess_bulk 61 on tau mean the group-level scale is not identified, so nothing that depends on shrinkage can be reported from this fit.",
"summary": "The four chains disagree about tau, the parameter every school-level estimate is shrunk towards, and 216 divergent transitions (5.4% of post-warm-up draws) in a centred parameterisation say why: the sampler cannot move through the neck of the funnel where tau approaches zero. The fit report's claim that all r_hat were below 1.01 is contradicted by the table it was written beside. mu is stable enough to quote to one decimal place with the interval hedged; tau, every theta[i], and all 94% HDI endpoints are not reportable. This is a first fit of a model whose geometry has a known fix, so the next step is the respec lane, not a longer run.",
"body": {
"sampler_review": {
"assessment": "Four chains, 1000 draws each after 500 warm-up, NUTS at the default target_accept of 0.8 and max_treedepth 10. The chain count is right and the settings are defaults that have not been matched to the model: nothing here was tuned for a hierarchical scale.",
"settings_judgement": "mismatched",
"issues": [
"target_accept left at 0.8 for a centred hierarchical model, where the step size needed near the neck of the funnel is much smaller than the one 0.8 will accept",
"500 warm-up draws against 1000 sampling draws (ratio 0.5) is short: the mass matrix is still adapting when the sampling phase begins",
"38 draws saturated max_treedepth at 10, which is a symptom of the same curvature rather than an independent problem"
]
},
"convergence": {
"assessment": "The chains do not agree on tau and the disagreement propagates into the school-level effects through the centred parameterisation. mu itself is the least affected parameter in the table.",
"worst_parameters": [
{ "param": "tau", "statistic": "r_hat", "value": 1.31,
"reading": "the four chains are exploring different regions of the scale; there is no single posterior for tau here, only four partial ones" },
{ "param": "theta[3]", "statistic": "r_hat", "value": 1.08,
"reading": "inherited from tau: theta[3] is the school whose estimate depends most on the amount of shrinkage" },
{ "param": "tau", "statistic": "ess_bulk", "value": 61,
"reading": "61 effective draws out of 4000 is a 1.5% efficiency, and it is the same fact as the r_hat seen from the other side" }
],
"verdict_basis": "r_hat 1.31 on tau, against the 1.01 threshold the report itself claims to have met"
},
"geometry": {
"assessment": "216 divergent transitions, 5.4% of the 4000 post-warm-up draws, concentrated where tau is small. The model spec on line 14 builds theta directly from mu and tau, which is the centred parameterisation and the textbook source of this pattern.",
"pattern": "funnel",
"implication": "The neck of the funnel is where the strongly pooled solutions live, so the divergences are not a random sampling failure: they are the sampler failing precisely on the hypothesis that between-school variation is near zero. Any statement about how much pooling the data support is therefore unsupported in both directions."
},
"efficiency": {
"assessment": "Bulk efficiency is adequate for mu and poor for the scale. The run produced 4000 draws and roughly 61 independent pieces of information about tau.",
"bulk": "lowest ess_bulk 61 (tau); mu is at 452, which is above the 400 rule of thumb but not comfortably",
"tail": "lowest ess_tail 92 (tau); mu is at 653. The tail figure is what the 94% HDI endpoints rest on, so the interval on tau is not estimable and the interval on mu is wider in truth than the table's endpoints suggest."
},
"precision": {
"assessment": "mcse_mean is 0.24 for mu against a posterior mean of 8.017 — the third decimal place is noise and the second is not defensible either. For tau the mcse_mean of 0.42 against a mean of 3.574 makes the printed value meaningless independently of the r_hat.",
"digits_defensible": 1
},
"trust_scope": {
"reportable": [
"mu, to one decimal place, with the interval described as provisional",
"the sampler settings and the diagnostics themselves, which is what the report lane will need"
],
"not_reportable": [
"tau: any point estimate, any interval, any statement about its magnitude",
"every theta[i]: the shrinkage they encode is not identified",
"all 94% HDI endpoints, which rest on ess_tail figures in the low hundreds and below",
"any claim that the chains converged"
],
"why": "The line falls where the centred parameterisation couples a quantity to tau. mu is the population mean and survives because it is estimated largely from the group means directly; everything whose posterior is defined through tau inherits tau's non-convergence. This is a partition of what the fit produced, not a ranking: a quantity is on one side or the other."
}
},
"findings": [
{
"id": "F-001",
"severity": "blocking",
"area": "convergence",
"title": "r_hat 1.31 on tau",
"detail": "The four chains have not mixed on the group-level scale. 1.31 is far above any working threshold — 1.01 is the modern convention and 1.05 the loosest defensible one — and tau is not a nuisance parameter here: every theta[i] is defined through it, so the failure is not local. This is blocking rather than high because the report makes claims about between-school variation, and those claims rest entirely on the parameter that did not converge.",
"evidence": "tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"line": 22,
"fix": "Rerun non-centred. Do not attempt to fix this by drawing more samples: a longer run of a centred funnel produces the same r_hat with a smaller standard error on it."
},
{
"id": "F-002",
"severity": "high",
"area": "convergence",
"title": "The report claims a convergence the table contradicts",
"detail": "Line 9 states that all chains converged with r_hat below 1.01. The table on line 22 gives 1.31 for tau. Whichever is stale, the pair cannot both be current, and the version that reaches a reader is the prose one. This is high rather than blocking on its own terms — it is a reporting failure rather than a modelling one — but it is the finding most likely to survive into a manuscript unnoticed.",
"evidence": "Convergence: all chains converged, r_hat < 1.01",
"line": 9,
"fix": "Regenerate the summary and the prose from the same InferenceData object in the same session, and quote the max r_hat with the parameter it belongs to rather than a threshold."
},
{
"id": "F-003",
"severity": "high",
"area": "geometry",
"title": "216 divergences in a centred hierarchical parameterisation",
"detail": "5.4% of post-warm-up draws diverged, and the model builds theta from mu and tau directly. This is the canonical funnel: as tau approaches zero the posterior narrows faster than any fixed step size can follow. The rate matters less than the location — divergences clustered at small tau invalidate the pooled end of the scale specifically.",
"evidence": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8)",
"line": 14,
"fix": "Non-centre: sample theta_raw ~ Normal(0, 1) and set theta = mu + tau * theta_raw. Raise target_accept afterwards if anything remains, not before."
},
{
"id": "F-004",
"severity": "medium",
"area": "efficiency",
"title": "ess_bulk 61 for tau",
"detail": "Below 400 by more than a factor of six, and below the 100 mark at which the ESS estimate itself becomes unreliable. Graded medium rather than high only because F-001 already makes tau unreportable: as an independent finding on a converged chain this would be high.",
"evidence": "ess_bulk 61",
"line": 22,
"fix": "The reparameterisation in F-003 is the fix; expect ess_bulk on tau in the high hundreds or thousands afterwards."
},
{
"id": "F-005",
"severity": "medium",
"area": "precision",
"title": "mcse_mean forbids the printed digits",
"detail": "mu is printed as 8.017 with an mcse_mean of 0.24. The Monte Carlo error is two orders of magnitude larger than the last printed digit, so quoting 8.017 asserts a precision the run did not produce. This is separate from the convergence problem and would remain after a clean rerun of the same length.",
"evidence": "mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"line": 21,
"fix": "Report mu as 8.0. To defend a second decimal place, raise ess_bulk to the point where mcse_mean is below 0.005 — roughly a hundredfold increase in effective draws."
},
{
"id": "F-006",
"severity": "medium",
"area": "sampler",
"title": "Warm-up is half the sampling length",
"detail": "500 tune against 1000 draws. For a geometry this awkward the adaptation phase is the part that has to succeed, and a ratio of 0.5 means the step size and mass matrix reached sampling in whatever state 500 iterations left them.",
"evidence": "Chains: 4 | Draws: 1000 | Tune: 500",
"line": 4,
"fix": "tune=1500 with draws=1000 on the rerun. The extra warm-up costs less wall time than a second failed run."
},
{
"id": "F-007",
"severity": "low",
"area": "reporting",
"title": "No seed, no versions, no energy diagnostic",
"detail": "The report does not state a random seed, the PyMC or ArviZ versions, or any energy/BFMI figure. None of this changes the diagnosis; all of it would be needed to reproduce the run or to confirm the funnel from the sampler side rather than by inspecting the model code.",
"evidence": "",
"line": null,
"fix": "Record the seed and the versions in the same cell that produces the summary, and print az.bfmi(idata) beside it."
}
],
"reconciliation": [
{ "flag_uid": "S-RHAT-BLOCK", "status": "confirmed",
"note": "Confirmed at the same severity, and the consequence is the one the flag does not state: because theta is defined through tau, the non-convergence is not confined to one row of the table." },
{ "flag_uid": "X-RHAT-CLAIM", "status": "confirmed",
"note": "Confirmed. The prose and the table cannot both be current; the prose is the version a reader gets." },
{ "flag_uid": "X-FUNNEL", "status": "confirmed",
"note": "Confirmed, and it is the mechanism behind S-RHAT-BLOCK rather than a separate problem. Reported separately because the fix attaches here." },
{ "flag_uid": "S-ESS-BULK", "status": "adjusted",
"note": "Adjusted from the flag's own reading to medium: real, and already subsumed by the blocking r_hat finding, so it does not independently raise the verdict. On a converged chain the same number would be high." }
],
"caveats": [],
"context_notes": [
{ "claim": "reviewer 2 asked about the funnel",
"status": "honoured",
"note": "Addressed directly: geometry.pattern is funnel, F-003 names the parameterisation and the line, and the fix is the standard non-centring. The answer to reviewer 2 is that the funnel is present and unresolved in this fit." }
],
"unassessable": [
{ "item": "Energy / BFMI",
"why": "Not reported, so the funnel is identified from the model code and the divergence pattern rather than confirmed from the sampler's own energy distribution." },
{ "item": "Model comparison",
"why": "No LOO, WAIC or ELPD was reported and only one model was sent, so no comparison check was made." },
{ "item": "Posterior predictive check",
"why": "Not mentioned in the fit report. Nothing here speaks to whether the model fits the data, only to whether the sampler explored it." }
]
}
Read three things off that reply before anything else. lane is diagnose, so
the body is the one documented above and the request arrived flat. reconciliation has
four entries for the four uids that went in, which is the whole reason to send
prescan_facts. And verdict is not_supported with exactly one
blocking finding behind it, so the verdict and the findings agree. Everything else on the
page is easier to trust once those three hold.
Check the reply before you trust it
The browser does not render a reply verbatim and neither should a caller. Seven assertions cover everything this app can get wrong in a way that still looks plausible, and all seven are cheap:
- The lane is the one you asked for. Compare
laneagainst thetaskyou sent. A mismatch meanstaskdid not arrive — which is what a wrappedinputkey looks like from the outside, and it is the only symptom that failure has. - Reconciliation covers the prescan exactly. Every
uidyou sent appears once inreconciliation; nouidyou did not send appears at all. This catches a fluent review that dropped your blocking fact. - The verdict matches the worst severity.
not_supportedneeds ablockingfinding,reviseahighone, andsoundneeds nothing aboveinfo. A verdict its own findings contradict is a broken reply, not a judgement call. - The body keys belong to that lane. Three shapes, never blended. A body carrying both
trust_scopeandrun_planis malformed even though it parses. - The enums are in range.
verdict,severityandareaare closed sets; an unrecognised value renders as an error rather than being coerced to something plausible. trust_scopepartitions. In thediagnoselane, no quantity appears in bothreportableandnot_reportable. A quantity in both is the one contradiction a reader will not notice and a downstream gate will act on.truncatedis false. A truncated reply is a prefix, not a review. Retry; do not repair. See below.
One more, and it is the cheapest of all: context_notes is non-empty exactly when you sent
notes. The assertion code for all of these is in step 4, in all
eight languages.
0. A tiny client
One helper that adds the two headers, unwraps data and raises on ok: false.
Two headers is the whole story: Content-Type and Authorization. If you find
yourself reaching for X-App-Slug, the token already carries the app. Every later step on
this page uses this helper.
# Every call is the same three things: the base URL, your bearer token, and a
# JSON body. Keep the token in a shell variable.
BASE="https://api.skillsafe.ai/v1/app-api"
TOKEN="$SKILLSAFE_TOKEN" # from https://posterior-desk.skillsafe.ai/tokens.html
call() { # call <path> [json-body]
if [ -n "$2" ]; then
curl -sS -X POST "$BASE/$1" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-d "$2"
else
curl -sS "$BASE/$1" -H "Authorization: Bearer $TOKEN"
fi
}
# Unwrap the envelope: print data, or exit non-zero with the API error code.
data() {
python3 -c '
import sys, json
env = json.load(sys.stdin)
if not env.get("ok"):
err = env.get("error") or {}
raise SystemExit("%s: %s" % (err.get("code"), err.get("message")))
json.dump(env["data"], sys.stdout)
'
}
call me | data
# {"subject_type": "user", "subject_id": "usr_...", "credits": 51234}
#
# No X-App-Slug header. The slug was only ever needed by POST /guest.
import json, os, urllib.error, urllib.request
BASE = "https://api.skillsafe.ai/v1/app-api"
TOKEN = os.environ.get("SKILLSAFE_TOKEN", "YOUR_TOKEN") # from /tokens.html
LANES = ("diagnose", "respec", "report")
class ApiError(RuntimeError):
def __init__(self, code, message, details=None):
super().__init__(f"{code}: {message}")
self.code, self.message, self.details = code, message, details or {}
def call(path, body=None, headers=None):
"""Returns the unwrapped `data`, or raises ApiError with the API error code.
Two headers only: Content-Type and Authorization. There is no X-App-Slug -
the token is already bound to posterior-desk.
"""
payload = json.dumps(body).encode() if body is not None else None
req = urllib.request.Request(
f"{BASE}/{path}", data=payload, method="POST" if body is not None else "GET")
req.add_header("Authorization", f"Bearer {TOKEN}")
if payload is not None:
req.add_header("Content-Type", "application/json")
for k, v in (headers or {}).items():
req.add_header(k, v)
try:
with urllib.request.urlopen(req) as res:
env = json.load(res)
except urllib.error.HTTPError as exc: # 4xx and 5xx carry the envelope too
env = json.loads(exc.read() or b"{}")
if not env.get("ok"):
err = env.get("error") or {}
raise ApiError(err.get("code", "INTERNAL"), err.get("message", "no message"),
err.get("details"))
return env["data"]
print(call("me"))
# {'subject_type': 'user', 'subject_id': 'usr_...', 'credits': 51234}
// Node 18+ or any browser. Paste a token from
// https://posterior-desk.skillsafe.ai/tokens.html, or mint a guest one in step 1.
const BASE = "https://api.skillsafe.ai/v1/app-api";
const TOKEN = "YOUR_TOKEN";
const LANES = ["diagnose", "respec", "report"];
class ApiError extends Error {
constructor(code, message, details) {
super(`${code}: ${message}`);
this.code = code;
this.details = details ?? {};
}
}
// call("me") -> GET; call("estimate", input) -> POST with the input object as
// the whole body. Extra headers are for Idempotency-Key on a run.
async function call(path, body, extraHeaders = {}) {
const res = await fetch(`${BASE}/${path}`, {
method: body === undefined ? "GET" : "POST",
headers: {
Authorization: `Bearer ${TOKEN}`,
...(body === undefined ? {} : { "Content-Type": "application/json" }),
...extraHeaders,
},
body: body === undefined ? undefined : JSON.stringify(body),
});
const env = await res.json();
if (!env.ok) throw new ApiError(env.error.code, env.error.message, env.error.details);
return env.data;
}
console.log(await call("me"));
// { subject_type: 'user', subject_id: 'usr_...', credits: 51234 }
// Two headers, and no X-App-Slug: the token already carries the app.
package main
// Imports used across every Go sample on this page:
// bytes, crypto/sha256, encoding/json, fmt, io, net/http, os, strings, time
const base = "https://api.skillsafe.ai/v1/app-api"
// From https://posterior-desk.skillsafe.ai/tokens.html, or minted in step 1.
var token = func() string {
if t := os.Getenv("SKILLSAFE_TOKEN"); t != "" {
return t
}
return "YOUR_TOKEN"
}()
var lanes = []string{"diagnose", "respec", "report"}
type apiError struct {
Code string `json:"code"`
Message string `json:"message"`
Details json.RawMessage `json:"details"`
}
func (e *apiError) Error() string { return e.Code + ": " + e.Message }
type envelope struct {
OK bool `json:"ok"`
Data json.RawMessage `json:"data"`
Error *apiError `json:"error"`
}
// call returns the raw `data` for the caller to unmarshal into its own struct.
// Two headers only - there is no X-App-Slug in this API.
func call(path string, body any, extra map[string]string) (json.RawMessage, error) {
method := http.MethodGet
var reader io.Reader
if body != nil {
method = http.MethodPost
raw, err := json.Marshal(body)
if err != nil {
return nil, err
}
reader = bytes.NewReader(raw)
}
req, err := http.NewRequest(method, base+"/"+path, reader)
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+token)
if body != nil {
req.Header.Set("Content-Type", "application/json")
}
for k, v := range extra {
req.Header.Set(k, v)
}
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
var env envelope
if err := json.NewDecoder(res.Body).Decode(&env); err != nil {
return nil, err
}
if !env.OK {
if env.Error != nil {
return nil, env.Error
}
return nil, fmt.Errorf("INTERNAL: no error body on HTTP %d", res.StatusCode)
}
return env.Data, nil
}
// java.net.http, single file. Imports: java.net.URI, java.net.http.*,
// java.security.MessageDigest, java.util.Map.
public final class PosteriorDesk {
static final String BASE = "https://api.skillsafe.ai/v1/app-api";
// From https://posterior-desk.skillsafe.ai/tokens.html, or minted in step 1.
static String token = System.getenv("SKILLSAFE_TOKEN") == null
? "YOUR_TOKEN" : System.getenv("SKILLSAFE_TOKEN");
static final HttpClient HTTP = HttpClient.newHttpClient();
static class ApiError extends RuntimeException {
final String code;
ApiError(String code, String message) { super(code + ": " + message); this.code = code; }
}
/** GET when body is null, POST otherwise. Returns the raw response text.
* Two headers: Content-Type and Authorization. There is no X-App-Slug. */
static String call(String path, String jsonBody, Map<String, String> extra) throws Exception {
var b = HttpRequest.newBuilder(URI.create(BASE + "/" + path))
.header("Authorization", "Bearer " + token);
if (jsonBody == null) {
b = b.GET();
} else {
b = b.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString(jsonBody));
}
for (var e : extra.entrySet()) b = b.header(e.getKey(), e.getValue());
HttpResponse<String> res = HTTP.send(b.build(), HttpResponse.BodyHandlers.ofString());
String text = res.body();
// These samples keep the envelope as text and use a real JSON library in
// production (Jackson, Gson). The only thing to get right is the check:
// an envelope with "ok":false carries error.code and never data.
if (text.contains("\"ok\":false")) throw new ApiError("API_ERROR", text);
return text;
}
}
require "json"
require "net/http"
require "uri"
BASE = "https://api.skillsafe.ai/v1/app-api"
# From https://posterior-desk.skillsafe.ai/tokens.html, or minted in step 1.
TOKEN = ENV.fetch("SKILLSAFE_TOKEN", "YOUR_TOKEN")
LANES = %w[diagnose respec report].freeze
class ApiError < StandardError
attr_reader :code, :details
def initialize(code, message, details = {})
super("#{code}: #{message}")
@code = code
@details = details
end
end
# call("me") -> GET; call("estimate", input) -> POST with the input object as
# the whole body. No X-App-Slug: the token already carries the app.
def call(path, body = nil, extra = {})
uri = URI("#{BASE}/#{path}")
req = body.nil? ? Net::HTTP::Get.new(uri) : Net::HTTP::Post.new(uri)
req["Authorization"] = "Bearer #{TOKEN}"
unless body.nil?
req["Content-Type"] = "application/json"
req.body = JSON.generate(body)
end
extra.each { |k, v| req[k] = v }
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |h| h.request(req) }
env = JSON.parse(res.body)
unless env["ok"]
err = env["error"] || {}
raise ApiError.new(err["code"], err["message"], err["details"] || {})
end
env["data"]
end
p call("me")
# {"subject_type"=>"user", "subject_id"=>"usr_...", "credits"=>51234}
<?php
const BASE = "https://api.skillsafe.ai/v1/app-api";
// From https://posterior-desk.skillsafe.ai/tokens.html, or minted in step 1.
define("TOKEN", getenv("SKILLSAFE_TOKEN") ?: "YOUR_TOKEN");
const LANES = ["diagnose", "respec", "report"];
class ApiError extends RuntimeException {
public string $code;
public array $details;
public function __construct(string $code, string $message, array $details = []) {
parent::__construct("$code: $message");
$this->code = $code;
$this->details = $details;
}
}
// call("me") is a GET; call("estimate", $input) POSTs $input as the whole body.
// Two headers only - there is no X-App-Slug in this API.
function call(string $path, ?array $body = null, array $extra = []): array {
$headers = ["Authorization: Bearer " . TOKEN];
if ($body !== null) $headers[] = "Content-Type: application/json";
foreach ($extra as $k => $v) $headers[] = "$k: $v";
$ch = curl_init(BASE . "/" . $path);
curl_setopt($ch, CURLOPT_HTTPHEADER, $headers);
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
if ($body !== null) {
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($body));
}
$raw = curl_exec($ch);
curl_close($ch);
$env = json_decode($raw, true) ?: [];
if (empty($env["ok"])) {
$err = $env["error"] ?? [];
throw new ApiError($err["code"] ?? "INTERNAL", $err["message"] ?? "no message",
$err["details"] ?? []);
}
return $env["data"];
}
print_r(call("me"));
// Array ( [subject_type] => user [subject_id] => usr_... [credits] => 51234 )
// .NET 8. Usings: System.Net.Http.Json, System.Security.Cryptography,
// System.Text, System.Text.Json.
static class PosteriorDesk
{
const string Base = "https://api.skillsafe.ai/v1/app-api";
// From https://posterior-desk.skillsafe.ai/tokens.html, or minted in step 1.
public static string Token =
Environment.GetEnvironmentVariable("SKILLSAFE_TOKEN") ?? "YOUR_TOKEN";
public static readonly string[] Lanes = { "diagnose", "respec", "report" };
static readonly HttpClient Http = new();
public class ApiError : Exception
{
public string Code { get; }
public ApiError(string code, string message) : base($"{code}: {message}") => Code = code;
}
/// GET when body is null, POST otherwise. Returns the unwrapped `data`.
/// Two headers: Content-Type and Authorization. There is no X-App-Slug.
public static async Task<JsonElement> Call(string path, object? body = null,
Dictionary<string, string>? extra = null)
{
var req = new HttpRequestMessage(body is null ? HttpMethod.Get : HttpMethod.Post,
$"{Base}/{path}");
req.Headers.Add("Authorization", $"Bearer {Token}");
if (body is not null)
req.Content = new StringContent(JsonSerializer.Serialize(body),
Encoding.UTF8, "application/json");
foreach (var kv in extra ?? new()) req.Headers.Add(kv.Key, kv.Value);
var res = await Http.SendAsync(req);
var env = await res.Content.ReadFromJsonAsync<JsonElement>();
if (!env.GetProperty("ok").GetBoolean())
{
var err = env.GetProperty("error");
throw new ApiError(err.GetProperty("code").GetString() ?? "INTERNAL",
err.GetProperty("message").GetString() ?? "no message");
}
return env.GetProperty("data");
}
}
var me = await PosteriorDesk.Call("me");
Console.WriteLine(me.GetProperty("credits").GetInt32());
1. Get a token
For a human, the shortest path is the token page — that is the link to follow if you would rather not touch DevTools. It shows the token this browser already holds, with a copy button and a ready-made shell export, and a sign-in button for a personal token. Nothing on it needs a developer tool: it reads the same storage the app itself uses and prints the token for you.
For a program, POST /guest mints one. The body is
{"slug": "posterior-desk"} — this is the one and only place the slug appears in this API
— and the call answers 201 Created:
HTTP/1.1 201 Created
{ "ok": true, "data": {
"token": "sk_guest_...",
"guest_id": "gst_...",
"expires_at": "2026-08-27T09:14:02Z"
} }
Three things follow from that shape. expires_at is real, so a long-lived worker re-mints
rather than caching forever; a 401 on a previously good token usually means it lapsed.
guest_id is worth keeping — it is what lets a later sign-in migrate the guest wallet, and
it is the only handle you have on an anonymous session. And a guest token is enough for
/me and /estimate but not for a metered run: a review is
metered, so /run and /run-stream want a personal token from signing in. A
guest attempting a run gets 403 FORBIDDEN, not a 402.
Watch the status code rather than the body when you wire this up. POST /guest is the only
call on this API that answers 201, and a client that tests status === 200 before parsing
will decide the mint failed while holding a perfectly good token. Treat any 2xx as success and read
ok.
# The slug goes in the BODY, not in a header. This is the only call that needs it.
TOKEN=$(curl -sS -X POST "https://api.skillsafe.ai/v1/app-api/guest" \
-H "Content-Type: application/json" \
-d '{"slug":"posterior-desk"}' \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["token"])')
echo "${TOKEN:0:12}..." # sk_guest_...
# Keep the whole object if you want guest_id and expires_at:
curl -sS -X POST "https://api.skillsafe.ai/v1/app-api/guest" \
-H "Content-Type: application/json" \
-d '{"slug":"posterior-desk"}' | python3 -m json.tool
# {
# "ok": true,
# "data": { "token": "sk_guest_...", "guest_id": "gst_...",
# "expires_at": "2026-08-27T09:14:02Z" }
# }
#
# 201 Created, not 200. A guest token can /me and /estimate; a metered /run
# needs a personal token from https://posterior-desk.skillsafe.ai/tokens.html.
import datetime, json, urllib.request
BASE = "https://api.skillsafe.ai/v1/app-api"
SLUG = "posterior-desk" # the ONLY place the slug appears in this API
def mint_guest():
"""POST /guest -> 201 with {token, guest_id, expires_at}. No auth header."""
req = urllib.request.Request(
f"{BASE}/guest", data=json.dumps({"slug": SLUG}).encode(), method="POST")
req.add_header("Content-Type", "application/json")
with urllib.request.urlopen(req) as res:
assert res.status == 201, f"expected 201, got {res.status}"
env = json.load(res)
if not env.get("ok"):
err = env.get("error") or {}
raise RuntimeError(f"{err.get('code')}: {err.get('message')}")
return env["data"]
tok = mint_guest()
print(tok["token"][:12], tok["guest_id"], tok["expires_at"])
# sk_guest_ gst_... 2026-08-27T09:14:02Z
# expires_at is real. A worker that runs for days re-mints rather than caching:
expiry = datetime.datetime.fromisoformat(tok["expires_at"].replace("Z", "+00:00"))
if expiry - datetime.datetime.now(datetime.timezone.utc) < datetime.timedelta(minutes=5):
tok = mint_guest()
TOKEN = tok["token"] # feed this to call() from step 0
const BASE = "https://api.skillsafe.ai/v1/app-api";
const SLUG = "posterior-desk"; // the ONLY place the slug appears
// POST /guest answers 201 Created. No Authorization header on this one call.
async function mintGuest() {
const res = await fetch(`${BASE}/guest`, {
method: "POST",
headers: { "Content-Type": "application/json" },
body: JSON.stringify({ slug: SLUG }),
});
// Test for 2xx, not for 200: this endpoint returns 201 and a strict
// `res.status === 200` check reads a successful mint as a failure.
const env = await res.json();
if (!env.ok) throw new Error(`${env.error.code}: ${env.error.message}`);
return env.data; // { token, guest_id, expires_at }
}
const guest = await mintGuest();
console.log(guest.guest_id, guest.expires_at);
// Re-mint when it is close to lapsing rather than caching forever.
function expiring(g, marginMs = 5 * 60 * 1000) {
return Date.parse(g.expires_at) - Date.now() < marginMs;
}
let TOKEN = guest.token;
if (expiring(guest)) TOKEN = (await mintGuest()).token;
// POST /guest is the one call with no Authorization header and the one call
// that answers 201. The slug goes in the body.
type guestToken struct {
Token string `json:"token"`
GuestID string `json:"guest_id"`
ExpiresAt string `json:"expires_at"`
}
func mintGuest() (*guestToken, error) {
body, _ := json.Marshal(map[string]string{"slug": "posterior-desk"})
req, err := http.NewRequest(http.MethodPost, base+"/guest", bytes.NewReader(body))
if err != nil {
return nil, err
}
req.Header.Set("Content-Type", "application/json")
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
if res.StatusCode/100 != 2 {
return nil, fmt.Errorf("guest mint: HTTP %d", res.StatusCode)
}
var env struct {
OK bool `json:"ok"`
Data guestToken `json:"data"`
Error *apiError `json:"error"`
}
if err := json.NewDecoder(res.Body).Decode(&env); err != nil {
return nil, err
}
if !env.OK {
return nil, env.Error
}
return &env.Data, nil
}
func mustToken() string {
g, err := mintGuest()
if err != nil {
panic(err)
}
// expires_at is real; a long-lived worker re-mints rather than caching.
if t, err := time.Parse(time.RFC3339, g.ExpiresAt); err == nil {
if time.Until(t) < 5*time.Minute {
if g2, err := mintGuest(); err == nil {
g = g2
}
}
}
return g.Token
}
// POST /guest: no Authorization header, slug in the body, 201 on success.
static String mintGuest() throws Exception {
HttpRequest req = HttpRequest.newBuilder(URI.create(BASE + "/guest"))
.header("Content-Type", "application/json")
.POST(HttpRequest.BodyPublishers.ofString("{\"slug\":\"posterior-desk\"}"))
.build();
HttpResponse<String> res = HTTP.send(req, HttpResponse.BodyHandlers.ofString());
// 201 Created, not 200. Accept any 2xx.
if (res.statusCode() / 100 != 2)
throw new ApiError("HTTP_" + res.statusCode(), res.body());
String body = res.body();
// Replace with Jackson in production; this keeps the sample dependency-free.
int i = body.indexOf("\"token\":\"") + 9;
token = body.substring(i, body.indexOf('"', i));
return token;
}
public static void main(String[] args) throws Exception {
if (System.getenv("SKILLSAFE_TOKEN") == null) mintGuest();
System.out.println(call("me", null, Map.of()));
// {"ok":true,"data":{"subject_type":"guest","subject_id":"gst_...","credits":1200}}
//
// A guest can /me and /estimate. A metered /run needs a personal token from
// https://posterior-desk.skillsafe.ai/tokens.html
}
# POST /guest: no Authorization header, the slug in the body, 201 on success.
def mint_guest
uri = URI("#{BASE}/guest")
req = Net::HTTP::Post.new(uri)
req["Content-Type"] = "application/json"
req.body = JSON.generate({ slug: "posterior-desk" })
res = Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) { |h| h.request(req) }
# 201 Created, not 200 - check the class, not the exact code.
raise "guest mint: HTTP #{res.code}" unless res.is_a?(Net::HTTPSuccess)
env = JSON.parse(res.body)
raise ApiError.new(env.dig("error", "code"), env.dig("error", "message")) unless env["ok"]
env["data"] # { "token" => ..., "guest_id" => ..., "expires_at" => ... }
end
guest = mint_guest
puts "#{guest['guest_id']} expires #{guest['expires_at']}"
# Re-mint near expiry rather than caching a token for days.
guest = mint_guest if Time.parse(guest["expires_at"]) - Time.now < 300
TOKEN_FROM_GUEST = guest["token"]
<?php
// POST /guest: no Authorization header, the slug in the body, 201 on success.
function mint_guest(): array {
$ch = curl_init(BASE . "/guest");
curl_setopt($ch, CURLOPT_HTTPHEADER, ["Content-Type: application/json"]);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode(["slug" => "posterior-desk"]));
curl_setopt($ch, CURLOPT_RETURNTRANSFER, true);
$raw = curl_exec($ch);
$status = curl_getinfo($ch, CURLINFO_RESPONSE_CODE);
curl_close($ch);
// 201 Created, not 200. Accept any 2xx.
if (intdiv($status, 100) !== 2) throw new RuntimeException("guest mint: HTTP $status");
$env = json_decode($raw, true) ?: [];
if (empty($env["ok"])) {
$err = $env["error"] ?? [];
throw new ApiError($err["code"] ?? "INTERNAL", $err["message"] ?? "no message");
}
return $env["data"]; // ["token" => ..., "guest_id" => ..., "expires_at" => ...]
}
$guest = mint_guest();
printf("%s expires %s\n", $guest["guest_id"], $guest["expires_at"]);
// Re-mint near expiry.
if (strtotime($guest["expires_at"]) - time() < 300) $guest = mint_guest();
// POST /guest: no Authorization header, slug in the body, 201 on success.
static async Task<JsonElement> MintGuest()
{
var res = await new HttpClient().PostAsync(
"https://api.skillsafe.ai/v1/app-api/guest",
new StringContent("{\"slug\":\"posterior-desk\"}", Encoding.UTF8, "application/json"));
// 201 Created, not 200 - IsSuccessStatusCode covers both.
if (!res.IsSuccessStatusCode)
throw new Exception($"guest mint: HTTP {(int)res.StatusCode}");
var env = await res.Content.ReadFromJsonAsync<JsonElement>();
if (!env.GetProperty("ok").GetBoolean())
throw new PosteriorDesk.ApiError(
env.GetProperty("error").GetProperty("code").GetString() ?? "INTERNAL",
env.GetProperty("error").GetProperty("message").GetString() ?? "");
return env.GetProperty("data");
}
var guest = await MintGuest();
PosteriorDesk.Token = guest.GetProperty("token").GetString()!;
Console.WriteLine(guest.GetProperty("expires_at").GetString());
// Re-mint near expiry rather than caching for days.
if (DateTimeOffset.Parse(guest.GetProperty("expires_at").GetString()!) - DateTimeOffset.UtcNow
< TimeSpan.FromMinutes(5))
PosteriorDesk.Token = (await MintGuest()).GetProperty("token").GetString()!;
2. Check the session and the balance
GET /me returns three fields and nothing else:
{ "ok": true, "data": {
"subject_type": "guest",
"subject_id": "gst_...",
"credits": 1200
} }
Read that literally, because the fields people expect are not there. There is no
user_id — the identifier is subject_id whichever kind of subject it
is, so a guest's subject_id is its guest_id and a signed-in person's is
their user id. There is no is_guest flag, so a truthiness test on it is
silently false for everybody, which reads as "this is a real user" for a guest token. Branch on
subject_type, which is guest or user. There is no
username, no email and no plan field either; if you need a display name, you
need your own.
credits is the wallet balance. Compare it against min_credits from step 3
before you run, so a shortfall becomes your own clear message instead of a 402 in the middle of a
batch of forty fit reports.
call me
# {"ok":true,"data":{"subject_type":"user","subject_id":"usr_...","credits":51234}}
# Branch on subject_type. There is no is_guest field to test, and no user_id -
# the id is subject_id for both kinds of subject.
call me | python3 -c '
import sys, json
me = json.load(sys.stdin)["data"]
kind = me["subject_type"] # "guest" or "user"
print(kind, me["subject_id"], me["credits"], "credits")
if kind == "guest":
print("guest: /me and /estimate only - a metered run needs a personal token")
'
me = call("me")
print(me)
# {'subject_type': 'guest', 'subject_id': 'gst_...', 'credits': 1200}
# Three fields. No user_id, no is_guest, no email, no plan.
assert set(me) == {"subject_type", "subject_id", "credits"}, me
if me["subject_type"] == "guest":
# A guest can price a run but not start one: /run answers 403 FORBIDDEN.
print("guest session:", me["subject_id"], "- sign in before a metered run")
else:
print("signed in as", me["subject_id"])
# Keep the balance for the step-3 comparison rather than calling /me twice.
credits = me["credits"]
const me = await call("me");
console.log(me);
// { subject_type: 'guest', subject_id: 'gst_...', credits: 1200 }
// The two mistakes this shape invites, written out so they are easy to avoid:
// me.user_id -> undefined, always. The id is me.subject_id.
// if (me.is_guest) -> never true, for anybody. Branch on subject_type.
if (me.subject_type === "guest") {
console.warn("guest token: /me and /estimate only, /run will 403");
}
const credits = me.credits; // compare against min_credits from step 3
type meResponse struct {
SubjectType string `json:"subject_type"` // "guest" | "user"
SubjectID string `json:"subject_id"` // NOT user_id
Credits int64 `json:"credits"`
}
func whoAmI() (*meResponse, error) {
raw, err := call("me", nil, nil)
if err != nil {
return nil, err
}
var me meResponse
if err := json.Unmarshal(raw, &me); err != nil {
return nil, err
}
return &me, nil
}
func main() {
token = mustToken()
me, err := whoAmI()
if err != nil {
panic(err)
}
fmt.Printf("%s %s: %d credits\n", me.SubjectType, me.SubjectID, me.Credits)
// There is no IsGuest bool in the payload to unmarshal - the string is the flag.
if me.SubjectType == "guest" {
fmt.Println("guest: /run will answer 403 FORBIDDEN; sign in for a metered lane")
}
}
String meJson = call("me", null, Map.of());
System.out.println(meJson);
// {"ok":true,"data":{"subject_type":"guest","subject_id":"gst_...","credits":1200}}
// Three fields only: subject_type, subject_id, credits. No user_id, no is_guest.
boolean isGuest = meJson.contains("\"subject_type\":\"guest\"");
if (isGuest) {
System.out.println("guest token: /me and /estimate only, /run will 403");
}
// With Jackson:
// var me = new ObjectMapper().readTree(meJson).get("data");
// long credits = me.get("credits").asLong();
// String kind = me.get("subject_type").asText(); // "guest" | "user"
me = call("me")
p me
# {"subject_type"=>"guest", "subject_id"=>"gst_...", "credits"=>1200}
# Exactly three keys. Anything you reach for beyond them is nil.
raise "unexpected /me shape: #{me.keys}" unless
me.keys.sort == %w[credits subject_id subject_type]
case me["subject_type"]
when "guest" then warn "guest token: a metered /run will answer 403 FORBIDDEN"
when "user" then puts "signed in as #{me['subject_id']}"
end
credits = me["credits"] # compare against min_credits from step 3
<?php
$me = call("me");
print_r($me);
// Array ( [subject_type] => guest [subject_id] => gst_... [credits] => 1200 )
// No user_id and no is_guest. $me["is_guest"] is null, which is falsy, which
// reads as "a real user" for a guest token. Branch on the string.
if ($me["subject_type"] === "guest") {
fwrite(STDERR, "guest token: /me and /estimate only, /run will 403\n");
}
$credits = $me["credits"]; // compare against min_credits from step 3
var me = await PosteriorDesk.Call("me");
var kind = me.GetProperty("subject_type").GetString(); // "guest" | "user"
var id = me.GetProperty("subject_id").GetString(); // NOT user_id
var credits = me.GetProperty("credits").GetInt64();
Console.WriteLine($"{kind} {id}: {credits} credits");
// GetProperty throws on a missing key, which is the behaviour you want here:
// there is no is_guest and no user_id to reach for by accident.
if (kind == "guest")
Console.Error.WriteLine("guest token: /run will answer 403 FORBIDDEN");
3. Price the run — free, but authenticated
POST /estimate creates no job and charges nothing. It is still an
authenticated call, which is the ordering trap: it has to come after step 1. A caller who
builds an input, prices it and only then goes looking for a token gets a 401
UNAUTHORIZED on the free call and reads it as a broken endpoint. Mint the token first,
price second.
The body is the input object from above, flat. What comes back:
| field | for this app | meaning |
|---|---|---|
model | gpt-5.6-terra | The exact model the run will bind to. |
model_alias | gpt-terra | The stable alias that binding came from — what to log, since the concrete model moves under it. Worth asserting: if this is not gpt-terra, you are not talking to this app. |
markup_bps | 1000 | The app's markup in basis points; 1000 is ten per cent. |
hold_credits | varies | What gets reserved when the run starts. Priced against the full output cap, so it is an upper bound, not the price. |
min_credits | varies | The balance you must clear for the run to start at all. Compare this against credits from /me. |
sponsor_enabled | varies | Whether the app is covering this run rather than your wallet. |
The hold is a reservation. The charged_credits you see on the settled job is normally far
lower, because a diagnose pass over a clean four-parameter fit says so in a few hundred
tokens while the cap allows for a report lane with a full diagnostics table and a drafted
methods paragraph. Budget against hold_credits; report against
charged_credits.
Estimates differ by lane, and they differ a lot here. report is the wordiest — it drafts
prose — and diagnose is the leanest on a clean fit and the fattest on a broken one, since
every failing parameter earns a finding. If you are running all three lanes over one fit, price all
three rather than multiplying the diagnose figure by three. And the
table field moves the estimate more than anything else in the input: 120 rows of
ten columns is a lot of text, every row is read, and a 3000-parameter model clipped to 120 rows still
prices roughly ten times a four-row eight-schools table. Price the real input, not a stub.
# The fit report and the summary table as shell variables, so the JSON stays
# readable. The table is TAB separated - printf, not echo, so \t survives.
FIT=$(cat <<'EOF'
Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Wall time: 412 s
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase `target_accept` or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec:
mu = pm.Normal('mu', 0, 5)
tau = pm.HalfCauchy('tau', 5)
theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred
y_obs = pm.Normal('y', mu=theta, sigma=sigma_known, observed=y)
EOF
)
TABLE=$(printf 'param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat\n'
printf 'mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01\n'
printf 'tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31\n'
printf 'theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08\n')
# Build the body with python3 so the tabs and newlines are escaped correctly.
# NOTE the shape: the input object IS the body. No "input" wrapper.
INPUT=$(FIT="$FIT" TABLE="$TABLE" python3 -c '
import json, os
print(json.dumps({
"task": "diagnose",
"fit": os.environ["FIT"],
"table": os.environ["TABLE"],
"goal": "publication",
"stage": "first_fit",
"notes": "reviewer 2 asked about the funnel",
}))')
call estimate "$INPUT" | data
# {"model":"gpt-5.6-terra","model_alias":"gpt-terra","markup_bps":1000,
# "hold_credits":2140,"min_credits":260,"sponsor_enabled":false}
#
# Free, but authenticated: without the step-1 token this is a 401, not a price.
FIT = """Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Wall time: 412 s
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase `target_accept` or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec:
mu = pm.Normal('mu', 0, 5)
tau = pm.HalfCauchy('tau', 5)
theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred
y_obs = pm.Normal('y', mu=theta, sigma=sigma_known, observed=y)
"""
# Tab separated, with the ten canonical columns. If you have an InferenceData
# object to hand, `az.summary(idata).to_csv(sep="\t")` produces this directly;
# the header tokens are matched whole, so n_eff, ess, ESS bulk all land in
# ess_bulk and any hdi_NN% pair becomes hdi_lo / hdi_hi.
TABLE = "\t".join(["param", "mean", "sd", "hdi_lo", "hdi_hi", "mcse_mean",
"mcse_sd", "ess_bulk", "ess_tail", "r_hat"]) + "\n" + "\n".join([
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
])
# The input object IS the body. Never {"input": {...}}.
INPUT = {
"task": "diagnose",
"fit": FIT,
"table": TABLE,
"goal": "publication",
"stage": "first_fit",
"notes": "reviewer 2 asked about the funnel",
}
est = call("estimate", INPUT)
print(est)
# {'model': 'gpt-5.6-terra', 'model_alias': 'gpt-terra', 'markup_bps': 1000,
# 'hold_credits': 2140, 'min_credits': 260, 'sponsor_enabled': False}
assert est["model_alias"] == "gpt-terra", est # wrong app or wrong slug
if not est["sponsor_enabled"] and credits < est["min_credits"]:
raise SystemExit(f"need {est['min_credits']} credits, have {credits}")
const FIT = [
"Model: eight_schools_centred",
"Sampler: NUTS (PyMC)",
"Chains: 4 | Draws: 1000 | Tune: 500",
"target_accept: 0.8 | max_treedepth: 10",
"Wall time: 412 s",
"Observations: 8",
"Free parameters: 10 (mu, tau, theta[8])",
"Convergence: all chains converged, r_hat < 1.01",
"Warnings:",
" There were 216 divergences after tuning. Increase `target_accept` or reparameterize.",
" 38 of 4000 iterations saturated the maximum tree depth of 10.",
"Model spec:",
" mu = pm.Normal('mu', 0, 5)",
" tau = pm.HalfCauchy('tau', 5)",
" theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) // centred",
" y_obs = pm.Normal('y', mu=theta, sigma=sigma_known, observed=y)",
].join("\n");
// Ten canonical columns, tab separated. Send what your tooling prints; the
// parser accepts to_string(), to_csv(), markdown pipes and Stan-style columns
// and normalises them itself.
const TABLE = [
"param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat",
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
].join("\n");
// The input object IS the body. Never { input: {...} }.
const INPUT = {
task: "diagnose",
fit: FIT,
table: TABLE,
goal: "publication",
stage: "first_fit",
notes: "reviewer 2 asked about the funnel",
};
const est = await call("estimate", INPUT);
console.log(est);
// { model: 'gpt-5.6-terra', model_alias: 'gpt-terra', markup_bps: 1000,
// hold_credits: 2140, min_credits: 260, sponsor_enabled: false }
if (est.model_alias !== "gpt-terra") throw new Error("not this app's model");
if (!est.sponsor_enabled && credits < est.min_credits) {
throw new Error(`need ${est.min_credits} credits, have ${credits}`);
}
// The flat input object. Every field is top-level; there is no wrapper struct.
type runInput struct {
Task string `json:"task"`
Fit string `json:"fit"`
Table string `json:"table,omitempty"`
Goal string `json:"goal,omitempty"`
Stage string `json:"stage,omitempty"`
Notes string `json:"notes,omitempty"`
Prescan map[string]any `json:"prescan_facts,omitempty"`
}
type estimate struct {
Model string `json:"model"`
ModelAlias string `json:"model_alias"`
MarkupBps int `json:"markup_bps"`
HoldCredits int64 `json:"hold_credits"`
MinCredits int64 `json:"min_credits"`
SponsorEnabled bool `json:"sponsor_enabled"`
}
const fitReport = `Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase target_accept or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec: theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) // centred`
// Tabs matter: these are the ten canonical columns.
var summaryTable = strings.Join([]string{
"param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat",
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
}, "\n")
func priceIt(in runInput) (*estimate, error) {
raw, err := call("estimate", in, nil) // free, but authenticated
if err != nil {
return nil, err
}
var est estimate
if err := json.Unmarshal(raw, &est); err != nil {
return nil, err
}
if est.ModelAlias != "gpt-terra" {
return nil, fmt.Errorf("unexpected model alias %q - wrong app?", est.ModelAlias)
}
return &est, nil
}
func example() {
in := runInput{
Task: "diagnose",
Fit: fitReport,
Table: summaryTable,
Goal: "publication",
Stage: "first_fit",
Notes: "reviewer 2 asked about the funnel",
}
est, err := priceIt(in)
if err != nil {
panic(err)
}
fmt.Printf("hold %d, min %d, markup %d bps\n",
est.HoldCredits, est.MinCredits, est.MarkupBps)
}
// The fit report and the table as text blocks. \t in a text block is a real tab.
static final String FIT = """
Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase target_accept or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec: theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) // centred
""";
static final String TABLE = String.join("\n",
"param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat",
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08");
/** Build the flat input object. In production use Jackson's ObjectNode; this
* keeps the sample dependency-free. The body IS this object - no wrapper. */
static String inputJson(String task) {
return "{"
+ "\"task\":" + q(task) + ","
+ "\"fit\":" + q(FIT) + ","
+ "\"table\":" + q(TABLE) + ","
+ "\"goal\":\"publication\","
+ "\"stage\":\"first_fit\","
+ "\"notes\":\"reviewer 2 asked about the funnel\""
+ "}";
}
/** Minimal JSON string escaper: quotes, backslashes, tabs and newlines. */
static String q(String s) {
StringBuilder b = new StringBuilder("\"");
for (char c : s.toCharArray()) {
switch (c) {
case '"' -> b.append("\\\"");
case '\\' -> b.append("\\\\");
case '\n' -> b.append("\\n");
case '\t' -> b.append("\\t");
case '\r' -> b.append("\\r");
default -> b.append(c);
}
}
return b.append('"').toString();
}
// Free, but authenticated: this line after step 1, never before it.
String est = call("estimate", inputJson("diagnose"), Map.of());
System.out.println(est);
// {"ok":true,"data":{"model":"gpt-5.6-terra","model_alias":"gpt-terra",
// "markup_bps":1000,"hold_credits":2140,"min_credits":260,"sponsor_enabled":false}}
if (!est.contains("\"model_alias\":\"gpt-terra\"")) throw new IllegalStateException(est);
FIT = <<~TEXT
Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase target_accept or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec: theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred
TEXT
# Ten canonical columns, tab separated.
TABLE = [
%w[param mean sd hdi_lo hdi_hi mcse_mean mcse_sd ess_bulk ess_tail r_hat].join("\t"),
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
].join("\n")
# The input object IS the body. Never { input: {...} }.
def build_input(task)
{
task: task,
fit: FIT,
table: TABLE,
goal: "publication",
stage: "first_fit",
notes: "reviewer 2 asked about the funnel",
}
end
est = call("estimate", build_input("diagnose"))
p est
# {"model"=>"gpt-5.6-terra", "model_alias"=>"gpt-terra", "markup_bps"=>1000,
# "hold_credits"=>2140, "min_credits"=>260, "sponsor_enabled"=>false}
raise "wrong app: #{est['model_alias']}" unless est["model_alias"] == "gpt-terra"
if !est["sponsor_enabled"] && credits < est["min_credits"]
abort "need #{est['min_credits']} credits, have #{credits}"
end
<?php
$FIT = <<<'TEXT'
Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase target_accept or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec: theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred
TEXT;
// Ten canonical columns. Note the double-quoted strings: "\t" is a real tab,
// '\t' is a backslash and a t, and the second one silently breaks the parse.
$TABLE = implode("\n", [
implode("\t", ["param", "mean", "sd", "hdi_lo", "hdi_hi", "mcse_mean",
"mcse_sd", "ess_bulk", "ess_tail", "r_hat"]),
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
]);
// The input object IS the body. Never ["input" => [...]].
function build_input(string $task, string $fit, string $table): array {
return [
"task" => $task,
"fit" => $fit,
"table" => $table,
"goal" => "publication",
"stage" => "first_fit",
"notes" => "reviewer 2 asked about the funnel",
];
}
$est = call("estimate", build_input("diagnose", $FIT, $TABLE));
print_r($est);
// [model] => gpt-5.6-terra [model_alias] => gpt-terra [markup_bps] => 1000
// [hold_credits] => 2140 [min_credits] => 260 [sponsor_enabled] =>
if ($est["model_alias"] !== "gpt-terra") throw new RuntimeException("wrong app");
if (!$est["sponsor_enabled"] && $credits < $est["min_credits"]) {
exit("need {$est['min_credits']} credits, have $credits\n");
}
const string Fit = """
Model: eight_schools_centred
Sampler: NUTS (PyMC)
Chains: 4 | Draws: 1000 | Tune: 500
target_accept: 0.8 | max_treedepth: 10
Observations: 8
Free parameters: 10 (mu, tau, theta[8])
Convergence: all chains converged, r_hat < 1.01
Warnings:
There were 216 divergences after tuning. Increase target_accept or reparameterize.
38 of 4000 iterations saturated the maximum tree depth of 10.
Model spec: theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) // centred
""";
// Ten canonical columns, tab separated.
static readonly string Table = string.Join("\n", new[]
{
"param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat",
"mu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01",
"tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31",
"theta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08",
});
// The input object IS the body - an anonymous object serialises flat.
static object BuildInput(string task) => new
{
task,
fit = Fit,
table = Table,
goal = "publication",
stage = "first_fit",
notes = "reviewer 2 asked about the funnel",
};
var est = await PosteriorDesk.Call("estimate", BuildInput("diagnose"));
Console.WriteLine(est);
// {"model":"gpt-5.6-terra","model_alias":"gpt-terra","markup_bps":1000,
// "hold_credits":2140,"min_credits":260,"sponsor_enabled":false}
if (est.GetProperty("model_alias").GetString() != "gpt-terra")
throw new InvalidOperationException("wrong app or wrong slug");
var min = est.GetProperty("min_credits").GetInt64();
if (!est.GetProperty("sponsor_enabled").GetBoolean() && credits < min)
throw new InvalidOperationException($"need {min} credits, have {credits}");
4. Run it, poll, and check the reply
POST /run returns {job_id}; poll GET /jobs/{job_id} until
status is succeeded or failed. The review is a JSON string at
data.output.output — one object, the envelope described in
the output contract. The terminal job also carries
charged_credits, the real price, and truncated.
The body is the input object itself — again, and for the last time. This is the call
where the wrapped-input mistake costs money: {"input": {…}} returns 200,
takes the hold, runs, and bills you for a review of an empty object. The reply will be fluent and it
will be about nothing — a convergence assessment of a fit with no chains, no divergences and no
table. There is no error code and no warning; the only signal is that lane comes back as
whatever the model guessed from an empty input, and every reconciliation entry you were
promised is missing. Send it flat, and assert lane.
Always send an Idempotency-Key. Derive it from the input the way the web
app does — a content hash plus the lane plus an attempt counter,
posterior-desk:<hash>:<lane>:a<attempt>. A retried request carrying the
same key returns the same job instead of billing a second run, which is what makes a retry safe after
a network blip on a fit you have already paid to review. Bump the attempt suffix whenever the input
actually changed — including when all that changed is task, because the lane is part of
the body, and a key replayed against a different body is rejected rather than silently answered from
the wrong job.
The polling interval that works here is one second with a cap: a diagnose pass over a
four-parameter table settles in a few seconds, a report lane drafting prose against 120
table rows can take most of a minute. Poll on a fixed short interval rather than an exponential
backoff — the job is not rate-limited and a backoff mostly adds latency to the common case.
# Always send an Idempotency-Key derived from the input, with the lane in it.
# A retried request with the same key returns the SAME job instead of billing twice.
KEY="posterior-desk:$(printf '%s' "$INPUT" | shasum -a 256 | cut -c1-16):diagnose:a1"
JOB=$(curl -sS -X POST "$BASE/run" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" \
-d "$INPUT" | python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["job_id"])')
echo "job $JOB"
# Poll on a fixed one-second interval, with a ceiling.
for i in $(seq 1 120); do
RES=$(curl -sS "$BASE/jobs/$JOB" -H "Authorization: Bearer $TOKEN")
STATUS=$(printf '%s' "$RES" | python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["status"])')
case "$STATUS" in
succeeded|failed) break ;;
esac
sleep 1
done
# Pull the review out of data.output.output and run the assertions.
printf '%s' "$RES" | TASK=diagnose python3 -c '
import json, os, sys
job = json.load(sys.stdin)["data"]
if job["status"] != "succeeded":
raise SystemExit("job %s: %s" % (job["status"], job.get("error")))
if job.get("truncated"):
raise SystemExit("truncated: retry, do not repair")
raw = job["output"]["output"]
review = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
assert review["lane"] == os.environ["TASK"], (review["lane"], "wrapped input?")
worst = {f["severity"] for f in review["findings"]}
if review["verdict"] == "not_supported": assert "blocking" in worst
if review["verdict"] == "sound": assert not (worst - {"info"})
print(review["verdict"], "|", review["headline"])
print("charged", job["charged_credits"], "credits")
'
import hashlib, time
LANE_BODY_KEYS = {
"diagnose": {"sampler_review", "convergence", "geometry", "efficiency",
"precision", "trust_scope"},
"respec": {"root_cause", "model_changes", "prior_changes", "sampler_changes",
"run_plan", "cost_note", "if_it_does_not_work"},
"report": {"methods_paragraph", "diagnostics_table", "results_wording",
"limitations", "reproducibility", "open_items"},
}
VERDICTS = {"sound", "sound_with_caveats", "revise", "not_supported"}
SEVERITIES = {"blocking", "high", "medium", "low", "info"}
AREAS = {"sampler", "convergence", "efficiency", "precision", "geometry",
"interval", "model", "comparison", "reporting"}
def idem_key(body, attempt=1):
"""posterior-desk:<hash>:<lane>:a<attempt> - the lane is part of the key
because it is part of the body."""
blob = json.dumps(body, sort_keys=True).encode()
return (f"posterior-desk:{hashlib.sha256(blob).hexdigest()[:16]}"
f":{body.get('task', 'auto')}:a{attempt}")
def run(body, attempt=1, timeout=180):
job = call("run", body, {"Idempotency-Key": idem_key(body, attempt)})
job_id = job["job_id"]
deadline = time.time() + timeout
while time.time() < deadline:
j = call(f"jobs/{job_id}")
if j["status"] in ("succeeded", "failed"):
return j
time.sleep(1)
raise TimeoutError(job_id)
def review_of(job, expected_task):
"""Unwrap data.output.output and assert everything worth asserting."""
if job["status"] != "succeeded":
raise RuntimeError(f"job {job['status']}: {job.get('error')}")
if job.get("truncated"):
raise RuntimeError("truncated reply: retry with a retry_note, do not repair")
raw = job["output"]["output"]
review = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
# 1. the lane you asked for (a mismatch is what a wrapped input looks like)
assert review["lane"] == expected_task, (review["lane"], expected_task)
# 2. the body belongs to that lane, and to no other
keys = set(review["body"])
assert keys <= LANE_BODY_KEYS[review["lane"]], keys - LANE_BODY_KEYS[review["lane"]]
# 3. enums in range
assert review["verdict"] in VERDICTS, review["verdict"]
for f in review["findings"]:
assert f["severity"] in SEVERITIES and f["area"] in AREAS, f
# 4. the verdict matches the worst severity
worst = {f["severity"] for f in review["findings"]}
if review["verdict"] == "not_supported":
assert "blocking" in worst, "not_supported with no blocking finding"
if review["verdict"] == "revise":
assert worst & {"blocking", "high"}, "revise with nothing above medium"
if review["verdict"] == "sound":
assert not (worst - {"info"}), f"sound with {worst}"
return review
job = run(INPUT)
review = review_of(job, INPUT["task"])
# 5. the reconciliation contract: one entry per prescan uid, exactly
sent = {f["uid"] for f in INPUT.get("prescan_facts", {}).get("flags", [])}
got = [r["flag_uid"] for r in review["reconciliation"]]
assert sorted(got) == sorted(sent), (set(got) ^ sent)
assert len(got) == len(set(got)), "a uid was reconciled twice"
# 6. trust_scope partitions, in the diagnose lane
if review["lane"] == "diagnose":
ts = review["body"]["trust_scope"]
both = set(ts["reportable"]) & set(ts["not_reportable"])
assert not both, f"quantity on both sides of trust_scope: {both}"
# 7. notes came back as context_notes
if INPUT.get("notes"):
assert review["context_notes"], "notes were sent and not read"
print(review["verdict"], "|", review["headline"])
print("charged", job["charged_credits"], "credits")
import { createHash } from "node:crypto"; // in a browser: SubtleCrypto
const LANE_BODY_KEYS = {
diagnose: ["sampler_review", "convergence", "geometry", "efficiency",
"precision", "trust_scope"],
respec: ["root_cause", "model_changes", "prior_changes", "sampler_changes",
"run_plan", "cost_note", "if_it_does_not_work"],
report: ["methods_paragraph", "diagnostics_table", "results_wording",
"limitations", "reproducibility", "open_items"],
};
const VERDICTS = ["sound", "sound_with_caveats", "revise", "not_supported"];
const SEVERITIES = ["blocking", "high", "medium", "low", "info"];
const AREAS = ["sampler", "convergence", "efficiency", "precision", "geometry",
"interval", "model", "comparison", "reporting"];
// posterior-desk:<hash>:<lane>:a<attempt> - bump the attempt when the input changes.
function idemKey(body, attempt = 1) {
const hash = createHash("sha256")
.update(JSON.stringify(body, Object.keys(body).sort()))
.digest("hex").slice(0, 16);
return `posterior-desk:${hash}:${body.task ?? "auto"}:a${attempt}`;
}
const sleep = (ms) => new Promise((r) => setTimeout(r, ms));
async function run(body, attempt = 1, timeoutMs = 180_000) {
const { job_id } = await call("run", body, { "Idempotency-Key": idemKey(body, attempt) });
const deadline = Date.now() + timeoutMs;
while (Date.now() < deadline) {
const job = await call(`jobs/${job_id}`);
if (job.status === "succeeded" || job.status === "failed") return job;
await sleep(1000);
}
throw new Error(`timed out polling ${job_id}`);
}
function reviewOf(job, expectedTask, sentUids = []) {
if (job.status !== "succeeded") throw new Error(`job ${job.status}: ${job.error}`);
if (job.truncated) throw new Error("truncated reply: retry, do not repair");
const raw = job.output.output;
const review = JSON.parse(raw.slice(raw.indexOf("{"), raw.lastIndexOf("}") + 1));
// The lane you asked for. A mismatch is what a wrapped input looks like.
if (review.lane !== expectedTask) throw new Error(`lane ${review.lane} != ${expectedTask}`);
// The body belongs to that lane and to no other.
const allowed = LANE_BODY_KEYS[review.lane];
const stray = Object.keys(review.body).filter((k) => !allowed.includes(k));
if (stray.length) throw new Error(`stray body keys: ${stray}`);
// Enums in range.
if (!VERDICTS.includes(review.verdict)) throw new Error(review.verdict);
for (const f of review.findings) {
if (!SEVERITIES.includes(f.severity) || !AREAS.includes(f.area)) throw new Error(f.id);
}
// The verdict matches the worst severity.
const worst = new Set(review.findings.map((f) => f.severity));
if (review.verdict === "not_supported" && !worst.has("blocking"))
throw new Error("not_supported with no blocking finding");
if (review.verdict === "sound" && [...worst].some((s) => s !== "info"))
throw new Error(`sound with ${[...worst]}`);
// The reconciliation contract, both directions.
const got = review.reconciliation.map((r) => r.flag_uid).sort();
const want = [...sentUids].sort();
if (got.join() !== want.join()) throw new Error(`reconciliation drift: ${got} vs ${want}`);
// trust_scope partitions.
if (review.lane === "diagnose") {
const { reportable, not_reportable } = review.body.trust_scope;
const both = reportable.filter((q) => not_reportable.includes(q));
if (both.length) throw new Error(`on both sides of trust_scope: ${both}`);
}
return review;
}
const job = await run(INPUT);
const review = reviewOf(job, INPUT.task,
(INPUT.prescan_facts?.flags ?? []).map((f) => f.uid));
console.log(review.verdict, "|", review.headline);
console.log("charged", job.charged_credits, "credits");
type job struct {
JobID string `json:"job_id"`
Status string `json:"status"`
Truncated bool `json:"truncated"`
ChargedCredits int64 `json:"charged_credits"`
Error string `json:"error"`
Output struct {
Output string `json:"output"` // the review, as a JSON string
} `json:"output"`
}
type finding struct {
ID string `json:"id"`
Severity string `json:"severity"`
Area string `json:"area"`
Title string `json:"title"`
Detail string `json:"detail"`
Evidence string `json:"evidence"`
Line *int `json:"line"`
Fix string `json:"fix"`
}
type review struct {
Lane string `json:"lane"`
Title string `json:"title"`
Verdict string `json:"verdict"`
Headline string `json:"headline"`
Summary string `json:"summary"`
Body json.RawMessage `json:"body"` // lane-specific; decode per lane
Findings []finding `json:"findings"`
Reconciliation []struct {
FlagUID string `json:"flag_uid"`
Status string `json:"status"`
Note string `json:"note"`
} `json:"reconciliation"`
Caveats []map[string]string `json:"caveats"`
ContextNotes []map[string]string `json:"context_notes"`
Unassessable []map[string]string `json:"unassessable"`
}
// posterior-desk:<hash>:<lane>:a<attempt>
func idemKey(in runInput, attempt int) string {
blob, _ := json.Marshal(in)
sum := sha256.Sum256(blob)
lane := in.Task
if lane == "" {
lane = "auto"
}
return fmt.Sprintf("posterior-desk:%x:%s:a%d", sum[:8], lane, attempt)
}
func runAndPoll(in runInput, attempt int) (*job, error) {
raw, err := call("run", in, map[string]string{"Idempotency-Key": idemKey(in, attempt)})
if err != nil {
return nil, err
}
var started job
if err := json.Unmarshal(raw, &started); err != nil {
return nil, err
}
deadline := time.Now().Add(3 * time.Minute)
for time.Now().Before(deadline) {
raw, err := call("jobs/"+started.JobID, nil, nil)
if err != nil {
return nil, err
}
var j job
if err := json.Unmarshal(raw, &j); err != nil {
return nil, err
}
if j.Status == "succeeded" || j.Status == "failed" {
return &j, nil
}
time.Sleep(time.Second)
}
return nil, fmt.Errorf("timed out polling %s", started.JobID)
}
// Slice from the first brace to the last, then decode and assert.
func reviewOf(j *job, expectedTask string, sentUIDs []string) (*review, error) {
if j.Status != "succeeded" {
return nil, fmt.Errorf("job %s: %s", j.Status, j.Error)
}
if j.Truncated {
return nil, fmt.Errorf("truncated reply: retry, do not repair")
}
raw := j.Output.Output
start, end := strings.Index(raw, "{"), strings.LastIndex(raw, "}")
if start < 0 || end < start {
return nil, fmt.Errorf("no JSON object in output")
}
var r review
if err := json.Unmarshal([]byte(raw[start:end+1]), &r); err != nil {
return nil, err
}
if r.Lane != expectedTask {
return nil, fmt.Errorf("lane %q != task %q - wrapped input?", r.Lane, expectedTask)
}
worst := map[string]bool{}
for _, f := range r.Findings {
worst[f.Severity] = true
}
if r.Verdict == "not_supported" && !worst["blocking"] {
return nil, fmt.Errorf("not_supported with no blocking finding")
}
if r.Verdict == "sound" && (worst["blocking"] || worst["high"] ||
worst["medium"] || worst["low"]) {
return nil, fmt.Errorf("sound verdict with findings above info")
}
if len(r.Reconciliation) != len(sentUIDs) {
return nil, fmt.Errorf("sent %d prescan uids, got %d reconciliation entries",
len(sentUIDs), len(r.Reconciliation))
}
return &r, nil
}
/** posterior-desk:<hash>:<lane>:a<attempt> - the lane is part of the body, so
* it is part of the key. */
static String idemKey(String inputJson, String lane, int attempt) throws Exception {
var md = MessageDigest.getInstance("SHA-256");
byte[] d = md.digest(inputJson.getBytes("UTF-8"));
var hex = new StringBuilder();
for (int i = 0; i < 8; i++) hex.append(String.format("%02x", d[i]));
return "posterior-desk:" + hex + ":" + lane + ":a" + attempt;
}
static String runAndPoll(String inputJson, String lane) throws Exception {
String started = call("run", inputJson,
Map.of("Idempotency-Key", idemKey(inputJson, lane, 1)));
int i = started.indexOf("\"job_id\":\"") + 10;
String jobId = started.substring(i, started.indexOf('"', i));
long deadline = System.currentTimeMillis() + 180_000L;
while (System.currentTimeMillis() < deadline) {
String job = call("jobs/" + jobId, null, Map.of());
if (job.contains("\"status\":\"succeeded\"")) return job;
if (job.contains("\"status\":\"failed\"")) throw new ApiError("JOB_FAILED", job);
Thread.sleep(1000);
}
throw new IllegalStateException("timed out polling " + jobId);
}
// The review is a JSON string at data.output.output. Slice from the first brace
// to the last before parsing - it costs nothing and survives small variations.
static String sliceObject(String s) {
return s.substring(s.indexOf('{'), s.lastIndexOf('}') + 1);
}
public static void check(String jobJson, String expectedLane) {
if (jobJson.contains("\"truncated\":true"))
throw new IllegalStateException("truncated reply: retry, do not repair");
// With Jackson:
// var job = mapper.readTree(jobJson).get("data");
// var review = mapper.readTree(sliceObject(job.get("output").get("output").asText()));
// if (!review.get("lane").asText().equals(expectedLane))
// throw new IllegalStateException("lane mismatch - wrapped input?");
// var worst = new HashSet<String>();
// review.get("findings").forEach(f -> worst.add(f.get("severity").asText()));
// var verdict = review.get("verdict").asText();
// if (verdict.equals("not_supported") && !worst.contains("blocking"))
// throw new IllegalStateException("not_supported with no blocking finding");
// if (verdict.equals("sound") && !worst.stream().allMatch("info"::equals))
// throw new IllegalStateException("sound with findings above info");
// if (review.get("reconciliation").size() != sentUids.size())
// throw new IllegalStateException("reconciliation does not cover the prescan");
}
String job = runAndPoll(inputJson("diagnose"), "diagnose");
check(job, "diagnose");
System.out.println(job);
require "digest"
LANE_BODY_KEYS = {
"diagnose" => %w[sampler_review convergence geometry efficiency precision trust_scope],
"respec" => %w[root_cause model_changes prior_changes sampler_changes run_plan
cost_note if_it_does_not_work],
"report" => %w[methods_paragraph diagnostics_table results_wording limitations
reproducibility open_items],
}.freeze
VERDICTS = %w[sound sound_with_caveats revise not_supported].freeze
# posterior-desk:<hash>:<lane>:a<attempt>
def idem_key(body, attempt = 1)
hash = Digest::SHA256.hexdigest(JSON.generate(body.sort.to_h))[0, 16]
"posterior-desk:#{hash}:#{body[:task] || 'auto'}:a#{attempt}"
end
def run(body, attempt = 1, timeout = 180)
started = call("run", body, { "Idempotency-Key" => idem_key(body, attempt) })
job_id = started["job_id"]
deadline = Time.now + timeout
while Time.now < deadline
job = call("jobs/#{job_id}")
return job if %w[succeeded failed].include?(job["status"])
sleep 1
end
raise "timed out polling #{job_id}"
end
def review_of(job, expected_task, sent_uids = [])
raise "job #{job['status']}: #{job['error']}" unless job["status"] == "succeeded"
raise "truncated reply: retry, do not repair" if job["truncated"]
raw = job.dig("output", "output")
review = JSON.parse(raw[raw.index("{")..raw.rindex("}")])
# The lane you asked for. A mismatch is what a wrapped input looks like.
raise "lane #{review['lane']} != #{expected_task}" unless review["lane"] == expected_task
# The body belongs to that lane and to no other.
stray = review["body"].keys - LANE_BODY_KEYS.fetch(review["lane"])
raise "stray body keys: #{stray}" unless stray.empty?
raise "bad verdict #{review['verdict']}" unless VERDICTS.include?(review["verdict"])
worst = review["findings"].map { |f| f["severity"] }.uniq
raise "not_supported with no blocking finding" if
review["verdict"] == "not_supported" && !worst.include?("blocking")
raise "sound with #{worst}" if
review["verdict"] == "sound" && !(worst - ["info"]).empty?
got = review["reconciliation"].map { |r| r["flag_uid"] }
raise "reconciliation drift" unless got.sort == sent_uids.sort
raise "a uid was reconciled twice" unless got.uniq.size == got.size
if review["lane"] == "diagnose"
ts = review["body"]["trust_scope"]
both = ts["reportable"] & ts["not_reportable"]
raise "on both sides of trust_scope: #{both}" unless both.empty?
end
review
end
body = build_input("diagnose")
job = run(body)
review = review_of(job, body[:task],
(body.dig(:prescan_facts, :flags) || []).map { |f| f[:uid] })
puts "#{review['verdict']} | #{review['headline']}"
puts "charged #{job['charged_credits']} credits"
<?php
const LANE_BODY_KEYS = [
"diagnose" => ["sampler_review", "convergence", "geometry", "efficiency",
"precision", "trust_scope"],
"respec" => ["root_cause", "model_changes", "prior_changes", "sampler_changes",
"run_plan", "cost_note", "if_it_does_not_work"],
"report" => ["methods_paragraph", "diagnostics_table", "results_wording",
"limitations", "reproducibility", "open_items"],
];
const VERDICTS = ["sound", "sound_with_caveats", "revise", "not_supported"];
// posterior-desk:<hash>:<lane>:a<attempt>
function idem_key(array $body, int $attempt = 1): string {
$hash = substr(hash("sha256", json_encode($body)), 0, 16);
$lane = $body["task"] ?? "auto";
return "posterior-desk:$hash:$lane:a$attempt";
}
function run_job(array $body, int $attempt = 1, int $timeout = 180): array {
$started = call("run", $body, ["Idempotency-Key" => idem_key($body, $attempt)]);
$jobId = $started["job_id"];
$deadline = time() + $timeout;
while (time() < $deadline) {
$job = call("jobs/$jobId");
if (in_array($job["status"], ["succeeded", "failed"], true)) return $job;
sleep(1);
}
throw new RuntimeException("timed out polling $jobId");
}
function review_of(array $job, string $expectedTask, array $sentUids = []): array {
if ($job["status"] !== "succeeded")
throw new RuntimeException("job {$job['status']}: " . ($job["error"] ?? ""));
if (!empty($job["truncated"]))
throw new RuntimeException("truncated reply: retry, do not repair");
$raw = $job["output"]["output"];
$slice = substr($raw, strpos($raw, "{"), strrpos($raw, "}") - strpos($raw, "{") + 1);
$review = json_decode($slice, true);
// The lane you asked for. A mismatch is what a wrapped input looks like.
if ($review["lane"] !== $expectedTask)
throw new RuntimeException("lane {$review['lane']} != $expectedTask");
$stray = array_diff(array_keys($review["body"]), LANE_BODY_KEYS[$review["lane"]]);
if ($stray) throw new RuntimeException("stray body keys: " . implode(",", $stray));
if (!in_array($review["verdict"], VERDICTS, true))
throw new RuntimeException("bad verdict {$review['verdict']}");
$worst = array_unique(array_column($review["findings"], "severity"));
if ($review["verdict"] === "not_supported" && !in_array("blocking", $worst, true))
throw new RuntimeException("not_supported with no blocking finding");
if ($review["verdict"] === "sound" && array_diff($worst, ["info"]))
throw new RuntimeException("sound with findings above info");
$got = array_column($review["reconciliation"], "flag_uid");
sort($got); sort($sentUids);
if ($got !== $sentUids) throw new RuntimeException("reconciliation drift");
if ($review["lane"] === "diagnose") {
$ts = $review["body"]["trust_scope"];
$both = array_intersect($ts["reportable"], $ts["not_reportable"]);
if ($both) throw new RuntimeException("both sides of trust_scope");
}
return $review;
}
$body = build_input("diagnose", $FIT, $TABLE);
$job = run_job($body);
$review = review_of($job, $body["task"]);
echo $review["verdict"], " | ", $review["headline"], "\n";
echo "charged ", $job["charged_credits"], " credits\n";
static readonly Dictionary<string, string[]> LaneBodyKeys = new()
{
["diagnose"] = new[] { "sampler_review", "convergence", "geometry", "efficiency",
"precision", "trust_scope" },
["respec"] = new[] { "root_cause", "model_changes", "prior_changes",
"sampler_changes", "run_plan", "cost_note",
"if_it_does_not_work" },
["report"] = new[] { "methods_paragraph", "diagnostics_table", "results_wording",
"limitations", "reproducibility", "open_items" },
};
static readonly string[] Verdicts =
{ "sound", "sound_with_caveats", "revise", "not_supported" };
// posterior-desk:<hash>:<lane>:a<attempt>
static string IdemKey(object body, string lane, int attempt = 1)
{
var json = JsonSerializer.Serialize(body);
var hash = Convert.ToHexString(SHA256.HashData(Encoding.UTF8.GetBytes(json)))
.ToLowerInvariant()[..16];
return $"posterior-desk:{hash}:{lane}:a{attempt}";
}
static async Task<JsonElement> Run(object body, string lane, int attempt = 1)
{
var started = await PosteriorDesk.Call("run", body,
new() { ["Idempotency-Key"] = IdemKey(body, lane, attempt) });
var jobId = started.GetProperty("job_id").GetString();
var deadline = DateTime.UtcNow.AddMinutes(3);
while (DateTime.UtcNow < deadline)
{
var job = await PosteriorDesk.Call($"jobs/{jobId}");
var status = job.GetProperty("status").GetString();
if (status is "succeeded" or "failed") return job;
await Task.Delay(1000);
}
throw new TimeoutException($"timed out polling {jobId}");
}
static JsonElement ReviewOf(JsonElement job, string expectedTask, string[] sentUids)
{
if (job.GetProperty("status").GetString() != "succeeded")
throw new InvalidOperationException("job did not succeed");
if (job.TryGetProperty("truncated", out var t) && t.GetBoolean())
throw new InvalidOperationException("truncated reply: retry, do not repair");
var raw = job.GetProperty("output").GetProperty("output").GetString()!;
var slice = raw[raw.IndexOf('{')..(raw.LastIndexOf('}') + 1)];
var review = JsonDocument.Parse(slice).RootElement;
// The lane you asked for. A mismatch is what a wrapped input looks like.
var lane = review.GetProperty("lane").GetString()!;
if (lane != expectedTask)
throw new InvalidOperationException($"lane {lane} != {expectedTask}");
foreach (var key in review.GetProperty("body").EnumerateObject())
if (!LaneBodyKeys[lane].Contains(key.Name))
throw new InvalidOperationException($"stray body key {key.Name}");
var verdict = review.GetProperty("verdict").GetString()!;
if (!Verdicts.Contains(verdict))
throw new InvalidOperationException($"bad verdict {verdict}");
var worst = review.GetProperty("findings").EnumerateArray()
.Select(f => f.GetProperty("severity").GetString()).ToHashSet();
if (verdict == "not_supported" && !worst.Contains("blocking"))
throw new InvalidOperationException("not_supported with no blocking finding");
if (verdict == "sound" && worst.Any(s => s != "info"))
throw new InvalidOperationException("sound with findings above info");
var got = review.GetProperty("reconciliation").EnumerateArray()
.Select(r => r.GetProperty("flag_uid").GetString()!).OrderBy(s => s);
if (!got.SequenceEqual(sentUids.OrderBy(s => s)))
throw new InvalidOperationException("reconciliation drift");
return review;
}
var job = await Run(BuildInput("diagnose"), "diagnose");
var review = ReviewOf(job, "diagnose", Array.Empty<string>());
Console.WriteLine($"{review.GetProperty("verdict")} | {review.GetProperty("headline")}");
Console.WriteLine($"charged {job.GetProperty("charged_credits")} credits");
5. Or stream it
POST /run-stream is the same call over server-sent events, with
Accept: text/event-stream added to the same headers and the same
Idempotency-Key. The events are job ({job_id}, first),
delta ({"text": "..."}, a chunk of the review JSON), done
(status, charged_credits, truncated) and error on
a failure. Two practical details: an idempotent replay of a key that already ran comes back as
plain JSON rather than a stream, so check the response Content-Type before you
start reading lines; and events are separated by a blank line, so split on \n\n rather
than assuming one data: line per event.
For a progress display, do not parse the partial JSON — watch for key names arriving in the
accumulating text. "findings" means the review is naming problems,
"reconciliation" means it has reached your prescan flags, "trust_scope"
means the diagnose lane is deciding what may be quoted, "run_plan" means
respec is down to the ordered steps, and "summary" means it is nearly done.
Substring matching on the quoted key name is enough and it costs nothing. The report
lane is the one where this matters: drafting a methods paragraph takes a while, and the accumulating
text is the only honest progress signal there is.
# Same body, same Idempotency-Key, one extra header.
curl -sSN -X POST "$BASE/run-stream" \
-H "Authorization: Bearer $TOKEN" \
-H "Content-Type: application/json" \
-H "Accept: text/event-stream" \
-H "Idempotency-Key: $KEY" \
-d "$INPUT"
# event: job
# data: {"job_id":"job_..."}
#
# event: delta
# data: {"text":"{\"lane\":\"diagnose\",\"title\":\"eight_schools_centred"}
#
# event: delta
# data: {"text":"\",\"verdict\":\"not_supported\",\"headline\":\"r_hat 1.31"}
#
# event: done
# data: {"status":"succeeded","charged_credits":915,"truncated":false}
# -N disables curl's buffering; without it the deltas arrive in one lump at the
# end and the stream was pointless. Reassemble the review from the delta texts:
curl -sSN -X POST "$BASE/run-stream" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-H "Accept: text/event-stream" -H "Idempotency-Key: $KEY" -d "$INPUT" \
| python3 -c '
import json, sys
text = ""
for line in sys.stdin:
if not line.startswith("data:"):
continue
payload = json.loads(line[5:].strip() or "{}")
if "text" in payload:
text += payload["text"]
for key in ("findings", "reconciliation", "trust_scope", "summary"):
if ("\"%s\"" % key) in text:
print("...", key, file=sys.stderr)
elif payload.get("status"):
if payload.get("truncated"):
raise SystemExit("truncated: retry, do not repair")
review = json.loads(text[text.index("{"):text.rindex("}") + 1])
print(review["verdict"], "|", review["headline"])
'
def stream(body, attempt=1, on_progress=None):
"""POST /run-stream. Yields nothing; returns the parsed review.
Events are separated by a blank line, so accumulate and split on \\n\\n
rather than assuming one data: line per event.
"""
payload = json.dumps(body).encode()
req = urllib.request.Request(f"{BASE}/run-stream", data=payload, method="POST")
req.add_header("Authorization", f"Bearer {TOKEN}")
req.add_header("Content-Type", "application/json")
req.add_header("Accept", "text/event-stream")
req.add_header("Idempotency-Key", idem_key(body, attempt))
with urllib.request.urlopen(req) as res:
ctype = res.headers.get("Content-Type", "")
# An idempotent replay of a key that already ran answers with plain JSON.
if "text/event-stream" not in ctype:
env = json.load(res)
job = env["data"]
return review_of(job, body["task"])
text, buf, done = "", "", None
for chunk in res:
buf += chunk.decode("utf-8", "replace")
while "\n\n" in buf:
raw, buf = buf.split("\n\n", 1)
name, data = "message", ""
for line in raw.splitlines():
if line.startswith("event:"):
name = line[6:].strip()
elif line.startswith("data:"):
data += line[5:].strip()
if not data:
continue
ev = json.loads(data)
if name == "delta":
text += ev.get("text", "")
if on_progress:
on_progress(text)
elif name == "done":
done = ev
elif name == "error":
raise ApiError(ev.get("code", "INTERNAL"), ev.get("message", ""))
if done and done.get("truncated"):
raise RuntimeError("truncated reply: retry, do not repair")
return json.loads(text[text.index("{"):text.rindex("}") + 1])
# Progress without parsing partial JSON: watch for key names arriving.
STAGES = ["findings", "reconciliation", "trust_scope", "summary"]
def progress(text):
for key in STAGES:
if f'"{key}"' in text:
print("reached", key)
STAGES.remove(key)
break
review = stream(INPUT, on_progress=progress)
print(review["verdict"], "|", review["headline"])
async function stream(body, { attempt = 1, onProgress } = {}) {
const res = await fetch(`${BASE}/run-stream`, {
method: "POST",
headers: {
Authorization: `Bearer ${TOKEN}`,
"Content-Type": "application/json",
Accept: "text/event-stream",
"Idempotency-Key": idemKey(body, attempt),
},
body: JSON.stringify(body),
});
// An idempotent replay of a key that already ran answers with plain JSON,
// not a stream. Check before you start reading lines.
const ctype = res.headers.get("content-type") ?? "";
if (!ctype.includes("text/event-stream")) {
const env = await res.json();
if (!env.ok) throw new ApiError(env.error.code, env.error.message);
return reviewOf(env.data, body.task);
}
const reader = res.body.getReader();
const dec = new TextDecoder();
let buf = "", text = "", done = null;
for (;;) {
const { value, done: eof } = await reader.read();
if (eof) break;
buf += dec.decode(value, { stream: true });
// Events are separated by a blank line - split on \n\n, not on \n.
let i;
while ((i = buf.indexOf("\n\n")) >= 0) {
const raw = buf.slice(0, i);
buf = buf.slice(i + 2);
let name = "message", data = "";
for (const line of raw.split("\n")) {
if (line.startsWith("event:")) name = line.slice(6).trim();
else if (line.startsWith("data:")) data += line.slice(5).trim();
}
if (!data) continue;
const ev = JSON.parse(data);
if (name === "delta") {
text += ev.text ?? "";
onProgress?.(text);
} else if (name === "done") {
done = ev;
} else if (name === "error") {
throw new ApiError(ev.code ?? "INTERNAL", ev.message ?? "stream error");
}
}
}
if (done?.truncated) throw new Error("truncated reply: retry, do not repair");
return JSON.parse(text.slice(text.indexOf("{"), text.lastIndexOf("}") + 1));
}
// Progress without parsing partial JSON.
const stages = ["findings", "reconciliation", "trust_scope", "summary"];
const review = await stream(INPUT, {
onProgress(text) {
while (stages.length && text.includes(`"${stages[0]}"`)) {
console.log("reached", stages.shift());
}
},
});
console.log(review.verdict, "|", review.headline);
// POST /run-stream: the same body and key, plus Accept: text/event-stream.
func streamRun(in runInput, attempt int) (*review, error) {
raw, _ := json.Marshal(in)
req, err := http.NewRequest(http.MethodPost, base+"/run-stream", bytes.NewReader(raw))
if err != nil {
return nil, err
}
req.Header.Set("Authorization", "Bearer "+token)
req.Header.Set("Content-Type", "application/json")
req.Header.Set("Accept", "text/event-stream")
req.Header.Set("Idempotency-Key", idemKey(in, attempt))
res, err := http.DefaultClient.Do(req)
if err != nil {
return nil, err
}
defer res.Body.Close()
// An idempotent replay answers with plain JSON, not a stream.
if !strings.Contains(res.Header.Get("Content-Type"), "text/event-stream") {
var env struct {
OK bool `json:"ok"`
Data job `json:"data"`
}
if err := json.NewDecoder(res.Body).Decode(&env); err != nil {
return nil, err
}
return reviewOf(&env.Data, in.Task, nil)
}
body, err := io.ReadAll(res.Body) // or read incrementally for progress
if err != nil {
return nil, err
}
var text strings.Builder
truncated := false
// Events are separated by a blank line.
for _, block := range strings.Split(string(body), "\n\n") {
name, data := "message", ""
for _, line := range strings.Split(block, "\n") {
switch {
case strings.HasPrefix(line, "event:"):
name = strings.TrimSpace(line[6:])
case strings.HasPrefix(line, "data:"):
data += strings.TrimSpace(line[5:])
}
}
if data == "" {
continue
}
switch name {
case "delta":
var ev struct{ Text string `json:"text"` }
if err := json.Unmarshal([]byte(data), &ev); err == nil {
text.WriteString(ev.Text)
}
case "done":
var ev struct{ Truncated bool `json:"truncated"` }
if err := json.Unmarshal([]byte(data), &ev); err == nil {
truncated = ev.Truncated
}
case "error":
return nil, fmt.Errorf("stream error: %s", data)
}
}
if truncated {
return nil, fmt.Errorf("truncated reply: retry, do not repair")
}
s := text.String()
start, end := strings.Index(s, "{"), strings.LastIndex(s, "}")
if start < 0 || end < start {
return nil, fmt.Errorf("no JSON object in the accumulated deltas")
}
var r review
if err := json.Unmarshal([]byte(s[start:end+1]), &r); err != nil {
return nil, err
}
return &r, nil
}
/** POST /run-stream with Accept: text/event-stream. Reads lines, accumulates
* the delta texts, and returns the review JSON. */
static String streamRun(String inputJson, String lane) throws Exception {
HttpRequest req = HttpRequest.newBuilder(URI.create(BASE + "/run-stream"))
.header("Authorization", "Bearer " + token)
.header("Content-Type", "application/json")
.header("Accept", "text/event-stream")
.header("Idempotency-Key", idemKey(inputJson, lane, 1))
.POST(HttpRequest.BodyPublishers.ofString(inputJson))
.build();
HttpResponse<java.util.stream.Stream<String>> res =
HTTP.send(req, HttpResponse.BodyHandlers.ofLines());
// An idempotent replay of a key that already ran answers with plain JSON.
String ctype = res.headers().firstValue("content-type").orElse("");
if (!ctype.contains("text/event-stream")) {
return res.body().reduce("", String::concat);
}
StringBuilder text = new StringBuilder();
boolean[] truncated = { false };
java.util.Set<String> seen = new java.util.HashSet<>();
res.body().forEach(line -> {
if (!line.startsWith("data:")) return; // event: lines and blanks
String data = line.substring(5).trim();
if (data.isEmpty()) return;
if (data.contains("\"text\"")) {
// Pull the delta text out; use Jackson in production.
int i = data.indexOf("\"text\":\"") + 8;
String piece = data.substring(i, data.lastIndexOf('"'));
text.append(piece.replace("\\n", "\n").replace("\\\"", "\""));
for (String key : new String[]{"findings", "reconciliation",
"trust_scope", "summary"}) {
if (text.indexOf("\"" + key + "\"") >= 0 && seen.add(key)) {
System.out.println("reached " + key);
}
}
} else if (data.contains("\"truncated\":true")) {
truncated[0] = true;
}
});
if (truncated[0]) throw new IllegalStateException("truncated: retry, do not repair");
String s = text.toString();
return s.substring(s.indexOf('{'), s.lastIndexOf('}') + 1);
}
String reviewJson = streamRun(inputJson("diagnose"), "diagnose");
System.out.println(reviewJson.substring(0, Math.min(200, reviewJson.length())));
# POST /run-stream: same body, same key, plus Accept: text/event-stream.
def stream(body, attempt = 1)
uri = URI("#{BASE}/run-stream")
req = Net::HTTP::Post.new(uri)
req["Authorization"] = "Bearer #{TOKEN}"
req["Content-Type"] = "application/json"
req["Accept"] = "text/event-stream"
req["Idempotency-Key"] = idem_key(body, attempt)
req.body = JSON.generate(body)
text = ""
truncated = false
seen = []
Net::HTTP.start(uri.hostname, uri.port, use_ssl: true) do |http|
http.request(req) do |res|
# An idempotent replay answers with plain JSON rather than a stream.
unless res["Content-Type"].to_s.include?("text/event-stream")
env = JSON.parse(res.body)
return review_of(env["data"], body[:task])
end
buf = ""
res.read_body do |chunk|
buf << chunk
# Events are separated by a blank line.
while (i = buf.index("\n\n"))
block = buf.slice!(0, i + 2)
name = block[/^event:\s*(.+)$/, 1] || "message"
data = block.scan(/^data:\s*(.*)$/).flatten.join
next if data.empty?
ev = JSON.parse(data)
case name.strip
when "delta"
text << ev.fetch("text", "")
%w[findings reconciliation trust_scope summary].each do |key|
next if seen.include?(key)
next unless text.include?(%("#{key}"))
seen << key
warn "reached #{key}"
end
when "done" then truncated = ev["truncated"]
when "error" then raise ApiError.new(ev["code"], ev["message"])
end
end
end
end
end
raise "truncated reply: retry, do not repair" if truncated
JSON.parse(text[text.index("{")..text.rindex("}")])
end
review = stream(build_input("diagnose"))
puts "#{review['verdict']} | #{review['headline']}"
<?php
// POST /run-stream. CURLOPT_WRITEFUNCTION gets the chunks as they arrive.
function stream_run(array $body, int $attempt = 1): array {
$text = "";
$buf = "";
$truncated = false;
$seen = [];
$ch = curl_init(BASE . "/run-stream");
curl_setopt($ch, CURLOPT_HTTPHEADER, [
"Authorization: Bearer " . TOKEN,
"Content-Type: application/json",
"Accept: text/event-stream",
"Idempotency-Key: " . idem_key($body, $attempt),
]);
curl_setopt($ch, CURLOPT_POST, true);
curl_setopt($ch, CURLOPT_POSTFIELDS, json_encode($body));
curl_setopt($ch, CURLOPT_WRITEFUNCTION,
function ($ch, $chunk) use (&$text, &$buf, &$truncated, &$seen) {
$buf .= $chunk;
// Events are separated by a blank line.
while (($i = strpos($buf, "\n\n")) !== false) {
$block = substr($buf, 0, $i);
$buf = substr($buf, $i + 2);
$name = "message";
$data = "";
foreach (explode("\n", $block) as $line) {
if (str_starts_with($line, "event:")) $name = trim(substr($line, 6));
elseif (str_starts_with($line, "data:")) $data .= trim(substr($line, 5));
}
if ($data === "") continue;
$ev = json_decode($data, true) ?: [];
if ($name === "delta") {
$text .= $ev["text"] ?? "";
foreach (["findings", "reconciliation", "trust_scope", "summary"] as $k) {
if (!in_array($k, $seen, true) && str_contains($text, "\"$k\"")) {
$seen[] = $k;
fwrite(STDERR, "reached $k\n");
}
}
} elseif ($name === "done") {
$truncated = !empty($ev["truncated"]);
} elseif ($name === "error") {
throw new ApiError($ev["code"] ?? "INTERNAL", $ev["message"] ?? "");
}
}
return strlen($chunk);
});
curl_exec($ch);
curl_close($ch);
if ($truncated) throw new RuntimeException("truncated: retry, do not repair");
$slice = substr($text, strpos($text, "{"),
strrpos($text, "}") - strpos($text, "{") + 1);
return json_decode($slice, true);
}
$review = stream_run(build_input("diagnose", $FIT, $TABLE));
echo $review["verdict"], " | ", $review["headline"], "\n";
// POST /run-stream with Accept: text/event-stream, read line by line.
static async Task<JsonElement> StreamRun(object body, string lane, int attempt = 1)
{
var req = new HttpRequestMessage(HttpMethod.Post,
"https://api.skillsafe.ai/v1/app-api/run-stream");
req.Headers.Add("Authorization", $"Bearer {PosteriorDesk.Token}");
req.Headers.Add("Accept", "text/event-stream");
req.Headers.Add("Idempotency-Key", IdemKey(body, lane, attempt));
req.Content = new StringContent(JsonSerializer.Serialize(body),
Encoding.UTF8, "application/json");
using var http = new HttpClient();
var res = await http.SendAsync(req, HttpCompletionOption.ResponseHeadersRead);
// An idempotent replay of a key that already ran answers with plain JSON.
if (res.Content.Headers.ContentType?.MediaType != "text/event-stream")
{
var env = await res.Content.ReadFromJsonAsync<JsonElement>();
return ReviewOf(env.GetProperty("data"), lane, Array.Empty<string>());
}
var text = new StringBuilder();
var truncated = false;
var stages = new List<string> { "findings", "reconciliation", "trust_scope", "summary" };
using var reader = new StreamReader(await res.Content.ReadAsStreamAsync());
string? name = null;
while (await reader.ReadLineAsync() is { } line)
{
if (line.StartsWith("event:")) { name = line[6..].Trim(); continue; }
if (!line.StartsWith("data:")) continue; // blank line = end of event
var data = line[5..].Trim();
if (data.Length == 0) continue;
var ev = JsonDocument.Parse(data).RootElement;
if (name == "delta" && ev.TryGetProperty("text", out var piece))
{
text.Append(piece.GetString());
while (stages.Count > 0 && text.ToString().Contains($"\"{stages[0]}\""))
{
Console.WriteLine($"reached {stages[0]}");
stages.RemoveAt(0);
}
}
else if (name == "done")
{
truncated = ev.TryGetProperty("truncated", out var t) && t.GetBoolean();
}
else if (name == "error")
{
throw new InvalidOperationException(data);
}
}
if (truncated) throw new InvalidOperationException("truncated: retry, do not repair");
var s = text.ToString();
return JsonDocument.Parse(s[s.IndexOf('{')..(s.LastIndexOf('}') + 1)]).RootElement;
}
var streamed = await StreamRun(BuildInput("diagnose"), "diagnose");
Console.WriteLine(streamed.GetProperty("headline").GetString());
6. One worked request per lane
Three bodies, one per lane, each a complete request you can POST as-is, followed by a trimmed but
structurally complete reply. They are deliberately small: a real fit is the whole
printed run plus the model spec, and a real table is every parameter. The only things
that change between these three are task, stage, goal and how
much supporting material is worth sending. The prescan_facts are trimmed here to keep
the examples readable; in the browser they carry every flag the free read raised.
diagnose — is the posterior trustworthy?
The lane to run first, and the one worth gating on: if the chains disagree about a parameter the
conclusion depends on, nothing downstream is worth reading. It is the only lane that can turn on
every per-parameter check, so this is the request where the table earns its place.
{
"task": "diagnose",
"fit": "Model: eight_schools_centred\nSampler: NUTS (PyMC)\nChains: 4 | Draws: 1000 | Tune: 500\ntarget_accept: 0.8 | max_treedepth: 10\nWall time: 412 s\nObservations: 8\nFree parameters: 10 (mu, tau, theta[8])\nConvergence: all chains converged, r_hat < 1.01\nWarnings:\n There were 216 divergences after tuning. Increase `target_accept` or reparameterize.\n 38 of 4000 iterations saturated the maximum tree depth of 10.\nModel spec:\n mu = pm.Normal('mu', 0, 5)\n tau = pm.HalfCauchy('tau', 5)\n theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred\n y_obs = pm.Normal('y', mu=theta, sigma=sigma_known, observed=y)\n",
"table": "param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat\nmu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01\ntau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31\ntheta[3]\t7.113\t5.512\t-2.900\t18.020\t0.31\t0.22\t318\t402\t1.08\n",
"goal": "publication",
"stage": "first_fit",
"notes": "reviewer 2 asked about the funnel",
"prescan_facts": {
"sampler": { "chains": 4, "draws_per_chain": 1000, "tune": 500,
"total_post_warmup_draws": 4000, "target_accept": 0.8,
"max_treedepth": 10, "wall_time_seconds": 412 },
"diagnostics": { "divergences": 216, "divergence_rate": 0.054, "treedepth_hits": 38,
"warmup_to_draws_ratio": 0.5, "worst_r_hat": 1.31,
"worst_r_hat_param": "tau", "lowest_ess_bulk": 61,
"lowest_ess_param": "tau" },
"table_present": true,
"verdict": "not_supported",
"severity_counts": { "blocking": 1, "high": 2, "medium": 1, "low": 0, "info": 0 },
"flags": [
{ "uid": "S-RHAT-BLOCK", "severity": "blocking", "area": "convergence",
"title": "r_hat 1.31 for tau", "line": 22,
"evidence": "tau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31" },
{ "uid": "X-RHAT-CLAIM", "severity": "high", "area": "convergence",
"title": "The report claims all r_hat < 1.01; the table says 1.31", "line": 9,
"evidence": "Convergence: all chains converged, r_hat < 1.01" },
{ "uid": "X-FUNNEL", "severity": "high", "area": "geometry",
"title": "Centred hierarchical parameterisation with divergences", "line": 14,
"evidence": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred" },
{ "uid": "S-ESS-BULK", "severity": "medium", "area": "efficiency",
"title": "ess_bulk 61 for tau, below 400", "line": 22,
"evidence": "ess_bulk 61" }
],
"unassessable": ["Energy / BFMI was not reported"]
}
}
The reply to that request is written out in full above, which is why it
is not repeated here. The two things to look at in it are the four reconciliation
entries against the four uids, and trust_scope, which is the block that
answers the question a caller actually has: what may I say?
respec — what do I change before the next run?
Run this when diagnose says revise or not_supported. The
input is the same fit and table, with stage unchanged — you have not rerun yet — and
the diagnosis carried in prescan_facts so the prescription is held to it. A
respec run with no diagnosis behind it will diagnose first and prescribe second, which
works and costs more than doing it in two lanes.
{
"task": "respec",
"fit": "Model: eight_schools_centred\nSampler: NUTS (PyMC)\nChains: 4 | Draws: 1000 | Tune: 500\ntarget_accept: 0.8 | max_treedepth: 10\nWarnings:\n There were 216 divergences after tuning.\nModel spec:\n mu = pm.Normal('mu', 0, 5)\n tau = pm.HalfCauchy('tau', 5)\n theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred\n",
"table": "param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat\nmu\t8.017\t5.086\t-0.958\t17.951\t0.24\t0.17\t452\t653\t1.01\ntau\t3.574\t3.281\t0.000\t9.190\t0.42\t0.30\t61\t92\t1.31\n",
"goal": "internal_review",
"stage": "first_fit",
"notes": "We have about two hours of compute before the meeting. The HalfCauchy prior on tau came from the paper we are replicating and I would rather not change it.",
"prescan_facts": {
"diagnostics": { "divergences": 216, "divergence_rate": 0.054,
"warmup_to_draws_ratio": 0.5, "worst_r_hat": 1.31,
"worst_r_hat_param": "tau", "lowest_ess_bulk": 61,
"lowest_ess_param": "tau" },
"table_present": true,
"verdict": "not_supported",
"severity_counts": { "blocking": 1, "high": 1, "medium": 0, "low": 0, "info": 0 },
"flags": [
{ "uid": "S-RHAT-BLOCK", "severity": "blocking", "area": "convergence",
"title": "r_hat 1.31 for tau", "line": 11, "evidence": "r_hat 1.31" },
{ "uid": "X-FUNNEL", "severity": "high", "area": "geometry",
"title": "Centred hierarchical parameterisation with divergences", "line": 9,
"evidence": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) # centred" }
],
"unassessable": []
}
}
The reply, trimmed to one entry per array:
{
"lane": "respec",
"title": "eight_schools_centred — reparameterisation plan",
"verdict": "revise",
"headline": "The centred parameterisation is the whole problem; non-centring is the fix and it costs one rerun.",
"summary": "The divergences and the r_hat on tau are one fact seen twice: a centred hierarchical model cannot be sampled through the neck of its funnel at a step size target_accept 0.8 will accept. Non-centre theta and rerun with a longer warm-up; that is the entire prescription and it fits inside your two hours. Your constraint on the HalfCauchy prior is honoured — it is not the cause, though it makes the tail harder — and the plan does not require changing it.",
"body": {
"root_cause": {
"diagnosis": "A centred hierarchical parameterisation creates a funnel in (tau, theta) space: as tau shrinks, the conditional scale of theta shrinks with it, so no single step size works over the whole posterior. The sampler diverges at small tau, which is exactly where the strongly pooled solutions live.",
"evidence": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8) on line 9, with 216 divergences and r_hat 1.31 on tau",
"confidence": "high"
},
"model_changes": [
{
"change": "non-centre the school-level effects",
"rationale": "Sampling a standardised theta_raw with a fixed unit scale removes the tau-dependence from the geometry the sampler sees; tau then enters only through a deterministic transform.",
"before": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8)",
"after": "theta_raw = pm.Normal('theta_raw', 0, 1, shape=8)\ntheta = pm.Deterministic('theta', mu + tau * theta_raw)",
"expected_effect": "divergences to zero or single digits; ess_bulk on tau from 61 into the high hundreds or better; r_hat on tau to 1.00-1.01"
}
],
"prior_changes": [
{
"parameter": "tau",
"current": "HalfCauchy(5) — stated in the model spec and constrained by your notes",
"proposed": "keep HalfCauchy(5) for this run; HalfNormal(5) only as a sensitivity check afterwards",
"rationale": "The prior is not the cause of the divergences and you have asked to keep it. Its heavy tail does make the upper end of tau harder to sample, so if ess_tail on tau is still poor after non-centring, a sensitivity run under HalfNormal(5) tells you whether the tail is doing any work — that is a comparison, not a replacement."
}
],
"sampler_changes": [
{
"setting": "tune",
"current": "500",
"proposed": "1500",
"rationale": "The warm-up is half the sampling length. Adaptation is the phase that has to succeed here, and 1000 extra warm-up draws cost about 40% more wall time than the 412 s you already spent."
},
{
"setting": "target_accept",
"current": "0.8",
"proposed": "0.9 — only if step 1 leaves divergences behind",
"rationale": "Secondary, and explicitly not a substitute for the reparameterisation. Raising it on a centred funnel buys smaller steps through a geometry that should not be there and hides the symptom while the cause remains."
}
],
"run_plan": [
{ "step": 1,
"action": "Non-centre theta exactly as in model_changes. Change nothing else — same seed policy, same chains, same draws, same tune, same target_accept — so the comparison is clean.",
"stop_if": "divergences do not fall below 10. The funnel is then not the whole story and the next suspect is the likelihood, not the geometry." },
{ "step": 2,
"action": "Raise tune to 1500, keep draws at 1000, rerun.",
"stop_if": "ess_bulk on tau is still below 400 after 4000 draws — that points at identifiability rather than adaptation." },
{ "step": 3,
"action": "Confirm with az.summary(): every r_hat at or below 1.01, ess_bulk and ess_tail above 400, and az.bfmi(idata) above 0.3. Record the seed and the versions this time.",
"stop_if": "any r_hat is above 1.01 — do not proceed to a write-up on a partial pass." }
],
"cost_note": "Step 1 is one rerun at roughly the original 412 s. Step 2 is about 580 s. Both fit inside two hours with room for a third run; step 3 is free. The sensitivity check on the prior is another full run and is not needed before the meeting.",
"if_it_does_not_work": "If divergences persist below 10 but ess_bulk on tau stays under 400 after step 2, the problem is identifiability rather than geometry: 8 observations carry very little information about a group-level scale. The honest fallback is to report mu with tau's prior stated explicitly and no point estimate of tau, or to fix tau at a defensible value and say so. Both are publishable; a non-converged tau is not."
},
"findings": [
{
"id": "F-001",
"severity": "high",
"area": "geometry",
"title": "Centred parameterisation is the root cause",
"detail": "216 divergences at 5.4% of draws in a model that builds theta directly from mu and tau. Graded high rather than blocking in this lane because it is fixable in one rerun and the fix is standard; the same fact is blocking in the diagnose lane, where the question is whether the current posterior can be reported.",
"evidence": "theta = pm.Normal('theta', mu=mu, sigma=tau, shape=8)",
"line": 9,
"fix": "Non-centre, per model_changes[0]."
},
{
"id": "F-002",
"severity": "medium",
"area": "sampler",
"title": "Warm-up is half the sampling length",
"detail": "500 tune against 1000 draws leaves adaptation unfinished for an awkward geometry. This is a contributing factor rather than the cause, and it is worth fixing in the same rerun because it costs less than a third failed run.",
"evidence": "Chains: 4 | Draws: 1000 | Tune: 500",
"line": 3,
"fix": "tune=1500."
}
],
"reconciliation": [
{ "flag_uid": "S-RHAT-BLOCK", "status": "confirmed",
"note": "Confirmed. In this lane it is the target of the prescription rather than a verdict on the posterior: the plan exists to make this number 1.00." },
{ "flag_uid": "X-FUNNEL", "status": "confirmed",
"note": "Confirmed and promoted to the root cause. Every change in the plan is ordered by how directly it addresses this." }
],
"caveats": [
{ "from_lane": "diagnose",
"fact": "tau and every theta[i] are currently not reportable (r_hat 1.31, ess_bulk 61)",
"why_it_matters": "Nothing in this plan licenses quoting the current numbers. The comparison in step 1 is between two runs' diagnostics, not between two posteriors' estimates." }
],
"context_notes": [
{ "claim": "about two hours of compute before the meeting",
"status": "honoured",
"note": "The plan is three actions totalling roughly 20 minutes of sampling; cost_note gives the arithmetic. The prior sensitivity run is explicitly deferred." },
{ "claim": "would rather not change the HalfCauchy prior on tau",
"status": "honoured",
"note": "Kept. It is not the cause and the plan does not depend on changing it; prior_changes says what a sensitivity check would tell you, without prescribing one." }
],
"unassessable": [
{ "item": "Whether the non-centred version will converge",
"why": "That is an empirical question about a run that has not happened. expected_effect is a prediction with a stated basis, and step 1's stop_if is what to do if it is wrong." }
]
}
report — write it up honestly
Run this last, on the fit you intend to publish. This example is the same model after the
reparameterisation worked: stage is after_respec, the diagnostics are clean,
and the remaining findings are about what was written down rather than what was sampled. Note what
the lane still refuses to do — the energy diagnostic and the seed were never reported, so they appear
in diagnostics_table with source: "not established" rather than being
quietly omitted.
{
"task": "report",
"fit": "Model: eight_schools_noncentred\nSampler: NUTS (PyMC)\nChains: 4 | Draws: 1000 | Tune: 1500\ntarget_accept: 0.9 | max_treedepth: 10\nWall time: 583 s\nObservations: 8\nFree parameters: 10 (mu, tau, theta[8])\nWarnings: none\nModel spec:\n mu = pm.Normal('mu', 0, 5)\n tau = pm.HalfCauchy('tau', 5)\n theta_raw = pm.Normal('theta_raw', 0, 1, shape=8)\n theta = pm.Deterministic('theta', mu + tau * theta_raw)\n y_obs = pm.Normal('y', mu=theta, sigma=sigma_known, observed=y)\n",
"table": "param\tmean\tsd\thdi_lo\thdi_hi\tmcse_mean\tmcse_sd\tess_bulk\tess_tail\tr_hat\nmu\t8.114\t5.041\t-1.204\t17.802\t0.06\t0.04\t3902\t3654\t1.00\ntau\t6.482\t5.703\t0.014\t16.911\t0.11\t0.09\t2418\t2201\t1.00\ntheta[3]\t7.902\t6.201\t-4.010\t20.114\t0.10\t0.07\t3611\t3402\t1.00\n",
"goal": "publication",
"stage": "after_respec",
"notes": "Target journal wants 95% intervals, not 94%. Two weeks to submission.",
"prescan_facts": {
"coverage": { "answered": 28, "total": 37,
"stated_none": ["divergences"],
"never_mentioned": ["seed", "energy_bfmi", "loo", "software_versions"] },
"sampler": { "chains": 4, "draws_per_chain": 1000, "tune": 1500,
"total_post_warmup_draws": 4000, "target_accept": 0.9,
"max_treedepth": 10, "wall_time_seconds": 583 },
"diagnostics": { "divergences": 0, "divergence_rate": 0.0, "treedepth_hits": null,
"warmup_to_draws_ratio": 1.5, "worst_r_hat": 1.0,
"worst_r_hat_param": "mu", "lowest_ess_bulk": 2418,
"lowest_ess_param": "tau" },
"table_present": true,
"verdict": "sound_with_caveats",
"severity_counts": { "blocking": 0, "high": 0, "medium": 1, "low": 1, "info": 1 },
"flags": [
{ "uid": "T-DIV-OK", "severity": "info", "area": "geometry",
"title": "No divergent transitions", "line": 8, "evidence": "Warnings: none" },
{ "uid": "T-CORE-MISSING", "severity": "medium", "area": "reporting",
"title": "4 core reporting items never mentioned", "line": null,
"evidence": "seed, energy_bfmi, loo, software_versions" },
{ "uid": "X-SHAPE-ABSENT", "severity": "low", "area": "reporting",
"title": "10 declared parameters, 3 rows in the summary table", "line": 7,
"evidence": "Free parameters: 10 (mu, tau, theta[8])" }
],
"unassessable": ["Energy / BFMI was not reported", "No LOO or WAIC was reported"]
}
}
The reply, trimmed:
{
"lane": "report",
"title": "eight_schools_noncentred — reporting pack",
"verdict": "sound_with_caveats",
"headline": "The fit is clean and reportable; what is missing is the record of it — no seed, no versions, no energy diagnostic.",
"summary": "After non-centring, every r_hat is 1.00, the lowest ess_bulk is 2418 and no divergences were reported, so the posterior supports the estimates being drawn from it. Two caveats bound the write-up: the summary table carries 3 of 10 declared parameters, so a claim about all parameters converging covers only what is shown, and the run is not currently reproducible. The intervals below are the 94% HDIs your table carries, not the 95% intervals your journal wants — regenerate them rather than relabelling.",
"body": {
"methods_paragraph": "We fitted a non-centred hierarchical normal model to the eight-school data using the No-U-Turn Sampler in PyMC. The population mean received a Normal(0, 5) prior and the between-school scale a HalfCauchy(5) prior; school-level effects were parameterised as theta = mu + tau * theta_raw with theta_raw ~ Normal(0, 1). We drew 1000 posterior samples per chain across 4 chains after 1500 warm-up iterations, with a target acceptance probability of 0.9 and a maximum tree depth of 10. No divergent transitions were reported. All parameters shown in the summary reached a rank-normalised split-R-hat of 1.00, with a minimum bulk effective sample size of 2418 and a minimum tail effective sample size of 2201.",
"diagnostics_table": [
{ "quantity": "chains", "value": "4", "source": "reported" },
{ "quantity": "draws after warm-up (per chain)", "value": "1000", "source": "reported" },
{ "quantity": "warm-up draws (per chain)", "value": "1500", "source": "reported" },
{ "quantity": "total post-warm-up draws", "value": "4000", "source": "computed" },
{ "quantity": "target_accept", "value": "0.9", "source": "reported" },
{ "quantity": "divergent transitions", "value": "0 (0.0% of draws)", "source": "computed" },
{ "quantity": "max r_hat", "value": "1.00 (across the 3 parameters shown)", "source": "computed" },
{ "quantity": "min ess_bulk", "value": "2418 (tau)", "source": "reported" },
{ "quantity": "min ess_tail", "value": "2201 (tau)", "source": "reported" },
{ "quantity": "energy / BFMI", "value": "not reported", "source": "not established" },
{ "quantity": "random seed", "value": "not reported", "source": "not established" },
{ "quantity": "PyMC / ArviZ versions", "value": "not reported", "source": "not established" }
],
"results_wording": [
{ "claim": "the population mean effect",
"wording": "The posterior mean of the population effect was 8.1 (94% HDI -1.2 to 17.8), giving substantial posterior mass on both positive and negligible effects.",
"hedge_required": false },
{ "claim": "between-school variation",
"wording": "The between-school standard deviation had a posterior mean of 6.5 (94% HDI 0.0 to 16.9). The interval reaches zero, so the data are consistent with complete pooling as well as with substantial heterogeneity.",
"hedge_required": true },
{ "claim": "that all parameters converged",
"wording": "All parameters reported in Table 2 reached an R-hat of 1.00. (Say \"reported in Table 2\", not \"all parameters\": the summary covers 3 of the 10 free parameters in the model.)",
"hedge_required": true },
{ "claim": "the interval mass",
"wording": "Report these as 94% highest-density intervals, or regenerate the summary with az.summary(idata, hdi_prob=0.95) and requote. Do not relabel a 94% interval as 95%.",
"hedge_required": true }
],
"limitations": "Three limitations bear on how far these results travel. First, with eight groups the between-school scale is weakly identified: its posterior is wide and reaches zero, so statements about heterogeneity are statements about a quantity the data constrain loosely. Second, the reported summary covers three of ten free parameters, so convergence is established for what is shown rather than for the model as a whole. Third, the run is not currently reproducible — no seed and no software versions were recorded — which does not affect the estimates but does affect what a reader can check.",
"reproducibility": {
"stated": ["model specification with priors", "sampler (NUTS, PyMC)",
"chains, draws and warm-up", "target_accept and max_treedepth",
"wall time"],
"missing": ["random seed", "PyMC and PyTensor versions", "ArviZ version",
"the data as used, or a checksum", "the az.summary() call that produced the table"],
"minimum_to_add": "the seed passed to pm.sample(), the PyMC and ArviZ versions, and one line giving the az.summary() call including hdi_prob — three lines in the same cell that printed the table"
},
"open_items": [
{ "item": "regenerate the summary at hdi_prob=0.95 and requote every interval in the results", "who": "analyst" },
{ "item": "print az.summary() for all 10 parameters, or state in the caption that Table 2 is a subset and why", "who": "analyst" },
{ "item": "record the seed and versions and add them to the supplement", "who": "analyst" },
{ "item": "confirm the journal's interval convention in writing before the intervals are typeset", "who": "corresponding author" }
]
},
"findings": [
{
"id": "F-001",
"severity": "medium",
"area": "reporting",
"title": "The run is not reproducible as reported",
"detail": "No random seed, no PyMC or ArviZ version, and no record of the data as used. For a publication goal this is the finding a methods reviewer raises first, and unlike a convergence problem it cannot be fixed after the fact if the environment has moved on.",
"evidence": "seed, energy_bfmi, loo, software_versions",
"line": null,
"fix": "Record the seed and the versions now, in the same session that produced the fit."
},
{
"id": "F-002",
"severity": "medium",
"area": "interval",
"title": "94% intervals against a 95% requirement",
"detail": "The table carries hdi_lo and hdi_hi at ArviZ's 94% default; the journal wants 95%. These are different intervals and the difference is not a rounding matter — relabelling is a misstatement of the reported mass.",
"evidence": "hdi_lo -1.204, hdi_hi 17.802",
"line": 12,
"fix": "az.summary(idata, hdi_prob=0.95), then requote every interval in the results."
},
{
"id": "F-003",
"severity": "low",
"area": "reporting",
"title": "3 of 10 free parameters are in the summary",
"detail": "The model declares mu, tau and theta[8]; the table shows mu, tau and theta[3]. Nothing suggests the others are problematic, and nothing establishes that they are not. A blanket convergence claim would cover parameters the reader cannot see.",
"evidence": "Free parameters: 10 (mu, tau, theta[8])",
"line": 7,
"fix": "Show all ten, or scope the claim to the table and say why it is a subset."
}
],
"reconciliation": [
{ "flag_uid": "T-DIV-OK", "status": "confirmed",
"note": "Confirmed and worth stating positively in the methods paragraph: zero divergences against a stated 4000 draws is a fact, not an absence of a warning." },
{ "flag_uid": "T-CORE-MISSING", "status": "confirmed",
"note": "Confirmed as F-001. For a publication goal the seed and the versions are the two that matter most, which is why minimum_to_add names them first." },
{ "flag_uid": "X-SHAPE-ABSENT", "status": "adjusted",
"note": "Adjusted to low. The gap is real but benign here: the missing rows are the remaining theta[i], all of which are transforms of parameters that did converge. It constrains the wording rather than the result." }
],
"caveats": [
{ "from_lane": "diagnose",
"fact": "the previous centred fit of this model had r_hat 1.31 on tau and 216 divergences",
"why_it_matters": "If any figure or number in the manuscript predates the reparameterisation, it is from the unusable fit. Regenerate every artefact from the current InferenceData object." }
],
"context_notes": [
{ "claim": "target journal wants 95% intervals, not 94%",
"status": "honoured",
"note": "Raised as F-002 and as an open item. The drafted wording quotes 94% because that is what the table carries, and says explicitly that the fix is regeneration rather than relabelling." },
{ "claim": "two weeks to submission",
"status": "honoured",
"note": "All four open items are minutes of work apiece; none needs another sampling run. The 95% regeneration is a summary call, not a refit." }
],
"unassessable": [
{ "item": "Energy / BFMI",
"why": "Not reported. Zero divergences is good evidence the geometry is now benign, but the energy diagnostic is the direct check and it was not run." },
{ "item": "Model comparison",
"why": "No LOO, WAIC or ELPD was reported and only one model was sent, so nothing here speaks to whether this model is preferable to an alternative." },
{ "item": "Posterior predictive check",
"why": "Not mentioned. Convergence says the sampler explored the posterior; it says nothing about whether the model describes the data." }
]
}
Running all three lanes over one fit
The pipeline, as a loop. Three things make it correct rather than merely sequential: the
Idempotency-Key includes the lane, so three runs over one input are three keys and not
one replayed job; lane is asserted against the task on every iteration,
which is the wrapped-input check; and the loop stops before report when
diagnose comes back not_supported, because a reporting pack for an
unusable fit is a document nobody should have.
# Three lanes over one fit. The lane is in the key, so these are three jobs.
for LANE in diagnose respec report; do
BODY=$(printf '%s' "$INPUT" | LANE="$LANE" python3 -c '
import json, os, sys
body = json.load(sys.stdin)
body["task"] = os.environ["LANE"]
print(json.dumps(body))
')
KEY="posterior-desk:$(printf '%s' "$BODY" | shasum -a 256 | cut -c1-16):$LANE:a1"
JOB=$(curl -sS -X POST "$BASE/run" \
-H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" \
-H "Idempotency-Key: $KEY" -d "$BODY" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["job_id"])')
until [ "$(curl -sS "$BASE/jobs/$JOB" -H "Authorization: Bearer $TOKEN" \
| python3 -c 'import sys,json; print(json.load(sys.stdin)["data"]["status"])')" \
!= "running" ]; do sleep 1; done
curl -sS "$BASE/jobs/$JOB" -H "Authorization: Bearer $TOKEN" \
| LANE="$LANE" python3 -c '
import json, os, sys
job = json.load(sys.stdin)["data"]
raw = job["output"]["output"]
review = json.loads(raw[raw.index("{"):raw.rindex("}") + 1])
assert review["lane"] == os.environ["LANE"], "wrapped input?"
print(review["lane"], review["verdict"], "|", review["headline"])
' > "review-$LANE.txt" || break
# Do not write a reporting pack for a fit that cannot be reported.
if [ "$LANE" = "diagnose" ] && grep -q not_supported "review-diagnose.txt"; then
echo "diagnose says not_supported - running respec, skipping report"
fi
done
def pipeline(base_input):
"""diagnose -> respec -> report, stopping where it makes sense to stop."""
out = {}
diag = review_of(run({**base_input, "task": "diagnose"}), "diagnose")
out["diagnose"] = diag
print("diagnose:", diag["verdict"], "|", diag["headline"])
# respec is worth running exactly when diagnose objected.
if diag["verdict"] in ("revise", "not_supported"):
resp = review_of(run({**base_input, "task": "respec"}), "respec")
out["respec"] = resp
print("respec:", resp["body"]["root_cause"]["diagnosis"])
for s in resp["body"]["run_plan"]:
print(f" {s['step']}. {s['action']} [stop if: {s['stop_if']}]")
# A reporting pack for an unusable fit is a document nobody should have.
if diag["verdict"] == "not_supported":
print("skipping report: the posterior does not support a write-up yet")
return out
rep = review_of(run({**base_input, "task": "report"}), "report")
out["report"] = rep
print("report:", rep["verdict"])
for row in rep["body"]["diagnostics_table"]:
if row["source"] == "not established":
print(" not established:", row["quantity"])
return out
results = pipeline({k: v for k, v in INPUT.items() if k != "task"})
# One fact must land at the same severity in every lane it appears in.
for lane, review in results.items():
for f in review["findings"]:
print(lane, f["id"], f["severity"], f["area"], f["title"])
async function pipeline(baseInput) {
const out = {};
const uids = (baseInput.prescan_facts?.flags ?? []).map((f) => f.uid);
const diag = reviewOf(await run({ ...baseInput, task: "diagnose" }), "diagnose", uids);
out.diagnose = diag;
console.log("diagnose:", diag.verdict, "|", diag.headline);
console.log(" not reportable:", diag.body.trust_scope.not_reportable);
// respec is worth running exactly when diagnose objected.
if (diag.verdict === "revise" || diag.verdict === "not_supported") {
const resp = reviewOf(await run({ ...baseInput, task: "respec" }), "respec", uids);
out.respec = resp;
console.log("root cause:", resp.body.root_cause.diagnosis);
for (const s of resp.body.run_plan) {
console.log(` ${s.step}. ${s.action} [stop if: ${s.stop_if}]`);
}
}
// A reporting pack for an unusable fit is a document nobody should have.
if (diag.verdict === "not_supported") {
console.log("skipping report: nothing here is writable up yet");
return out;
}
const rep = reviewOf(await run({ ...baseInput, task: "report" }), "report", uids);
out.report = rep;
for (const row of rep.body.diagnostics_table) {
if (row.source === "not established") console.log(" not established:", row.quantity);
}
return out;
}
const { task, ...baseInput } = INPUT;
const results = await pipeline(baseInput);
// One fact, the same severity in every lane it appears in.
for (const [lane, review] of Object.entries(results)) {
for (const f of review.findings) {
console.log(lane, f.id, f.severity, f.area, f.title);
}
}
// diagnose -> respec -> report over one fit, with the lane in every key.
func pipeline(in runInput) (map[string]*review, error) {
out := map[string]*review{}
in.Task = "diagnose"
j, err := runAndPoll(in, 1)
if err != nil {
return out, err
}
diag, err := reviewOf(j, "diagnose", nil)
if err != nil {
return out, err
}
out["diagnose"] = diag
fmt.Printf("diagnose: %s | %s\n", diag.Verdict, diag.Headline)
// respec is worth running exactly when diagnose objected.
if diag.Verdict == "revise" || diag.Verdict == "not_supported" {
in.Task = "respec"
j, err := runAndPoll(in, 1) // different body -> different key
if err != nil {
return out, err
}
resp, err := reviewOf(j, "respec", nil)
if err != nil {
return out, err
}
out["respec"] = resp
// body is lane-specific: decode it into the shape for this lane.
var b struct {
RootCause struct {
Diagnosis string `json:"diagnosis"`
Confidence string `json:"confidence"`
} `json:"root_cause"`
RunPlan []struct {
Step int `json:"step"`
Action string `json:"action"`
StopIf string `json:"stop_if"`
} `json:"run_plan"`
}
if err := json.Unmarshal(resp.Body, &b); err != nil {
return out, err
}
fmt.Println("root cause:", b.RootCause.Diagnosis)
for _, s := range b.RunPlan {
fmt.Printf(" %d. %s [stop if: %s]\n", s.Step, s.Action, s.StopIf)
}
}
// A reporting pack for an unusable fit is a document nobody should have.
if diag.Verdict == "not_supported" {
fmt.Println("skipping report: the posterior does not support a write-up yet")
return out, nil
}
in.Task = "report"
j, err = runAndPoll(in, 1)
if err != nil {
return out, err
}
rep, err := reviewOf(j, "report", nil)
if err != nil {
return out, err
}
out["report"] = rep
return out, nil
}
/** diagnose -> respec -> report over one fit. Three lanes, three keys, three
* jobs: the lane is part of the body and therefore part of the key. */
static void pipeline() throws Exception {
String diagJob = runAndPoll(inputJson("diagnose"), "diagnose");
check(diagJob, "diagnose");
System.out.println("diagnose done");
// Cheap verdict read without a JSON library; use Jackson in production.
boolean objected = diagJob.contains("\"verdict\":\"revise\"")
|| diagJob.contains("\"verdict\":\"not_supported\"");
boolean unusable = diagJob.contains("\"verdict\":\"not_supported\"");
// respec is worth running exactly when diagnose objected.
if (objected) {
String respecJob = runAndPoll(inputJson("respec"), "respec");
check(respecJob, "respec");
System.out.println("respec done - read body.run_plan for the ordered steps");
}
// A reporting pack for an unusable fit is a document nobody should have.
if (unusable) {
System.out.println("skipping report: nothing here is writable up yet");
return;
}
String reportJob = runAndPoll(inputJson("report"), "report");
check(reportJob, "report");
System.out.println("report done - rows with source \"not established\" are the gaps");
}
public static void main(String[] args) throws Exception {
if (System.getenv("SKILLSAFE_TOKEN") == null) mintGuest();
pipeline();
}
# diagnose -> respec -> report over one fit.
def pipeline(base)
out = {}
uids = (base.dig(:prescan_facts, :flags) || []).map { |f| f[:uid] }
diag = review_of(run(base.merge(task: "diagnose")), "diagnose", uids)
out["diagnose"] = diag
puts "diagnose: #{diag['verdict']} | #{diag['headline']}"
puts " not reportable: #{diag.dig('body', 'trust_scope', 'not_reportable').join('; ')}"
# respec is worth running exactly when diagnose objected.
if %w[revise not_supported].include?(diag["verdict"])
resp = review_of(run(base.merge(task: "respec")), "respec", uids)
out["respec"] = resp
puts "root cause: #{resp.dig('body', 'root_cause', 'diagnosis')}"
resp.dig("body", "run_plan").each do |s|
puts " #{s['step']}. #{s['action']} [stop if: #{s['stop_if']}]"
end
end
# A reporting pack for an unusable fit is a document nobody should have.
if diag["verdict"] == "not_supported"
puts "skipping report: the posterior does not support a write-up yet"
return out
end
rep = review_of(run(base.merge(task: "report")), "report", uids)
out["report"] = rep
rep.dig("body", "diagnostics_table")
.select { |r| r["source"] == "not established" }
.each { |r| puts " not established: #{r['quantity']}" }
out
end
base = build_input("diagnose").reject { |k, _| k == :task }
results = pipeline(base)
# One fact, the same severity in every lane it appears in.
results.each do |lane, review|
review["findings"].each { |f| puts "#{lane} #{f['id']} #{f['severity']} #{f['title']}" }
end
<?php
// diagnose -> respec -> report over one fit.
function pipeline(array $base): array {
$out = [];
$uids = array_column($base["prescan_facts"]["flags"] ?? [], "uid");
$diag = review_of(run_job($base + ["task" => "diagnose"]), "diagnose", $uids);
$out["diagnose"] = $diag;
echo "diagnose: {$diag['verdict']} | {$diag['headline']}\n";
echo " not reportable: "
. implode("; ", $diag["body"]["trust_scope"]["not_reportable"]) . "\n";
// respec is worth running exactly when diagnose objected.
if (in_array($diag["verdict"], ["revise", "not_supported"], true)) {
$resp = review_of(run_job($base + ["task" => "respec"]), "respec", $uids);
$out["respec"] = $resp;
echo "root cause: {$resp['body']['root_cause']['diagnosis']}\n";
foreach ($resp["body"]["run_plan"] as $s) {
echo " {$s['step']}. {$s['action']} [stop if: {$s['stop_if']}]\n";
}
}
// A reporting pack for an unusable fit is a document nobody should have.
if ($diag["verdict"] === "not_supported") {
echo "skipping report: nothing here is writable up yet\n";
return $out;
}
$rep = review_of(run_job($base + ["task" => "report"]), "report", $uids);
$out["report"] = $rep;
foreach ($rep["body"]["diagnostics_table"] as $row) {
if ($row["source"] === "not established") {
echo " not established: {$row['quantity']}\n";
}
}
return $out;
}
$base = build_input("diagnose", $FIT, $TABLE);
unset($base["task"]);
$results = pipeline($base);
// One fact, the same severity in every lane it appears in.
foreach ($results as $lane => $review) {
foreach ($review["findings"] as $f) {
echo "$lane {$f['id']} {$f['severity']} {$f['area']} {$f['title']}\n";
}
}
// diagnose -> respec -> report over one fit. The lane is in every key.
static async Task<Dictionary<string, JsonElement>> Pipeline()
{
var results = new Dictionary<string, JsonElement>();
var noUids = Array.Empty<string>();
var diag = ReviewOf(await Run(BuildInput("diagnose"), "diagnose"), "diagnose", noUids);
results["diagnose"] = diag;
var verdict = diag.GetProperty("verdict").GetString();
Console.WriteLine($"diagnose: {verdict} | {diag.GetProperty("headline")}");
foreach (var q in diag.GetProperty("body").GetProperty("trust_scope")
.GetProperty("not_reportable").EnumerateArray())
Console.WriteLine($" not reportable: {q.GetString()}");
// respec is worth running exactly when diagnose objected.
if (verdict is "revise" or "not_supported")
{
var resp = ReviewOf(await Run(BuildInput("respec"), "respec"), "respec", noUids);
results["respec"] = resp;
var body = resp.GetProperty("body");
Console.WriteLine($"root cause: {body.GetProperty("root_cause").GetProperty("diagnosis")}");
foreach (var s in body.GetProperty("run_plan").EnumerateArray())
Console.WriteLine($" {s.GetProperty("step")}. {s.GetProperty("action")} " +
$"[stop if: {s.GetProperty("stop_if")}]");
}
// A reporting pack for an unusable fit is a document nobody should have.
if (verdict == "not_supported")
{
Console.WriteLine("skipping report: nothing here is writable up yet");
return results;
}
var rep = ReviewOf(await Run(BuildInput("report"), "report"), "report", noUids);
results["report"] = rep;
foreach (var row in rep.GetProperty("body").GetProperty("diagnostics_table")
.EnumerateArray())
if (row.GetProperty("source").GetString() == "not established")
Console.WriteLine($" not established: {row.GetProperty("quantity")}");
return results;
}
var all = await Pipeline();
foreach (var (lane, review) in all)
foreach (var f in review.GetProperty("findings").EnumerateArray())
Console.WriteLine($"{lane} {f.GetProperty("id")} {f.GetProperty("severity")} " +
$"{f.GetProperty("title")}");
What the browser does before you pay
Everything described so far costs credits. The web app does a substantial amount of work before it
asks for any, in the page, with no model call and no network request, and an API caller can either
reproduce it or simply send its output as prescan_facts and let the lane be held to it.
It is worth understanding either way, because it is what makes the reconciliation contract more than
a formality: these are facts, established by arithmetic and string comparison, that a model cannot
argue with.
Thirty-seven checklist items, in three states
The free read resolves thirty-seven reporting items against the text you pasted — the
sampler settings, the chain and draw counts, the warm-up, target_accept,
max_treedepth, the seed, the software versions, the priors, the likelihood, the observed
data size, the parameter count, the divergence count, the treedepth saturation count, the energy or
BFMI figure, the r_hat and ESS summaries, the interval mass, the posterior and prior
predictive checks, LOO or WAIC and their Pareto k, and the rest. Each lands in exactly one of three
states, and the distinction between the last two is the point of the whole exercise:
| state | means | where it goes |
|---|---|---|
stated | The report gives a value. It is parsed, kept, and cross-checked against everything else that touches it. | counts toward coverage.answered |
declared_none | The report explicitly says there is none — Divergences: 0, Warnings: none, no posterior predictive check was run. This is a positive fact and it is the strongest thing a report can say about an absence. | coverage.stated_none |
missing | The report never reaches the item. Nothing is known, in either direction. | coverage.never_mentioned |
Collapsing the last two into "missing" is the mistake this design exists to prevent. "Zero divergent
transitions" and "we did not look at divergences" are opposite statements about the same quantity, and
a review that treats them alike is worthless on precisely the fits where the distinction decides
whether a paragraph can be written. Four items never mentioned raises the coverage flag at
high; fewer raises it at medium; a report that explicitly declares core
quantities absent raises a different flag with a different reading.
The draw count, and every rate computed against it
The free read computes total_post_warmup_draws as chains ×
draws_per_chain and then insists on it as the denominator. That single number is what makes a
divergence count into a divergence rate: 216 divergences is meaningless on its own and 5.4%
of 4000 draws is a diagnosis. Where the report states a rate itself, the two are compared, and a
mismatch is a flag — most often because the report divided by the total including warm-up, which
understates it, or by one chain's draws, which overstates it by the chain count.
The warm-up ratio is tune / draws_per_chain and it is checked against
the geometry: a ratio at or below 0.5 in a model with divergences is the short-warm-up flag, because
adaptation is exactly the phase that a difficult posterior defeats. Both numbers travel in
prescan_facts.diagnostics, rounded — the rate to six places, the ratio to three — so a
reply that restates them cannot restate them differently.
Declared shapes against the rows the table carries
A model that says Free parameters: 10 (mu, tau, theta[8]) declares ten scalar quantities.
A summary table with three rows carries three of them. The free read adds up the declared shapes into
model.declared_array_elements, counts model.summary_table_rows, and compares
— and when the table is a strict subset, every blanket claim in the prose ("all parameters converged",
"no r_hat exceeded 1.01") is unsupported for the parameters that are not shown. It flags the gap at
high when the table is missing most of the model and at medium when it is
missing a few, and the reply is expected to scope the claim rather than to assume the absent rows are
fine. This is also the check that catches the opposite error: a table with more rows than the model
declares parameters, which usually means a transformed variable or a deterministic is being counted as
a free parameter somewhere.
Every place the prose contradicts the table
The X- family of checks exists only because you sent both artefacts, and it is the most
valuable thing the free read does. Each one takes a claim in the prose and the corresponding cell in
the table and asks whether they can both be true:
| check | the contradiction it names |
|---|---|
X-RHAT-CLAIM | The prose states an r_hat bound the table exceeds. The commonest and the most consequential, because the prose is what a reader sees. |
X-CONVERGED-CLAIM | The prose asserts convergence with no numbers behind it, and the table disagrees with the assertion. |
X-RHAT-STALE | The prose quotes a maximum r_hat that no row of the table matches in either direction — the summary and the sentence came from different runs. |
X-DIV-CONTRADICT | The warning block reports divergences and the prose says there were none, or the reverse. |
X-ESS-CLAIM | The prose claims an effective sample size the table's lowest row does not support. |
X-FUNNEL | Divergences plus a centred hierarchical parameterisation visible in the model spec: the geometry is named from the code rather than guessed. |
X-SHAPE-SHORT / X-SHAPE-ABSENT | The declared shapes and the table's rows do not reconcile, as above. |
X-PARAMCOUNT | The declared free-parameter count and the shapes do not add up to each other. |
X-COMPARE-UNCONVERGED | A LOO or WAIC comparison is reported over fits whose own diagnostics say they did not converge — a ranking of numbers that do not mean anything yet. |
X-NO-PARETO / X-NO-DSE | A model comparison with no Pareto k diagnostic, or an ELPD difference quoted with no standard error on the difference. |
X-NO-NUMBERS | A convergence claim with no diagnostic number anywhere in the input to support or refute it. |
Per-row table checks run alongside them — the S- family: r_hat against the
1.01 and 1.05 thresholds and against NaN, ess_bulk and ess_tail against 400
and against the total draw count (an ESS above the draw count means antithetic chains, which is
legitimate and worth naming), mcse against the printed digits, and the interval columns
against each other and against the mean. A flat interval, a reversed one, a mean outside its own HDI
and an infinity anywhere in the row each have their own uid.
All of it is free, all of it is deterministic, and none of it needs a token. The lanes exist for the judgements it cannot make: why the geometry is what it is, what to change, and what a sentence in a paper is allowed to claim.
Truncation, retries and partial results
When the balance sits between min_credits and hold_credits, the run is not
refused: it executes with a reduced output cap and comes back with truncated: true on the
finished job and on the streaming done event. What you hold then is a prefix — the
findings may be complete while reconciliation, body and summary
are missing or cut mid-string. In this app that is worse than an error, because a prefix of a
diagnose lane can read as a clean sampler review with nothing after it — the
trust_scope block that says what may not be quoted is the last thing written —
and a prefix of respec is a run plan that stops before the step that would have caught
the mistake.
Check the flag before you treat a review as complete, and treat a truncation as a retry rather than a
repair. Send the same input with a concrete retry_note and the attempt suffix on the
Idempotency-Key incremented, so the new body is not a replay of the old key:
"retry_note": "The previous reply was truncated after findings[]. Return the same
findings, keep reconciliation complete for all four prescan uids, and shorten the
per-parameter readings in convergence.worst_parameters to one clause each."
The same route handles a reply that fails your own checks in the verification
list: a missing reconciliation entry, a verdict its findings contradict, a body
carrying another lane's keys, a quantity on both sides of trust_scope. Name the defect in
retry_note — it is obeyed exactly — and bump the attempt. Do not append closing braces to
truncated JSON; that produces something that parses and is not what the model meant.
A last note on grounding, because it changes how you read a clean review. Every finding names the
line, the warning string or the table cell that produced it, and nothing is invented — not a number,
not a threshold, not a line number, not an ess_bulk for a parameter that is not in the
table. So an empty findings array with a full unassessable array is not a
pass; it is a statement that the input did not contain enough to judge. Read the two together, and
read unassessable before you tell anyone the fit is sound. And read
context_notes too: a contradicted entry there is the app telling you that
something you asserted is not what the run supports.