Observability
Nothing an LLM does
goes unrecorded.
Every LLM call any atom, the Compose orchestrator or an agent team makes goes through one entry point, so every one of them is measured. See what a question cost, where the time went, which step failed, and whether the answer was any good.
Measure everything
One dashboard for what the models cost you.
KPIs and trends
Total calls, cost, tokens, error rate, and p50 / p95 latency across 7, 30 or 90 days, with daily cost and daily call charts.
By provider and model
Where the spend actually goes, sorted by cost, so a routing change is a decision with numbers behind it.
Recent calls
The latest individual calls with atom, kind, model, tokens, cost, latency and status. One click jumps to the run that made the call.
Every provider
One path for Anthropic, OpenAI, Gemini, AWS Bedrock and local Ollama models, measured the same way.
Spend caps that bite
A monthly AI budget can be set per Refinery. When it is reached the request stops with a clear message instead of failing silently.
OpenTelemetry out
Opt-in OTLP export sends traces to whatever collector you already run.
A real trace hierarchy
Session, trace, step, generation.
Runs nest the way work actually happens, so a slow or failing call is found by looking, not by grepping logs.
Nested spans
Every atom step expands to the LLM calls it made, with prompt, completion, tokens, cost, latency and any error.
Gantt waterfall
One row per span on a shared time axis, so parallel work and the long pole are obvious at a glance.
Group-aware
forEach, retry, branch and agent-team groups are tagged in the trace, not flattened into a list.
External calls linked
A non-LLM interaction, such as a SQL query or a REST call, is hard-linked into the trace with its own detail.
Capture you control
Metrics are on by default. Raw prompt and completion capture is a separate switch, off by default, and it never records the content of a System classified restricted.
Same run page
The trace lives on the ordinary run-detail page, next to inputs, outputs and the risk breakdown.
Is it any good?
Score it. Version it. Try it before you ship it.
Human scores
Thumbs up or down, with a comment, on any trace, step or generation.
LLM-as-judge
A judge atom scores outputs programmatically, so a Formula can grade its own work or another's.
Score rollups
Scores roll up onto the dashboard, next to cost and latency, so quality and spend are read together.
Versioned prompts
Prompt templates carry the same publish, roll and rollback discipline as Formulas, Skills, Views and Chains. Tenants can override a template.
Live playground
Edit a prompt and run it against real inputs before publishing the new version.
Rollback in a click
A bad prompt version is one rollback away, and earlier versions are kept rather than overwritten.
Multi-agent orchestration
A supervisor plans. Specialists run in parallel.
An agent-team step lets a Formula hand the plan to a supervisor at run time instead of fixing the sequence in advance.
Plan
A supervisor reads the objective and the description of each candidate agent, then chooses whom to dispatch.
Dispatch
Each chosen agent is an existing Formula, run as its own child run, in parallel up to a concurrency cap.
Synthesize
A built-in synthesizer merges the members' outputs into one result, even when some members failed.
Trace
The run page shows the objective, every agent's status, a link to its own run, and the raw plan.
A team run finishes as completed when every agent succeeded, partial when some did, and failed when none did. The supervisor only ever picks from the agents the author offered it.
Open the hood on your own runs.
We'll run a real workflow and walk the trace together, from the first token to the synthesized answer.