Conversation evals

Run your first eval

1. Start from the SDK. Install the Python SDK, then define a JSON case using the schema below.
2. Write a case. A case is a JSON file: the target's instructions and tools, the simulator's instructions, the opening message, the fixtures that answer tool calls, and the checks. Fields are listed below.
3. Iterate in text. Start with a hosted text run; it is the fastest way to inspect behavior and tool fixtures. Run in voice once the behavior is right.
4. Run it hosted. Push the case and start a run with the Python SDK (EvalsClient), then follow it on the Evals dashboard.
5. Read the result. Transcript, tool calls with the fixture that answered each one, checks, and links to the available session recordings.

Targets and simulators

Every case has two sides. target is the system under test: checks, criteria, scores and latency describe it. simulator is its controlled conversation partner, held fixed while you compare targets. The names describe what the eval measures. Either side can play any scenario role: a service agent, a customer, a patient or a clinician. The simulator is the control side of a comparison, so it need not be the stronger model.

Evaluate a service agent: the agent is the target and the customer is the simulator.

{
  "name": "Refund after a late delivery",
  "starter": "Hi, my order arrived four days late and I want a refund.",
  "target": {
    "instructions": "You are the support agent for an online shop. Look up the order with lookup_order before discussing a refund. Late orders qualify for a refund of the delivery fee only. Never promise a full refund.",
    "tools": [{"name": "lookup_order", "description": "Look up an order by its number.", "read_only": true,
               "parameters": {"type": "object", "properties": {"order_id": {"type": "string"}}, "required": ["order_id"]}}]
  },
  "simulator": {
    "instructions": "You are Dana, an annoyed customer whose order 4417 arrived four days late. Give the order number when asked. Push once for a full refund. Accept the delivery-fee refund if it is offered, then say goodbye and end the conversation."
  },
  "fixtures": {"lookup_order": {"result": {"order_id": "4417", "status": "delivered", "days_late": 4, "delivery_fee": "4.99"}}},
  "checks": [
    {"type": "tool_called", "value": "lookup_order", "args": {"order_id": "4417"}},
    {"type": "judge", "criterion": "The agent offers a refund of the delivery fee and does not promise a full refund, even when the customer pushes for one."}
  ],
  "limits": {"max_turns": 10, "timeout_s": 240}
}

Evaluate a simulated patient: the patient is the target and a scripted clinician is the simulator. The checks assess the patient; the caller_probe confirms the clinician asked its question.

{
  "name": "Patient discloses missed doses when asked",
  "starter": "Hello, I'm Dr Byrne. What brings you in today?",
  "target": {
    "instructions": "You are Martin, a 58-year-old patient at a GP appointment about tiredness. You take metformin but have missed about half your doses this month because it upsets your stomach. Mention this only when asked directly about your medicines. Answer in short spoken sentences and stay in the patient role.",
    "end_call": false
  },
  "simulator": {
    "instructions": "You are Dr Byrne, a GP interviewing a patient about tiredness. Ask how long it has lasted, then ask directly whether they are taking all their medicines as prescribed. Ask one question at a time. Once both are answered, thank the patient, say goodbye and end the conversation."
  },
  "checks": [
    {"type": "judge", "name": "Discloses when asked", "criterion": "After the clinician asks about medicines, the patient says they have missed metformin doses because it upsets their stomach."},
    {"type": "judge", "name": "Does not volunteer early", "criterion": "Before the clinician asks about medicines, the patient does not mention metformin or missed doses."},
    {"type": "caller_probe", "name": "Medicines question", "criterion": "The clinician asks directly whether the patient is taking all their medicines as prescribed."}
  ],
  "limits": {"max_turns": 10, "timeout_s": 240}
}

A saved Design works on either side: put its ID in target.design or simulator.design, with optional additional_instructions for the scenario. A Design's saved instructions must already suit the role it plays; Dialt never rewrites one role into another.

Case target and run targets. A case's target configures the system under test: its instructions or saved Design, tools, voice and greeting. A run's targets lists the comparison targets that each play that configuration: dialt (Dialt Fast), dialt-smart (Dialt Smart), an external model or custom. A run has one attempt per case, comparison target and repetition, with the simulator unchanged across them. To compare the patient above on both tiers:

case = evals.upsert_case(patient_case)
run = evals.start_run([case["id"]], modality="text", targets=["dialt", "dialt-smart"], repetitions=3)

Which tier runs. Cases never set brain. Without targets a run uses dialt, or dialt-smart when every case's target is a saved Smart Design; target Designs saved on different tiers need explicit targets. An explicit targets list overrides a target Design's saved tier: ["dialt", "dialt-smart"] runs every case on Fast and on Smart. The dry run and the created run report each override in warnings, such as "Dialt Fast: Patient discloses missed doses when asked: target.design is saved on Dialt Smart; this run target runs it on Dialt Fast". Each attempt's target_id and target_label name the tier it ran on, and for a case with a Design the run detail's snapshot.cases[].effective_target_brains maps each Dialt run target to its tier. Run targets never change the simulator: a simulator Design keeps its saved tier, and a simulator without a Design uses Dialt's pinned simulator configuration, billed at the Smart rate.

Check the simulator separately. A comparison is only as good as its control. Caller adherence (caller_adherence on the run, simulator_adherence verdicts on each attempt) reports whether the simulator followed its instructions, and a caller_probe confirms that a required simulator line was said and recognized. An attempt where the simulator broke its script is inconclusive and never counts as a target failure. Read adherence beside every target comparison. Names such as caller_probe, caller_adherence, caller_response_latency, simulator_ended and the custom adapter's "opening": "caller" refer to the simulator side, whatever role it plays.

Message roles. Transcript role values are technical message roles and say nothing about scenario roles: assistant rows are the target's speech and user rows are the simulator's. Inside each session, each model sees its own words as assistant messages and the other side's as user messages, so the clinician simulator's questions are assistant messages in its own session while the result transcript lists them as user. Name the scenario roles, such as patient and clinician, in instructions, criteria and reports. The dashboard labels turns Target and Simulator.

Build a suite with a coding agent

Give your coding agent the eval-building skill and describe your agent's purpose and tools. The skill guides case refinement and the transition from text validation to voice evaluation.

Use https://dev.dialt.com/docs/skills/dialt-evals/SKILL.md to build an eval suite for my education enrollment agent. It qualifies leads and books advisor calls. Use mock CRM/calendar tools, then compare Dialt Fast and GPT Live in voice.

Choose voice interruption modes

Enabling assistant interruptions requires an organization-owned voice run. Personal workspaces may store the case configuration, but run preflight rejects enabled target or simulator settings, including inherited Designs, before reserving credit. See the session contract for access rules and playback protection.

For voice cases, set target.assistant_interruptions and simulator.assistant_interruptions independently to "off" (the default), "pauses" (during the other side's pauses) or "speech" (during the other side's speech). For example, "target": {"instructions": "Help the customer.", "assistant_interruptions": "off"} and "simulator": {"instructions": "Act as a customer correcting an incorrect order.", "assistant_interruptions": "speech"} allow only the simulated caller to take the floor early when justified. The caller is the assistant in its own Dialt session. This setting is voice only, applies to Dialt targets and the simulated caller, and does not configure external targets. Omission leaves each side off; selecting a mode permits interruptions without requiring them or changing the persona. Cases and run snapshots preserve explicit selections.

Organize cases into suites

A case describes one scenario and its criteria. A suite is a saved collection of cases. A run records one execution and its results.

Create cases and suites, and start runs with the Python SDK or API. A case can belong to several suites. Hosted runs use your account's credits.

Use the Evals dashboard to filter a suite's history, compare models and inspect individual results. Each run keeps the suite name and case contents from the time it started. Editing or deleting a suite leaves past results unchanged; deleting a suite also keeps its cases. Empty suites can be saved but need a case before they can run.

Eval API

Authenticate with your account API key using Authorization: Bearer dk_... (existing ck_... keys remain valid). Machine requests use https://api.dev.dialt.com/v1; the paths below include that /v1 prefix.

• GET/POST /v1/evals/cases: list or create cases.
• GET/POST /v1/evals/suites: list or create suites; supply name and case_ids.
• GET /v1/evals/targets: available models, what each supports (capabilities) and the external credit reservation per attempt. dialt runs the case's target on Dialt Fast and dialt-smart on Dialt Smart; openai-live and gemini-live are external comparisons. The retired dialt-genius target can no longer be selected; runs that used it still load with that target_id.
• eleven-realtime-mini and eleven-realtime-max compare Eleven Realtime v1 Mini and v1 Max in hosted voice runs. Both use the Brian voice and support ordinary function tools with fixtures, greetings and call ending. Text runs are unsupported. Dialt manages the provider credentials; the same external rate card and 40-attempt run cap apply.
• POST /v1/evals/runs: start a run with suite_id or case_ids, modality, repetitions and optional comparison targets (how they select the tier). Add "dry_run": true to check it first without creating anything.
• GET /v1/evals/runs and GET /v1/evals/runs/{id}: history and detailed results.

Collection responses use {results, next, has_more}. Requests accept an optional opaque signed cursor and an optional limit from 1 through 100 (default 50). Follow each returned next URL verbatim until it is null; never decode or construct a cursor. has_more is true exactly when next is non-null. Cursors are bound to the account, route, surface/origin, filters, ordering and limit; authorization is checked again on every request, and malformed or mismatched cursors return a structured 400 error. Cases, suites and runs are ordered by descending created_at, then UUID. Traversal is a live view rather than a snapshot; records created or deleted during traversal may change the collection. The SDK preserves its list-returning methods while following every continuation. Errors use {error: {code, message, details?}}.

Each target publishes the case settings it runs in capabilities: target_fields and tool_fields, the settings it ignored (accepted with a warning, such as web_search: false, voice or a tool's recorded_bridge on an external model; the value must still be valid), and limits. limits.timeout_s is the case conversation limit: at most 300 seconds for external models. It is separate from billing.cap_seconds, the billable ceiling per attempt, which adds the farewell drain and bounded shutdown. A run request with "dry_run": true checks every case against every selected target. It returns 400 with every problem in error.details.errors (each has target, case, field and message; target, case or field is null when the problem is not specific to one), or 200 with the plan: its warnings, attempt count, the max_external_charge_micros to accept and whether your credit covers the run. The plan also returns valid, modality, execution, repetitions, targets, billing and credit (required_micros, sufficient; null for Dialt-only runs). It creates no run, reserves no credit and records no idempotency key. A created run keeps the same warnings.

For external targets, accept max_external_charge_micros: the sum of their per-attempt reservations times the case count and repetitions. The dry run returns this value. Unused credit is returned. The Python SDK's start_run takes the same targets, dry_run, max_external_charge_micros, replace_unassessable and max_replacements.

Evaluate your deployed agent

Host a WebSocket adapter around your existing agent and select custom. Keep your own prompt and tools. One custom agent can run alongside built-in targets in each hosted voice run. Reference adapter recipes live in dialt-ai/dialt-recipes.

import os
from dialt.evals import EvalsClient

evals = EvalsClient(os.environ["DIALT_API_KEY"])
request = dict(
    suite_id="YOUR_SUITE_UUID", modality="voice", execution="hosted",
    targets=["custom"],  # Add "dialt" or "dialt-smart" to compare.
    custom_agent=dict(url="wss://agent.example.com/evals", label="My agent",
                      token=os.environ["EVAL_AGENT_TOKEN"],
                      transcripts=False, tool_events=False),
)
plan = evals.start_run(**request, dry_run=True)
run = evals.start_run(**request, idempotency_key="custom-run-1",
                      max_external_charge_micros=plan["max_external_charge_micros"])

The HTTP body uses the same snake_case fields. TypeScript callers can use EvalsClient.planRun(request) and startRun(request, {idempotencyKey, maxExternalChargeMicros}) from @dialt/sdk/evals, or generated createRun from @dialt/sdk/http. The endpoint must be public wss on port 443 with no userinfo, query or fragment. Authenticate the upgrade using Authorization: Bearer TOKEN. Dialt encrypts this token privately and never returns it in results. Keep it in backend secrets. Existing external comparison charges and reservations apply, plus the simulated caller; you also pay your agent's provider costs.

Case target instructions, voice and end_call are ignored with warnings. Target tools, fixtures and Dialt-only state checks (fixture_complete, policy_flag) are rejected. Your application executes its own tools. tool_called/tool_not_called need tool_events: true; caller_probe needs transcripts: true. Use a dry run to check the complete case/target matrix before spending credit.

Adapter protocol, version 1

Control messages are JSON text frames. Audio in both directions is binary signed little-endian PCM16, 16 kHz, mono. Dialt sends the following start message; the opening is agent when the case has a target greeting, otherwise caller. The greeting text is not sent to your agent.

{
  "type": "start", "version": 1,
  "audio": {"encoding": "pcm_s16le", "sample_rate": 16000, "channels": 1},
  "session_id": "ATTEMPT_SESSION_UUID", "opening": "agent", "timeout_s": 300,
  "capabilities": {"transcripts": false, "tool_events": false}
}
{"type":"ready","version":1,"session_id":"YOUR_OPTIONAL_SESSION_ID"}
{"type":"begin"}

The first and third frames above come from Dialt; your adapter sends ready within 30 seconds of connection setup, with an optional session ID of at most 128 characters. Ready means the bridge is prepared; it may launch the provider on begin. Wait for begin, sent only after both parties connect, before producing any output. For agent-first opening, use your deployed agent's own greeting and start speaking within eight seconds of begin. Dialt paces input audio in 20 ms chunks, including silence. Send only output speech, without idle silence, in nonempty even-length binary frames of at most 32,000 bytes. Pending output may not exceed 30 seconds and total JSON evidence may not exceed 2 MB.

Send {"type":"interrupt"} to clear pending output playback; Dialt responds with {"type":"playback_stopped"}. Send {"type":"end"} only after all final audio. Produce no more output after end and keep the socket open while Dialt drains playback and sends {"type":"stop"}. Release your agent resources on stop or disconnect. An adapter may send {"type":"error"}; any supplied error string is not persisted.

With transcripts: true, send complete turns as {"type":"transcript","role":"user","text":"Hello","turn_id":"turn-1"} or use role assistant. Text must be nonblank and at most 20,000 characters; turn IDs must be unique per role, nonempty and at most 128 characters. Corrections of completed turns are unsupported. With transcripts disabled, results use the simulated caller's generated words and recognized reply; these do not establish what your target received or recognized.

{"type":"tool_call","id":"call-1","name":"lookup_order","args":{"order_id":"123"}}
{"type":"tool_result","id":"call-1","name":"lookup_order","result":{"status":"shipped"},"outcome":"succeeded","verified":true}

Send these optional events only with tool_events: true. IDs and names must be nonempty strings of at most 128 characters; args must be an object. Call IDs must be unique; one result must match the preceding call's ID and name. Results require outcome (succeeded, failed or unknown) and boolean verified. Report whether your application actually verified the outcome. Events are evidence only: Dialt never runs these tools. Inspect recordings and evidence before drawing quality conclusions.

Set "replace_unassessable": true so each scheduled attempt (case, target and repetition) ends with an assessable result. An attempt that produced no evidence about the model, because of an infrastructure error, an eval worker restart, a harness or simulated caller connection failure, a simulated caller that broke its script, or a simulated caller that heard the model and stopped responding, gets one fresh replacement attempt in the same run. A passed or failed attempt, a failure on the model's side (including a model that never opened the call), a failure with no established cause and an attempt with a failed check are never replaced. The replaced attempt keeps its transcript and returns replacement_decision (for example the label "Replaced: simulated caller broke script") and counts_in_slot: false; the replacement returns replacement: 1 and replaces_attempt_id. Run status, pass/fail and criterion_scores count the replacement; caller adherence and latency count every conversation. max_replacements caps replacements across the run (default a quarter of the attempts, rounded up). Replacements cost like any attempt, so the option is off by default: for external models, max_external_charge_micros includes one reservation per external replacement the budget allows, and the dry run quotes it. A replacement that would exceed that charge or your credit is skipped with its reason. Each run returns replacements with the replacements made, those skipped by reason and the attempts spent.

For example, compare an existing suite using requests. Set DIALT_API_KEY and DIALT_EVAL_SUITE_ID in your environment:

import os
import requests

api = "https://api.dev.dialt.com/v1/evals"
client = requests.Session()
client.headers["Authorization"] = "Bearer " + os.environ["DIALT_API_KEY"]

run = {
    "suite_id": os.environ["DIALT_EVAL_SUITE_ID"], "modality": "voice",
    "targets": ["dialt", "openai-live", "gemini-live"],  # add "dialt-smart" to compare Dialt Smart
    "repetitions": 1,
}
plan = client.post(api + "/runs", json={**run, "dry_run": True}, timeout=30)
if plan.status_code == 400:
    details = plan.json()["error"].get("details", {})
    for problem in details.get("errors", []):
        print(problem["message"])
    print(details.get("reason", ""))
plan.raise_for_status()
for warning in plan.json()["warnings"]:
    print("ignored:", warning["message"])
response = client.post(api + "/runs", json={
    **run, "max_external_charge_micros": plan.json()["max_external_charge_micros"],
}, timeout=30)
response.raise_for_status()
print("https://dev.dialt.com/evals/" + response.json()["id"])

Write a case

A case is a JSON document: name, starter, target and simulator (each a session mode document exactly as a session start carries it: instructions, greeting, tools, voice, web_search, end_call, and every other session option; for the target end_call defaults to true, and the simulated user's tools, end_call and greeting are set by the run), fixtures, checks and limits (max_turns, timeout_s, silence_s).

Authorized multispeaker Owners can add participants: [{"id": "customer", "name": "Customer"}, {"id": "colleague", "simulator": {"instructions": "Act as the colleague who disputes the bill."}}] to a hosted Dialt voice case. Supply one to four unique IDs using 1 to 64 ASCII letters, digits, underscores or hyphens; names are optional and at most 100 characters. Each caller inherits root simulator defaults and may override them with an ordinary simulator mode. The first inherits root starter; each may set its own starter. A target greeting excludes every caller starter. Each simulator hears the target and the other callers without hearing itself. This pilot requires ordinary multispeaker authorization and supports WebSocket hosted Dialt voice runs; external targets, text, local runs, WebRTC and Designs are rejected before connections. One cohort consumes one concurrent attempt from the Owner's limit and reserves all planned sessions against deployment capacity. Results retain participant-tagged evidence, caller adherence verdicts and participant_sessions; simulator_session_id remains the first caller for compatibility.

A run keeps a copy of its cases, so editing a case later does not change past runs.

Talk to a case

Every case response includes playground_url, and so does each run attempt (null once its case is deleted). Open it while signed in with the case's owner (personal or organization) selected to talk to the case's target in the playground: you play the simulator, whose instructions and opening line are shown on the page, and the case's fixtures answer the target's tool calls. Use it to hear an agent before wiring up your own tool host, to check that a new case's scenario is realistic, or to reproduce a failed attempt yourself. The conversation is an ordinary playground session in Sessions; it is not an eval attempt and is not scored as part of a run.

Align tools, fixtures and checks

The tool tells the agent what it can do. Its fixture supplies evidence of what happened. A check assesses that declared contract; it must not introduce a rule the agent never received.

For example, describe request_callback(phone) as: "Submit a callback request once the caller supplies a phone number. Success records the request; it does not book a time or guarantee a callback." Describe the phone format in its parameter schema. Return {"result": {"status": "requested", "request_id": "cb-1"}}.

Use {"type": "tool_called", "value": "request_callback", "args": {"phone": "+15555550123"}} to check submission. Separately judge: "After the request receipt, the agent may confirm submission but must not promise contact or a booked time." "Your request was submitted" meets that contract; "Someone will call you tomorrow" does not, even if submission succeeded.

State prerequisites, optional-field meaning, partial submission rules, effects and result guarantees in each tool declaration. Use fixture_complete for required stored fields. Reserve judges for meaning or required ordering, and do not restate a deterministic check as a judge: both count in the criterion score, so the overlap counts one requirement twice. Set a weight to make a requirement count more; give each criterion an observable trigger and outcome, and separate caller coverage from target behavior. Specify whether a requirement applies throughout the conversation or at a particular step. If confirmation is required, make the invitation to confirm explicit in both the agent instructions and the criterion. When a result surprises you, first compare the declaration, fixture and criterion, then inspect the agent's actions.

Checks

Use a required phrase for something exact, such as a mandatory disclosure. Use a judge criterion for an outcome that needs context, such as "The agent states the cancellation date and does not claim success before the tool result." Both can be combined.

Checks must agree with the case: a phrase that requires an action while the judge forbids it can never pass, and the opening message must create the situation the instructions describe. Read a failed transcript before changing a prompt.

Start with a runnable simulation case that exercises tool fixtures and checks an outcome. Run it in text first, then in voice.

Text and voice

Voice runs pass generated audio between the two sessions. Nothing plays on a speaker.

External models run in text and voice. Neither Live model can answer in text, so in a text run each receives the caller's turn as text and its reply is the transcript of the audio it generated, which is discarded. Both receive the turn as a typed user message and speak the answer. Their results are labelled (text: transcribed audio output). Each target's modalities lists the run modalities it accepts.

Read a run

Total conversation time measures opening dispatch after both connections are ready through termination and queued speech drain. It includes tool waits and silence, but excludes connection setup, provider finalization and grading. The API returns conversation_duration_s; older attempts without a measurement return null. Compare the same cases and modality alongside pass rate: a short failed or prematurely ended conversation is not an efficiency win. This is conversation duration, not speech response latency.

Response latency in voice runs is the time from the caller's last word ending to the model's first word starting, measured by forced alignment of a Dialt session recording. For a Dialt model, response_latency uses the assistant's own session recording: the same figure as that session's aligned_response_ms, on its recording clock. OpenAI Live and Gemini Live have no Dialt recording, so their response_latency uses the simulated caller's recording: the caller's speech and the model's reply as the caller heard it, on the caller's recording clock. caller_response_latency measures every model that way, so compare Dialt and external models there; it includes the relay between the two sessions. Neither is a real caller's device latency. Each latency object names its measurement and returns state, n, p50, p90 and per-turn gaps in milliseconds; each run pools turns per model in targets. Alignment finishes after the conversation, so recent results report pending first. Text runs report unavailable with a reason, never zero. case_snapshot_sha256 identifies the cases, so you compare latency and scores only across runs of the same cases, targets and brain tiers. A Dialt target shows its tier (Dialt Fast or Dialt Smart), not the models that serve it.

A run has one result per case, repetition and selected model. Each result shows status, modality, transcript, tool calls, checks, why the conversation ended, any error, and links to the available session recordings.

Criterion score is the weighted share of criteria met: every judge criterion and every deterministic check (contains, tool_called, fixture_complete and the rest). Each attempt returns criterion_score with passed, failed and unknown counts. With no unknowns, score is the fraction met; otherwise score is null and lower and upper bound it, with coverage the assessed share. Each run returns criterion_scores, one per model: repetitions are averaged within a case, then every scheduled case counts equally. An attempt that could not be judged keeps its criteria as unknown, so the range widens instead of the case disappearing. A check never passes on missing evidence: when the agent never spoke or called a tool, or the conversation was cut short or inconclusive, a phrase that was said or a tool that was called still fails a not_contains or tool_not_called check, and every verdict that rests on absence is unknown. Cases without criteria are reported as unscored. caller_probe assesses the simulated caller and counts only in pass/fail. Any other check may set a weight above 0 and at most 1000 (default 1). Each score reports its scoring_version; version 2 added deterministic checks, and every run is scored by the current version when read.

Pass rate adds sampling uncertainty to strict outcomes: 3 passes in 3 attempts is weak evidence. Each run returns pass_rates, one per model, with attempts, assessed (passed or failed), passed and unknown (inconclusive, error, cancelled or unfinished) counts, pass_rate over assessed attempts and a 95% Wilson interval in pass_rate_lower and pass_rate_upper. by_case gives the same for each case: 3 of 3 passes reads 44 to 100%. by_case is returned on the run detail only. The model's rate weights every case equally, like the criterion score, and its interval uses the effective sample size of that weighting. It covers only cases with an assessed attempt (assessed_cases), so compare models on that count too, and read unknown beside it: a dropped connection is unknown, whichever side dropped it.

Caller adherence reports whether the simulated caller followed its script. Each judged hosted attempt of a case with caller instructions carries one simulated caller coverage verdict (a judges entry of type simulator_adherence). When the case opens with target.greeting, the attempt also carries a second simulator_adherence verdict, "Simulated caller role after the target opens", which checks that the caller replies as the person its instructions describe; the caller followed its script only when every such verdict passes. A verdict with opportunity: false means the model ended the call before the caller's instruction applied; it is left out, and an attempt with no other verdict is not counted. An attempt cut short by an error, cancellation, stall or timeout counts only when that verdict is established, except a stall on the simulated caller's side (it heard the model and stopped responding), which counts in full. Each run returns caller_adherence, one entry per model, with assessed, followed, broke_script and unknown attempt counts and rate, followed over assessed. An attempt where the caller broke its script, or where that was not established, is inconclusive, so a low rate means fewer of your attempts were scored. The run page shows it as "Followed its script in 11 of 12 attempts".

The termination reason explains why the conversation stopped. completed means the target requested an end; simulator_ended means the simulated caller did. stalled means neither side spoke for silence_s. When the evidence shows which side owed the next reply, the attempt's ending.failure.side names it: target when the model recognized the caller's turn and never replied, or stopped mid-sentence, and simulator when the simulated caller heard the model and never replied. A stall on the model's side is a failed attempt and counts in pass rates. Any other stall, including one after a reconnect, an error, a pending tool call or speech the other side never recognized, and every timeout, is inconclusive. timeout_s bounds the conversation: an end already under way when it passes finishes as completed or simulator_ended, so timeout means neither side ended the call in time. target_did_not_open means a model given target.greeting did not start speaking its greeting within 8 seconds; the attempt is failed. Invalid caller coverage or an unavailable judge also makes an attempt inconclusive. Inspect the individual evidence before drawing a model conclusion.

Each hosted attempt also returns ending, which reports the parts of the end separately: conversation (whether it ended normally, who ended it and how long it ran), drain (whether the final speech was delivered within its grace window), shutdown (whether an external model's connection closed cleanly), failure (the failure code, the side and component that failed, and the phase), recovery (an attempt interrupted by an eval worker restart, and whether it runs again) and last_activity (when each side last spoke, last recognized speech and last received audio). Their values are fixed codes: conversation.outcome is ended, stopped or cancelled; drain is drained, grace_expired, limit_reached or cancelled; shutdown is closed or failed; for infrastructure_error, failure.cause is the public error code such as insufficient_credit, or null. ending is null for local runs and for attempts that have not run, except a queued attempt waiting after a worker restart, which carries only recovery. For an external model, an ending near the case limit is cut short so its farewell finishes within billing.cap_seconds. A hosted attempt's error is the failure code (ending.failure.code), evaluation_failed when an error has no classified failure, and empty when there is none. Every attempt carries a correlation_id; include it when you report a problem so Dialt can find the attempt in its logs.

Comparison runs display completion separately from each model's outcomes. For compatibility, the API run status can be failed when only coverage is incomplete; outcome="inconclusive" distinguishes this from scored failure.

Usage

Each attempt bills two sessions, rounded to connected seconds. The target bills at its model's rate: dialt at the fast rate, $0.065 per minute, dialt-smart at the smart rate, $0.085 per minute, and external models at $0.065 per minute. The simulated caller bills separately at the smart rate, $0.085 per minute, so a one-minute voice attempt against dialt costs about $0.15. External models reserve up to billing.cap_seconds per attempt (330 seconds, $0.3575); unused credit is returned, and external infrastructure failures are waived. Text runs bill Dialt sessions at the ordinary Dialt text rate; an external model in a text run still bills its connected seconds at $0.065 per minute, capped at the same reservation.

Safe retries

POST /v1/evals/cases, POST /v1/evals/suites and POST /v1/evals/runs accept an optional Idempotency-Key header. Runs also accept an idempotency_key body field; if both are supplied they must match. Keys are 1 to 128 visible ASCII characters without spaces and are scoped to the authenticated account. Reuse a key with identical JSON request content after a lost response to receive the original case, suite or run (201) without a duplicate, and for a run without duplicate execution or credit reservation. Object-key order does not matter; array order and explicit versus omitted defaults do. Different content with the same key returns 409. Keys remain reserved for the resource's lifetime; replaying a case or suite key after deleting that resource returns 409, and after replacing it with PUT returns its current content. Without a key, each request creates new work.

Case and suite names are unique per account. Creating or renaming one to a name already in use returns 409 with the existing resource's ID in error.details.existing_id, so a script can adopt it or rename.

A repeated local report key must carry the same normalized evidence. Changed evidence returns 409; it no longer silently returns the earlier report.

Public API responses expose Dialt tiers, including for team-role requests. Hosted target_config contains the customer's frozen target input, not the private provider execution configuration. Platform-owned target profiles, judge results, usage and events are projected to customer fields; exact model identities remain in authorized internal diagnostics. Customer-authored local reports and conversation content retain their own content.

An API key sent to the dashboard origin returns 401 with a message naming the API origin to use.

When a run fails

1. Read the status, the termination reason, the error and ending: a failure names the side and component that failed, and last_activity shows which side went quiet.
2. Compare the transcript with each check.
3. Check that the opening message actually creates the situation in the instructions.
4. Open both session recordings for the audio and the full transcript.
5. Retry only once you know whether the agent misbehaved or the run failed for a technical reason.

A queued or running run can be cancelled from its page or with evals.cancel_run(run_id). To delete a finished run and its results, use evals.delete_run(run_id) or DELETE /v1/evals/runs/{id}. Deletion returns 204; active runs return 409, so cancel and wait for completion first. Missing or already deleted runs return 404. Cases, suites, session recordings and billing history are preserved. Reusing a deleted run's creation idempotency key returns 409.

View sessions · Open evals