Accuracy reporting
Outcome signal coverage
Outcome coverage100%
300/300 sessions have score evidence
Selected source100%
reward coverage
Observed success56.7%
56.7%
Low-confidence scores550
3,880 total score records
reward
LLM judge
Workflow evaluation
Sample size
Total estimate—Claude Sonnet 4.6
Discovery——
Final run——
Range—pricing source
Selected score source
Outcome distribution
Success170
Failure130
Partial0
Unknown0
Analyzed sample
Quality by workflow
WorkflowCoverageSuccessOutcome gapTop signal
Flight Reservation Modificationderived-flight-reservation-modification100%52.6%47.4 ppOutcome cohort gapFlight Cancellation Refundderived-flight-cancellation-refund100%60.9%39.1 ppOutcome cohort gapOtherderived-other100%59.6%40.4 ppOutcome cohort gapFlight Bookingderived-flight-booking100%45.8%54.2 ppOutcome cohort gapFlight Delay Compensationderived-flight-delay-compensation100%66.7%33.3 ppOutcome cohort gapAccuracy opportunities
What could improve outcomes
Tool misuse
55 of 116 runs fail at full cost. Adding early exit checks before heavy tool calls could cut waste fast.
NextAdd a preflight prompt step to validate inputs before any tool calls run.
Flight Reservation Modification47.4 pp gaphigh riskmedium confidenceFlight Reservation Modificationaccuracy signalmedium riskmedium confidenceFlight Cancellation Refund39.1 pp gaphigh riskmedium confidenceFlight Cancellation Refundaccuracy signalmedium riskmedium confidenceFlight Booking54.2 pp gaphigh riskmedium confidenceOther40.4 pp gaphigh riskmedium confidenceOtheraccuracy signalmedium riskmedium confidenceFlight Bookingaccuracy signalmedium riskmedium confidenceFlight Delay Compensation33.3 pp gaphigh riskmedium confidenceFlight Delay Compensationaccuracy signalmedium riskmedium confidence
Outcome cohort gap
47% of runs fail or partially pass. Failed runs average 19.8 tool calls and 934 input tokens versus 9.9 calls and 565 tokens for successful runs.
NextSet a tool call limit per run to stop runaway loops early.
Tool error loop
The agent retried a broken tool call 193 times across 2 threads without changing its approach, burning context and making no progress.
NextAdd a circuit breaker that stops retrying after 3 identical errors.
Output contract mismatch
Users are fixing incomplete or incorrectly shaped responses from your assistant. This suggests the assistant isn't following its output rules consistently. A validator can catch these problems before users see them.
NextDefine exact output shape: required fields, data types, and format rules
Flight Reservation Modification8.6 pp correctedmedium riskhigh confidenceFlight Cancellation Refund4.6 pp correctedmedium riskhigh confidenceOther9.6 pp correctedmedium riskhigh confidenceFlight Booking33.3 pp correctedmedium riskhigh confidenceFlight Delay Compensation28.6 pp correctedmedium riskhigh confidence