Agent Quality Loop

What To Review

Signal	What it tells you
Confidence	Whether the agent believed it had enough context to answer.
Citations	Which sources and chunks shaped the answer.
Handoffs	Where the agent decided a human should take over.
Feedback	Operator or customer signals that an answer was useful or weak.
Skips	Questions the agent could not answer from current context.

Improvement Workflow

Review low-confidence conversations

Look for missing endpoints, unclear auth rules, stale examples, or ambiguous product language.

Fix the source

Update the external docs, upload better reference text, or add a repository with examples.

Rescan context

Re-ingest the source from Sources so the retrieval layer has the latest material.

Re-run learning

Run Context Learning when source coverage or agent routing changed materially.

Measure again

Use Analytics to watch answer rate, handoffs, feedback, and survey outcomes.

Good Agent Behavior

Answer only from available workspace context.

Ask a focused clarifying question when details are missing.

Hand off to an operator when the question is account-specific, unsafe, or outside the knowledge base.

Cite relevant source material when available.

Avoid inventing endpoints, request fields, auth rules, schemas, limits, or production behavior.

Common Fixes

Problem	Fix
Agent knows the endpoint but not auth	Add auth examples and configure source auth testing.
Agent answers from the wrong product	Use agent-specific widget keys and source attachments.
Agent gives vague SDK answers	Ingest SDK examples or a GitHub repository with sample code.
Agent hands off too often	Improve source coverage and review confidence settings.
Agent answers too confidently	Tighten guardrails and review retrieval evidence.

Failure Layers

Layer	What to inspect
Retrieval	Did the right source, endpoint, or chunk appear?
Context pack	Did the prompt receive enough selected evidence to answer?
Prompt or guardrail	Did instructions conflict with safe behavior?
Confidence policy	Was the threshold appropriate for the evidence?
Answer contract	Did the final answer cite API facts and avoid unsupported claims?
UI/debug	Did operators see the skip reason, citations, and confidence clearly?
Provider failure	Did the provider error, timeout, or BYO configuration fail safely?

​Agent Quality Loop