Module 8 of 11
Monitor, analyze, and tune AI agents
Instrument agents, read the telemetry, and run a continuous improvement loop that keeps quality rising.
Exam questions Q71–Q80 8 units on Learn
Open module 8 on Microsoft Learn ↗ Watch the episode (40 min)
Why this module matters
An agent is never finished. Content ages, models are updated beneath you, and user behaviour shifts. This module is about the operational loop that keeps quality from quietly eroding.
The exam focuses on metric interpretation: given a symptom, identify the likely cause and the correct corrective action. Learn to read the *pattern*, not just the number.
The one big ideaMonitor → analyse failures → adjust knowledge, instructions, or tools → re-evaluate → release. Continuously. Tuning once at go-live is not an operating model.
What Microsoft Learn covers
Units in this module, as published on Microsoft Learn:
- Introduction
- Recommend process tools for monitoring agents
- Analyze backlog and user feedback for AI agent usage
- Apply AI-based tools to analyze, identify issues, and perform tuning
- Monitor AI agent performance metrics
- Interpret telemetry data to tune AI performance
- Module assessment
- Ensure reliable AI agent operations
Watch this module
Microsoft Learn publishes a free video course for AB-100 on YouTube. This module is covered by one 40 min episode. An instructor walking through the module is often faster than reading it twice.
Key concepts and definitions
| Containment / resolution rate | The share of interactions resolved without human escalation. The clearest single measure of whether the agent actually helps. |
| Engagement vs. resolution | High engagement with low resolution means people try the agent but leave unsatisfied — typically knowledge gaps or unclear instructions. |
| Groundedness | Whether responses are supported by retrieved sources. Confident answers with no citation support indicate hallucination. |
| Evaluation dataset | Representative inputs with expected outcomes, used to measure quality consistently over time. It tests the agent; it does not train the model. |
| Telemetry / tracing | Logged reasoning and tool-selection detail — inputs, chosen tool, parameters, outcome — the primary evidence for diagnosing wrong-tool behaviour. |
| Escalation / fallback rate | How often the agent gives up or hands off. A spike is an early warning of regression. |
| Agent scorecard | A balanced view across quality, adoption, efficiency (cost and latency), and safety/compliance. Any single metric can be gamed. |
| A/B testing | Running two variants — for example two instruction sets — to determine empirically which performs better before full rollout. |
| Content decay | The gradual staleness of grounding sources that silently degrades answer quality. |
Design guidance
Reading the symptoms
- High engagement, low resolution → knowledge coverage gaps or ambiguous instructions.
- Confident answers without citations → grounding failure; tighten retrieval and require citation.
- Wrong tool chosen → inspect tracing logs, then improve tool descriptions and parameter schemas.
- Rising fallback rate after a release → regression; compare against your evaluation baseline.
- Good quality, poor adoption → discovery, training, or workflow-fit problem, not a model problem.
Improving retrieval
- Curate, chunk, and maintain authoritative content with accurate metadata. Adding more unrelated documents makes retrieval worse, not better.
- Retire superseded content actively.
- Narrow retrieval scope where possible — it improves both accuracy and cost.
Alerting and cadence
- Alert on spikes in fallback or escalation rate, error rate, latency, and content-filter triggers — not on ordinary successful responses.
- Re-run the evaluation suite whenever the model version, instructions, knowledge, or tools change.
- Feed user feedback and the backlog into a prioritised improvement queue rather than ad-hoc fixes.
Good practice vs. anti-pattern
Do
- Maintain a curated evaluation dataset with expected outcomes.
- Trace tool selection so failures are diagnosable.
- Score quality, adoption, efficiency, and safety together.
- A/B test instruction changes before rolling out.
Avoid
- Judge success on session counts or token averages.
- Add documents indiscriminately to fix accuracy.
- Wait for loud complaints before tuning.
- Alert on every successful interaction.
Exam traps to watch for
TrapAn evaluation dataset measures the agent; it does not train the base model.
TrapMore documents is not better grounding. Curation, chunking, and metadata are what improve retrieval.
TrapTuning is a continuous loop, not a go-live activity and not an annual ritual.
TrapA balanced scorecard needs all four dimensions. Cost-only, latency-only, or satisfaction-only answers are distractors.
Readiness checklist
You are ready to move on when you can do all of these without notes.
- Map four common symptoms to their likely causes and fixes.
- Explain the purpose of an evaluation dataset in one sentence.
- Name the four dimensions of a balanced agent scorecard.
- Say which telemetry you need to diagnose a wrong-tool error.
- List the conditions that justify a production alert.