Module 9 of 11
Manage testing AI-powered business solutions
Test probabilistic systems properly: evaluation metrics, validation criteria, adversarial testing, and quality gates.
Exam questions Q81–Q90 8 units on Learn
Open module 9 on Microsoft Learn ↗ Watch the episode (32 min)
Why this module matters
Testing generative systems breaks the habits of deterministic QA. The same input can produce different valid outputs, so assertions must target behaviour and quality ranges rather than exact strings.
The module also covers end-to-end scenario testing across multiple Dynamics 365 apps, and using Copilot itself to help generate test cases — with human review of what it produces.
The one big ideaYou cannot assert equality against a probabilistic output. You assert groundedness, relevance, task success, safety, and consistency — measured against a curated dataset with agreed thresholds.
What Microsoft Learn covers
Units in this module, as published on Microsoft Learn:
- Introduction
- Recommend process metrics for testing AI agents
- Create validation criteria for custom AI models
- Validate effective Copilot prompt best practices
- Design end-to-end test scenarios for AI solutions using multiple Dynamics 365 apps
- Build a strategy for creating test cases using Copilot
- Module assessment
- Summary
Watch this module
Microsoft Learn publishes a free video course for AB-100 on YouTube. This module is covered by one 32 min episode. An instructor walking through the module is often faster than reading it twice.
Key concepts and definitions
| Probabilistic output | Valid responses vary between runs. Testing must therefore measure quality characteristics, not exact matches. |
| Groundedness | Is the answer supported by the retrieved sources? A primary metric for retrieval-grounded solutions. |
| Relevance | Does the answer actually address the question asked? The natural companion metric to groundedness. |
| Task success rate | Did the interaction achieve the user's goal end to end? The closest proxy for business value. |
| Regression testing | Re-running known-good cases to detect breakage after changes to prompts, knowledge, models, or tools. |
| Safety / red-team testing | Adversarial testing that the agent refuses harmful, out-of-policy, or manipulative requests, including prompt injection attempts. |
| Validation criteria | The agreed, measurable definition of "good enough" for a model or agent before it may be promoted. |
| Quality gate | A threshold — groundedness, task success, safety — that must be met for a release to proceed. |
| End-to-end scenario test | Following a complete business process across the agent, its tools, and downstream systems. The best evidence of real business value. |
| User acceptance testing | Representative end users and business process owners performing genuine tasks, not the delivery team clicking through. |
| Synthetic / masked test data | Test data that preserves realism without exposing personal information, protected to the same standard as production. |
Design guidance
Building the test strategy
- Assemble a curated evaluation dataset covering common paths, edge cases, and known past failures.
- Layer the testing: component tests for tools and connectors, quality evaluation for generative behaviour, end-to-end scenario tests for business value, and adversarial tests for safety.
- Use Copilot to help draft test cases, then have a human review them for coverage and correctness — generated cases are a starting point, not the final suite.
Gates and promotion
- Define thresholds that must pass before promotion. "Zero defects" is not a workable gate for probabilistic systems; agreed thresholds are.
- Re-run the evaluation suite whenever the platform updates the model version — behaviour can shift even with no change on your side.
- Automate evaluation pipelines so the suite runs on every significant change; manual-only testing does not scale.
Data and participants
- Use masked, synthetic, or minimised data in test environments, with production-grade protection.
- Involve real users and process owners in UAT — they surface workflow mismatches no internal tester will.
- Test the fallback and escalation paths deliberately; they are where poor agents are exposed.
Good practice vs. anti-pattern
Do
- Measure groundedness, relevance, task success, and safety.
- Automate evaluation runs on every significant change.
- Re-test after platform model updates.
- Use masked or synthetic data with full protection.
Avoid
- Assert exact-string equality on generated text.
- Treat UAT as a developer walkthrough.
- Copy production personal data into test environments.
- Assume a model update is behaviour-neutral.
Exam traps to watch for
TrapDeterministic testing fails here because outputs are probabilistic — not because tooling is unavailable or automation is impossible.
TrapFor retrieval-grounded answers the metric pair is groundedness and relevance, not infrastructure metrics like CPU, storage, or uptime.
TrapA platform model update requires re-running evaluation and regression tests. Behaviour is not guaranteed identical, and rebuilding from scratch is not required either.
TrapRelease decisions hinge on agreed quality gates, not on zero defects and not on a single maker's judgement.
Readiness checklist
You are ready to move on when you can do all of these without notes.
- Explain in one sentence why exact-match assertions fail for generative output.
- Name the metric pair for retrieval-grounded answers.
- Describe the four layers of an AI test strategy.
- State what must happen when the platform updates the model version.
- Define a quality gate and give two example thresholds.