# Controlled Skill A/B validation

Use A/B testing to measure the Skill's creative effect instead of judging MCP plumbing.

1. Call `create_skill_ab_test` once. Freeze one source brief, World/reference set, model route, aspect ratio, duration, resolution, attempt budget, and scoring rubric.
2. Keep the baseline run free of ElserStudio Skill modules. It may use the same MCP capabilities but must not consume Skill prompt or directing guidance.
3. Load and record every relevant module in the Skill run. Keep all non-Skill inputs identical.
4. Do not silently replace a failed generated script or media result. Record the degradation and preserve the failed evidence.
5. Score both variants only after comparable outputs exist. Use the same reviewer and rubric.
6. Call `score_skill_ab_variant` for both variants. The desktop validates metric ranges and computes a normalized weighted score.
7. If media generation or review cannot occur, finish both paired runs explicitly, then call `finish_skill_ab_test` with `blocked`, `failed`, or `canceled` and preserve the concrete reason. Never leave an impossible comparison as `running`, and never fabricate scores.

Recommended 0–5 metrics for a two-shot video comparison:

- `story_legibility`: the intended cause, change, and endpoint are readable without explanation;
- `identity_consistency`: canonical characters, scene, props, and style remain stable;
- `action_completion`: each Shot performs one visible job and reaches its declared endpoint;
- `camera_intent`: framing and movement serve the narrative intention;
- `continuity`: state, prop ownership, direction, light, and audio handoff remain coherent;
- `audio_dialogue`: speaker, wording, timing, voice reference, and lip-sync are usable;
- `artifact_rate`: anatomy, morphing, flicker, text, and material failures remain controlled.

Do not claim a Skill win from static asset generation alone. For Seedance direction, test at least two connected or deliberately comparable video Shots when credits and inspection access permit.
