1. How should you structure the eval pyramid for a customer-support agent?
A Everything goes through an LLM judge, since support quality is subjective — deterministic assertions can't capture tone or empathy, so one well-prompted judge scoring every case end to end is the most consistent single instrumentB Deterministic assertions for checkable behavior (right tool, valid JSON, refund-limit routing), a validated LLM judge for faithfulness/tone, and sampled human review for the subtle cases and judge driftC Only human review, because customers deserve human judgmentD A single 1–10 quality score from a frontier model, averaged across all cases — one holistic number is easier to trend across releases than juggling separate assertion, judge, and human-review results
2. Why must an LLM judge be validated before you trust it, and how concretely?
A It doesn't need validation if you use a frontier modelB Because an unvalidated judge may be confidently noisy; validate by hand-labeling ~30+ examples, running the judge on them, computing judge-human agreement, and tuning the rubric until it clears a bar you set in advanceC Validate by asking the judge to rate its own confidence on each verdict and discarding the low-confidence ones; a frontier model's self-reported certainty is a reliable stand-in for measured agreement with human graders, so no hand-labeling is neededD Run the judge twice on the same examples and check that it agrees with itself; high self-consistency across reruns proves the rubric is unambiguous, which is the property validation is really measuring
3. Which set correctly names three LLM-judge biases and a valid mitigation for each?
A Position bias → run comparisons in both orders and require consistency; verbosity bias → anchor the rubric on correctness/relevance; self-preference bias → use a different model as judge plus human samplingB Recency bias → shorten the prompt so nothing important falls out of the window; anchoring bias → raise the temperature so the judge samples more diverse verdicts; halo bias → force JSON mode so structure keeps the score honestC Cost bias → cache the judge's verdicts by input hash; latency bias → stream the judge's reasoning; token bias → truncate long answers before grading so length can't sway the scoreD There are no reliable biases in LLM judges; they are objective
4. Why track both task success rate and per-step correctness?
A They're the same metric measured twice for reliabilityB Per-step correctness is only a debugging aid; task success is the metric users actually experience, so once task success is high you can stop collecting per-step data to save tracing overhead and dashboard noiseC A large gap reveals fragile recoveries: high task success with low per-step correctness means the agent is limping to the finish on lucky recoveries that break under distribution shiftD Task success is cheaper to compute at scale, so per-step correctness is optional — reserve it for the small subset of runs that fail outright, where the failing step is worth localizing
5. Explain the lethal trifecta and apply it to an email-assistant agent.
A It's three prompt-engineering techniques — role assignment, few-shot examples, and chain-of-thought reasoning — and the email agent should layer all three in its system prompt so the model reliably separates content to process from commands to obeyB Private-data access + untrusted-content exposure + external communication = exfiltration risk; the email agent has all three (reads inbox, reads incoming mail, can send), so cut a leg — e.g., require human approval to sendC It's the three most expensive model calls; batch them to save costD It refers to misconfiguring temperature, top_p, and max_tokens at once; the email agent should pin all three to conservative defaults so sampling variance can never produce a harmful send
6. Why is 'just filter the input for injection phrases' insufficient, and what's the layered alternative?
A It's actually sufficient if the blocklist is maintained well — keep a regularly updated list of known injection phrases across major languages, encodings, and paraphrases, run it on every fetched document before it enters context, and coverage converges toward complete over timeB Attacks come in infinite paraphrases, other languages, encodings, and markdown-image exfil, so a blocklist is a speed bump; the alternative is defense in depth: privilege separation, input demarcation, output filtering, and HITL for consequential actionsC Filtering is fine but slow; cache the filter resultsD Replace filtering with a frontier model as the agent, since larger models are specifically trained to ignore embedded instructions and therefore can't be injected the way smaller open-weights models can
7. What makes a good HITL approval UX — what does the approver need to decide in ten seconds?
A The full model transcript, every system prompt, and all intermediate tool results, since an approver can't make a genuinely informed call without the complete context the model sawB Just an approve/reject button with no other context, because any detail you surface slows the human down and the entire point of a ten-second gate is speedC The raw token counts and latency of every prior spanD The action, its target, the reason, and the cost or blast radius — on one line — with a default-reject timeout and an audit log behind it
8. Your agent's cost doubled week-over-week with flat traffic. Using traces, how do you diagnose it?
A Assume the provider raised prices and move onB Turn off tracing first, since the observability overhead itself — extra spans, logged tokens, metadata writes on every call — is billed like everything else and is the usual hidden culprit when cost climbs while traffic stays flatC Since traffic is flat, cost-per-run rose; slice per-span to find the culprit — cache misses re-billing the prefix, extra loop turns from an erroring tool, ballooning output tokens, or one heavy userD Retry every failed run and re-average, since transient tool errors inflate apparent cost and a clean re-run establishes the true baseline before any per-span digging is worthwhile
9. What belongs in a prompt-change CI pipeline?
A Only a manual eyeball check on a few known-good prompts before merge, since judged suites cost real API money per run and prompt diffs are short enough to review carefully by handB Deterministic suite on every commit, judged suite on prompt/tool/model changes, a cost-budget check, pinned model versions, and a diff-friendly PR report — with a non-zero exit gating the mergeC Just the judged suite, since deterministic assertions only catch trivial breakage that code review sees anyway, and judged cases are the only ones sensitive to the subtle behavior shifts a prompt or model change actually causesD Nothing automated; prompts aren't code
10. In an HITL gate, what should happen when an approval request times out?
A Default to reject — fail closed — and record the timeout in the audit logB Default to approve so overnight work isn't blocked, and rely on the append-only audit log to reconstruct and reverse anything harmful after the factC Retry the action automaticallyD Escalate by executing with reduced permissions, since a scoped-down version of the action preserves the agent's autonomy while shrinking the blast radius to an acceptable level
11. What is the correct discipline for regression testing an agent?
A Re-run the demo occasionally and eyeball itB Every fixed bug becomes a permanent case in a suite that runs on every prompt, tool, or model change, exiting non-zero to block mergesC Only test after a customer complainsD Retire cases once they've passed for several consecutive releases — a suite that only grows slows CI, and a bug is unlikely to recur once the prompt that caused it has been rewritten
12. What are the essential elements of a blameless postmortem for an agent failure?
A The name of whoever wrote the prompt, a review plan for their recent changes, and a sign-off step so individual accountability is unambiguous the next time something like this shipsB A one-line apology and a promise to be more carefulC Timeline, root cause (not symptom), detection gap, fix, and the regression test that now makes recurrence catchable — framed as a system property, not a person's faultD A full unedited transcript of the failing run with no analysis attached, so every reader can draw independent conclusions untainted by the author's framing of events
13. Your team tracks a single blended number that averages a fixed regression suite with a capability benchmark. Why is this a mistake, and what should you do instead?
A It's not a mistake — one number is simpler for stakeholders to trackB Regression evals answer 'did we break something that used to work' and capability evals answer 'are we getting better'; blending them into one average can hide a change that raises capability while quietly regressing a previously-fixed case — report the two deltas separatelyC Drop the capability benchmark entirely: the regression suite encodes every real failure you've ever fixed, so it's strictly more informative, and capability scores mostly track the provider's model updates rather than your own changesD Keep the blended number but run it twice as often and alert on its derivative — with enough repeated samples the noise averages out, and a single trended KPI is easier for stakeholders to act on than two separate deltas
14. When is a judge ensemble (multiple judge calls, majority vote) worth its added cost and latency, versus a single judge call?
A Ensembles should always replace single judges — they're strictly more accurateB Ensembles are worth it where a single flipped verdict has outsized consequence (gating a merge or an autonomous refund); for aggregate trend tracking over many cases, single-judge noise averages out and the budget is better spent on more itemsC Ensembles are a bootstrap tool for the period before you have a validated rubric — the majority vote across several judge models substitutes for human calibration until judge-human agreement data exists, after which a single validated judge is always the right choiceD Ensembles remove the need for human calibration entirely: three independent judges agreeing is stronger evidence than thirty hand-labels, so teams that run ensembles can skip the labeling step
15. A regression suite improves from 82% to 84% (41/50 → 42/50) after a prompt change. What's the senior response?
A Ship it immediately — any improvement is good newsB Check whether the delta is distinguishable from noise: at n=50 the confidence intervals around 82% and 84% overlap heavily, so look at which specific cases flipped pass/fail (a paired comparison) rather than trusting the aggregate delta, and consider growing the suite before decidingC Revert immediately: any aggregate move under five points is meaningless by definition regardless of suite size, so the only safe policy is an automatic rollback whenever a delta is that smallD Set temperature to 0 on the agent and judge to eliminate the variance, then re-measure
16. What's the difference between offline evals and online production monitoring, and why do you need both?
A They're redundant; whichever is cheaper to run should be keptB Offline evals run a fixed, curated suite before shipping to catch known failure modes; online monitoring (canary sets replayed on a schedule, drift metrics, user-feedback signals treated as weak labels) watches live traffic after shipping to catch drift and unknown-unknowns like a provider-side model update — neither substitutes for the otherC Online monitoring replaces offline evals once you have enough production traffic — live users exercise far more inputs than any curated suite ever will, so a mature product can retire its pre-ship suite and rely on drift alerts to catch regressionsD Offline evals are for tracking cost and latency budgets under controlled load; online monitoring is where correctness gets measured, since only live traffic reveals whether answers are actually right
17. Why is indirect prompt injection considered more dangerous than direct injection, and what defense specifically targets it that input filtering and 'just prompt it not to' do not?
A Indirect injection is actually less dangerous, because the attacker never interacts with the agent directly and so can't adapt the payload to the agent's responses the way a live attacker probing a chat session canB In indirect injection, the attacker plants instructions in third-party content (a web page, email, or file) that the agent reads on behalf of an unwitting victim, who never sees the attack; because a system-prompt instruction is just more competing text with no mechanical enforcement, the effective defense is structural — e.g. tool gating that disables dangerous tools for the rest of the turn once untrusted content has entered contextC Indirect injection only works against open-weights models; frontier hosted models ship instruction-hierarchy training that mechanically guarantees fetched content can never outrank the system promptD The defense is to increase max_tokens so the model has more room to reason past the injection