How to ship AI features to production: evaluation, privacy, and failure modes

Start with the job, not the model

A production AI feature begins with a user decision, not a provider comparison. “Add an LLM” is an implementation preference. “Help a revenue operator decide which prospect deserves attention, with evidence they can inspect” is a product job. The second statement can be evaluated even if the underlying model changes.

Write the job in observable language:

• who is making the decision;
• what input they provide or authorize;
• what output changes their next action;
• how quickly the result must arrive;
• what an acceptable result contains;
• which mistakes are harmless, costly, or unacceptable;
• whether the action is advisory, reversible, or automatic.

This framing prevents model capability from silently expanding product scope. A model may be able to draft, classify, search, call tools, and act on third-party systems. The product should expose only the capabilities required for the job. Every additional permission creates evaluation, security, privacy, support, and incident-response work.

The public Eryx product position illustrates a narrow job: combine prospect search, enrichment, scoring, and qualification into a maintainable workflow for revenue teams. VStok focuses on measuring how AI systems represent and cite brands, then connecting evidence to prioritized improvements and comparable re-audits. These examples describe product responsibilities, not claims that a model is always correct.

Write the production contract

Before prompt work expands, write a short production contract between the feature, its user, and the operating team. This is not a legal agreement. It is the shared definition of what the system may do and how it behaves when reality does not match the happy path.

The contract should distinguish assistive output from automated action. An assistive system proposes, explains, and lets the user decide. An automated system changes state—sending a message, editing a record, publishing content, approving an application, or moving money. Automation requires a higher evidence threshold, narrower permissions, idempotency, and a reliable way to stop or reverse the action.

Define acceptance and refusal together. A feature that is expected to answer every request will fabricate certainty at the boundary of its data. Give it permission to decline, return a partial result, request more information, or route the task to a deterministic flow.

Build an evaluation system

Evaluation is the product’s quality specification made executable. Without it, prompt changes, model upgrades, retrieval changes, and new tools are judged by memorable examples. Teams then optimize for the latest demo instead of the user distribution.

Start with a versioned dataset drawn from the intended workflow:

1. Normal cases: representative inputs that should succeed.
2. Ambiguous cases: incomplete, conflicting, or underspecified requests.
3. Edge cases: uncommon formats, languages, long inputs, empty results, and stale sources.
4. Adversarial cases: prompt injection, malicious documents, attempts to reveal sensitive context, and requests outside authority.
5. High-impact failures: examples where a confident error could cause material harm.
6. Known regressions: every important production incident that should never silently return.

Do not start with an arbitrary target such as “1,000 eval cases.” Coverage matters more than volume. Twenty carefully selected examples across the real failure classes can guide an early product better than thousands of synthetic variations with identical structure. Grow the set as user behavior and incidents reveal missing classes.

Evaluate components and the whole workflow

An AI feature may contain query interpretation, retrieval, ranking, prompt assembly, generation, tool selection, output validation, and UI presentation. A good final answer can hide a weak retrieval step; a good retrieval result can be damaged by generation.

Use three layers of evaluation:

• Component evaluation: Did retrieval find the authoritative source? Did classification select the correct category? Did the validator reject malformed output?
• End-to-end evaluation: Did the user receive the required outcome with correct evidence and acceptable latency?
• Operational evaluation: Did the system respect permissions, retention, cost, logging, fallback, and rollback rules?

Automated scoring is useful for stable properties: JSON validity, citation presence, classification labels, forbidden phrases, latency, token usage, and tool-call permissions. Human review remains necessary for nuanced correctness, usefulness, tone, risk, and whether evidence actually supports a conclusion. Model-based graders can help at scale, but they also need calibration against human judgments and versioned prompts.

Use a scorecard, not one average

A single “quality score” hides trade-offs. A faster model may reduce cost but lower citation precision. A new retrieval source may improve recall while increasing prompt-injection exposure. Release decisions need the dimensions separately.

Data and retrieval quality

Many apparent model failures are data failures. The system retrieved an outdated page, merged two entities, lacked a required document, or presented marketing text as independent proof. Improving the prompt cannot repair evidence that never entered the context.

Define a source policy:

• which repositories, APIs, documents, websites, and user fields are allowed;
• which source is authoritative when facts conflict;
• freshness requirements and update signals;
• ownership classification—first party, competitor, independent, user-provided;
• access-control checks applied before retrieval;
• data that must be redacted or excluded;
• how citations map back to exact source locations.

Retrieval should preserve provenance. Store the source identifier, retrieval timestamp, relevant excerpt or span, ownership class, and any transformation applied before the content reached the model. When a user challenges an answer, the team must be able to determine whether the fault came from the source, retrieval, generation, or display.

Do not present a link as proof merely because it was retrieved. Validate that the cited passage supports the claim. Citation quality needs at least three checks: the source is relevant and authoritative for the fact, the referenced content entails the statement, and the source is not misrepresented as independent when it is owned by the subject or a competitor.

Freshness is domain-specific. A company registration fact may change rarely. Pricing, store availability, software documentation, and model behaviour can change quickly. Assign review or expiry rules based on the decision impact rather than one global cache duration.

Privacy, security, and tool boundaries

AI features enlarge the application’s trust boundary. User text, retrieved documents, web pages, model output, and tool results may all contain untrusted instructions or sensitive data. Treating the model as a trusted orchestrator bypasses security controls that the rest of the product would never delegate to generated text.

The OWASP Top 10 for LLM and generative AI applications identifies risks including prompt injection, sensitive-information disclosure, supply-chain weaknesses, data and model poisoning, improper output handling, and excessive agency. These are application risks, not problems that a better system prompt can fully solve.

Use ordinary security architecture around the model:

• enforce authentication and authorization outside the prompt;
• grant tools the minimum functionality and permissions;
• validate structured output before it reaches an interpreter, database, browser, or API;
• keep read and write capabilities separate;
• require explicit confirmation for consequential writes;
• make write operations idempotent where possible;
• isolate secrets and never rely on the model to redact them;
• apply rate, cost, and concurrency limits;
• audit tool calls and permission decisions;
• disable risky capabilities independently of the rest of the product.

The OWASP Application Security Verification Standard remains relevant because the AI component does not replace web application security. Session management, access control, input handling, logging, secrets, and data protection still require normal verification.

Minimize data before model selection

Create a data-flow map covering collection, transport, provider processing, storage, logs, evaluation datasets, support access, and deletion. For each field ask:

• Is it necessary for the user outcome?
• Can it be transformed or minimized before leaving the trusted system?
• Is the user authorized to submit it?
• Is it retained by the application or provider?
• Can it enter analytics, traces, or human review queues?
• How is deletion propagated?
• Are regional or contractual restrictions relevant?

Do not promise “private AI” without defining the claim. It may refer to no training on customer data, a retention setting, regional processing, isolated infrastructure, client-side processing, or contractual controls. State the actual mechanism and its limitations.

Design the failure experience

AI systems fail differently from deterministic interfaces. They may return fluent but unsupported answers, partially complete a task, select the wrong tool, time out after expensive work, or produce different results for similar inputs. The interface must help the user recognize and recover from these states.

Design at least the following:

• No evidence: say that the required source was not found; do not fill the gap with a plausible claim.
• Conflicting evidence: show the conflict and source dates rather than silently choosing.
• Low confidence: communicate what is uncertain and what the user can verify.
• Partial completion: preserve completed work and label missing sections.
• Policy refusal: explain the boundary without leaking internal instructions.
• Provider failure or timeout: offer a safe retry and avoid duplicate actions.
• Invalid output: fall back to a deterministic state rather than rendering broken generated content.
• Write failure: show whether the action occurred and provide an idempotent recovery path.

Uncertainty should be actionable. A coloured badge with no explanation does not help. Show why confidence is limited—missing sources, entity ambiguity, stale evidence, or conflicting data—and let the user resolve the cause.

User correction is product data, but it must be handled responsibly. Separate a correction used to fix the current result from consent to retain it for evaluation or product improvement. Record what changed, which evidence justified the correction, and whether it exposes a missing eval case.

Observability that explains quality

Traditional uptime is necessary but insufficient. An AI endpoint can return HTTP 200 while producing unusable answers. Observability must connect system behaviour to product quality without collecting more sensitive content than the team can govern.

For each run, consider recording:

• feature and workflow version;
• model and provider configuration;
• prompt-template and retrieval-policy version;
• source identifiers, freshness, and ownership classes;
• latency by component;
• input/output size and cost;
• validation and safety outcomes;
• tool calls, permissions, and side effects;
• fallback, retry, correction, and abandonment events;
• links to applicable eval cases or incident classifications.

Avoid logging raw prompts and outputs by default. Use redaction, structured metadata, sampling, short retention, and access controls. If raw content is required for debugging or evaluation, document the purpose and obtain the appropriate permission.

Build dashboards around decisions: Is grounded task success degrading? Did the new model improve normal cases while regressing another language? Are low-confidence results increasing because a source stopped updating? Is cost per completed task rising because users retry?

Alerts should identify an owner and an action. Useful conditions include a validation failure spike, tool-call denial spike, missing-source rate, latency or cost threshold, provider errors, sudden output distribution change, and a high-impact eval regression.

Human control and reversibility

“Human in the loop” is not a universal safety solution. A reviewer who sees hundreds of plausible suggestions will eventually approve by habit. Human control works when the person has enough context, time, authority, and evidence to make the decision.

Choose the control based on impact:

Reversibility must be concrete. A kill switch should disable the risky feature without requiring a deploy or taking down unrelated product functions. A fallback should preserve the user’s data and core journey. A rollback should identify whether generated writes need compensation or cleanup.

NIST’s AI Risk Management Framework organizes continuous risk work around govern, map, measure, and manage. Its Generative AI Profile expands actions for risks specific to generative systems. A small product does not need enterprise ceremony, but it does need named owners, documented risk tolerance, testing, incident handling, and evidence that controls work.

Release gates

Do not release because the demo looks convincing. Release when the defined job passes explicit gates.

Roll out gradually when exposure can be segmented. Start with internal use or selected design partners, then expand by account, region, workflow, or feature flag. Compare canary quality and operational signals with the control. Do not use production users as the first evaluation set.

Model and provider updates are releases even when application code does not change. Pin versions where supported, rerun evals, inspect safety and cost changes, and retain a rollback option.

A practical delivery sequence

1. Frame

Define the job, user, risk tier, decision owner, sources, forbidden behaviour, latency, and success measure. Remove capabilities that do not support the job.

2. Prototype the complete thin slice

Build one end-to-end path: authorized input, retrieval, model call, validation, visible evidence, correction, and deterministic fallback. Do not optimize a prompt in isolation from the workflow.

3. Establish evals and threat cases

Version representative and adversarial examples. Add component, end-to-end, and operational checks. Calibrate automated graders with human review.

4. Harden data and tools

Enforce source policy, provenance, authorization, output schemas, permission boundaries, idempotency, rate limits, and audit events.

5. Complete failure UX and operations

Implement uncertainty, no-evidence, conflict, partial completion, timeout, provider failure, and write recovery. Add dashboards, alerts, runbook, kill switch, and rollback.

6. Canary and learn

Release to a controlled group. Review task success, corrections, missing evidence, safety events, latency, cost, and support load. Convert important failures into permanent eval cases.

A trustworthy AI product is not one that never fails. It is one whose useful behaviour is measured, whose authority is bounded, whose failures are visible and recoverable, and whose operators can explain what happened. That discipline survives model changes and turns an impressive prototype into maintainable software.