Key takeaways

  • Define the business metric and baseline before selecting a model.
  • Bound the first production workflow by process, users and permitted actions.
  • Design evaluation, human approval, logging and fallback before launch.
  • Assign a business owner and an operational owner — the AI system will not manage itself.

Pilots and production systems answer different questions

A proof of concept asks whether a model can perform a task. Production must answer harder questions: is it reliable enough, when does it fail, who can see the data, who approves actions, what happens during an outage and what does it cost at real volume?

A successful demo is therefore not a reliable readiness signal. Architecture, integration, risk controls, workflow redesign and user adoption still stand between the demo and daily work.

1. There is no specific business outcome

“Use AI” is not an objective. A useful objective describes a workflow change: shorter handling time, fewer manual checks, faster knowledge retrieval or a higher share of correctly routed requests.

Measure the baseline before development: volume, time, cost, error rate and exceptions. Select one leading metric and two guardrail metrics. Speed must not improve at the expense of errors or an unacceptable operating cost.

2. The workflow is too broad or unclear

“AI for customer service” contains many intents, channels, systems and policies. The first production release needs explicit boundaries: accepted requests, available sources, permitted actions and mandatory human handoff points.

A strong first scope is narrow enough to measure in six to eight weeks but valuable enough to have a business owner. One document type, one user group or one class of service desk requests is often better than a universal assistant.

3. Data exists but is not ready

A SharePoint site, file server or database does not make knowledge trustworthy. Duplicate versions, obsolete procedures, missing ownership and broad permissions can turn a capable model into a confident source of errors.

Map sources, ownership, sensitivity, quality and retention. AI should inherit the access boundaries of the source, not become a new route to broader information.

4. Integration is left until the end

Value rarely comes from generated text alone. It appears when AI retrieves the right context, creates a structured request, updates an approved system or prepares a decision for review. This requires APIs, identity, network connectivity, error handling and observability.

Design system dependencies during discovery. Validate rate limits, service accounts, environments, audit logs and legacy constraints before promising automation.

5. Evaluation is anecdotal

A handful of impressive prompts does not prove quality. Build an evaluation set from real, representative and difficult examples. Define the expected outcome, acceptable variation and critical failures. Run it whenever the model, prompt, retrieval, tool or policy changes.

Measure the complete task, not only the language response. For document processing, this includes extraction, rules, matching, escalation and the successful write to the target system.

6. Controls are added after the build

Human in the loop does not mean a person checks everything. It means approval points are set in advance based on risk, confidence, value or irreversibility. The system must know when it is not allowed to continue.

A minimum production control layer includes least-privilege access, environment separation, decision and action logs, input/output checks, cost and rate limits, escalation and an effective stop mechanism.

7. Nobody owns the system after launch

AI systems change even when code does not: inputs, user behaviour, models, pricing and upstream services move. Assign a business owner for the outcome and an operational owner for reliability.

Track task success, human override, critical error rate, latency, cost per completed task and adoption. Establish a regular review, an incident process and controlled version management.

A practical path to production

  • Select one workflow with provable value and a reachable owner.
  • Measure the baseline and agree success criteria.
  • Map data, systems, permissions, risks and exceptions.
  • Build a bounded production release, not an open-ended demo.
  • Validate it with an evaluation set and real users.
  • Roll out in stages with monitoring, human approval and rollback.
  • Review business outcomes and operational signals every month.

Frequently asked questions

How long does it take to move an AI pilot into production?

A bounded workflow with available integrations can often be deployed in 6–8 weeks. Unclear processes, poor data, legacy integrations and higher regulatory risk extend the timeline.

When is an AI pilot ready for production?

When it has a measurable outcome, representative evaluation, clear permissions and accountability, controlled actions, monitoring, fallback and named business and operational owners.

Should we choose the AI model first?

No. Define the workflow, value, data and constraints first. Select and test the model against the task, risk, latency, privacy and total cost.