Most organizations that experiment with generative AI end up with a collection of pilots: an assistant in customer support, a summarization tool in legal, a coding assistant in engineering. Each one may work well in a demo. Few reach production, and those that do are hard to operate. When you look closely, the problem is rarely the model. It is that every pilot rebuilt the same foundations in its own way, and none of them were built for production.
This article explains why pilots stall, walks through a hypothetical example of a pilot meeting production requirements, describes the shared building blocks that turn pilots into maintainable AI services, and suggests an order for putting them in place.
In short: Pilots stall because each one solves access, data, safety, evaluation and cost on its own. Standardize those once, as a shared platform, and every new use case becomes a small project instead of a large one. Start with model access and logging, then data access and guardrails, then evaluation and cost attribution.
Why AI pilots stall
A pilot is built to answer one question: can this work? That is the right question at the start, and pilots are a good way to answer it. The trouble begins when a successful pilot is asked to become a production service, because production asks different questions.
Which data is the model allowed to see, and on whose behalf? What happens when it gives a wrong or harmful answer, and who notices? How do we know quality did not get worse after a model or prompt change? Who pays for it, and what does each request cost? Can we show an auditor what the system did and why?
None of these questions is specific to one use case, yet each pilot team has to answer them from scratch. Security and legal reviews start from zero every time, and the organization ends up with several incompatible ways of doing the same thing. After a few rounds of this, many organizations conclude that AI is harder to productionize than expected, when the real issue is that they never built the shared foundations.
A worked example
Consider a hypothetical legal team that builds a contract review assistant. An engineer connects a model to a folder of sample contracts, writes a prompt that flags unusual clauses and shows the result in a simple web page. The demo is impressive, and the legal team wants to use it every day.
Then the production questions arrive. The assistant uses a personal API key, so it is unclear who is responsible for the account and the bill. Real contracts contain confidential information, so security wants to know where the data goes and who can see the prompts. Legal asks what happens when the assistant misses a risky clause, and how anyone would know whether a model update made it worse. Finance asks what it will cost when the whole department uses it.
In an organization with shared AI foundations, most of these questions already have answers. The assistant calls models through the approved access path in the company’s AWS environment, so data stays in the approved region and invocations are logged centrally. A default guardrail configuration applies. The team only has to define what is specific to its use case: which contracts may be processed, who reviews the output and a set of test contracts with known answers. The path to production becomes a short checklist instead of a new project.
Key idea: The second AI use case should be much cheaper than the first. If it is not, you are building pilots, not a platform.
The shared building blocks
An AI service is a use case built on shared, governed building blocks. On AWS they typically look like this.
Model access
Provide one approved way to call models, for example through Amazon Bedrock in your own AWS accounts, with a defined list of allowed models and regions. Teams no longer need their own vendor contracts or API keys, and adding a model becomes a platform decision rather than a separate review for every team. A service control policy can enforce the list across all accounts:
{
"Version": "2012-10-17",
"Statement": [
{
"Sid": "DenyUnapprovedModels",
"Effect": "Deny",
"Action": [
"bedrock:InvokeModel",
"bedrock:InvokeModelWithResponseStream"
],
"NotResource": [
"arn:aws:bedrock:*::foundation-model/amazon.nova-*",
"arn:aws:bedrock:*::foundation-model/anthropic.claude-*"
]
}
]
}This example denies invoking any foundation model outside the approved families. If you use cross-region inference profiles, the policy must also allow the inference profile ARNs, so test policies like this in a sandbox organizational unit before rolling them out.
Identity and data access
Applications should access data with the permissions of the user or service they act for, using IAM and your identity provider, rather than with a broad service account that can read everything. This matters most for retrieval-augmented generation, where the model answers questions using your documents: the knowledge base, whether Amazon Bedrock Knowledge Bases or your own vector store, should respect the same access rules as the source systems, or the assistant becomes a way around them.
Guardrails
Content filters, denied topics and filters for sensitive information should be applied consistently, for example with Amazon Bedrock Guardrails. Equally important are clear rules about when a person must review the output before it is used. Guardrails reduce risk; they do not replace human judgment where mistakes are expensive.
Logging and traceability
Enable model invocation logging centrally. Amazon Bedrock can deliver invocation logs to Amazon CloudWatch Logs or Amazon S3, so prompts, responses and metadata can be reviewed when something goes wrong. Decide early who may read these logs and how long they are kept, because they can contain the same sensitive data as the documents being processed.
Evaluation
Each use case needs a small, versioned test set with expected answers and a repeatable way to score outputs, whether by people, by rules or by a model acting as a judge. Evaluation is what makes it safe to change models, prompts or retrieval sources. Without it, every change is a guess, and teams end up afraid to touch a system that works.
Cost attribution
Track usage per use case and team, for example through separate accounts, application inference profiles with cost allocation tags, or your own metering. Cost per use case is what lets you decide which services are worth scaling and where a smaller model would be good enough.
What to standardize first
You do not need all of this before the first use case goes live. A practical order is to start where the risk and the waste are highest.
First, model access and logging. One approved way to call models, with invocation logging switched on. This slows the spread of unmanaged tools and gives you visibility into what is actually being used.
Second, data access and guardrails. Define which classes of data may be used with which models, and apply a default guardrail configuration that use cases can tighten but not weaken.
Third, evaluation. Require a test set before any use case goes to production. This is often the step teams resist most, and the one they appreciate most later.
Fourth, cost attribution and reuse. Track cost per use case, and turn repeated patterns, such as document question answering, summarization and classification, into templates that new teams can start from.
A use-case intake template
A short intake template helps teams think through a use case before building it, and gives the platform team what it needs to support them. Here it is filled in for the hypothetical contract assistant:
| Question | Why it matters | Example answer |
|---|---|---|
| What task does it support, and who uses the output? | Defines success and the audience | Flag unusual clauses for lawyers reviewing supplier contracts |
| What does a wrong answer cost? | Sets the level of review and guardrails | A missed clause could create legal risk |
| Who checks the output? | Determines whether a human stays in the loop | A lawyer reviews every flagged and unflagged contract |
| Which data does it need? | Drives data access and model choice | Supplier contracts, classified as confidential |
| How will we measure quality? | Makes changes safe | A set of past contracts with known issues |
| What volume do we expect, and who pays? | Enables cost attribution | Legal department budget |
| What is the fallback? | Keeps the process running during outages | Manual review as before |
Choosing good first use cases
Two questions separate good first use cases from risky ones: how expensive is a wrong answer, and does a person review the output before it has an effect? Use cases where mistakes are cheap and a person reviews the output, such as drafting internal summaries or suggesting replies that an agent edits, are good places to build experience. Use cases where the system acts on its own and mistakes are expensive, such as sending messages to customers or changing records, need strong guardrails and evaluation before they go live. Starting with the first group lets the organization build its foundations while the stakes are low.
People and ownership
A platform does not create adoption on its own. Name an owner for the shared AI platform and an owner for each use case, publish clear rules about what is allowed and share working examples internally. Clear rules make adoption faster, not slower: when teams know what they may do, they stop waiting for permission, and they stop building around the rules because nobody told them what the rules were.
Try this: List the AI pilots and tools in use across your organization today. For each, note how it handles model access, data access, guardrails, logging, evaluation and cost. Every cell where a team built its own solution is a candidate for the shared platform, and the cells that are empty show where your risks are.
Key takeaways
- AI pilots stall on shared foundations, not on models.
- Model access, data access, guardrails, logging, evaluation and cost attribution should be built once and reused.
- Start with model access and logging, and add the rest as use cases move toward production.
- Use an intake template and start with use cases where mistakes are cheap and a person reviews the output.
Running several AI pilots without a clear path to production? We help design shared AI foundations on AWS, so each new use case is faster and safer than the last.
Talk to us