Key takeaways
- The challenge of enterprise AI isn't launching the first agent; it's governing the 10th, 50th, or 100th consistently.
- A repeatable operating model needs three layers: a managed runtime, an automated on-ramp, and an operating layer for tools, observability, security, and cost.
- Self-service onboarding and declarative manifests simplify AI agent scaffolding, while CI/CD and policy-as-code automate security and governance.
- Evaluations and observability answer two different questions: was the right agent built, and is it behaving correctly in production?
- Governance should be built into the platform itself; not left to individual teams to remember.
The real bottleneck
Building the first AI agent on AWS is relatively straightforward. A team selects a foundation model, builds the agent, connects a few tools or data sources, and deploys it.
The real challenge starts when an enterprise wants to move from a few experiments to tens or hundreds of AI agents across multiple teams, AWS accounts, and business units.
At that point, the questions change. Which models can teams use? Where can inference run? Which enterprise systems can an agent access? How are identities and permissions managed? How is sensitive data protected? Who owns the cost? How do security teams audit what an agent did? And how does a team know a new model, prompt, or tool change hasn't reduced agent quality?
Solving these questions separately for every agent creates AI infrastructure sprawl. The challenge, therefore, isn't simply providing access to Amazon Bedrock; it's creating a repeatable operating model for enterprise AI.
TO THE NEW treats this shared foundation as an AI Enterprise Core: a governed platform pattern that gives teams a standard way to build, deploy, connect, evaluate, secure, observe, and operate AI agents on AWS.
The objective is simple: let teams ship agents, not rebuild AI infrastructure every time.
The anatomy of AI sprawl
AI sprawl often starts quietly. Different teams make different design choices, and each agent may still work individually. The problem appears when the enterprise needs to govern all of them consistently.
Enterprise question | Why it matters | What happens without a shared platform |
Which models can agents use? | Models differ in cost, capability, risk, and regional availability. | Model sprawl and inconsistent approvals. |
Where can workloads and inference run? | Enterprises may have security, regulatory, and data residency requirements. | Teams create different region and network patterns. |
Who owns the agent? | Every production workload needs clear technical and business ownership. | Orphaned agents and unclear accountability. |
What can the agent access? | Agents can interact with APIs, databases, repositories, and enterprise systems. | Different permission models and direct integrations. |
Which tools can it execute? | Tool access determines what an agent can actually do. | Unmanaged MCP servers and uncontrolled tool access. |
How is sensitive data protected? | Prompts, responses, and tools may handle confidential information. | Inconsistent PII and content controls. |
How does a team know the agent still performs correctly? | Model, prompt, tool, and orchestration changes can affect quality and behaviour. | Regressions can reach production without being noticed. |
Who owns the AI cost? | Model usage can increase quickly across teams. | Spend cannot be easily attributed to a use case or team. |
Can the team trace what the agent did? | Troubleshooting requires visibility across model and tool calls. | Fragmented logs and weak audit trails. |
None of these questions is difficult for one agent. The challenge is answering them the same way for every agent.
The 3-layer blueprint
A scalable enterprise AI platform can address these questions through three connected layers.
Layer | What it provides | What problem it solves |
Managed runtime & harness | Agent execution, orchestration, session isolation, memory, identity, and networking | Removes the need to rebuild agent infrastructure repeatedly |
Automated on-ramp | Manifest, CI/CD, infrastructure as code, policy-as-code, and deployment controls | Ensures every agent passes the same checks before deployment |
Operating layer | Gateway, MCP/tool governance, guardrails, evaluations, observability, and FinOps | Keeps quality, security, access, and cost controls consistent |
Layer 1 – Managed runtime & harness
Amazon Bedrock AgentCore provides managed capabilities for running enterprise AI agents.
AgentCore Runtime supports teams that want to bring their own agent code and orchestration framework. AgentCore Harness goes a level higher by simplifying orchestration around the model, instructions, tools, skills, memory, identity, networking, and observability.
This gives enterprises flexibility: standardized use cases can use a managed harness, while teams that need more control over agent behaviour can use Runtime. Both can still follow the same enterprise governance model.
The platform can also define standard patterns for approved models, IAM, container images, networking, logging, tracing, and agent configuration.
For larger organizations, agents don't need to run in one central AWS account. Workloads can stay within the appropriate business-unit or workload accounts while the platform centrally governs policies and controls.
Layer 2 – Automated on-ramp
The second layer controls how an agent enters the platform. Instead of requiring every team to understand the complete infrastructure, developers provide a simple declarative manifest. For example:
Agent + Owner + Environment + Cost Centre + Model + Tools + Memory + Data Classification
The manifest becomes the contract between the development team and the platform. A developer submits it through Git or an internal developer portal. CI/CD and policy-as-code then check:
- Is the model approved?
- Is the region allowed?
- Are ownership and cost tags present?
- Is observability enabled?
- Are the requested tools permitted?
- Are the required guardrails attached?
- Are evaluation checks defined for this agent?
Only after these checks pass does infrastructure as code provision the agent environment. This creates a controlled front door for enterprise AI.
Layer 3 – Operating layer
Governance must continue after deployment.
AgentCore Gateway can provide a controlled path to enterprise APIs, tools, and MCP-based integrations instead of every agent creating its own point-to-point connections. Guardrails help manage sensitive information, undesirable content, and prompt-based risks.
IAM and authorization policies control what an agent can access. Model controls define which foundation models teams may use. Observability provides evidence of what the agent actually did.
Evaluations add another important control: is the agent still behaving as expected? A platform can evaluate response quality, task completion, tool selection, safety, and regression across different agent versions.
The result is a layered operating model:
Identity + Network + Approved Models + Guardrails + Governed Tools + Evaluations + Observability + Cost Controls
Provisioning in under 5 minutes
Governance only works at scale when the governed path is also the easiest path. If secure onboarding takes weeks, teams will naturally look for shortcuts.
A target onboarding experience can look like:
Developer Portal / Git → Manifest → CI/CD → Policy Checks → Provision → Evaluate → Governed Agent
The developer describes the agent once. The platform handles repetitive work.
Developer provides | Platform provides |
Agent name and owner | Runtime or Harness |
Business unit and cost centre | IAM and identity pattern |
Approved model | Inference configuration |
Required tools | Gateway and MCP connectivity |
Memory requirement | Memory configuration |
Data classification | Guardrails and policies |
Evaluation criteria | Standard evaluation workflow |
Environment | Account, region, and networking |
Application or container | Deployment, logging, tracing, and tagging |
The platform therefore applies security, networking, observability, evaluation, and cost controls during the delivery lifecycle, not after go-live.
AgentCore Harness can simplify this further for suitable use cases, since teams can define much of the agent behaviour through managed configuration rather than building orchestration from scratch.
The goal is to automate repeatable engineering controls while keeping human review for sensitive data, new models, new tools, or higher-risk use cases.
Operational governance
The questions raised by AI sprawl continue throughout the agent lifecycle.
Question | How the platform answers it |
Which models can agents use? | Maintain an approved model catalogue and enforce model and region policies. |
Where can workloads run? | Define approved regions, accounts, network boundaries, and inference patterns centrally. |
Who owns the agent? | Make owner, business unit, environment, and cost-centre metadata mandatory. |
What can the agent access? | Use least-privilege IAM and governed enterprise identity patterns. |
Which tools can it execute? | Expose approved tools through Gateway and apply authorization policies. |
How is sensitive data protected? | Combine guardrails, classification, private connectivity, and tool authorization. |
Is the agent performing correctly? | Run evaluations for response quality, task completion, tool usage, safety, and regression. |
Can the team trace what happened? | Standardize logs, metrics, and OpenTelemetry-based tracing across the agent flow. |
Who owns the cost? | Carry ownership metadata into model and infrastructure usage for FinOps attribution. |
Evaluations become especially important as agents evolve. Changing a prompt may improve one use case while reducing quality in another. A different model may lower cost but change response behaviour. A new tool may improve capability but also introduce unexpected execution paths.
A governed platform should therefore treat evaluation as part of the release lifecycle, not as a one-time test. Teams can maintain representative test datasets and evaluate areas such as response relevance, task success, tool selection, safety, latency, and regression. Teams can then compare new versions with previous versions before promotion.
This creates a more controlled lifecycle:
Build → Evaluate → Deploy → Observe → Improve
Observability then provides the production view:
User → Agent → Model → Tool → Enterprise System → Model → Response
Together, evaluations and observability answer two different questions: was the right agent built, and is it behaving correctly in production?
The key principle is simple: governance shouldn't depend on every development team remembering every control. The platform should build it in.
Key takeaway: repeatability is the real benchmark
The difference between an AI proof of concept and an enterprise AI platform is repeatability.
A proof of concept proves that an AI agent can work. A platform proves that teams can deploy the next 10, 50, or 100 agents without rebuilding security, networking, governance, evaluation, observability, and cost controls every time.
A practical enterprise AI platform on AWS combines managed runtime and orchestration, declarative onboarding, infrastructure as code, policy-as-code, approved foundation models, governed tools and MCP integrations, guardrails, evaluations, centralized observability, and clear cost ownership.
The real measure of success isn't whether the first agent can reach production. It's whether the 100th agent meets the same security, quality, governance, observability, and compliance bar as the first.
That is how enterprises move from building individual AI agents to operating AI at scale.