What is a governed AI agent?

In the era of agentic engineering, it’s vital that AI agents perform safely and securely. It’s easy to say, “we’ll bring AI agents into our production flow,” but incorporating agents into your software development lifecycle (SDLC) needs to be done carefully and thoughtfully. Governed AI agents are likely what you really want.

A governed AI agent is an engineering agent that’s been integrated into a production environment in a controlled way. Its data access, operations, and actions are managed by frameworks and guardrails that ensure actions are taken safely and successfully.

What makes an AI agent governed?

A number of key tools and principles shape whether or not an AI agent is governed. There may be more elements of agent governance that come into play for your organization’s SDLC than what’s included here, but at a minimum, these are essential to enable your agents to behave safely and successfully.

Managing AI agents

It’s vital to manage agents from specific, consistent places. In the case of internal development platforms (IDPs) and agentic engineering platforms (AEPs), this is largely handled from the outset. AEPs provide the context lake agents use to perform tasks, and AEPs and IDPs are both used to set up guardrails for the agents. 

Applying guardrails

Guardrails are essential, full stop. Making sure an AI agent doesn’t take destructive actions is key to integrating them into your SDLC. There are a few key ways to automate guardrails, all of which work with one another to keep your agents on the right track. 

• Role-based access control (RBAC): Like human developers, AI agents should only have access to tools that are relevant to the tasks they’re designed to accomplish. Anything beyond that increases the risk of the agent taking destructive actions. For example, you can give an incident response agent read access to logs and monitoring data and allow it to execute rollback actions, but not give the agent privs to modify production infrastructure configurations.
• Context scope: Limit the scope AI agents have to just the context needed to successfully complete the tasks expected of them. This will help keep them in line, improve outputs by prioritizing fewer, higher-quality inputs, and minimize chances of hallucinations or incorrect outputs. If an agent needs to access specific databases or blueprints to do a task, that context should be included; if not, that context should be excluded (e.g., a vulnerability remediation agent should be given an affected service’s dependencies, recent deployment history, and related security policies, but shouldn’t be given the entire catalog of your organization’s services).
• Policy enforcement: You should expect AI agents to stay within organizational standards and guidelines the same way you expect your human developers to. 

It’s crucial to enforce organizational standards when applying guardrails to keep your agents on golden paths. These enforcements can be done in two ways:

1. Soft enforcement: Giving AI agents specific instructions and prompts that tell the agent how to behave within expectations. For example, when provisioning a new service, an agent can be told: “Service names must follow the pattern: team-domain-service (e.g., payments-api-processor). Use kebab case, max 50 characters.”
2. Hard enforcement: Setting hard barriers that trigger an error and stop the agents from proceeding. For example, if an agent attempts to create a service without required tags (such as owner, cost-center, or compliance-tier), the action can be blocked and instead produce the error message: “Error: Missing required tags. Action cannot proceed.”

Including human oversight

Human oversight of AI agents is a must. There should always be multiple human-in-the-loop (HITL) steps as part of your SDLC, to make sure agents are performing tasks safely and successfully.

• Review: Human developers should review and approve actions agents take (such as creating a pull request) before those actions go live, particularly those involving merges to production. 
• Explainability: Before taking an action, agents should be able to explain what action it plans to take, why that specific action is the solution, and what alternatives it considered before deciding on that action.
• Auditability: Governed agents should always provide audit logs and audit trails, so humans reviewing can tell what the agent did, interpret what reasoning led to the agent taking the actions it took, and track token usage. This can and should also include the ability to revert to a previous state (e.g., a rollback to a previous version of production, or restoring to a previous state from audit logs) in case something breaks.

FAQs

How do we govern AI agents?

AI agents are governed by applying guardrails (through RBAC, context scope, and policy enforcement) and including human oversight of their actions (through HITL review, explainability, and auditability). Management of AI agents is handled through a centralized entry point, usually an IDP or AEP like Port.

What are the main risks and governance challenges posed by AI agents?

The main risks and governance challenges posed by AI agents include loss of control and accountability, security and compliance breaches, and data privacy issues. To mitigate these risks, it’s vital to apply clear guardrails, include human oversight, and manage AI agents from a central gateway like an AEP. 

What is the difference between orchestrating agents and governing them?

Orchestrating agents involves coordinating and chaining multiple different AI agents together to complete multi-step tasks and workflows, where governing agents involves defining guardrails and oversight to fortify pipelines and automate software and security standards.

Should AI agents always operate under human supervision, or can they be deployed fully autonomously?

Human oversight of AI agents is a must. There should always be multiple human-in-the-loop (HITL) steps as part of your SDLC, including code and action review, explainability, and auditability, to make sure agents are performing tasks safely and successfully.

Should all AI agents within an organization have the same equal level of freedom or access?

No, role-based access control should be applied to AI agents based on the tasks they’re meant to perform. Like human developers, AI agents should only have access to tools that are relevant to the tasks they’re designed to accomplish. Anything beyond that increases the risk of the agent taking destructive actions.