How to Manage Agents and Skills in Your Software Factory

If your engineers use Claude Code or Cursor, your software factory already runs agents and skills, whether or not your catalog knows about them. A software factory exists to make delivery predictable, with standard paths, known owners, and guardrails that hold whether or not anyone is watching. An agent that skips that discipline is a privileged actor nobody registered, and teams usually discover it during an incident.

Take an engineer who sets up a coding agent with a long-lived service-account token and write access to the deployment repos. Months later, they leave the company. Offboarding revokes their personal accounts, but the token was never tied to them and the agent was never in any inventory, so it keeps running. When it merges a dependency bump that breaks checkout, the incident channel spends its first hour working out what the agent is, who owns it, and how to shut it off.

Gartner expects more than 40% of agentic AI projects to be cancelled by the end of 2027, and inadequate risk controls is one of the three reasons it names. Closing that gap means being able to answer five questions about every agent and skill in your factory:

• Which tools can this agent call?
• Which data can it read?
• Was this skill certified before other teams started using it?
• Did this agent pass evals before it reached production?
• Can you trace what it did, run by run?

The rest of this guide covers why AI agent management is the layer most software factories skip, why agents and skills need different controls, and how platform teams build those answers.

Why Agent and Skill Management Is the Layer Most Software Factories Are Missing

The main reason is how agents arrive and there's no procurement step and no architecture review. An engineer adds a coding agent to a repo and writes a few skills that encode how their team works. It saves them a day, so the setup spreads from team to team before the platform team hears about it.

Paul Iusztin draws a useful line in his O'Reilly piece, Inside a Software Factory. A small team can get very far with a handful of skills and agents layered on a coding harness. The problem changes once engineers you don't personally supervise are running agents, because tracing, cost tracking, and control stop being optional.

The second reason is that the software factory was designed around people. Golden paths, scaffolding templates, scorecards, and self-service actions all assume a human does the work with tooling in support. The catalog models services, teams, and environments, and it has no entry for the dozen non-human actors changing those services every day.

What breaks as a result is very much predictable. A skill that upgrades a dependency and opens a PR gets copied by the payments team, edited, and forked again by a third team. Six weeks later, three versions run inside agents with different credentials, and when one breaks a tier-1 service, nobody can say which version ran or who approved it. Permissions sprawl the same way, because scoping a credential properly is fiddly and handing over a broad token takes thirty seconds.

A better model wouldn't fix any of these failures. Each one traces back to a missing control: an inventory, an owner, a version, or a permission boundary.

The Difference Between Managing Agents and Managing Skills

An agent is a non-human identity that makes decisions. It has credentials, a runtime, an owner, a cost profile, and a set of environments and actions it's allowed to touch. The closest thing you already manage is a privileged service account, except this one picks its own next step.

A skill is a capability, usually packaged as a SKILL.md file with a name, a description, instructions, and the tools it needs. Each one does a single job well: upgrade a dependency and open a PR, triage an alert against the service runbook, turn a thin ticket into a spec. Skills behave like libraries, so they need versions, owners, and a deprecation path.

That's why they can't share one control loop. Bundle them and every wording fix to a skill becomes a change to a privileged identity, while one bad skill version hits every team whose agents load it. The relationship also runs many-to-many: your security team can own the secret-scanning skill without operating a single agent, while ten agents across ten teams call it. MCP servers sit alongside both as the source of the tools agents reach for, so each of the three needs its own inventory.

How to Structure Agent and Skill Management in Your Software Factory

Your software catalog already solved this problem for services. Teams that manage AI agents well extend the same discipline to new kinds of entity, one area at a time.

In Port, you can create agents to automate your engineering/dev workflows. One essential point to note is, you can group your agents and skills to the categories they belong. 

Similarly, you can create skills to support your software engineering activities. 

Below are some things you need to know about agents and skills. 

Discover what's running before you ask anyone to register it

Registration mandates decay - nothing stops a team from shipping an agent without filling in a form. Discovery holds up better because it reads from where assets already live.

Skills sit in Git as SKILL.md files under conventional paths: .claude/skills, .cursor/skills, .codex/skills, .github/skills, .opencode/skills, .agents/skills, or a top-level skills/ folder. Point an ingestion mapping at those paths and every SKILL.md across every repo lands in one registry automatically - with its description, version, and a link back to the source file. Merging to the default branch is the only registration step required.

The same principle applies to agents on AWS Bedrock, Azure AI Foundry, and homegrown setups; to agent plugins exposing capabilities to Claude Code, Cursor, and Codex; and to MCP servers. One mapping per asset type, one registry per category. The assets nobody can account for are exactly where to start.

Give every agent and skill a named owner

Without an owner, nobody has the authority to retire an agent or skill, so it keeps running until something breaks. Agents belong to whoever operates them, and skills belong to whoever owns the domain knowledge inside them, which is often a different team. Record both on the entity, then use group assignments to scope who can see and install each skill, so a half-finished team skill doesn't show up for the whole org.

Scope each agent's tools and data, and enforce the limits before it acts

Instructions in a prompt can be misread, ignored, or argued away on the next turn. In July 2025, a Replit agent deleted a live production database during a code freeze its user had declared in the tool more than once. Replit's fix was separating development and production databases, a control enforced by the platform.

Give each agent its own identity, bound to specific environments and a specific set of actions, with read, propose, and act as separate levels. Propose-only in production is a sensible default, and you widen access per agent once there's a track record. Route those actions through the same developer self-service layer your engineers already use, so one policy change covers humans and agents together.

Scope data the same way. An agent triaging an alert needs that service's runbook, owner, and recent deploys, not read access to every repo and dashboard, so limit which catalog entities and MCP servers it can query.

Watch for shared credentials in your own automation. A hook that syncs skills to developer machines using a service-account token bypasses per-user access control and installs every published skill on every laptop. A personal credential carries only that user's permissions, so the same hook respects the groups you set without extra filtering.

Ship skills through one golden path and version them

A prompt edit ships a behavior change to something with write access. Give developers one standard way to propose a skill, such as submitting a SKILL.md that opens a pull request automatically, and track each skill's status as pending, published, or deprecated. Every skill gets a version number, and consumers either pin a version or track latest on purpose.

Certify skills and gate agents on evals before production

A scorecard handles the deterministic checks: whether the skill has a version, an owning team, and a group. Description quality needs judgment. 

An AI review can read the skill's name, description, and full instructions, rate the description as good, too vague, too verbose, or missing, and write its reasoning back onto the entity with a recommendation to approve, iterate, or reject.

The same kind of check extends to flagging a new skill that duplicates one already in the registry, or one that hardcodes dates and model versions that will go stale in a quarter.

Agents need the same gate one level up. Before an agent is promoted to production, run it against a fixed set of known tasks and require every skill it loads to be certified at its current version. A useful rule: if one of those skills drops back to iterate or reject, the agent drops back to propose-only until it's fixed.

Measure cost and audit every run

A skill can clear every quality bar and still cost more to run than the work it replaces, so connect real usage and spend data back to each skill entity. 

For agents, record every run as a structured event: which agent, which skill version, the inputs, the tools called, what changed, and who approved it, linked to the services it touched. When something breaks, that record traces the incident back to a single run.

What Platform Teams Must Put in Place to Scale Agent and Skill Management

The areas above only hold up at scale when four foundations sit underneath them.

Port's AgentOps solution applies this to the agents and skills running inside your engineering org, as opposed to the AI features you ship in your product. It works in four parts:

• Traceability is the baseline: a registry and dependency tree mapping every internal agent and skill to its owners, permissions, and dependencies.
• Certification runs skill evals that catch gaps before an agent reaches production, so only approved versions ship.
• Enforcement checks each agent action against certified permissions and cost before it runs.
• Optimization keeps tightening each asset and its boundaries once it's live.

AgentOps runs on Port's agentic engineering platform, so agents acting through it read ownership, dependency, and environment data from the Context Lake. Once agents handle real volume, agentic work management gives people one view of what's in flight, what's waiting on approval, and what shipped. AgentOps doesn't build your agents, issue their credentials, or run the model infrastructure underneath.

If you're starting from zero, skip the grand design. Run discovery, put an owner on every agent and skill, and certify the five skills with the most obvious reuse. That's enough to show you which agents you'd struggle to explain to an auditor, and which skills nobody has called in a month. The AgentOps guides include the blueprints, mappings, and workflow JSON for each step.

FAQ

What is the difference between an agent and a skill in a software factory?

An agent is an actor with its own identity, credentials, runtime, and owner. A skill is a reusable capability, usually a SKILL.md file, that an agent loads to do one job, like upgrading a dependency or triaging an alert. Agents are managed like service accounts. Skills are managed like libraries: versioned, certified, shared, and deprecated.

How do platform teams manage agent permissions in a software factory?

They give each agent its own identity, bound to specific environments and a defined set of allowed actions, and enforce those limits in the platform before an action runs instead of relying on the prompt. Most teams start agents on propose-only access in production, route actions through self-service with approvals, and widen access once there's a track record.

What is a skill registry in a software factory context?

A skill registry is the central, searchable inventory of every reusable agent skill in the organization. Each entry records what the skill does, its version, publish status, owner, the groups allowed to use it, and its certification result. Ingesting it automatically from Git keeps it accurate and stops teams from rebuilding the same capability.

How do you audit what agents and skills do in a software factory?

Log every run as a structured event instead of free text. Each record should capture the agent, the skill version, the inputs, the tools called, what changed, and who approved it, linked to the services it touched. That lets you trace an incident to one specific execution and tie usage and spend back to each skill.

‍