How to Evaluate a Software Factory Platform During a POC

If you work in software development, you’ve probably already heard about software factories. They are systems in an SDLC where agents do the work and people manage it, running across intake, code, review, deploy, and incidents. Anthropic and OpenAI have both published how theirs work, and the pattern is familiar enough now that most platform teams can put one together in a few weeks. Currently, very few teams have run one across a whole engineering organization because building a factory at scale brings unique challenges. So before you sign anything, it’s important to understand how to evaluate a software factory platform during a POC.

Here is how to scope a POC around the hard questions, including what to evaluate, and how to score it.

{{download_survey}}

Why software factory POCs fail before they actually start

Most software factory POCs are designed to test whether the factory works. They’re opinionated and quick to get going. But in a quick POC, the question worth asking yourself is how it will hold up when your whole organization is inside it. Getting a software factory up and running and scaling it are two very different tests.

Building the first line is easy on purpose. You already have CI/CD, GitOps, gradual deployments, and observability, so connecting an agent to them takes just weeks and succeeds on any platform you try. A demo line probably runs on clean data, is run by one cooperative team, and works on a service with no tier-1 constraints. Every vendor should pass this test easily, which means you finish the month with a yes from all of them and no way to choose.

On the test of scale, you should predict breaking points in a number of areas. We think these breaks fall into four categories:

• Governance. Agents spread from the bottom up (which is how it should work). An engineer writes a skill that helps, points an agent at a system with an MCP server, and ships. Across a few hundred engineers they become agents that nobody owns, no production-ready bar on any of them, agents running on a developer's credentials, and no single record of who decided what.
• Unique needs and constraints. The “factory lines” your business depends on cross several teams, tools, and stages, and they need entities, data sources, and steps from you. Without them, and if the platform cannot reach certain areas, a team will automate on its own.
• Quality. The volume of things running in a demo factory allow you to read every agent decision. At full volume nobody can, and nothing tells you how good those decisions were or which of them to trust more the next time it runs.
• Operations. A factory keeps needing more workflows, skills, and context, most of it requested by teams the platform team does not sit with. It also moves at the speed of its slowest queue, which is often pull request review.

If you were to score a POC factory line against these four tests of scale, it will likely fail every one.

How to define the right scope for a software factory POC

When it comes to scoping out a POC, we think of it in two dimensions: the building blocks the platform gives you, and the assembly lines you can build out of them.

The building blocks are what everything else is made from: the context lake and the data model, standards and scorecards, workflow orchestration, human-in-the-loop controls, the agent and skill registries with MCP, permissions, audit, measurement, and the interfaces each group works in. During a POC, each one should be included in a working factory line rather than demoed on its own.

The assembly lines are where scale actually gets tested. Therefore, you should pick ones that mirror your actual complexity instead of a clean and simple one. We recommend choosing these three:

• Idea to production. Take a real ticket from intake to enrichment, to a routing decision made by policy, and then on to a plan, a coding agent, review, and finally a gated deploy.
• Self-healing incidents. When an alert comes in, diagnose it, and plan remediation for it that runs under an approval policy, then verify it afterwards. Encode it in a standard that stops the same incident from happening again.
• A cross-cutting initiative. Choose one change that many services need and let agents do the work. Engineers should be able to direct them, and the manager running should see where it stands.

When you’re setting them up, they should be loaded with standard and non-standard, risky things. What’s non-standard? Could be a team that doesn’t fit the classic CI/CD model. Or a tool that has no native connector.  And then do potentially risky things, like at least one tier-1 service in scope. Have a product team, not the platform team, build one workflow end to end.

In the lead up to the POC, you’ll want to capture as much data as possible about the workflows when they aren’t running through a software factory. Thirty days is enough for that, running on your own repositories, tickets, and alerts, with your engineers at the keyboard. Capture the baseline before the first agent runs: deployment frequency, lead time for changes, rework, review load, and the spread between teams. Once you have a baseline, the POC can start.

The evaluation criteria to POC against

Each criterion below exists because of a specific way factories break. Test them inside the three factory lines rather than as separate demos.

Governed

Require at least one control that actively blocks an action rather than just logging it after the fact. Reporting alone relies on people reviewing logs in time, which doesn't work at high volume. Ensure the same governance standards cover both agent-generated changes and the agents themselves, giving each agent its own identity and tool set, while recording all human and agent actions in a single audit log. Security teams routinely balance security with delivery speed, and that same principle applies here, as covered in how platform teams balance security and agility.

Flexible

You cannot automate a whole agentic SDLC on a platform you cannot extend. Add an entity type and a relation yourself, ingest from a tool with no connector, and build a workflow the vendor never shipped. If any of that needs a vendor engineer, your coverage will stop at their roadmap.

Covers all AI-SDLC stages

All stages of the SDLC, intake, build, deploy, run, and security should use the same triggers, conditions, and approvals. If an SDLC stage doesn’t make it into the AI-SDLC, it will get automated by whoever needs it first, and you end up governing several factories instead of one.

Measurable

Your POC should run enough volume through it that you cannot possibly review every decision by hand. It should be able to give you a record of what each agent read and concluded, and the ability to mark the decisions that were wrong and trace them back to the context the agent lacked. When an agent gets something wrong the cause is usually the data, not the model, and the platform should let you fix the data and re-run.

Agnostic

For each tool in your stack, ask what would have to be rebuilt if you replaced it next quarter. Your git provider, CI, scanners, observability, issue tracker, coding agents, and model provider all change on their own schedules.

Contributable

A factory is a living project, and the platform team cannot be the only ones adding to it. If teams cannot contribute a workflow or a skill according to your standards, they will build their own factory on the side, and those automations sit outside your audit log and your scorecards. Test it directly by having a product team ship something during the POC.

Serves everyone

Developers, security, SRE, engineering managers, and people who never write code each need a view and actions scoped to them. Anyone the platform does not serve keeps working in their own tools, and what they do there is never gated or recorded.

{{download_survey}}

What to measure during the POC to make a decision

We recommend measuring two things: whether each factory line completed, and how the platform scored against the seven criteria. Then combine them with a scoring model you commit to before the POC starts, so the debate happens over weights rather than over results.

There are seven criteria. Each should be given a weight for a total of 100. For example:

Then each criterion should be scored between 0 and 3

Calculate the total by multiplying each weight by the score (0-3) and then divide by three. Then add them up for a number out of 100.

You can also add a question of self-sufficiency. When you count everything the POC produced, meaning entity types, integrations, workflows, standards, and views, how many did you built yourself vs built for you? then divide the ones your team built unaided by the total. Below half predicts a services-heavy first year whatever the score says, and it is the number that most often contradicts a good demo.

Above 80 looks like a buy signal. 

Between 60 and 80 usually means you can extend what you already run and revisit in two quarters, especially when your current platform engineering tools already cover half the blocks 

Anything less than 60 is a sign you should probably look at another vendor.

FAQ

How long should a software factory platform POC run?

Thirty days is enough for three end-to-end lines on your own repositories and tickets. Shorter tests demos rather than delivery. Past forty-five days the scope was never defined, and the evaluation drifts into an unpaid pilot with no decision date and no baseline to compare against.

What is the most common mistake teams make during a software factory POC?

Testing whether the factory works instead of whether it scales. Connecting an agent to existing CI/CD succeeds everywhere, so it decides nothing. The expensive mistakes are skipping the baseline, letting vendor engineers build the lines, and never putting a tier-1 service or a non-standard team in scope.

How do you involve platform engineering in a software factory POC?

Put them at the keyboard and cap vendor hours. Platform engineers should model an entity, connect a tool with no native connector, and ship a workflow the product does not include. Have a product team build one too, since a factory only its platform team can extend will fragment within a quarter.

What does a good software factory POC output look like?

A weighted score across the seven criteria, a self-sufficiency ratio, before and after numbers for each line, and the audit evidence for one release in a single view. That is enough for a buy, extend, or build decision without a second round of vendor demos.

{{download_survey}}