How to prepare a legacy codebase for agentic engineering - Part 2

Cleaning up your legacy codebase feels like the obvious first step. Here is what our numbers said, and what moved them instead.

In part 1 we built a small frozen eval to answer one question about our legacy codebase: will an AI agent working here follow our rules? Writing brand-new code, it passed five of our seven rules every time and failed two every time. Both failures were architectural rules rather than local coding habits.

So we did the obvious thing. We cleaned up the code.

The cleanup that did nothing

We hand-cleaned a large batch of existing violations. Then we ran the same eval again.

The new code the agent wrote scored exactly the same as before.

Why it did nothing

Cleaning code you have already written (mostly) teaches the agent nothing about code it is about to write.

There are two separate channels here, and they don't cross over.

• What the agent reads as guidance shapes the new code in new modules it writes. That is your documentation and your rules files.
• What the agent copies from the code around it shapes the edits it makes to existing modules. That’s why we used the word “mostly” above.

"Clean everything first" pours a lot of effort into the second channel and expects the first one to improve. It won't.

What actually worked

Write the rules down for new code

For new code, documentation is the highest-leverage thing you own. Writing down the missing mechanism is what fixed our two failing rules, including the one case that nobody had ever documented anywhere.

The lever was docs. Not cleaner code.

Where to clean to extend existing code?

This question has two important points:

1. How impactful are the existing violations of your rules?
2. How can you predict which parts of the code are changed next?

So, if your rule violations are impactful and not just style (e.g. security rules), then the general cleanup has to come anyways - and AI can be very helpful here.

If they are less critical, you have the unthankful task to predict where your agent might work next and do the cleanup there - or have a look at the next step which would block violations from getting merged.

Add a gate to hold the line

Docs make the agent write it right. They do nothing to stop a regression from slipping in six weeks later. A small, deterministic check that blocks the main failure mode from merging keeps the codebase from drifting back.

Controls that work together. Guidance or the right examples at generation time, and a gate at merge time.

The playbook

1. Don't start by cleaning everything. Start by measuring whether agents follow your rules. A small frozen eval turns a vague worry into a number you can move.
2. Expect the architectural rules to be the hard ones. They fail by omission, they need system knowledge, and they cut against the grain of the public code these models learned from.
3. Invest in docs first. They are the lever for all new code.
4. Be mindful about what to clean. Your codebase is where agents copy from, and the code nearby is the example it copies.
5. Back it with an automated gate so regressions can't merge.

One thing we didn't expect

The invariant our agents kept missing was the same one our people missed most often.

Preparing the codebase for AI didn't only make it safer for agents. It made the rules clearer for everyone. So overall our initial instinct of cleaning everything was not wrong, but way more subtle than expected.