Did your agent learn, or did it get lucky?

A few weeks ago, an AI leader at a Fortune 500 technology company told me about an HR agent they were using. Since being in production, they had asked it four times to approve a PTO request, something near and dear to all our hearts, and three times it said yes. On the fourth run, with the request still inside policy, it said no. They were not happy.

What bothered the team was not simply that the answer changed. They could not see what had changed in the reasoning, so they had no obvious way to improve the system. They also could not explain why the first three answers had been right. Had the agent understood the policy, or had it landed on the correct decision by chance?

Around the same time, a product leader building an AI travel experience described a more complex version of the same problem. Give the agent a request combining company policy, flight preferences, timing and hotel location, and it may produce a sensible itinerary. Run it again, and it can make different trade-offs without making clear why.

Those examples have stayed with me because they turn what looks like a model-quality problem into a systems question: when an agent is supposed to be learning a process, how do you know it learned rather than got lucky?

Learning has to live somewhere

When people talk about continual learning, they often go straight to post-training or fine-tuning. In practice, an enterprise system learns in two places, and each is suited to a different kind of knowledge.

Some learning belongs in memory. This is where you keep explicit facts, policies, exceptions and previous decisions, such as an approval threshold or a rule that documents with “draft” in the title should not be processed. That knowledge should be inspectable, traceable and easy to update (or delete).

The problem is that memory does not stay useful by itself. Facts go stale, instructions contradict one another, and a correction for one unusual case quietly becomes a universal rule. Anyone who has maintained a personal agent has probably seen its context turn into a crowded drawer where nothing has been thrown away.

Other learning belongs in the model’s parameters. This is closer to skill than recorded knowledge. A skilled dispatcher does not make every decision by looking up a written rule; over time, they learn how to combine weak signals and make judgments that are difficult to put into words. Some of that know-how may need to be learned through post-training or specialist models.

Neither approach is enough alone. Put changing facts into model weights and they become hard to inspect or update. Try to express every piece of tacit expertise as another prompt instruction and the system becomes brittle. The better question is what kind of knowledge we are trying to preserve, and where it should live.

Separate the planner from the executor

The travel conversation helped make another part of the problem clearer for me. When the same model both invents the process and carries it out, it may create a different plan every time it runs.

On one travel request, it may begin with company policy, then consider timing and finally optimize for preference. On the next, it might start with the traveller’s preferred airline and treat policy as a later constraint. An HR approval can drift in the same way, with the agent checking the policy first on one run and the employee record first on another.

Each chain of reasoning may sound plausible on its own. From the company’s point of view, however, the process is changing without anyone deciding that it should.

We have found it more useful to separate the roles. A planning, or architect, agent designs and stores the process: which signals matter, what order the steps should follow, where another agent can act and where a person needs to approve. Execution agents then carry out that plan rather than rebuilding it. Exceptions can still change the plan, but those adaptations become visible and reviewable, similar to pull requests in code.

In one of our systems, when the planning agent reaches an ambiguous fact, it can create three roles: one gathers the signals, another makes the decision, and a third checks the first two. In another recent build, the planner told the executing agents not to process documents with “draft” in the title, even though nobody had explicitly written that rule.

That looked like common sense emerging from the process, but I would still hesitate to call it proof of learning. The real test is whether the rule appears in the next relevant case, improves the process and avoids creating new mistakes elsewhere.

Three ways to grade a learner

For now, I think about evaluation at three levels.

The first is controlled replay and disagreement testing. Run the same task several times and compare the decisions, not just the wording. An operator agent can also send the task to two models and ask a third to judge whether their conclusions materially disagree. It is only a smoke detector, but when unchanged facts repeatedly produce different actions, the system has not earned much autonomy.

The second level is human feedback, with more context than a thumbs-up or thumbs-down. Did the person accept the recommendation, edit it or reject it, and does the same correction appear when a similar case arrives? This is not perfect ground truth, but preserving the correction lets you test whether the next comparable case is handled better.

The third level is the one we are still researching: observe what happens in the world. Did the staffing decision create a problem later? Did the travel plan lead to a missed meeting? Did an AP exception recur?

This is where luck becomes especially difficult to separate from judgment. An airline can lose $150,000 after making the best available decision, when every realistic alternative would have cost more. A good result can also hide poor reasoning that happened not to fail. To learn from outcomes, the system has to retain what was known when the decision was made and compare enough similar cases to distinguish a pattern from hindsight.

We do not yet have a clean outcome-based reward function for every long-running enterprise process we work on, and I do not think it helps to pretend otherwise. What we can build is the record needed to keep learning: what the agent knew, which plan it followed, why it chose an action, what a person changed and what eventually happened.

For me, that is where learning begins. A correction should survive into the next relevant case, generalize beyond the original example and improve the decision in a way the people responsible for the process can understand. One correct answer or one successful outcome is not enough, and even consistency can mislead us when the system is consistently wrong.

If you are running agents on a real process, how are you measuring whether they are actually learning, and has a good outcome ever fooled you into thinking the decision was good?