Measure what the agent really brings in

Judge an agent on what it changed, never on how much it produced. Pages written, posts drafted and leads enriched are output; clicks, replies, signups and pipeline are outcome, and only the second pays for the first. So measure both, side by side, every week, run every change as a bounded test with a verdict date set in advance, and check the measuring instrument before you believe any number it gives you.

That last part is the one I learned the hard way. The examples below come from the agents that run Derrick’s marketing (I co-founded Derrick), including a week when the agent in charge of SEO found that its own numbers were wrong.

Put output and outcome on the same weekly scorecard

Output is easy to count and always goes up, which makes it dangerous. A scorecard that shows only output will tell you the agents are doing great while nothing moves. Put one line per area of work, with the output next to the outcome it is supposed to drive and the human time it cost.

Area	Output this week	Outcome it should move	Source of the outcome	Human time spent
SEO	Pages created, pages updated	Clicks and signups from those pages	Search Console, product database	Review of drafts
LinkedIn	Posts published	Profile visits, inbound conversations	LinkedIn analytics, CRM	Edits before publishing
Outbound	Leads pushed, messages sent	Positive replies, meetings booked	Sending tool, CRM	Sequence setup, reply handling
Newsletter	Issues sent	Clicks to product pages, signups	Email tool, product database	Final read

The last column is the one people forget. An agent that saves the writing but needs an hour of corrections per piece has moved the work, not removed it. Read the scorecard as a trend over four to eight weeks, never one week alone, and look for the lines where output grows and outcome stays flat: those are the agents to fix or stop.

For the outcome columns, the rule from making the data reliable applies unchanged: one source of truth per metric, agreed before you read it.

Run every change as a bounded test with a dated verdict

An agent can change a lot of things quickly, which makes it easy to lose track of what caused what. In my setup, an experimenter agent runs every change as a test on a bounded set, for example 10 pages, and every test gets a written verdict at four weeks. The spec is written before the change ships.

test_id: seo-2026-w40-faq-blocks
hypothesis: Adding a FAQ block from real support questions raises clicks on feature pages.
pages_under_test: 10        # listed in under_test.yaml
control: 10 similar feature pages left unchanged
metric: clicks per page, Search Console, web search
baseline: 4 weeks before start
verdict_on: start + 4 weeks
decision:
  keep_if: test pages gain at least 20% more than control
  revert_if: test pages lose against control
  otherwise: extend 4 weeks once, then revert

The figures in this spec are illustrative; pick thresholds that fit your volumes. What matters is that the decision rule is written before the data comes in. Otherwise the agent, or you, will find a reading of the results that supports whatever was already decided. If your pages come from a template, measure by group of pages rather than page by page, as explained in tracking programmatic SEO performance.

Check the instrument before you trust the number

On 5 November 2026, the Head of SEO agent noticed that two exports which should have matched did not. One of its own measurement exports was broken down by query, and it only counted clicks on queries the tool names, which was about a third of the real total. The agent confirmed it with a second, independent instrument, the site’s GA4 data.

Relative rankings between pages held. Absolute levels were wrong, and some pages showed zero clicks when they had traffic. A decision to stop production had already been taken on one of those false zeros.

Two habits came out of it, and both are cheap to add to any agent that reads metrics.

def check_totals(export_a, export_b, tolerance=0.05):
    """Two exports of the same metric, same period, must agree before you combine them."""
    a, b = sum(export_a.values()), sum(export_b.values())
    gap = abs(a - b) / max(a, b)
    if gap > tolerance:
        raise ValueError(f"Totals disagree by {gap:.0%}: {a} vs {b}. Do not combine.")

First, before combining two exports, check that their totals match, and stop if they do not. Second, when a number looks surprising, confirm it with a second instrument that measures the same thing a different way. Running the same check a third time on the first tool will only give you the same wrong answer with more confidence. A zero deserves the most suspicion: it is the number most likely to trigger a decision to stop, and the one most likely to be an artefact of how the data was cut.

Protect what is under test

The same week brought a second lesson. A routine update had touched pages that were part of a running test, and spoiled it. The pages were declared as under test, in a file. The routine simply never read that file.

Declaring a test is useless unless every routine that can change those pages reads the declaration first. So the check goes into the routines themselves, as an early step:

STEP 1. Read under_test.yaml. Remove every page listed there from today's
        work list. If the file cannot be read, STOP and alert.

This connects to the routines chapter: the file is one more thing step 0 must be able to read before the routine is allowed to start.

Replay old verdicts when an instrument changes

When the measurement was fixed, the experimenter agent went back and replayed three earlier verdicts on the corrected data. One earlier fix had been wrong too: a decision that looked justified on the broken numbers did not hold on the real ones.

So keep every verdict with the data it was based on: the export, the date range, the query used. When an instrument changes, list the verdicts that used it and rerun them. It is the only way to know which past decisions you are still standing on for good reasons.

Verdict log field	Why it is there
Test id and hypothesis	What was tested
Instrument and export used	So it can be replayed when the instrument changes
Date range, baseline, control	So the comparison can be rebuilt
Decision rule written beforehand	So the verdict is a rule applied, not a reading
Verdict, date, who accepted it	So someone owns it
Replayed on, result	So you know which verdicts survived a correction

Stopping is a valid verdict

Measuring honestly means some agents will lose. In October 2026, a Reddit lever that had been started for Derrick was cancelled. That is the system working: an area of work that does not earn its keep should stop, and the time goes to the ones that do. The chapter on the full agent team shows how the remaining levers are organised.

Decide in advance what would make you stop an agent: outcome flat for eight weeks while output grows, review time higher than the time it saves, or a test that keeps ending in “extend”. Write it on the scorecard next to the line, so the decision is already made when the numbers arrive.

Your next step: build the scorecard for one agent this week, with output, outcome, source and human time, and pick the two exports it relies on. Check that their totals match before you read a single trend.