Hey there,
Earlier this month, the AI steering group at an architecture firm I've worked with for the past year showed its executive team what they'd been building. One of the first examples came from the staffing coordinator. Every Monday morning she exports the week's staffing data from Deltek, the project and resource planning software many architecture firms run on, hands it to Claude with a template, and gets back a utilization report: who's booked on which projects, who has room, and who's overcommitted. Claude groups people by discipline, flags anyone new or missing, asks her about holidays, and checks its own math against the raw data before she reviews it and sends it out.
Building even a basic version by hand took more than two hours, and the result was of limited use to the project managers. With AI it takes about fifteen minutes, and the report has grown to three tabs: an executive summary, a view by discipline and person, and a view by project, color-coded so anyone can see who's under 90% booked and who's over 103%.
That's the number that goes on the slide. Two hours down to fifteen minutes, every week.
The more interesting detail, however, came when someone asked whether a report like that would even be worth building by hand. Not really, she said. By the time it was done, the week would be over. Even now, she runs it at 9 on Monday morning, and by 4pm nobody's looking at it anymore. People have seen who's short or overbooked and fixed their project plans. If a designer has nothing planned, someone plans him onto a project. By the staffing meeting, the report is mostly a conversation starter about who can take on what.
Judge that report by accuracy and it's failing. Judge it by time saved and it's a nice win worth about ninety hours a year. Both miss what changed. Before, the data lived in several places in Deltek, looked different depending on whose screen you used, and took expertise to read. Now it arrives in one place, in plain terms, on the morning people plan their week. The report goes stale because it's working. Its real value is the plans people changed, and I've yet to see an AI dashboard capture that.
The report goes stale because it's working.
Last November I wrote about why 85% of your team isn't using the AI you bought them. That issue treated usage as the problem to solve. This one asks whether usage should be the goal at all.
Most firms measure AI the way they'd measure any software rollout: licenses assigned, weekly active users, prompts sent, hours saved on a survey. Those numbers are easy to collect and they make for a clean board update. The trouble is that people respond to whatever you count.
I heard recently about companies that set token-consumption targets for AI. Some employees hit them by writing loops that keep the model running on nothing. It's an old pattern. When bars installed breathalyzers so patrons could check whether they were safe to drive, patrons turned them into a drinking competition to see who could get the highest number. Richard Rumelt put it plainly in The Crux: arbitrary goals don't motivate achievement, they "motivate cynicism and fabrication."
Gaming is the obvious version; the quieter version is building to the metric. A colleague recently told me about a software company where a team had an AI adoption target to hit, so they added a "Run AI" button that generates a summary. The summary wasn't saved and came out different every time you clicked, so nobody used it. When adoption came in low, the conclusion was that users don't want AI summaries. The metric did its job: it produced a button. Nobody checked whether anyone's work got better.
Self-reports fail differently. At another architecture firm I work with, the team described themselves as "comfortable" with their AI tool in the discovery survey before our training program. So we aimed the first session accordingly, and started at prompting technique and checking AI output. The first session and the debrief afterward told a different story. They felt solid on prompting, but were lost on setup. Almost nobody had custom instructions configured, and only one person had used the tool's notebook feature. During the call, a team member's memory setting turned out to be on and capturing nothing. We rebuilt the next session around setup for everyone.
They weren't wrong to call themselves comfortable. They used AI every day. But "comfortable" measured familiarity, and familiarity isn't the same as knowing what you don't know.
"Comfortable" measured familiarity, and familiarity isn't the same as knowing what you don't know.
I have seen a usage number work in the past. When I was on the developer relations team at HERE Technologies, our goal was to double sign-ups to our freemium developer plan every year. Everyone knew the number, every tactic was judged against it, and we successfully doubled our developer numbers three years in a row. Sign-ups alone would have been easy to inflate. What kept them honest was that we watched how many free developers became paying ones, and that rate held steady the whole time. The sign-up count was the target while the conversion rate told us the target still meant something. Economists call the usual outcome Goodhart's law: when a measure becomes a target, it stops being a good measure. Ours didn't, because something else was watching it.
Most firms tracking AI have the sign-up count but nothing playing the part of conversion. Usage measures the input. Time saved measures the input more precisely. What you really want to know is what people did differently because the AI was there, which is harder to see and almost never shows up on its own. Misha Sulpovar makes a similar point in The AI Executive's Handbook: ROI comes from productivity aligned to your goals, not from raw productivity.
Three changes would make that easier to see.
The first is to pair every usage number with the behavior it's supposed to produce, and watch the ratio between them. For the utilization report, that's how many project plans changed after it went out. If usage climbs and the behavior doesn't, someone is feeding the number.
The second is to replace opinions with artifacts. "How comfortable are you with AI?" invites a feeling. "Show me your custom instructions" invites evidence. So does "which projects do you open every week?" and "what's in them?" You learn more in five minutes of looking than from any survey.
The third is to treat strange results as a signal before you treat them as a defect. A report that goes stale by afternoon looks like a quality problem. It turned out to be the best evidence the workflow was doing something. When an AI output stops matching reality quickly, ask whether that's because someone acted on it.
The coordinator's report will never stand out on a usage dashboard. It's one person running one workflow once a week. By the numbers most firms track, it's a rounding error, and yet it might be the most useful AI workflow in that building today.
So here's the question I'd put to any leadership team: what would you measure if you couldn't measure usage?
Break a Pencil,
P.S. What's one AI metric your firm tracks that you suspect isn't telling you the truth? Reply to this email, or if you're reading on the web, write me at [email protected]. I read every one.
P.P.S. Know someone building an AI adoption dashboard right now? Share this with them.
