A property management office has had an AI Employee answering tenant emails for three months. The dashboard says it handled 1,400 messages. The owner asks the office manager whether it's working, and neither of them can answer, because 1,400 messages could mean 1,400 good replies or 300 good ones and a pile of tenants who gave up and called.
I've spent the last four years building AI systems and agents for businesses, and this is the question I hear most once the first month is over. The volume number is always the first thing anyone looks at, and it's almost useless on its own. An AI Employee should be measured the way a good manager measures a person in the same seat: by the quality of the work, the work it finished, and how much of everyone else's time it took to get there.

Why the activity count misleads
An AI Employee is AI set up to hold one job on a team, with a manager and written rules about what it can do alone. Because it logs everything, the easiest numbers to pull are activity numbers, like messages sent or invoices entered. Those count effort.
A front desk person who answers 80 calls a day and books the wrong time for ten of them isn't doing better than one who answers 60 and gets every booking right. The same holds for an AI Employee, with one difference. It works faster than a person, so a small error rate turns into a lot of errors quickly. Activity numbers belong on the scorecard only as context next to the numbers that measure quality.
The scorecard: six numbers
These are the six I'd track for almost any role. Most can be pulled from its log and the systems it works in.
Accuracy. The share of a weekly sample that was correct by the handbook's standard. This is the most important number and the only one that needs a person to grade it.
Next is the share of work it finished without a person: bookings made, invoices coded, questions answered with nothing handed back. This is the work you're actually paying for.
Escalations come third, counted and checked. Count how many it handed to a person, then look at whether each one was the right call. Two kinds of mistake hide here: handing back things it could have done, and keeping things it should have handed back.
Speed is the time from a request coming in to a reply or a finished task. For a law firm's intake role, the hours between an inquiry and a booked consultation matter more than almost anything else.
Mistakes that reach a customer get tracked separately from the sample, because they have a cost the sample doesn't show. Each one should have a written fix next to it.
Last is the manager's time: minutes a week spent reviewing its work and answering its escalations. If this doesn't fall after onboarding, something is wrong with the setup, whatever the other numbers say.
How to check accuracy each week
Accuracy only comes from a person reading the work. I'd pull about 20 pieces at random every week, from across the duties, not just the easy ones. For a dental clinic's front desk, that's a mix of booking replies, reminder texts, insurance questions and anything it escalated.

The person who manages the job grades each one against the handbook as right, right but badly worded, or wrong. For every wrong one, write down the cause, because the cause decides the fix. In my experience it's nearly always one of four things: a missing instruction, an out-of-date document, a bad example, or a limit that's too loose. I went through those fixes in more detail in how to train an AI Employee.
Twenty items takes about half an hour. That's small enough to happen every week, and big enough that a mistake repeating itself will show up within a couple of weeks.
The numbers that matter in each role
The six-line scorecard stays the same, but each job adds one or two numbers that tell you whether the work did what it was for.

A receptionist role at a clinic should be judged on missed calls and open slots filled from the waitlist. A bookkeeping role is judged at month-end, by how many transactions the accountant had to recode before closing. For an online store's customer emails, I'd watch how many conversations get reopened within a week, since a reply that sounded fine but didn't fix the problem comes back. An inside sales role lives on how fast a new lead gets a first reply and how many of those replies turn into a booked meeting.
Pick these with the manager before the numbers start coming in. If you choose them afterwards, it's tempting to pick whichever ones look good.
Measure the job before it starts
None of these numbers mean much without a baseline, and the baseline has to come from before the AI Employee starts. How many calls went to voicemail last month? How long did a lead wait for a reply? How many transactions got recoded at the last month-end? Most businesses have never measured the job, so the first comparison ends up being against a guess.
Spend two weeks counting before switching anything on. A spreadsheet with one row per day is enough. If you're still deciding which role to start with, this is also the point to check whether it suits an AI Employee at all. Some websites offer free assessments that check whether a role can be filled by an AI Employee.
The monthly review
Once a month, the manager and the owner should sit down for half an hour with four weeks of scorecards and decide three things.
The first is which duties change level. A duty with a clean sample for a month can move from "reports" to "does it alone". A duty where the same kind of mistake keeps coming back moves down to "asks first" until it's fixed. I covered the levels and how a duty earns its way up in AI Employee onboarding.
The second is what changed in the business. A new price list or a switch to new software is the most common reason performance drops. If accuracy fell in the same week the fee schedule changed, the fix is in the documents, not the AI Employee.
Third, decide whether it's worth what it costs. That's the next section.
Cost per finished task
Divide what the AI Employee costs in a month by the work it finished without a person. When it's built from software you assemble yourself, the cost is the subscriptions plus the hours someone on staff spends maintaining it. When a provider builds and maintains it, it's usually a monthly fee. I went into how those are priced in what an AI Employee costs.
Then do the same for the person in the seat. The US median wage for a receptionist was $38,010 in May 2025, according to the Bureau of Labor Statistics. Wages are about 70% of what private employers spend on compensation (BLS Employer Costs for Employee Compensation), which puts the full cost of the seat closer to $54,300 a year, or about $4,500 a month.

The comparison is only fair when the work is the same quality. An AI Employee that finishes twice as many bookings with three times the mistakes isn't cheaper once someone has to clean up after it, which is why cost per task belongs at the end of the review and not the start.
Signs performance is slipping
Three patterns show up before the problem reaches customers.
Escalations drop to zero. Some work always belongs with a person, like a refund outside policy or a tenant reporting a leak at midnight. When escalations stop entirely, it has usually started handling things it should hand back.
The same mistake keeps coming back after it was corrected. That usually means two documents disagree, or an instruction is too vague to follow the same way twice.
The manager stops doing the weekly sample. This is the most common one and the most dangerous, because it's how the other two go unnoticed for months. I listed the work that should never leave a person's hands in what an AI Employee can't do.

Where to start
Open a spreadsheet with six columns, one for each line on the scorecard, and one row per week. Add the one or two numbers specific to the role. Block 30 minutes on the manager's calendar every Friday for the sample of 20, and one longer meeting at the end of each month. If the AI Employee hasn't started yet, spend the next two weeks filling in the baseline, so the first review has something real to compare with.
Wage data: US Bureau of Labor Statistics, Occupational Outlook Handbook, receptionists, May 2025 median; full-cost figure uses the BLS Employer Costs for Employee Compensation share of wages in total compensation. The opening example is illustrative.
