How to test a borrowed benchmark for AI marketing tools

What to take away

A marketing manager at a Leeds retailer, drafting a board paper on whether last year's spend on AI marketing tools paid off, usually reaches for a vendor benchmark report first. The instinct is sound, provided the figure travels with its method.

What separates benchmark research from a supplier's case study?

Vendor research often blends the two. A case study follows one customer. A benchmark aggregates many accounts and reports a distribution, so ask which one you are reading. Ten brands from a single sector is a set of examples with a chart on top.

The difference matters when the number travels upward. A board will treat the word benchmark as the market, not the supplier's book of business.

How do you judge the sample behind a published figure?

Look for four details: who was measured, how many, over what period, and what was counted. If any are missing, the figure is marketing copy rather than evidence. An undefined metric cannot be replicated, and an unreplicated metric cannot be trusted.

Vendors borrow vocabulary too: fairness, transparency, accuracy. Where a report leans on those words, check the standard definition before you accept the number beside it. Google's Machine Learning Glossary: Responsible AI sets out how such terms are described in machine learning work, so you can see whether a supplier means the ordinary thing or one of its own.

What makes a benchmark AI-specific?

A generic marketing benchmark can be judged on its sample and period alone. An AI benchmark carries more moving parts: which model and version were tested, which prompt or workflow produced the output, and which tier of the tool the vendor ran it on. Ask which of those was held fixed and which changed. A single number that averages every model and every plan describes no tool you can actually buy.

Which benchmarks hold up in a board paper?

Prefer sources that publish a method and a questionnaire. Trade bodies tend to do this better than vendors. The Data & Marketing Association publishes research reports on data and marketing, including work on AI adoption (DMA research on data and marketing). Its reports carry methodology notes you can attach to a board paper.

Anything built on cookie-based measurement needs a compliance check too. The ICO's guidance on the use of storage and access technologies sets out the consent required under the Privacy and Electronic Communications Regulations (PECR) before you read from or write to a user's device.

What do the benchmark terms actually mean?

How do you test a borrowed benchmark on your own data?

Start by fixing your metric definitions before any comparison begins, following our guide to AI marketing tools measurement and reporting. Then take one metric the benchmark reports and rebuild it exactly as described for a single channel.

Run the comparison for a full quarter before drawing a conclusion. Differences usually come from attribution windows rather than the tool itself. A quarter is long enough to cover a full campaign cycle and a seasonal peak. Ask the vendor for their window in writing, then state yours on the same line of the report.

What should your own benchmark record keep?

Note the capture date, tool version, channel and the person who ran the query. Version changes and dashboard edits quietly break comparability, so a monthly snapshot exported to a fixed file protects the audit trail. Store that file outside the dashboard, because a deleted view takes its history with it.

Teams rebuilding their own reporting face the same discipline: lock the definitions before the charts are built, because that is where most comparability problems start.

Common questions

Is vendor benchmark research ever worth reading?

Yes, as a source of hypotheses. It shows what a comparable account achieved and gives you a starting range. It rarely proves what will happen in yours.

How many data points make a benchmark meaningful?

There is no fixed threshold. It depends on the spread: a tight distribution across thirty accounts can be more useful than a wide one across three hundred. Ask for the variance as well as the count.

Should we publish our own benchmark figures?

Only with the method attached: sample, period, definitions and exclusions. Without those details the number travels further than the caveat.

Can AI tools produce the benchmark for US?

They can assemble figures you already hold and speed up the reporting cycle. They cannot supply a baseline you never recorded, so fix the comparison period first.