One bit of my old life as a broadcaster that I seem unable to leave behind is the ratings. TV ratings – and specifically those from Nielsen – are the currency of the TV ad business and billions of dollars of media are bought and sold based on these numbers. What caught my eye this morning was the reporting of last week’s late night ratings and the analysis connected to the report. The writer did a good job dissecting the numbers except that they conveniently failed to mention one thing that should be instructive to any of us in business: the margin of error.
(Photo credit: Wikipedia)
What the author failed to mention is that there was no statistical significance between the reported audiences in any of the numbers that Nielsen was reporting. Since the numbers discussed in the piece were Adult 18-49 numbers, the reporting is based on a subsample of Nielsen’s panel, meaning that the margin of error is wider than on all the ratings as a whole. While I don’t have a rating book in front of me, I know there always used to be a disclaimer in every book explaining that the numbers it contains are only accurate up to a point. They’re estimates. When we’re looking at number this small (and the late-night numbers are in tenths of a point), it’s just as possible that the network reported in third place could, in fact, have more viewers than the network reported as in first place.
The point here isn’t to denigrate the ratings system (I’ll save that for another screed). The point is to remind each of us that almost every piece of data that we look at needs to be taken in context and with appropriate disclaimers. What I find helpful is to pay attention to trends and not to absolutes. The only numbers without a margin of error are those pertaining to actual money received and actual money spent, and even those are generally only snapshots of a moment in time.
The next time someone comes to you with a data point, ask about the margin of error or about any factors that could affect that data. New visitors to your website are up? What percentage of people routinely delete cookies and, therefore, seem to be new when they’re not. App installs are up? How many people deleted the app last week, was that an increase, and could the new installs, in fact, be reinstalls? See what I mean?
Consult Keith 