Tag Archives: Opinion poll

Dead Wrong

There are a lot of lessons to be learned from the events of the last couple of days and not all of them pertain to our politics. One question I think I hear from people who fall all across the political spectrum is “how could almost every prediction be so wrong?” After all, putting aside the prognostications of loyalists on either side, none of the pollsters and data-based predictors got this right.

I’m not going to go into the politics but we can learn some valuable business lessons here. I’m not sure this was a case of garbage in, garbage out. That said, it’s clear that the legacy systems from which samples are drawn such as calling landline telephones are not accurate anymore. The real issue is one that I think we have in business, though, which is the inability to tell the difference between “good” data and noise. More importantly, we tend to rely on faulty data to the exclusion of both external factors and our own common sense. We like to tell stories that can be believed, and that happens when the stories echo popular beliefs. We focus on things that have happened already and in so doing we often miss subtle undertones that tell us what went before may not indicate what will come next.

We also suffer from the echo chamber in business. We talk to our coworkers and reinforce faulty information. We tell the tales that our tribe shares and miss those from the outside – the other tribes.

I was just as bad as many of you on Tuesday. I said more than once “unless every poll is dead wrong, it’s going to be a short night.” Well, they were and so was I. While the pollsters will have to wait 4 years to show they’ve learned how data can’t be the only thing we consider as we make decisions about the right path, you get that chance the next time some information crosses your desk. Take it!

Leave a comment

Filed under Consulting, Huh?

The Right Question

We’re filling out a survey from our homeowner’s insurance company. I guess they want to make sure that we’ve got ample coverage in case a party gets out of hand and we need to rebuild Rancho Deluxe. One of the questions reads as follows: 

Percentage of the interior walls that are plaster

Hmm. Would that be the percentage based on the number of walls, the percentage plaster represents of square wall footage, or something else? After all, in a rectangular room, if one long wall is plaster, then the right answer may be 25% or it may be 40%. How accurate does this response need to be?

There’s actually an excellent business point contained in that silliness. It’s not enough to ask the right questions. We also need to ask them in the right way so we get the expected, actionable data. In the example above, while my answer isn’t a huge data set, when aggregated into the other data the company is pulling together, the sampling error will be larger than it needs to be since half the respondents can respond using one way to look at the question and half the other.

Obviously, it’s not just a lack of clarity that can affect the outcome and usefulness of your research.  Asking leading questions which are almost certain to elicit a particular response is bad as well (do you do XYZ every day?).  So can asking open-ended questions since there is no guarantee that anyone will focus on the specific area you’re researching.  Then there are the folks who overlap responses (how old are you – 18-21, 21-30 – how does a 21-year-old respond?).  Or ask loaded questions (how long ago did you stop beating your spouse?).

Asking questions is really important but asking badly structured questions is a waste of time. Clear?

Leave a comment

Filed under Consulting, Huh?

Political Polls And Your Business

It’s election season and there are multiple new polls released every day.

Description=Opinion polling by state for the U...

(Photo credit: Wikipedia)

It’s a lot of data and the media jump on each tidbit to proclaim the outcome of an event that’s still many weeks away.  Each new glimpse of the situation is treated as if today was Election Day until the next piece of information surfaces and the cycle begins again.  Sound like your business?

It’s a very instructive situation and points to something I tell my clients all the time.  A data point isn’t really important in and of itself.  What IS important is the trend of that data and what it reflects about your business.  Continuing the political example, Romney’s poll numbers over time have been declining while Obama‘s have been trending up.  Many polls are still within the statistical margin of error about which candidate is winning (a fact often overlooked since these are results based on samples).  Still, the data points to a sense of things, especially when you get past the top-line numbers and dig into segments and qualitative questions such as readiness for job, feelings about the person, etc.

If these were your web analytics, you’d want to do the same sort of thing.  Get past the top line stuff – visits, page views – and dig into segments.  Are visits from organic search up or down?  Are we seeing new and relevant search terms creating visits or is everything limited to searches on your company name indicating a narrow audience?  What are the different usage trends among new and repeat visitors?  How is social impacting your audience?  Are entrances coming to many pages or just one?

Don’t worry about any one day, worry about trends and you’ll be on a much more actionable track.  Rolling averages and trends don’t cause daily fire-drills to fix something that might be a blip.  Panic over a one-time occurrence usually does way more harm than good, whether it’s a poll, an analytics report, or a financial statement.

Enhanced by Zemanta

Leave a comment

Filed under Helpful Hints