The Problem with Opinion Polls

Back in December, when people actually had debates about whether or not Chrysler and GM would go bankrupt, one of the claims made by the anti-bankruptcy camp was that 80% of people would not buy a car from a bankrupt automaker. That number came from a CNW survey; here’s a summary from Motor Trend (hat tip Jane Hamsher):

A recent study from automotive market research firm CNW surveyed 6000 people intending to buy a new car within six months, and discovered that more than 80 percent of them would switch brands if the vehicle they wanted came from an automaker that went bankrupt. Breaking it down by company, Americans were more likely to abandon domestic automakers than foreign ones, with Chrysler faring the worst — a full 91 percent of buyers wouldn’t take home an Auburn Hills product if the company went bankrupt.

The theory was that bankruptcy would lead to an immediate collapse in sales which would lead to liquidation. (A later study, cited here, said that if the government were involved in the bankruptcy, the number of people who wouldn’t consider buying from GM would be 51%.)

This is what I said in December:

I strongly suspect that 80% is just a poorly worded and interpreted poll question. If you ask people in the abstract if they would buy cars from a bankrupt car company, of course they will say no. But in the real world, if the car they want is made by a bankrupt company, and they get a good deal, they will buy it. Just look at the November auto sales. GM was down 41%; Toyota, Honda, and Nissan were down 34%, 32%, and 42%, respectively. And everyone buying a car in November must have been aware that bankruptcy for GM was a serious possibility. (Besides, haven’t we been talking about a GM bankruptcy on and off for years?) Sure, bankruptcy will hurt sales a little. But 80% is just not credible.

Well, now we know. In May – during which Chrysler was in bankruptcy – Chrysler sales were down 47% from the year-ago period. Overall sales were down 34%, which means non-Chrysler sales were down around 33%. So as a crude estimate, if Chrysler were like the average automaker, for every 100 cars it sold last May, it would have sold 67 cars this May. Instead, it sold 53. That’s a 21% decrease – a lot less than the 91% predicted.

I’m not claiming I can predict auto sales better than other people – I know virtually nothing about auto sales. I’m just saying you shouldn’t rely on polls that ask people what they would do under some hypothetical scenario, since they don’t know what they would do.

By James Kwak

18 responses to “The Problem with Opinion Polls

  1. Well, it could also be due to some confusion over what “bankruptcy” entails in the mind of the polled public.

    Bankruptcy last year as “substantial likelihood of failure at reorganization in an free fall economy leading to near-term liquidation, voiding of warranties, and in unavailability of parts ” is quite different from how we currently understand it as “Too big to fail, implicit endless government support, stabilizing expectations for the marketplace, and the consumer need not worry all that much”

  2. I just wonder why people don’t mind taking flights from bankrupted airlines but do mind to buy cars from bankrupted automakers. If bankruptcy makes things unreliable, it induces a much higher risk on your life to take the flight!!

    Can it only be because they may lose their warrenty with a bankrupted automaker?

  3. i agree that many opinion polls can be misleading, but your argument is flawed. as a first step, you should only ask people who want to buy a car. it could well be that 80% never buy a car from a bankrupt company, but those are not the people who fork out their hard-earned money for a new Hummer.

  4. Wayne Martin

    So how many major US automakers have gone bankrupt in the past, and been effectively bought and run by the US Government–so that there is a history of what service and resale issues for a bankrupt Auto company.

    A second question that asks something like: “Would you buy a new (or used) DeLorean car today?” might help to provide a little perspective in helping to understand the original question.

  5. I think the average person wouldn’t even understand what you were asking. Bankruptcy mean the company is out of business right? I can’t buy a car from a company that’s out of business! So, no.

    But when they see a screaming deal on a new Dodge Ram, and they see that the dealership is open, it never occurs to them that Chrysler is bankrupt.

    This was a red herring from the beginning. The government was also going to ensure warranties and parts were available.

  6. Everyone I know tells me their next car will be anything BUT Government Motors..Not a poll, not scientific…but true none the less. Chrysler has the same problem. I now own Jeep..I will next buy Ford as they avoided government ‘help’.

  7. In fact I found out about the bankruptcy when I saw a friend looking at their website searching for fire sale prices.

    That question about whether you want to buy the car from a bankrupt company is totally invalid as Pete Muldoon points out. At the bare minimum you need to add an element of price and brand into the equation.

  8. I think when something or someone fails it is easier to question the polls than admit that the plan sucks.

  9. Tippy Golden

    So after the bankruptices …

    How might the American auto industry transform itself to be (ideally) non-polluting, economically-relevant and a world leader in innovation?

  10. I hear through the grapevine that dealerships that do GM warranty work are having a lot of trouble getting paid (delays of 90 days or more on payment). Some are now refusing to do GM warranty work if GM doesn’t pre-approve the work and, even with pre-approval, they are still having trouble getting paid.

    If this isn’t straightened out quickly, there really are going to be problems getting the warranties honored.

    So I wouldn’t buy a GM vehicle just yet …

  11. I read a poll (I’m sorry to say I can’t remember the source) that said 54% of Catholics voted for President Obama last November. I DO believe that number, but it’s quite amazing when you think about it. Either way, when a company is going bankrupt, how many people will buy the cars is kind of a moot point at that stage isn’t it? The real question which should have been asked around 1982 when the Japanese were making FAR superior cars than America, why wasn’t anyone listening to W. Edwards Deming??? If you asked the CEO of Ford or the CEO of GM now who is W. Edwards Deming could they tell you??? Could they tell you Deming’s 14 points, or about Deming’s “red beads test”. I doubt any of American auto manufacturers have figured out the problem yet. They’re still trying to divert blame on “legacy costs” instead of admitting they had their head in the sand for the last 30+ years.

  12. johnlocksmith

    most opinion polls are misleading.
    bankruptcy does make things unreliable and i wouldnt take part in that

  13. James Twiner

    The real problem, James, is that most of these surveys are very poorly constructed and performed. Even (or perhaps especially) “scholarly” surveys done by major think tanks are typically very seriously flawed. It is an open secret that political surveys and polling are frequently intentionally biased toward the survey author’s opinions. What is less commonly known is that sociological and marketing surveys are frequently unconsciously (?) biased towards the opinions of their authors. Not to mention that there is a whole industry (which I am sadly part of) that exists solely to massage data out of poorly constructed surveys. Everyone thinks they can construct a survey … until they try to get results, then they call me and people like me and expect us to get meaningful results from bad data.

  14. The poll questions, as you point out, are about hypothetical situations. In reality, the economic decline brought opportunities to many Americans that would not have had them otherwise. Domestic automakers offer deep discounts and 0% financing in times like these, making a new car a possibility to buyers that might not be buying new otherwise. In the housing market, first time home buyers are turning out in record numbers.

    Auto makers are also trying new things, like buyer assurance plans. Some will make payments for you if you loose you job in the next year, others seem to be giving money away. Those programs didn’t exist last December.

  15. James Twiner

    JIMV, I would suggest that you are suffering from what we in the business call “selection bias”. You are talking to people you know, who mostly share your values and opinions (after all they are your friends, or at least you coworkers in the same industry). You should not be surprised that they (largely) share the same opinion about what is, after all, a fiscal matter. This is a survey selection trap that even the most carefully constructed professional instrument can suffer from … obviously an informal “poll” is much more vulnerable to this phenomenon.

    Imagine you were extrapolating national abortion rights opinions from a random sample of frequent churchgoers in Mississippi. Obviously, even if your questionnaire is a masterfully constructed instrument of complete impartiality, your results will be fundamentally unreliable, because your sample is totally corrupt.

  16. People are terrible judges of their own motivations and reactions. Perhaps the very worst way to predict people’s future behavior is to ask them. Economists have a hard time getting around that, because it seems to strike at the heart of all they hold sacred, but any psychologist or social scientist will tell you that people have no idea how or why they will react to anything. These sorts of survey questions are worse than worthless, and always have been.

  17. Never really had a run on the long list of bankrupt banks.

    Either people don’t know or believe in government guarantee.

    Detroit has been denying the eventual bankruptcy of the big 3 for several decades now.

  18. Jim Summers

    As soon as I see the letters CNW, I am suspicious. A few years ago CNW research did a “study” that showed a Hummer was greener than a Prius. That study was total garbage, but a lot of people believe it to this day.

    This outfit seems to be really good at making baseless claims that the auto companies want to hear. They are also inexplicably good at getting their claims to be taken seriously. I enjoyed reading the comments and discussion, but it seems like a waste of time to speculate about how their claims could possibly be true; maybe they just made it up.