Comments: Consumer Reports/JD Power Rankings

andres3 · June 2008

Well, I've sought out many a fellow Neon owners over the years and found out the following:

Everyone had rattles galore that made it sound like the car was disintegrating on the road as you drove.

Everyone had glue leakage seeping around the outer edges and trims of all the windows.

Everyone had faulty head gaskets that couldn't make it past 50K.

Some had faulty auto trannies that couldn't make it past 65K.

Some had dashboard that became rattle traps and unseamed.
Some had faulty parking brakes.
Some had faulty O2 sensors, O-rings, battery terminals, starter cables, wires, belts, fuel pumps.

Some had all of these issues in a single vehicle as was a complete POS. :lemon:

tidester · June 2008

Everyone had ...

Would you share with us how large that sample was and can you say that the people you talked with were randomly selected?

tidester, host
SUVs and Smart Shopper

62vetteefp · June 2008

Why do that? Anybody can say anything on the net.

andres3 · June 2008

I'd say my sample size of 3 Neon owners was fairly randomly selected, being as one was myself, and the other 2 were friends/acquaintances I made years apart and years after purchasing my own Neon.

The fact that none in the sample size differed from the norm of all Neon's being lemons shows up as quite a strong case. Though some lemons are more sour than others.

backy · February 2009

In #3127 of Hyundai Elantra Maintenance and Repair, BambuListener said:

Whether a problem is "minor" or "major" may be subject to opinion. What is not, is that there is a problem in the first place - whether it's major or minor is secondary. As such, if I see very few complaints about problems for brand X, but a lot of complaints about brand Y, then that's all I need to know... a dispute may break out about how much of the brand Y problems are "minor" vs "major", but I'd rather stick with NO problems, and so plump for brand X. So the overall number is what matters. Now, CR does break down problems into major and into minor - fine, let's disregard that. What we can't disregard is that there IS a problem (major or minor).

As to the overall verdict of 2004 vs 2005 - I don't know about the 2008 Auto issue. I'm going by what's on their website right now - and anyone who has access to CR online is welcome to verify it. And there it is: on the overall verdict, which they call "Used Car Verdicts", it shows 2004 as "better than average" (red half circle), and for 2005 it shows "average" (plain black circle). Here a link, but you may have to be a subscriber:

http://www.consumerreports.org/cro/cars/models/used/hyundai/elantra/reliability.- - - htm

Thus there is deterioration from 2004 to 2005 (then picks up again for 2006) - according to CR. I'm just reporting on what I'm seeing.

As to the one person's experience with the tranny - the only reason I cited it, is not to say "that proves that 2000 trannys are bad", rather "that's why it can be useful to see what is being said by surveys", because in *this case* it mirrors the findings. And that's a fact - the mirroring; now what that implies is up to your choice as a consumer. I'm merely reporting on the FACT that this particular experience *mirrored* the survey finding, which is an objective fact (after all, it did).

I think you misunderstood me. I'll try again. CR in its auto survey doesn't ask subscribers to rate their problems as "major" or "minor". It asks them to decide whether a problem they had with a car is "significant". If it is significant in the mind of the subscriber, they are to report it on the survey. If it's not significant in the mind of the subscriber, they are to leave it off the survey. So whether a problem shows up on CR's survey or not is entirely up to the person taking the survey. The entire survey is therefore subjective. That doesn't make it some evil thing, but it's important to understand how the survey results were derived.

The link you posted is only for members. In a few days the 2009 Auto Issue will come out, and then the data will be publicly available.

Also, your logic on the use of a single example to prove that surveys are useful is flawed. That one example does NOT mirror the findings of CR's survey. In this one example of someone driving an Elantra, there was a transmission failure rate of 100%. CR's findings were that the transmission problem rate of the 2000 Elantra was worse than average. It did not say the failure rate was 100%. In order to show a "mirroring", you would need to prove that every subscriber who reported on the 2000 Elantra in CR's surveys for every year since 2000 reported at least one transmission failure. (See how that "proving" thing works? Fun, isn't it?

)

BambuListener · February 2009

Re: CR survey. I have not studied how CR conducts their surveys. Regardless, even if we accept your version, the conclusion still stands. Let us assume that the individual feedback is subjective. Since it is a random distribution, the margin of error should be the same across each polling. There is no reason for the users of 2005 to en masse have a negative perception, while 2004 users have en masse a positive perception. Thus this is still a valid statement: "in the view of car users, the Elantra 2005 had more problems than the Elantra 2004".

Re: my example of one user. I am exactly right in my example. Here's a simple usage example.

1)The Backy Survey indicates a very high number of reported problems with Yugos (here we can indicate transmission, or engine, or fuel system or whatever).

2)If I have a Yugo which never develops a problem, I can accurately say: "My experience was atypical given what one may expect from the Backy Survey results".

3)If I have a Yugo which does develop a lot of problems, I can accurately say: "My experience mirrored, or was to be expected in light of the Backy Survey results".

In mathematical terms, from probability and statistics: it is a question of odds. If I have a car that odds say will have a bad transmission, and it does have a bad transmission, then I say: par for the course. The expected happened. The event/car conformed/mirrored the odds/expectation. If the transmission is OK, then I'd say: the car beat the odds, that was unusual. Just as if I buy a lottery ticket and don't win, well, it conforms to the poor odds of winning. Same here. CR told us to expect trouble with the transmission. Trouble developed - and so, was in accordance with survey. That's why surveys are useful, even for one car... backy, that is the ENTIRE POINT OF THESE SURVEYS!!! They are meant as guides for the INDIVIDUAL buyer, not a fleet buyer as you seem to imply by saying that one car examples don't matter. And the reason they can do that is because of a very powerful tool: statistics and probability. Unsurprisingly, most of the time the "expected" happens - the high odds. That's why we use them.

I'll be happy to discuss this as long as you wish - I have a background in math and statistics, so I naturally like to share my knowledge and isn't this what this site is all about?

mickeyrom · February 2009

I think this site is for people to share an interest in cars.Nothing more, and nothing less.

backy · February 2009

Unless you know what CR considers "Average" or "Below Average" in terms of number of problems per car, it isn't possible to say that a single problem mirrors their survey. Here is a quote from Consumer Reports, April 2008, p. 86:

Models that score a (black dot) are not necessarily unreliable, but they had a higher problem rate than average models. Similarly, models that score a (red dot) are not necessarily problem-free, both they had relatively few problems compared with other models.

For some trouble spots with very low problem rates, we do not assign a (black dot) or (half black dot) unless a model's problem rate exceeds 3 percent. If the problem rate is between 1 and 2 percent, it is assigned a (half red dot). If the problem rate is below 1 percent, it is assigned a (red dot).

Now let's use this transmission problem example to illustrate what the statement above means and why your analysis above is incorrect. In order to get "Below Average" on transmission, it means that a little over 3% (minimum) of owners of that model year who responded to the survey reported a "significant" problem with the transmission. That means that nearly 97% of the owners of that car model who responded to the survey did NOT report a problem with the transmission. So now take this individual who drives a 2000 Elantra and finds it has a transmission problem. Does that experience have a positive correlation to the survey? No, because it goes against the behavior predicted by the survey. Now take another 2000 Elantra in which the transmission is fine. Does that experience have a positive correlation to the survey? Yes, because nearly 97% of the respondents for that particular model and year of car had a similar experience--no problem with the transmission.

Therefore, the CR survey predicts that most owners of 2000 Elantras should expect no problems from their transmissions, since that is the "high odds" outcome.

Hopefully you see now that it is important to know how these surveys are conducted and reported before making assertions based on them. Statistics and probability can be powerful tools, when applied correctly.

dtownfb · February 2009

Every year at this time we get a discussion about CR and JD Powers survey. Give it a week or so and it will get back to the normal car discussion.

backy · February 2009

Read the title of this discussion. We are discussing the CR and JD Power rankings on Elantras. To see the full context, you need to follow the thread back into the Elantra Maintenance and Repair discussion where is started.

There were no posts in this discussion since last June. If you or mickeyrom wanted to discuss cars here, there was plenty of opportunity.

tidester · February 2009

I have not studied how CR conducts their surveys.

There's a possible problem with sample bias. I understand that CR surveys subscribers to their publications so that all Elantra owners are not equally represented in the sample, i.e. not all Elantra owners subscribe to CR. The surveys also suffer from nonresponse bias as people choose to participate or not participate. Typically, people with strong feelings one way or the other will choose to submit responses. At best, the survery results are a guide, as you suggested, so one should not try to read too much into them. JMHO.

In mathematical terms, from probability and statistics: it is a question of odds.

In mathematical terms, a corollary to the Law of Large Numbers is that a large sample of skewed data will produce skewed probabilities. :shades:

tidester, host
SUVs and Smart Shopper

backy · February 2009

There seems to be an assumption, implied by you and others, that the CR subscriber base is not representative of car owners in general. I can understand how that could be true, but I've never seen any hard data on it one way or the other--have you?

On the non-response bias, I could see how it could skew the results, depending on what the "middle group" would say, and how it would be represented in the results, or potentially they could cancel each other out and we'd be left with a result similar to what would have happened with a broader base of responses.

As for reading too much into survey results, I am with you there. I think there is a lot of room for misinterpreting both the JD Power and CR surveys. For example, I don't know if many people realize that a large part of the IQS study has nothing to do with reliability, but with the respondees' opinions on the design, performance, styling, and feature content of the cars. Also, on CR's survey, I think it's easy to focus on those red dots and black dots and not realize that, as CR states, a black dot doesn't necessarily mean that a car is unreliable, and a red dot doesn't mean a car is problem-free. It's all about which cars are more reliable or less reliable than the CR survey average for cars of the same model year. Numbers-wise, in terms of problems per vehicle, we are talking about relatively small differences except in the very worst (or best) cases.

dtownfb · February 2009

So basically it's the same discussion that has happened about a dozen times regarding CR and JD but this time it's focused on the Elantra.

My two cents is the Elantra is one of the most overlooked car in this segment.

Happy debating!

backy · February 2009

Well, I don't think the Elantra is the most overlooked car in the segment any more, not since CR elevated it to "Top Pick" status in the class some time ago. But personally, I think the current Elantra deserves to be overlooked, if for nothing else than Hyundai can't seem to make small cars with good crash protection. But that is another subject... at least until the CR April Auto Issue comes out. Then I wouldn't be surprised to see CR take away the Elantra's "Top Pick" status because they don't like to recommend cars with substandard crash protection.

As for the ongoing discussion on the JD Power and CR scores for the older Elantras... feel free to use the ol' Page Down key until you find something more interesting to you.

tidester · February 2009

There seems to be an assumption, implied by you and others, that the CR subscriber base is not representative of car owners in general.

Polls based on CR subscribers are representative of CR subscribers. Why would you assume otherwise? At the very least, CR subscribers are more concerned about "quality" than the rest of the population since they are going out of their way to obtain information about quality.

And, lest we forget, the classic example of sample bias was the Literary Digest debacle during the 1936 presidential election (Alf Landen was supposed to beat FDR by a landslide). That poll was based on LD readership who were generally better off financially than the general population during the Great Depression.

Here's another example of sample bias. Do a Google search with the words consumer reports sample bias. I got 190,000 hits when I did it - including this very page! Is that sample representative of internet users and sites? I don't think so.

Do I know that CR subscribers are not representative of the general population of car owners? It is fundamental to statistics that polling can be valid only for a population when every member of the population has the same likelihood of being polled. Nonsubscribers have zero chance of participating in a CR subscriber poll. Therefore, the results say absolutely nothing about the general population of nonsubscibing car owners. So, the answer is "The question is moot."

By the way, I am not singling out CR. Any organization doing similar polling is subject the same criticism and I mention this only because the current thread is focussed on CR. And, to be fair, CR does conduct certain national polls not restricted to their subscriber list. There's nothing wrong with using subscriber polls as a guide. Just be aware of the limitations.

We now return to our regularly scheduled program ...!

tidester, host
SUVs and Smart Shopper

backy · February 2009

Do I know that CR subscribers are not representative of the general population of car owners? It is fundamental to statistics that polling can be valid only for a population when every member of the population has the same likelihood of being polled. Nonsubscribers have zero chance of participating in a CR subscriber poll. Therefore, the results say absolutely nothing about the general population of nonsubscibing car owners. So, the answer is "The question is moot."

Actually, the answer is, "You don't know."

I wouldn't be surprised to learn that the CR subscriber base is not representative of the entire population of U.S. car owners in some way that significantly affects the CR survey results. But I have yet to see any facts on that, as on the 1936 poll where it was known how the survey sample differed in a material way from the general voting public.

If it is true that it's fundamental to statistics that polling can be valid only for a population when every member of the population has the same likelihood of being polled, then there are few if any valid polls/surveys that are conducted. For example, any survey conducted by phone is invalid, since not everyone has the same likelihood of being reached via phone. Certainly any survey conducted via computer is invalid. As is any survey conducted via mail (especially in these tough times with more and more people without permanent addresses).

I think there are ways to adjust for sample bias in polls/surveys, are there not? Do you know for a fact that CR does not employ those kind of measures?

tidester · February 2009

then there are few if any valid polls/surveys that are conducted

That is absolutely correct and it's why I take most poll reports with a very large grain of salt.

I think there are ways to adjust for sample bias in polls/surveys, are there not?

Yes, there are. Pollsters do that all the time in public opinion surveys. But it is only possible when they have at least some sampling from the sub-populations available to them possibly from historical data.

Do you know for a fact that CR does not employ those kind of measures?

It is a fact that subscriber surveys exclude nonsubscribers. CR identifies many of its polls as subscriber surveys. There is no way to adjust for that exclusion.

tidester, host
SUVs and Smart Shopper

backy · February 2009

Here is at least a partial answer to my question, with some additional information:

Any survey has some sort of sampling frame that limits the people being surveyed. We choose our subscribers as our sampling frame. On average, CR subscribers tend to be more educated and affluent than the general population. With the growth of Consumer Reports online, a wider demographic range of individuals has been surveyed in recent years. However, our reliability questions do not ask respondents about their attitudes or opinions about the reliability of their cars, where one might expect different groups of individuals to have different perspectives. Instead, we ask for factual information about whether specifically defined problems occurred; these types of questions are less sensitive to the nature of the characteristics of the sample itself.

Further, our results track well with other sources of reliability information available on the market, and auto manufacturers have not formally disputed our survey findings, which often correspond to problems that the manufacturers see in the warranty experiences of the population of car owners at large.

There's a lot of other details on how CR's survey is done, here:

http://www.consumerreports.org/cro/cars/new-cars/auto-test/consumer-reports-car-- reliability-faq-8-06/overview/0608_consumer-reports-carreliability-faq_ov.htm

imidazol97 · February 2009

CR's polling also does nothing to verify the car ownership of those reporting to their questionnaire. The difference in age groups respondency rate would make the information much less useful. A lot of people don't respond and some respond to anything. Also many people occasionally read CR (not for car buying, thank you) but do it at the library.

If CR were to use the list automobile registrations and do a phone solicitation of a number of each car brand dispersed statistically around the country, that would be more useful, assuming proper corrections for those refusing to talk to a poll caller were made. That's what JD Powers is doing essentially, but theirs is by mail.

CR's methods are essentially convenience polling. That would be equivalent to standing at the entrance to your Kroger store and asking people to respond to a written questionnaire on groceries and buying as they come by and they select to cooperate or not to cooperate.

tidester · February 2009

Thanks for digging that up, Backy. The last paragraph you quoted is a clear indication that CR's statisticians recognize the inherent shortcomings of subscriber based polling and their conclusions cannot be extended to the general population.

That other sources of reliability information tend to the same conclusions is not validation. It is inferential and basically says "We hope you don't casually dismiss our findings," which is fine. But just don't assume that their conclusions are valid over the nonsubscriber population based on coincidental agreement.

As I said before, use the information as a guide but be aware of those limitations.

tidester, host
SUVs and Smart Shopper

tidester · February 2009

Those are excellent points. I suppose we could extend the convenience polling right here by simply asking people whether they agree with the ratings rather than debating methodology!

tidester, host
SUVs and Smart Shopper

backy · February 2009

As you said... there are NO valid surveys.

But I don't think CR recognizes that "...their conclusions cannot be extended to the general population." I think they have laid out their case as to why the survey results are useful outside the survey population. If they really believed their conclusions cannot be extended to the general population, they wouldn't have been publishing the results to the general public all these years.

I think CR's point about no automaker formally challenging the results of these surveys is interesting, though. Some automakers have been trashed pretty good by CR over the years by posting low scores in the survey. With the litigious society we have today, I think the lack of a formal complaint is rather remarkable.

tidester · February 2009

But I don't think CR recognizes that ...

If they don't then their statisticians are guilty of malpractice! :P

I don't believe their statisticians are guilty of malpractice.The paragraph I cited is a very clear indication they are aware of the "problem." That you remain unconvinced indicates it's time to move on so we'll just leave it there.

tidester, host
SUVs and Smart Shopper

backy · February 2009

I recognize CR is aware of the limitations of surveying only their subscriber base. I think we disagree as to how big the problem is and whether it makes the survey useful at all.

If what you say is true, that the survey is not extensible to the general population, then CR's statisticians ARE guilty of malpractice--or their Board is guilty of misrepresentation. Take your pick.

bolivar · February 2009

I would think the surveys have 'accuracy levels' attached which will show the 'bounds' of accuracy that are covered.

In other words, a certain survey's results are accurate within plus or minus certain percentage points. This is standard survey procedure. Within these bounds, the survey is accurate.

And, at one time a telephone survey was the most accurate. Having a phone in the home was one of the most general, cross-population things there were. With the advent of cell phones, this is probably somewhat less true. But, I would still think a telephone survey, set up with statistical bounds, is still about the very best that can be done.

backy · February 2009

With a phone survey, good luck: A) finding someone at home during the day (or night for that matter--with more people working nights, and with families busy with after-school activities);

getting someone to answer the phone (e.g. I screen my calls via caller ID and don't answer those I don't recognize); C) getting people to sit on the phone for 15-20 minuntes to complete a survey that is of no direct benefit to them (that's about what it would take for a survey like CR's, especially for multiple vehicles in the household); D) accounting for the segment of the population without a published phone number.

It would provide a broader segment of the car owning population than just going to CR subscribers, but would still mis-represent the general population. There's problems no matter how you reach out to people in a survey.

jipster · February 2009

whether it makes the survey useful at all.

They seem to be fairly accurate in their reported problem areas, of vehicles I have owned or know well. i.e Honda Ody transmissions, GM intake manifolds, clunky/rattleing Nissan Quests, coil problems in the MPV . You see a persistent problem on any of the various vehicle boards here on Edmunds, it will usually show up in Consumer Reports as being a problem too. That shows to me their "survey" is doing its job as being a useful tool.

bolivar · February 2009

I didn't say it did not have problems. I said "I would still think a telephone survey, set up with statistical bounds, is still about the very best that can be done." And I still stand by this.

What is better. Standing on a street corner? Print survey in a newspaper/magazine? The bias in these two approaches is very high. A corner - how do you chose a corner to represent the general population. Print a survey - this can be manipulated so easily by people that have their own bias. The same can be said about 'online' surveys. These are, in my opinion, totally useless because they can be manipulated so easily.

Telephone and mailout surveys are probably the very best. They can be tailored to reach across the population to get as unbiased a response as possible. Of course, if someone wants to construct a highly biased survey, it can also be easily done.

But, thinking about it, with telephone numbers now able to follow people when they move and prefixes no longer being restricted to a geographic area, it makes a phone survey more 'general' and also more difficult if someone does want to survey a specific area, for a valid or biased reason.

Designing a valid survey is the most important thing that is done. The number crunching is rather straight-forward, and is just statistical analysis.

BambuListener · February 2009

CR: "For some trouble spots with very low problem rates, we do not assign a (black dot) or (half black dot) unless a model's problem rate exceeds 3 percent. If the problem rate is between 1 and 2 percent, it is assigned a (half red dot). If the problem rate is below 1 percent, it is assigned a (red dot)."

Backy: Now let's use this transmission problem example to illustrate what the statement above means and why your analysis above is incorrect. In order to get "Below Average" on transmission, it means that a little over 3% (minimum) of owners of that model year who responded to the survey reported a "significant" problem with the transmission.

No. The way I read CR's statement, is that this applies only to "trouble spots with very low problem rates". In other words, the above 3% is only for the very low problem rates trouble spots - read the CR statement again. There is no reason to assume that transmission problems represent an area of "very low problem rates" for the Elantra 2000. Therefore, your subsequent analysis is based on a faulty assumption.

It may very well be the case that the rate of failure is 99% over a given time span - I have no idea.

Getting back to use of language - if the following statement is true:

"Elantra 2000 transmissions experience a higher than average rate of failure", then my experiencing a tranny failure in an Elantra 2000 is a confirming instance of the statement. Of course it may happen that the average rate of failure is only something like 1% over a given period of time (i.e. very low) - but it may also happen that the Elantra 2000 exceeds the average by a significant margin (say, resulting in a 90% failure rate). For the record, the transmission problems in CR for Elantra 2000 show a full black circle for significant problems - and nothing said so far shows that this may not be a very large failure rate indeed.

Re: various biases wrt. CR surveys. Yes, there are no doubt all sorts of biases, as in any survey. The key point however, is: do these biases selectively impact one year vs another year. If the answer is no - then the biases don't matter. No matter how much or how little the sample is biased, there is no reason why they all should suddenly show skewed results in 2005 vs 2004. So unless someone can show a methodological bias which cause the 2005 users to be significantly more critical than 2004 users, the results still apply within the 2004-2005 cohort.

backy · February 2009

I think mail would provide the broadest coverage, of the typical survey methods. (BTW, that is one of the approaches CR uses.) It doesn't require that someone be home at the time the survey arrives, or has time to complete the survey based on the survey taker's schedule, or even be able to speak English (forms could be provided in multiple languages). It would avoid only people who do not have a permanent address. Since car owners need a permanent address of some sort to receive notices on license renewals etc., that should affect few people.

One method that would guarantee a response from nearly all car owners annually is to require that car owners fill out a reliability survey when they apply for their renewal tabs on their licenses. That has its own problems, of course, and it would never be implemented. Just a thought...

backy · February 2009

I have no idea.

That's right. Until you do have some idea of what that black dot represents, it's premature to base an assertion on it. And a single individual experiencing a problem with a car tells us nothing about the validity of CR's survey. (CR actually addresses that point in the FAQs I posted the link to earlier.)

Once I get the 2009 Annual Auto Issue and can see the data that only you and other subscribers can see, I will be in a better position to comment on the changes in the 2004-5 reliability data from the 2008 survey results.

imidazol97 · February 2009

Mail would be a definite improvement. CR however mails only to certain or most of its subscriber base (I was a subscriber but didn't get a survey one year).

The real solution is to use the state's automobile registration databases and mail to a randomly selected portion of those. Ideally would be a followup telephone call if there is no response. This is essentially what JD Powers does without the phone followup. Statisticians have methods for adjusting for the nonresponses.

This method adjusts for the idea that of the small percentage of the car-owning population who subscribe to CR certain cars may be over represented or under represented. Being able to mail to a certain number say 2000 of each model would allow for non responses and still give a good cross section of what those people experience with their 1999 Hupmobile.

I like your idea of requiring a survey as part of license renewal. I can just imagine the huge government program that would be needed to handle.

BambuListener · February 2009

Well, I do know one thing - it says that the quality of that component is definitely below average. That much is clear, if there's a solid black circle it's worse than average. You don't have the exact number without CR publishing it, so you can't say how bad the average is, or how much worse than the average the particular case is, but you DO know that it is worse than average. That, friends, is not a positive. That, friends, is a negative. So it's inappropriate to assume a "neutral" position "oh well, we don't know one way or another". The appropriate position is "that is NOT GOOD, that is BAD". How bad, you don't know - ranging from "slightly" to "catastrophic". And so, caution is advised, not neutrality. And when the bad happens (failure), you can say - "aha, there was a caution note from CR with regard to this". This, is indisputable.

backy · February 2009

Since we don't know what "average" means here, I don't know how we can say "worse than average" is a "BAD" thing. It is certainly worse than the average. But if the average is high, somewhat less than that may not be "BAD". CR itself says that a black dot does not necessarily mean a car is unreliable. It's all relative to what the "average" is. It's not a good idea to extrapolate or interpret based on an unknown number.

Analogy: suppose at the next Summer Olympics, in the finals of the high bar, the average score is 9.96. There are five contestants, with scores of 9.92, 9.94, 9.96, 9.98, and 10.00. In your way of thinking, 9.92 is a "BAD" score. It is actually a very good score for the event, and may in fact have been the best score that contestant ever achieved on the high bar. I don't think you'll hear any commentator there say, "Oh, what a BAD score!! That contestant should retire from competition! He/she is a disgrace to world gymnastics!" :sick:

BambuListener · February 2009

Backy: Since we don't know what "average" means here, I don't know how we can say "worse than average" is a "BAD" thing.

Well, by generally agreed standards down here on Earth, "worse than average" is ALWAYS bad - by definition. Not really a point one can dispute - other than by those who prefer worse than average products in any given class.

Backy: It is certainly worse than the average.

And in what world is that a GOOD thing?!

Backy: But if the average is high, somewhat less than that may not be "BAD".

No. Here's how it works definitionally: Better than average is better than average, and worse than average is worse than average. We can re-define terms, or preferences (in deference, for example to masochists), but otherwise, sorry, the terms are set - look it up in a dictionary. So, worse than average is never GOOD. On the contrary it is BAD. Now, you may still wish to purchase a worse than average product due to other considerations (f.ex. price), but you don't get to re-define the meaning of common terms. A worse than average product doesn't suddenly become re-defined as GOOD. Same as in your Olympics example. Worse than average performer is not suddenly going to be getting the gold medal, no matter how sympathetic he/she is or what other good qualities they may have. And no need to quit gymnastics or manufacturing cars - somebody has got to be the last and the worst. As a consumer you may wish to avoid them, but that's up to you - what is not up to you, is to redefine terms wherein the car/athlete suddenly vaults to the top of the rankings because "cars/athletes in general on average are so good".

So in that case, when we see the transmission on the car go pear shaped, and we check the CR ratings, and find that it has been flagged as a problem area - we are well justified in saying "aha! not a surprise", whereas if it was flagged as much better than average, we'd say "what a surprise!". See? Therefore, there is sense in checking CR ratings for individual cars - WHICH IS EXACTLY THE WHOLE POINT of Consumer Reports - to guide individual consumers! If it wasn't, and there was nothing one could tell from such rankings, why issue them - answer this question! What exactly is the point of CR rankings then?

tidester · February 2009

Better than average is better than average, and worse than average is worse than average.

How about adopting more neutral terms such as "Higher than average" vs. "Lower than average?" We'll never resolves matters of judgment as in "good" versus "bad."

tidester, host
SUVs and Smart Shopper

backy · February 2009

What exactly is the point of CR rankings then?

That's a good question. Here is how CR answers that question (from the 2008 April CR issue):

The 17 trouble spots pinpoint a model's strengths and weaknesses. Ratings are based on the percentage of survey respondents who reported problems for that trouble spot, compared with the average of all vehicles for that year. Each model year requires a minimum sample size of 100 responses to be included.
Models that score a (black dot) are not necessarily unreliable, but they had a higher problem rate than the average model. Similarly, models that score a (red dot) are not necessarily problem-free, but they had relatively few problems compared with other models.

They go on to talk about the fact that a car could be assigned a (black dot) if the problem rate is a little over 3 percent, while a (half red dot) would then mean 1-2 percent and a (red dot) less than 1 percent. So a car could have 1 more failure over a year compared with another car, and get a (black dot) while the other car gets a (half red dot). You would say the car with 3 failures per 100 is BAD, and the car with 2 failures per 100 is GOOD. If that's the way you want to think of it, great. To me, it's really not that big a difference, or deal.

Now, if I see a chart in the survey report with almost ALL black dots for several years, vs. another car that has all red dots for many years, then I would probably avoid the car with all the black dots in favor of the red-dots car unless I had a real good reason not to do that. But without more information, I would not use one dot for one model year as a decision guide on a car. Nor would I be surprised if a single car with a red dot on a trouble spot had a breakdown in that area, or a car with a black dot had no problem there. You can't make a valid correlation of a sample size of one to a larger sample, if you don't know the probability numbers behind the larger sample. That's my story and I'm sticking to it.

P.S. Maybe in some of your upper-division math classes in college, there were all really sharp students and the professor was nice enough not to grade on a curve, and the average score in the class was around 98 and the range 96-100, and everyone got an A. Which means "EXCELLENT". Which is the opposite of "BAD". I saw that a few times (relatively small school, small upper-division math classes). If you want to tell me the students who got a 96 got a "BAD" score, go ahead but you'll be wasting your fingers.

BambuListener · February 2009

I don't know about you, but personally, I prefer to pick a car that's better than average rather than worse than average. And to address your other point directly:

Backy: Now, if I see a chart in the survey report with almost ALL black dots for several years, vs. another car that has all red dots for many years, then I would probably avoid the car with all the black dots in favor of the red-dots car unless I had a real good reason not to do that. But without more information, I would not use one dot for one model year as a decision guide on a car.

It so happens, that according to CR, for Elantras in years: 1999 (earliest year listed), 2000, 2001, 2002, 2003 - there's not a spot of red when it comes to "Transmission Major" - all solid black or black circle. Only with 2004 does it go to half red. Certainly looks to me, like major transmission issues are a problem area of concern for pre-2004 Elantras. Subsequently, having a 2000 Elantra driven by the OP exhibit transmission problems does not surprise me in the least - having seen the CR surveys.

In general, you can get lucky, even with a "bad" reputation car. Certainly, that's just basic statistics. However, as a rule, I don't tell myself "well, Yugos show black dots in CR surveys, but hey, I don't know the precise number, lemme go ahead and buy this fine vehicle". A black dot, is not a positive, any way you slice it. Yes, you may get lucky. And yes, you may even win the lottery. But that STILL doesn't make the black dot a positive.

backy · February 2009

Do you realize that a black circle means "Average"? How that can be an indication of an area of concern for cars that are from 7-10 years old, I can't fathom.

Also, try to see the big picture here. For example, go take a look at some other cars of that same vintage. My CR mag doesn't go back 10 years like your subscriber-only data does, but I can see back to 2002. Here's a few examples for Transmission-Major:

2002 Honda Accord V6: Black Circle
2002 Honda Accord I4: Half-Black Circle
2003 Honda Accord V6: Black Dot

2002 Acura TL: Black Circle
2003 Acura TL: Black Circle

2002 Honda Odyssey: Black Dot
2003 Honda Odyssey: Black Circle

2003 Honda Civic Hybrid: Half-Black Circle
2004 Honda Civic Hybrid: Black Circle

So for some of the vehicles with the best reliability reputation on the planet--Honda and Acura--they have no better than and in some cases a worse score for 2002 and 2003 as the Elantra! And note that in the tables I'm looking at, from April 2008, the Elantra actually shows a BETTER score than those Hondas and Acuras for 2002 and 2003 on Transmission-Major: Black Circle and Half-Red Circle, respectively.

This whole thing on labelling data as "BAD" is your idea. It's not CR's--they don't use "BAD" and "GOOD" to describe their results. Because they know better. Remember, they designed the survey and calculate the results. They know what "Average" means, numerically, and they know what "Above Average" and "Below Average" mean.

BTW... stay away from those BAD Hondas and Acuras!

BambuListener · February 2009

Backy: Do you realize that a black circle means "Average"? How that can be an indication of an area of concern for cars that are from 7-10 years old, I can't fathom.

Yes, of course, I realize a black circle means average. It's just that in picking cars, I try not to pick cars of average reliability - but better than average. You perhaps prefer worse than average - and that's fine, there's room for everyone in the market.

But really, the second part of your statement is just bizarre. What does the age of the car have to do with the issue of average as an indication of concern? What do you mean by the statement above? Is there some kind of inflation going on, so that a car of "average" reliability suddenly transform into "fabulous" at the 7-10 year mark? If not, what is the point of linking "average" with "age"? Average is average - whether for a 1 year car or a 25 year car. The "average" is across a given year of cohorts, not in comparison from one year to another year.

Backy: [bunch of cites of Honda transmissions as showing black dots and circles in CR survey]... So for some of the vehicles with the best reliability reputation on the planet--Honda and Acura--they have no better than and in some cases a worse score for 2002 and 2003 as the Elantra! [...] BTW... stay away from those BAD Hondas and Acuras!

I love it! Sometimes when people are wrong in an argument, the more they argue, the deeper hole they dig for themselves. And you got very unlucky

Today during lunch, I had a chance to peruse the CR online car articles that were updated. What a delight of information - sadly punching huge holes in your thesis.

They did a big write up on reliability, several articles based on a ten year study of surveys. They had feedback from over 1.4 million users. Here's a sample of quotes from their conclusions, and how they address your various assertions:

"the average Hyundai of five years and older is among the least reliable."

Note, that this would include the 2000 Elantra. Incidentally, according to a graph on that page, Hyundais have some 130+ problems per 100 vehicles, with only Chrysler and VW doing worse. You are practically guaranteed a problem with an older Hyundai - in comparison, a Toyota would have about 55 problems per 100 vehicles, and Honda 75 per 100. So almost 3 times as many problems on Huyndais as on Toyotas and twice as many as on Hondas.

So, you probably think you've made great points about the Honda transmissions, given how reliable Hondas are. And this is where you got unlucky. Because there was a patch of time in older Hondas, where their transmissions were notoriously unreliable. That black dot you dismissed? That "bad Honda" you laughed about? Well, dear Backy, Honda made the dishonor list of CR WORST USED CARS - see the article "Ten Years of Trouble" where they find the worst by years, and Honda is shamed for major transmission problems for 2001 Acura CL. THE WORST. And I feel very right in calling that BAD!

ZING!

So now your defense of how we don't know the numbers, maybe very tiny, blah, blah. So, what are the numbers according to CR on that Honda Acura? How bad? How about 17% of reported problems? That's almost one in five! That's "BAD" by anyone's definition! Hope I used the word correctly here

ZING! ZING!

So then you claim that it's all subjective and maybe major is actually overly sensitive consumers misreporting. So how bad is "major transmission" problems? According to CR "Transmission rebuild or replacement." HA! Nothing subjective about replacing a transmission! That's "BAD" by anyone's definition! You don't begrudge me using that term here, do you?

ZING! ZING! ZING!

And so it goes. What is the lesson from this? It is that going by reputation only takes you so far. That you really cannot go by hearsay ("Hondas are good"), but have to look at the numbers. And that you'd better pay attention to those black dots on CR - contrary to what you seem to think, they are not a mark of excellence... even in a Honda! I don't even want to think about how bad the transmission could be on those 2000 Hyundais since older Hyundais are reported by CR to be "among the least reliable."

All these gymnastics, Backy, to justify why a black dot on 2000 Elantras are the bees knees? The plain truth is that older Hyundais (which include the 2000), have a low resale value for a reason. Now, nobody says cars can't improve - and newer Hyundais certainly seem to promise vastly better quality, but let's let the truth in reporting (such as CR surveys) prevail - no sense in trying to make a silk purse out of a sow's ear. A black dot on the 2000 Elantra major problem transmissions - is a caution, not a medal of excellence! In other words - let's not avoid this word - BAD!

backy · February 2009

All these gymnastics, Backy, to justify why a black dot on 2000 Elantras are the bees knees?

Uh, no. You seem to have lost the point of this thread in your gleeful attacks. So I'll remind you what it was. The point was that you asseted that it is valid to look at the behavior of one component (transmission) of a single vehicle and make a correlation to the behavior of a larger group of vehicles, when you had no idea what that group's behavior was other than it was below some average value. I asserted that was not correct, mathematically speaking. You also asserted that it is valid to label vehicles that score "below average" in CR's reliability survey "BAD" and those that score above the average "GOOD". Since CR itself avoids these terms, and in fact states that a car that scores "below average" may not be unreliable at all, I maintain that it's inaccurate to use those terms with respect to survey results when the average, and what is below and above the average, is an unknown.

You of course are entitled to your opinion, which you have made very clear. But the zingers really aren't necessary, are they?

I am curious about one thing, though. Didn't you say you own a 2004 Elantra? If so, I am surprised, based on your comments above.

BambuListener · March 2009

BackyThe point was that you asseted that it is valid to look at the behavior of one component (transmission) of a single vehicle and make a correlation to the behavior of a larger group of vehicles, when you had no idea what that group's behavior was other than it was below some average value. I asserted that was not correct, mathematically speaking.

Yes, that is exactly what I asserted. I asserted, that when you have just this piece of information "the transmission on this vehicle brand shows serious problems with worse than average frequency", then absent any other information (such as what the average represents etc.), it is absolutely valid to be wary of that vehicle. Since you bring up "mathematically speaking", there are mathematical reasons to draw the exact conclusion that I did. There is a branch of math called game theory. For a simple wikipedia description, see;

http://en.wikipedia.org/wiki/Game_theory

Such mathematical principles, btw. are used in decision making algorithms for systems where there is varying degrees of completeness of information.

So, when you are faced with a situation where you have incomplete information, you cannot make a decision with 100% guarantee of outcome. That does not mean, that therefore there are no rational decisions to be made on incomplete information. This is where you are confused. You reason like this: because we don't have the number that represents what "average" is, nor "how much below average the failure rate is" therefore one cannot make a decision as to whether it is rational to avoid a given vehicle. Here's where you are wrong. What is true is the following: we dont' have the number that represents "average", nor "how much below average" the failure is - however, be precise as to what exactly that does not allow us to mathematically compute - the EXACT odds of failure. That is correct. That does NOT mean that you cannot still make rational decisions based on this lack of information, or only having partial information, based on game theory. The mistake you make is in assuming that since we lack an exact number for odds of failure, the odds of a negative outcome are just as good as the odds of a positive outcome - and that is MATHEMATICALLY WRONG. To give you an intuitive insight into this mathematical reality, think of it this way: There are stack of dollar bills out there. you get two closed boxes (red and blue) each with an a stack of bills inside. The information you have is the following: the red boxes, you only know that generally they tend to have stacks of bills which are lower in sum, but you don't know what the average is, or how much below average those stacks are. In the blue box, you know only that the blue boxes tend to have stacks of bills of higher than average amount of money. Which box would you choose? Once it's put in these terms you understand: IN ABSENCE OF OTHER INFORMATION, it is rational to choose the blue box, even though it is also possible that had you chosen the red box that particular box might actually have more bills than the blue box. Your argument is that since we don't know how much the other variables are, therefore the mere fact that one is below some unknown average, (and by an unknown number), tells us absolutely nothing about which is the rational choice here. That is WRONG MATHEMATICALLY (you can do a logic tree that will actually show this graphically).

Backy: You also asserted that it is valid to label vehicles that score "below average" in CR's reliability survey "BAD" and those that score above the average "GOOD". Since CR itself avoids these terms, and in fact states that a car that scores "below average" may not be unreliable at all, I maintain that it's inaccurate to use those terms with respect to survey results when the average, and what is below and above the average, is an unknown.

See above. And also: that CR avoids terms "BAD" and "GOOD" is only to point out that they cannot give you exact odds of failure, because sometimes the average is very good, or the "below" average is very small. Indeed - and that is simply a shorthand for them, rather than go into lengthy mathematical explanations; however, for the consumer, the takeaway is still this - it does not mean that rational decisions cannot be made - and the rational decision is TO AVOID vehicles that fall below average in the absence of any other information.

Re: my owning a 2004 Elantra. Funny you should ask. And funny you should think that based on my comments, you'd think I'd avoid it. The exact opposite is true. Based on my remarks, you'd expect me to buy a vehicle that has an overall score of "above average" - which the Elantra 2004 does. And when you look at the particulars, for the subsystems, the ones that matter most to me have the better scores (engine and transmission), while where it scores lower, it matters less to me (paint/trim). So, now factoring in money, I think that it is a good buy. Not the Elantra 2003, or earlier though.

backy · March 2009

OK, now you are putting words on my fingers.

You reason like this: because we don't have the number that represents what "average" is, nor "how much below average the failure rate is" therefore one cannot make a decision as to whether it is rational to avoid a given vehicle.

That is not what I "reason" at all. I said nothing about what rational decision someone might make about a car purchase based on survey results. There are many factors that might enter into that decision. And often decisions are made with incomplete data. But that's not what this was about--not until you changed the subject in your last post, anyway. It was about the incorrect application of statistics, and making correlations (not decisions, but correlations, two different things). Since you either don't understand what I am talking about there or choose to ignore what I'm talking about, I'll simply stop talking/typing about it.

You are forgetting about one very important aspect of decision-making, though: significance. Suppose I am choosing between two cars (let's try to keep this discussion at least somewhat related to cars, OK?) and they each have Characteristic X. Suppose there is no doubt that Car A is superior in this characteristic compared to Car B. We would say that, in the absence of other information, I would choose Car A, right? But what if Car A's advantage in Characteristic X were insignificant? For example, suppose Characteristic X is interior volume. Car A has 95 cubic feet. Car B has 94.9 cubic feet. In the big picture, who cares? Or what if Characteristic X is "availability of a maroon interior". Car A is available with a maroon interior. Car B is not. Car A is superior to Car B in that regard. Yet I don't even LIKE maroon interiors, so that advantage has no significance to me.

Significance is one reason why it's important to know what Average, Above Average, and Below Average really mean before making assumptions about them ("GOOD" vs. "BAD" for example), or comparing single examples to them... or making decisions based on them.

And I would hope that anyone who looks at CR's reliability survey, or ANY reliability survey, doesn't make a decision on a car based only on that data, i.e. in the absence of any other information.

The reason I was wondering why you would own a five-year-old Hyundai is because of some comments you made a few posts back, such as:

It's just that in picking cars, I try not to pick cars of average reliability - but better than average.

and, quoting a CR report you found online (bolding was yours):

"the average Hyundai of five years and older is among the least reliable."

So why would someone like you who puts store into CR's surveys want to own a five-year-old Hyundai? Especially a model that CR's survey results say is prone to decline in reliability beyond year 5? Would it be a rational decision to keep it, knowing what you know about CR's survey data on Hyundais and the Elantra in particular?

tidester · March 2009

Game theory? Hmm.

So, what if Honda customers are five times as likely to complain about problems with their cars than Hyundai customers and Hyundai ends up with above average ratings - even though (hypothetically) Hondas are, in fact, more reliable? You go for "above average" and you lose.

The variations are virtually endless and a resolution resorting to technical points of game theory or statistics is unlikely. I think we all win if we just move on.

tidester, host
SUVs and Smart Shopper

PatMatHat · March 2009

Backy: That is not what I "reason" at all. I said nothing about what rational decision someone might make about a car purchase based on survey results.

But that's exactly what the original argument was over - how to make a rational decision about a specific car (Elantra 2000). The subsequent rabbit hole of statistical discussion was your attempt to justify a claim that CR (and JD Powers) survey results should be dismissed as a consideration. Fact is, that what interested the OP was addressed best by how to make a rational decision - and that included in my opinion, survey results from respected organizations devoted to addressing just this kind of need. Surprisingly, you objected to this rather commonsensical and indeed orthodox recommendation; consequently you were reduced to arguing increasingly tenuous and unsupportable arguments based on an incomplete understanding of statistics and decision-making algorithms.

Backy : You are forgetting about one very important aspect of decision-making, though: significance. [example of interior volume and color as being insignificant].

I most certainly am not forgetting that. Quite the opposite. I cited for the OP the very significant areas of concern where the 2000 Elantra had the worst ratings: transmission, drive system, body hardware or below average (engine, minor). I don't know about you, but transmission and drive system are about as major as you get. Which consideration was also a factor for me in picking the 2004 Elantra.

You are puzzled about how I could pick an Elantra 2004 given that CR pronounced older Hyundais as among the least reliable? Easy peasy. I look to individual ratings for particular models, particular years and the significance of areas of concern (see significance?). Perhaps the majority of old Hyundais are unreliable - but I look for exceptions. Indeed, older Elantras (including the 2000, don't rise above average), but the 2004 DOES. Bingo. The overall score for the 2004 Elantra is "above average". Further, the significant areas: transmission, engine, drive system, fuel, electrical, exhaust (I live in California) - were all above average. Only in body paint trim and audio did they do worse than average, which are not areas of significance to me. So I picked a car that was ABOVE AVERAGE. Unlike the 2000 Elantra. Lesson - look at a specific car... that way you don't commit boners like plunking for a Honda, only to discover that a particular model has catastrophic problems with transmission (ha!)... I don't have that problem - I take a careful look at all the ratings. So that's how I came to pick the Elantra 2004.

As to the post by the other person about Honda owners possibly being 5 times more likely to complain than Hyundai owners... why would that ever be the case? If we were to accept such peculiar concerns as reality, then I guess we could NEVER make any comparisons between brand reliability (hey, Yugo owners happen to be super critical, OK). No way to compare a GM product to a Toyota product. Or any other brand. Neat trick - and would completely obliterate the need for any survey or the existence of CR and JD Powers. Somehow, that's not how reality works. There are excellent reasons why reliability surveys work. Anyhow, I can take it or leave it as to whether it makes sense to pursue this further - but I thought the whole point of this forum "Consumer Reports/JD Power Rankings" was to examine just such issues... but what do I know. Either way, fine with me.

tidester · March 2009

Please stick to using only one user ID - that's required by our Terms of Use.

tidester, host
SUVs and Smart Shopper

backy · March 2009

The subsequent rabbit hole of statistical discussion was your attempt to justify a claim that CR (and JD Powers) survey results should be dismissed as a consideration.

Please give us the # of my post in which I said that--plus the direct quote from that post would be handy also. Thanks.

As you seem to like mis-stating other people's posts, I am done with this particular thread.

In closing, I will say you did have one advantage over me when it came to buying your 2004 Elantra. You were able to buy it used, with a known, long-term relilability record as published by CR. I bought mine new 5 years ago. The long-term reliability of the Gen 3 Elantra was not established at that time. But I had excellent ownership experience with my 2001 Elantra (which when purchased in the fall of 2000 was really a leap of faith), which is the same geenral design as the 2004, and there were indications from sources like CR and JD Power that Hyundai's reliability was improving, especially for their post-2000 models. So I decided to go with the 2004. It's been excellent in reliability also. Even the paint and trim.

PatMatHat · March 2009

My apologies. When I first tried to register at home, I had some problem signing on, and then at work, registered again (with a different ID). As a result, when I'm at work, my browser automatically signs me in with one ID while at home I'm signed in with the other ID. No nefarious intent at all

Probably best to try to sign on with one, but I'll have to somehow sync my home computer with my work computer. Anyhow, you are welcome to delete one of the ID's (don't care which). I'm the same person, with the 2004 Elantra etc.

PatMatHat · March 2009

Backy, I don't want this to deteriorate into a personal battle of no interest to anyone else. I don't think that would be fair to the readers of this site. Let's try to keep to the topic and totally avoid personal attacks ("like mis-stating other people's posts"). I'll quit the thread, but I won't engage in flame wars and personal attacks, sorry.

You certainly dismissed JD Powers in post 3120 "Anyway, if you look closely at the JD Power reliability scores that you posted the link to earlier, you will see how suspect they are." or all surveys in 3126: It seems you are putting a lot of faith into two surveys in which you have no idea what methodology is used to obtain the results. I pointed out, that you have no valid grounds for doubting their methodology (which was a whole long argument over several posts) - and that the burden of proof is on you, since these are experienced organizations who have been in this business for a long time.

We can argue over this till the cows come home. My point is very simple - in the position most consumers find themselves (limited information), surveys such as CR and JD Powers are a valuable factor for rational decisions, even if they don't (and can't have) 100% predictive powers. As such, I was totally justified in telling the OP that he should be cautious about the transmission in the Elantra 2000 (indeed based on many other problem areas, he should avoid the vehicle). I felt that I was taken to task by you for specious reasons - after all, I at least don't rely on anecdotes but on the best evidence we have access to - the surveys. I feel that partisanship should not compel one to defend car brands in face of objective data - it is not grounds for rational decision.

And kudos on your choice of the 2004 Elantra - quality trends are going up for Hyundais. Not consistently, but enough so it is a rational decision to go for the brand all other things being equal.

mickeyrom · March 2009

Why in the world would Honda drivers be more likely to complain than Hyundai owners? I have had a Hyundai and currently have a KIA and I sure you that I take problems in my Optima (virtually none to date) very seriously.A new car is a new car.Either it's reliable or it's not.The only criticism about the Hunkias that I see in the "expert" reviews which I agree with is the undercarriage noise.Everything else is as good as anyone could wish for.

Comments: Consumer Reports/JD Power Rankings

Comments

Helpful Tools

Most popular vehicles