Polls Must Be Viewed With Extreme Caution

More than 100 years ago, writer H.G. Wells said: “Statistical thinking will one day be as necessary for efficient citizenship as the ability to read and write.”

Polling of public opinion has become an industry of its own, one of our more thriving growth industries.

Today we ask people what they think about the 2020 presidential election including questions related to their views on Democratic Socialism... illegal immigration... ever-growing budget deficits and its relationship to increasing taxes and inflation... Medicare For All... systemic racism... levying a wealth tax on millionaires and billionaires... defunding the police... reliability of media reporting... ever-increasing protests and riots, and dozens of other subjects receiving 24/7 news coverage.

The Point?

We're bombarded almost daily with the result of polls. But─and this is a big "but"─in the public's “opinion” on almost any issue will be a function of three factors, namely, the questions asked, the responses and the analysis.

Many books and articles have been devoted to the subject of misuses and abuses of polling and sample surveys. But in today's fast-paced media world, the science of correctly collecting, summarizing, analyzing and using data seems bothersome. Ignorance is bliss!

Lawmakers and media pundits seem more interested in the results of polls, not in how they were obtained. Until recently, it didn't occur to them that the results of polls and how they're designed, implemented and analyzed are inseparable.

A Neglected Truth About Opinion Polls

Neil Postman, in a marvelous book entitled Technopoly, delightfully exposed an inherent weakness in interpreting survey results:

How to Guarantee Wrong Conclusions From Polls or Surveys

Let's get right to it. Classically trained statisticians call it the statistical universe or population.

What is a statistical universe? The statistical universe consists of all things about which conclusions are to be drawn.

For example, in a study of price fluctuations in the price of heating oil in New York City from 2016 to 2019, the statistical universe would include every price change which had occurred during the specified time interval.

If the scope of the study were expanded to cover a larger territory or a longer period of time, the statistical universe is correspondingly enlarged.

Obviously the term statistical universe is an elastic one and varies in its precise connotation every statistical undertaking.

Take-home message: In general, the statistical universe may be defined as a totality embracing every item which might've been brought under observation had a complete enumeration been effected.

The Need for Sampling

Lack of time and money render it impossible to make a complete survey of most statistical universes. Thankfully, it's not necessary to survey the entire statistical universe.

Why? Because hard-working, brilliant statisticians discovered how to get the same information from carefully selected, relatively small samples.

Making the correct inferences on small size samples taken from large or sometimes infinite statistical universes is the subject matter of basic and advanced statistics.

If the sampling process is properly carried out, an analysis of the samples makes it possible to infer information about the statistical universe within the limits of unavoidable chance errors of sampling–the so-called margin of error.

Non-Sampling Errors: A Major Reason for Flawed Polling Results

Unfortunately, many courses in basic statistics fail to emphasize one critical point, namely, if the statistical universe is improperly defined, the powerful techniques of inferential statistics (making inferences on the basis of samples) are of little or no value.

The term for making inferences from an improperly defined statistical universe is called non-sampling error.

Statistical techniques designed to measure sampling error are valueless if the group conducting the poll/survey has committed the biggest statistical error of them all─non-sampling error.

In a nutshell, non-sampling error occurs when the group sampled is improperly defined, that is, samples are drawn from what is called an improperly or poorly defined statistical universe.

The Classic Example of a Poorly Defined Statistical Universe

Undoubtedly, the most widely publicized illustration of a poorly defined statistical universe is the one concerning the Literary Digest's error in predicting the winner of the presidential election of 1936. (Indeed, this was the example most often cited by W. Edwards Deming–a statistical sampling guru and quality management pioneer).

During the 1936 election campaign between Democrat Franklin D. Roosevelt and Republican Alfred M. Landon, The Literary Digest magazine sent mock ballots to a large list of people whose names appeared in telephone directories and automobile registration records. (Their lists also included their own magazine subscribers, country club members and the like).

Over 10 million mock ballots were sent out; 2.4 million ballots were returned by respondents; 7.6 million were not returned.

On the basis of 2.4 million returned ballots, The Digest predicted Landon would win by a comfortable margin–indeed, a landslide.

As it turned out, however, Roosevelt received 61% of the votes cast, a proportion representing one of the largest majorities in American presidential history.

How Could They Be So Wrong?

Polls only represent the people who are in a statistical universe and who respond to them. Despite the sample's huge size, this election became a textbook case of a biased sample: all the sample's component groups were heavily Republican.

Let's get more specific. There were two important reasons for the erroneous prediction–namely:(1) an incorrectly defined statistical universe and;(2) non-response bias.

Everyone with telephones and automobiles in 1936 were in a higher economic group than those people without these two " luxuries.” There was a bias inherent in the statistical universe.

A large percentage of the voting population would not show up in telephone directories, automobile registrations and club memberships.

The statistical universe was improperly defined─it tended to be biased in favor of higher income groups. Higher income groups tended to be Republican.

In the 1936 election there was a strong relationship between income and party preference. Lower income groups tended to be Democratic.

Bias in the Sample Selection Process

Classically trained statisticians define a bias as a persistent error in one direction. What does this mean?

No matter who you sampled from The Literary Digest's statistical universe, there was a high probability a relatively affluent person would be selected.

To repeat: the statistical universe selected was slanted towards middle and upper-income voters and excluded most lower-income voters. And, in reality, there were a great many low income voters in 1936.

Nine million people were unemployed in 1936.

Take-home message: With regard to economic status, The Literary Digest poll was far from being a representative cross-section of the population. Then, as now, voters are generally known to vote with their pocketbooks.

It should be mentioned–indeed, emphasized–that George Gallup was able to predict a victory for Roosevelt using a much smaller sample of about 50,000 people.

Gallup's statistical universe consisted of a representative cross-section of the population.

The Literary Digest poll sample size was 2.4 million people. This illustrates that a poorly defined statistical universe cannot be compensated for by increasing the size of the sample, which in fact just compounds the mistakes.

Non-Response Bias

The second problem with The Literary Digest poll was that out of the 10 million people whose names were on the original mailing list, only 2.4 million responded to the survey.

It was then a fact that individuals of higher educational and higher economic status were more likely to respond to mail questionnaires than those of lower economic and educational status.

Therefore, the non-response group–7.6 million people–contained a high percentage of the lower economic status group. The 2.4 million people who responded to the questionnaire tended to be from a higher educational and economic status group.

A case study involving the Roosevelt/Landon poll from the University of Pennsylvania's Wharton School describes the situation as follows:

“When the response rate is low (as it was in this case, 24%), a survey is said to suffer from non-response bias. This is a special type of selection bias where reluctant and non-responsive people are excluded from the sample.”

A Quick Look at the Erroneous Predictions of the 2016 Presidential Election

The 2016 presidential polls was a startling upset. Yet several savvy practicing statisticians/polling organizations called attention to the fact in an era in which few people answer landline calls, pollsters would have difficulty in reaching selected target audiences. In short, non-response error could severely bias polling results.

Indeed, several major polling organizations predicted total response ratios could be in 9 to 10% range–probably closer to 9% or less.

That could make them increasingly likely to under-represent key voting blocs. Non-response bias was a key factor in why many of the polls turned out to be incorrect.

Successfully Predicting the Upcoming Presidential Election Is No Easy Task

Do you answer your cell phone or landline phones if they are not from people on your contact list or from numbers you don't recognize? Probably not. That's the legacy of all the spam calling we're bombarded with. Simply put, this leaves plenty of room for non-sampling/non-response errors to creep in.

Additionally, who does not believe that most people–because of political correctness–if contacted won't say what they really think or perhaps even lie? This was a problem in the 2016 election and is even more likely to occur in the 2020 election.

Our point? The results of presidential polls must be viewed with extreme caution.

Summary and Conclusions

It's a major error to take a sample from a statistical universe which differs considerably from the “ideal” statistical universe about which you want to draw valid conclusions.

Considerable time must be spent in defining the ideal statistical universe and every attempt must be made to draw a representative sample from that universe.

Always be on the lookout for a poorly defined statistical universe. Fancy calculations relating to sampling errors, interval estimates and highfalutin statistical tests–including mathematical polling models–are meaningless if the statistical universe from which conclusions are drawn is incorrect.

Finally, beware of non-response bias. Said the Wharton School case study: “People who respond to surveys are different from people who don't, not only in the obvious way (their attitude toward surveys) but also in more subtle and significant ways.”

Part II of this article continues the discussion of the colossal blunders that can be made by improperly defining the statistical universe.