Warren Buffett: 3 steps to pass the test

The results of a Financial Times/Harris poll resulted in a great headline: Europeans favour charity over tax for the super-rich. But can this attention-grabbing statement be trusted?

Another example that made me wonder was a Gallup poll containing at least one statement that confused me: You or another household member was the victim of computer/Internet crime, such as fraud or computer hacking, while using your home computer.

Whether respondents inferred that getting spyware, viruses or other malware on their PCs means being hacked is unclear (read more at Stay safe and share the moment).

Article source: Warren Buffett: 3 steps to pass the test

We focus on 3 steps to take before acting on research findings.

1. Is the information in context?

To illustrate, the Financial Times/Harris poll suggests that the public in the UK, France, Italy, Spain and Germany believes that charity is “…the right way to go about the super-rich giving back.” However, this does not necessarily mean the public wants to see

a) tax breaks for the super-rich, or
b) abolition of the inheritance tax.

In Germany or France, the context might be a direct income tax rate of 60 percent or higher, as well as a hefty inheritance tax. Therefore, respondents in those areas may be feeling charitable and suggest that the super-rich should not be taxed higher than they already, and instead give more to charity, as reported by the pollster.

Tip 1: Try to understand the context in which data are collected (e.g., data taken from Fortune 500 companies or micro-firms?), and check whether the study’s funding source(s) and possible conflicts of interest are disclosed – (see FTC regulations), since that might help explain certain interpretations.

2. Is your source trustworthy?

Always remember that ignorance is NOT bliss. Another recent study stated,

“… [we] analyzed more than a billion tweets generated by over 20 million Twitter users in 2010. This included bio, web site address and location information (both textual and lat / long) disclosed in their profile.”

Turns out, only English tweets were analyzed and which Twitter accounts were selected, and how, was not explained. Nor was I able to find information about the coding system used to categorize them.

Pollsters like Harris Interactive often point out procedural or findings weaknesses in the methodology section:

“All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with non-response, error associated with question wording and response options, and post-survey weighting and adjustments.”

Unfortunately, making attention-grabbing headlines and sweeping inferences based on such data is dangerous.

Tip 2: Check the methodology section on how data was collected (e.g., how were subjects selected) and what analysis was done (e.g., sentiment analysis is still looking for a way to obtain valid data).

Why not indulge for 2011 and enter your email in the field below, so you can get the next post about social media trends and metrics first. You’ll be glad you did!

3. What do the results mean for you?

If we look at context and how data are collected, it may no longer be so obvious whether we can trust the findings. At the very least, we may question how applicable it really is for our needs.

For instance, US internet users receive more than 6,000 display ads in a quarter. The Gallup press release states,

“Internet users send a mixed message about how much these ads are influencing them. On the one hand, more than 6 in 10 say they have noticed that some ads are targeted specifically to them based on websites they have previously visited. On the other, 9 in 10 say they pay little or no attention to online ads. These assessments vary little by age and income.”
(See U.S. Internet Users Ready to Limit Online Tracking for Ads).

Since I cannot afford billboards or TV ads, will spending money on Google Ads make a difference to my bottom line?

More information about research and social media: Customer relationship management and fair disclosure

Tip 3: Completing a quick cost-benefit analysis while reading a study’s findings will help you decide whether choosing the strategy recommended by the report is within your budget.

Bottom line

Studies and surveys are done all the time, but it is up to you to ask the tough questions before you report the results or – heaven forbid – act on them.

If things do not make sense to you after carefully reading a report, do not trust it. And taking the word of a journalist without getting your hands on the original data is outright irresponsible, if not stupid.

Take a page from Warren Buffett’s book and use a short checklist to evaluate research:

ComMetrics Intel – Raise your hand if you hate research

More from Warren Buffett:

Warren Buffett says social media can work for you
If it is good enough for Warren Buffett, it’s good enough for you!
Warren Buffett: What works with social media

If you like this post, please share it with your friends. How about asking them to comment after they have read this post? I love to hear what people think!

Are you with me on these research methodology matters and trust issues, or have I lost my marbles? Will people continue to base important decisions on research reports that may not be worthy of their trust?

And since this is our last post before the new year, we would like to wish you and your family a safe and happy holiday!

Tagged as: business ethics, corporate governance, research methodology, risk management, sentiment analysis, social media best practice, social media ROI, tools for sentiment analysis fail validity test, validity of research findings