What We Want and What We Get: Error Bias in Investing

As we observe events in realms such as financial markets, politics, or weather, we tend to form beliefs — be they explicit or implicit beliefs — about cause and effect, or whether the events were positive or negative, good or bad. Science has formalized this process: testing a hypothesis with empirical data. One of the tradeoffs when evaluating beliefs in light of evidence, or a hypothesis in light of data, is the type of error we would prefer if our beliefs turn out to be wrong. In the investment realm, this bias can affect our beliefs and behaviors, such as our tolerance for risk and our allocation choices. Several examples will help illustrate this point.

Innocent until proven guilty

In criminal law, we have the presumption that the accused are “innocent until proven guilty.” The presumption of innocence (our “belief”) implies that we would prefer to allow some of the guilty to escape conviction, in error, as opposed to presuming guilt and ensnaring some of the innocent along the way. We could call this presumption of innocence a bias toward false negative errors — those who are guilty yet not convicted.

Better safe than sorry

During an October weekend last year, western Washington state, where Saturna Capital’s offices are located, bolted down for the forecast of an unusually powerful fall windstorm. Several factors conflated to make the forecast particularly dire: the storm involved the remnants of a violent typhoon that was swept up in the fast-moving jet stream east of Japan and rapidly regaining strength as a mid-latitude cyclone. Early models suggested the storm’s minimum pressure — a gauge of its intensity — could be as low or lower than the infamous October 1962 Columbus Day storm, which was also the progeny of a typhoon. Finally, the local news media played its role in drawing considerable attention to the approaching storm. In the end, the actual storm’s peak intensity was lower than expected, and it rapidly swept through a narrow area away from population centers, minimizing the impact. In short, the dire forecast and the considerable attention it was given could be characterized as a false positive error.

The forecast for such a severe storm, drawn from the meteorological models, was perhaps not discounted appropriately considering the actual rarity of such powerful storms in the historical record. Meanwhile the news media appear biased toward giving attention to potential newsmaking events, rather than ignoring or minimizing them. But still, the biases that may have led to this false positive probably serve as a public good. It is almost certainly better to have children cooped up indoors when a violent storm is suspected of approaching, but never shows up, than to have those children playing outdoors amid violent gusts, falling branches, and crashing trees because meteorologists “appropriately discounted” the likelihood of their forecast based on the historical record. It is in this sense that we prefer false positives in forecasts for severe weather: better safe than sorry.

In the cases of criminal law favoring false negative errors, or severe weather forecasts favoring false positive errors, these biases are consistent with a shared moral and cultural framework. In other situations, however, the prevalent error bias may be morally neutral or even objectionable. One example that has come to attention recently is the so-called “replication crisis” in psychology.

False-Positive Psychology

We should certainly expect social science researchers to have a natural bias for their experiments to yield novel results worthy of publication, attract popular attention, lead to mass market book deals, and develop into a long-lasting project with many paths and collaborators. At the same time, we expect highly trained research scientists to incorporate safeguards that insulate experimental results from the influence of these biases, so that reportedly significant results truly are significant. It appears, however, until recently at least, that standard experimental design frameworks have allowed researchers substantial latitude to run experiments in ways that permit considerable biases to creep in.

In 2011, Simmons et al. published an article called False-Positive Psychology¹ in the journal for the Association of Psychological Science. Five years later, they were asked to write a follow-up article to what has since become one of the most cited articles in the journal’s history. In their follow-up, which is hot off the press, they write, “In 2010 or thereabouts, we stopped believing that many published findings were true.”² Looking deep into methodology and experimental design, their original article coined the phrase “researcher degrees of freedom.” The phrase basically refers to the many ways in which psychology researchers had flexibility to design and refine their experiments, choose among variables, test results from various sample sizes, and other considerations. But the researchers frequently did not discuss all the iterations of the experiment’s design and would instead focus solely on the version of the experiment that produced significant results.

False-Positive Psychology went as far as to test an obviously false hypothesis that the ages of participants in a study would decrease as a result of playing them certain songs! By selectively disclosing details about the data included in the final experiment and excluding other details about decisions made to get there, the authors “proved” their hypothesis. Of course, the listeners left the experiment slightly older than when they started it, but the authors demonstrated how standard experimental design decisions could lead to absurd (and obviously false) but “statistically significant” results.

As researchers in the field thought critically about False-Positive Psychology they began planning and executing a series of replications of widely accepted psychology studies. Unfortunately, the replications have generally confirmed the suspicions of Simmons and co-authors, as well as the broader research community, about the prevalence of false positive results in social psychology.

Error bias in markets and investing

The collective investment industry makes numerous claims about performance, causes of market gyrations, reasons why stocks went up or down, and predictions about future performance. The biases involved are numerous and entrenched; and while there are some safeguards to prevent obviously false claims from biased commercial sources, it is all too easy to make false claims based on data selected and massaged in ways similar to the questionable practices uncovered in social psychology.

We can further categorize investing biases into types likely to produce false positives or false negatives. False positive biases have in the past arguably pervaded the realm of investments and society as a whole to a much larger degree than false negatives. Many of these biases have been well documented, from the skew of buy-to-sell ratings among Wall Street analysts, to the long-enduring success of the value factor for stock returns. The value factor may result from investors optimistically bidding-up the share prices of firms with strong growth trajectories and momentum, while pessimistically selling firms with poor outlooks. But not all firms with rosy outlooks will meet their high expectations (false positives), while even those that do meet expectations may have traded at such a premium that many investors do not recover an adequate return. Alternatively, not all the firms that investors pessimistically sold will fail, and indeed some may recover due to cyclical factors, a change in management, or simply a recovery from bad luck.

Passive investing: a pendulum swing from false positive to false negative bias?

A bias toward false negatives seems evident in the more recent and ongoing cascade of investment dollars into passive, low-cost index products in lieu of actively managed investments. In fact, it may be that a reckoning with the historical prevalence of false positive biases is one of the drivers behind this trend. The presumption here is that a passive, low-cost index product will outperform an actively managed (and higher fee) alternative on average. It is difficult to prove above-average active results were derived from skill as opposed to luck, and investors who choose active products must also bear the up-front and ongoing costs of selecting and monitoring the active managers. Therefore, for passive investors, the consequence of failing to find the truly skilled investment managers among those presumed mediocre and missing the opportunity for above-average returns is seen as an acceptable risk, just as is failing to find the guilty among those presumed innocent.

In the US in particular, it was not the willingness to accept mediocrity that propelled the risk-taking settlers, and millions of immigrants who followed, to our distant shores. The bias toward American exceptionalism has fueled a vibrant economy where the enormous wealth created by those whose risky bets have paid off has far more than offset the many bets that have failed. Passive investing and an increasing bias toward false negative rather than false positive outcomes run contrary to these animal instincts. It remains an open question whether the flow of investment dollars into the “safe bet” will limit economic vibrancy and growth even as passive investors do relatively better than their active peers, on average.

As an investor, it may be helpful to take stock of your own investing beliefs and behaviors. Do you tend toward the “innocent until proven guilty approach,” a false negative bias that assumes you can’t accurately determine the winners from the losers? Do you tend toward the “better safe than sorry” approach, a false positive bias that might lead you to “predict a typhoon” of exaggerated potential losses and view a potential market downturn as a Category 5 hurricane on the horizon? Do you selectively emphasize the importance of information that supports your investment hypothesis? Biases are ingrained and difficult to overcome, but awareness is the first step.

Footnotes

¹ Simmons, J.P., Nelson, L.D., and Simonsohn, U. False-Positive Psychology: Undisclosed Flexibility in Data Collection and Analysis Allows Presenting Anything as Significant, Psychological Science, Vol 22, Issue 11, 2011. http://journals.sagepub.com/doi/pdf/10.1177/0956797611417632

² Simmons, J.P., Nelson, L.D., and Simonsohn, U. False-Positive Citations, Perspectives on Psychological Science, Forthcoming, April 2017. https://papers.ssrn.com/sol3/Papers.cfm?abstract_id=2916240

Important Disclaimers and Disclosures

This material is for general information only and is not a research report or commentary on any investment products offered by Saturna Capital. This material should not be construed as an offer to sell or the solicitation of an offer to buy any security in any jurisdiction where such an offer or solicitation would be illegal. To the extent that it includes references to securities, those references do not constitute a recommendation to buy, sell, or hold such security, and the information may not be current. Accounts managed by Saturna Capital may or may not hold the securities discussed in this material.

We do not provide tax, accounting, or legal advice to our clients, and all investors are advised to consult with their tax, accounting, or legal advisers regarding any potential investment. Investors should not assume that investments in the securities and/or sectors described were or will be profitable. This document is prepared based on information Saturna Capital deems reliable; however, Saturna Capital does not warrant the accuracy or completeness of the information. Investors should consult with a financial advisor prior to making an investment decision. The views and information discussed in this commentary are at a specific point in time, are subject to change, and may not reflect the views of the firm as a whole.

All material presented in this publication, unless specifically indicated otherwise, is under copyright to Saturna. No part of this publication may be altered in any way, copied, or distributed without the prior express written permission of Saturna.

Saturna Capital does not share subscriber information with third parties.

Copyright 2017 Saturna Capital Corporation and/or its affiliates. All rights reserved. Vol. 11 · No. 1