Imagine the following…You are walking back to work after lunch, and pass by a small building that houses some local University offices. You’ve seen signs out front advertising research studies before, and today there is a sign out front that you don’t quite have the energy to read. A tall woman in a white lab coat with a badge of some sort steps in front of you and says,
“May I speak with you for a moment?…I’m a Professor at the University working on a grant from the National Institute of Self-Awareness and we are collecting data on the potential impact of life stressors on subjective well-being…Would you consent to answering a few questions to aid in National research?…Great! Let’s begin…
On a scale from 1 to 7, with 1 being the least happy a person can be, and 7 being the happiest a person can be, how happy are you, in general?…
How many times have you forgotten where you parked in the last year?
When you were 18, did you experience any signs of depression?…
Do you ever question your partner’s honesty?…
Do you use illegal drugs?…”
Asking people questions – in surveys and interviews – is one of the quickest, most information-rich means of understanding their thoughts, feelings, beliefs, and understanding of the world. It is also intrusive, tiring, and prone to a host of biases and other skewing factors, both methodological and psychological. Researchers and applied scientists have investigated different ways of collecting such valuable information in ways that ameliorate these flaws.
So, what can go wrong in a survey? We can refer to the interview that started this blog, and begin with the demand characteristics of the experiment. Demand characteristics are factors related to the experimental context – such as the behavior or appearance of the experimenter(s), aspects of the experimental methodology, or other external factors – that may lead to misassumptions and behavior changes unrelated to the experimental manipulation (1).
People want to find meaning in the situations they find themselves in, and will “create meaning” when it cannot be easily deduced, as in a sterile, experimental setting. For example, some people are overly submissive to people in authority (2). So, because our experimenter has a badge and white coat, a person might answer more positively, or in a way that shows them in the best light. Of course I’ve never done illegal drugs…
Another set of data-muddying factors are psychological, and include recall bias, which is the deviation of one’s memory of what happened from what actually happened, a deviation that increases with time and psychological distance (3, 4). Recall bias is something that necessarily happens as a result of the human memory system’s constructive (and re-constructive) nature (5). That is, our brains don’t map a verbatim version of reality into our memory store, but actively construct a set of associations in memory based in part on the external world, and based in part with our pre-judgements and past knowledge.
At certain points and with certain memories, these factors are inextricably linked. This constructive memory process is likely to occur many times over, as the research on reconsolidation processes in memory has begun to explore (6, 7).
Recall bias is behind many of the confusing, and spurious, findings in empirical literature. Given gaps in time as little as a month, people have proven unable to reliably recall how intense their headache pain was (8), whether they had been unemployed (9), or accurately report the number of days a week they smoke cigarettes (10).
Another set of psychological factors that effects data validity are a set of heuristic biases (11). Heuristics are short-cuts in thinking used to reach a decision, action, or outcome in a practical, potentially non-optimal way. For example, assuming I don’t need sunscreen is a practically useful heuristic in my hometown of Pittsburgh, Pennsylvania. I’d be wrong sometimes, but not often. Applying this heuristic in Dubai…
There are a litany of heuristics, just as with biases, with varying degrees of generalization and validity.
One well known, robust heuristic is the recognition heuristic, which is the phenomenon that if one of two (or potentially more) items is known, or recognized, and the other(s) is not, the recognized item will be assigned a greater value (12). For example, if you are recognize the name of one city and not another, you are likely to judge the one you recognize as having a larger population. Related biases include availability heuristic, which leads us to estimate things that come to mind easier as having a greater value or frequency of occurrence (13, 14).
So, with all these issues, how do we ever approach the truth?
The way we do this at Thrive is through a core belief in integrating experiential sampling techniques, such as Ecological Momentary Assessment (EMA), into our bi-directional cloud-based platform. Ecological Momentary Assessment refers to a set of gold-standard data collection techniques developed in clinical and experimental research. These techniques are designed around holding brief (Momentary) interactions in an individual’s current, real-life (Ecological) environment, that not only inform but also collect (Assessment) information (15, 16).
Ecological Momentary Assessments offers a means to reducing the impact of each of the factors discussed above. For example, after initial reactivity common to almost every new process or technique, Thrive experiences are responded to in much the same way as people do in emails, slack channels, and text messages. No white lab coats, less demand characteristics.
Additionally, as information can be collected in-situ, as an individual is working and thinking in the moment, recall bias can be greatly reduced. Because Thrive experiences are easy to create and edit on the fly, information can be provided to reduced heuristic biases such as recognition and availability heuristics.
It should be noted that there is no perfect assessment, or set of questions, or method of interviewing, that will avoid the impact of heuristics and biases. The best course of action is incrementally increasing in our probabilistic understanding of actions, beliefs, decisions, and outcomes through a continual dialogue over time. In such a dialogue, common ground can be defined over-time, and the information collected and distributed can be adapted to fit changing contexts and psychology.
Thanks for reading – JZ
- Orne, M. T. (1962). On the social psychology of the psychological experiment: With particular reference to demand characteristics and their implications. American psychologist, 17(11), 776-783.
- Nichols, A. L., & Maner, J. K. (2008). The good-subject effect: Investigating participant demand characteristics. The Journal of general psychology, 135(2), 151-166.
- Coughlin, S. S. (1990). Recall bias in epidemiologic studies. Journal of clinical epidemiology, 43(1), 87-91.
- Raphael, K. (1987). Recall bias: a proposal for assessment and control. International journal of epidemiology, 16(2), 167-170.
- Hyman Jr, I. E., & Loftus, E. F. (1998). Errors in autobiographical memory. Clinical psychology review, 18(8), 933-947.
- Nader, K., & Einarsson, E. Ö. (2010). Memory reconsolidation: an update. Annals of the New York Academy of Sciences, 1191(1), 27-41.
- Alberini, C. M., & LeDoux, J. E. (2013). Memory reconsolidation. Current Biology, 23(17), R746-R750.
- Kikuchi, H., Yoshiuchi, K., Miyasaka, N., Ohashi, K., Yamamoto, Y., Kumano, H., … & Akabayashi, A. (2006). Reliability of recalled self‐report on headache intensity: investigation using ecological momentary assessment technique. Cephalalgia, 26(11), 1335-1343.
- Horvath, F. W. (1982). Forgotten unemployment: recall bias in retrospective data. Monthly Labor Review, 105(3), 40-43.
- Shiffman, S., Hufford, M., Hickcox, M., Paty, J. A., Gnys, M., & Kassel, J. D. (1997). Remember that? A comparison of real-time versus retrospective recall of smoking lapses. Journal of consulting and clinical psychology, 65(2), 292-300.
- Hertwig, R., Fanselow, C., & Hoffrage, U. (2003). Hindsight bias: How knowledge and heuristics affect our reconstruction of the past. Memory, 11(4-5), 357-377.
- Goldstein, D. G., & Gigerenzer, G. (2002). Models of ecological rationality: the recognition heuristic. Psychological review, 109(1), 75-90.
- Tversky, A., & Kahneman, D. (1973). Availability: A heuristic for judging frequency and probability. Cognitive psychology, 5(2), 207-232.
- Schwarz, N., Bless, H., Strack, F., Klumpp, G., Rittenauer-Schatka, H., & Simons, A. (1991). Ease of retrieval as information: another look at the availability heuristic. Journal of Personality and Social psychology, 61(2), 195-202.
- Shiffman, S., Stone, A. A., & Hufford, M. R. (2008). Ecological momentary assessment. Annu. Rev. Clin. Psychol., 4, 1-32.
- Stone, A. A., & Shiffman, S. (1994). Ecological momentary assessment (EMA) in behavorial medicine. Annals of Behavioral Medicine, 199-202.