Today, I’d like to introduce you to Melanie Bretts, who wrote our blog today. Melanie joined Mobile Digital Insights in early 2019, bringing with her a wealth of experience and helpful blog-writing skills. You’ll hear more from Melanie going forward.
Research the research? Is this redundant? Not at all. As researchers, we know it’s important to constantly learn. Recently, we did a study to explore how similar the results were if we covered the same topic with two types of surveys: recall versus in-the-moment diaries. Here’s what we learned about research with distracted consumers, the big gaps between recall and in-the-moment findings and their similarities.
How Fragmented Attention Spans Affect Research
Instant gratification: Once digital cameras and email hit the scene, people no longer had to wait hours or days to see photos or receive a letter in the mail. Waiting became unacceptable. Instagram, Snapchat and Twitter built upon instant gratification and fast-paced media consumption. In this environment, it became imperative to put research tools on mobile devices, where consumers are living.
Simply a swipe away: Wandering consumers now jump from one app to a website to another app, checking in, responding and rapidly posting. They are more likely to complete a survey if it is simply a swipe away, which is why research that captures insights in the act of the behavior (or just after) is more effective. It’s more convenient for people and it generates more accurate results. People are exposed to hundreds of daily ads and information wherever they turn, so the sooner insights about their behavior is captured, the less cloudy the recall will be.
Comparing Recall Diaries versus In-the-Moment Surveys
To do a side-by-side study, we picked something simple - breakfast and lunch behavior. We asked recall respondents about the breakfast and lunches they have eaten in the last week: where they ate their meals, the time of day they ate, how much they spent on them, etc. (n=150) We followed in-the-moment diary participants for four days, and asked them to share one to two minutes of detail about each meal-time event. It captured the same data as the recall study, but because it was in-the-moment, we also explored reasons for the meal in more detail. (n=150)
The Differences: Perhaps a Reason Many Americans Are In Debt!
In the recall survey, people said they spent about $3.42 for breakfast, but the actual spend on the four-day Dairy was 3x higher. There was a similar pattern for lunch, where actual spend was nearly 2x above the recalled amount.
Why this big difference? Is it possible that recall respondents wrote what they ‘hoped’ or ‘should have’ spent on their meals as opposed to what they actually spent? Regardless, there was a consistently higher response from those indicating the amount spent in-the-moment.
The Similarities: Routine Behaviors
This research-on-research study shows minimal differences when comparing habitual behaviors, such as where people eat breakfast and lunch in a typical week.
Recall surveys do quite well at capturing the relative order of routine behaviors.
That said, people tended to overstate at-home meals and understate meals away from home, particularly for lunch. Relative responses were similar, but actual responses differed quite a bit.
$ for Research Study + Inaccurate Data = MISGUIDED BUSINESS DECISIONS
We have used and developed many recall studies over the years. They can be useful. However, we’re learning to be careful with them in certain situations, some of which are shared in this post. Recall surveys may reliably capture habitual behaviors, but when questions dig deeper or need to be specific, diary solutions provide superior accuracy and reliability. In-the-moment designs can focus on a particular incident and accurately capture details as well as feelings at the time.
We encourage you to thoughtfully determine the needs of your clients and the reliability they need. It may be that an ‘in-the-moment diary’ is the way to go with today’s distracted consumers.
Want to see the full findings of this Research-on-Research study? Just CLICK HERE to download your free copy.