Internal research done over the past few years validated the industry-wide hypothesis that the shopping experience is non-linear. We had observed that a purchase would happen over multiple visits by the customer. There was no single reason behind this; it was simply a result of the culmination of various factors largely related to customer behavior. There was an opportunity to understand this behavior and improve conversion.
We had observed that customers curated their cart over several browsing sessions. This peculiar behavior led to cart abandonment scenarios where a customer added items to their cart but did not complete the purchase in that session, leading to a loss in revenue. Items left in the cart for a prolonged period of time also risked becoming sold out/ not available leading to frustrations at the customer's end. Although the marketing department sent cart reminder emails to customers periodically, most customers did not engage with the emails, as seen by the low click-through rate.
Previous research had shown that there were 4 kinds of customers shopping with us. These were very similar to the 5 shopper types put together by the Nielsen Norman Group. These 4 types had distinct behaviors as shown below
Checking analytics and comparing these types against visitor behavior, we found that the majority of our customers were either Browsers, Researchers or Bargain-hunters, with Product-focused customers representing a small set of daily visitors. Combing through multiple qualitative studies done by our researchers helped me understand the correlation between customer type and intent to purchase. The image below shows customer types being arranged on a scale showing intent to purchase.
We can clearly see that the Browsers' intent to purchase isn't as high as the other three types of customers. In other words, Browsers were not contributing much to the revenue earned from online sales. Combining this with the fact that Product-focused customers represented a minority of our daily visitors, it became evident that designing a solution that catered towards Researchers and Bargain-hunters would deliver the maximum value and increase conversion.
I explored a range of design solutions varying in fidelity, flexibility and functionality. A simple solution was to remind customers about items in their cart via an informational message. This message was intended to live in the form of a banner among the content but it also risked being overlooked by the customer. Another solution explored how we could show the actual products in any customer's cart. This solution provided the highest level of fidelity but it also required heavy backend integration for showing the exact product + swatch the customer had added to cart. A third solution was displaying a notification with an ephemeral nature such as a toast.
An advantage of the toast was that it appeared on top of the content and thus it could work on any page the customer viewed. It also provided a lot of flexibility in implementation — the toast could either auto-dimiss or it could be dismissed by clicking/ tapping on a button, it could appear from the top or the bottom, etc. With all of these qualities, the toast emerged as a lean and flexible solution that could serve as the building block of a robust experience.
For questions that could not be answered because of lack of data, we decided to approach them with a test-and-learn strategy. For example, when it came to deciding if the toast should dismiss after a customer intitated an action or if it should dismiss automatically, we decided to test both, one at a time, and then compare engagement. Since we were introducing new functionality, we wanted to cause minimum distraction. So we decided to roll with the auto dimiss variant with the plan to test a persistent toast, if engagement was low.
We had to find an optimal way of capturing the customers' attention without distracting them or disturbing their flow. For this reason, I decided to make the toast slide in, after the page had finished loading. The hypothesis was that the slight movement caused by the toast would catch a customer's eye. The toast would then last on the screen for a total duration of 4 seconds. This timing was established after referring to WCAG best practices and ensuring that screen readers could parse through the toast contents in that duration. These parameters were subject to change as we tested the toast experience.
The next question we set out to answer was - how frequently should the toast be displayed? We took a look at our analytics dashboard and found that most customers returned to the website in about 24-36 hours. This seemed like a good opportunity to capitalize on – remind customers about lingering items in their cart when they return via a new session. Thus, we decided to display the toast in every session that was initiated after a minimum of 24 hours from the latest session.
The cart reminder toast was tested across desktop, tablet, mobile and the VS app platforms using a 50-50 traffic split in March 2020. The two variants were labeled as enhancement (with toast) and control (without toast). The success of this test was measured by monitoring two primary metrics- Orders and Revenue. A few secondary metrics we planned to analyze were Units per Transaction (UPT), Revenue per Visitor and Average Order Size (AOS). Capturing secondary metrics was important to validate alternative hypotheses and get granular data.
The toast performed well on mobile, with the enhancement variant showing an increase of 3% in Orders. Surprisingly, the enhancement variant had a 2% drop in Orders on desktop. Did this happen because customers missed out on seeing the toast? Would removing the auto dismiss nature of the toast help it to be more prominent thus driving up engagement and orders placed?
On comparing the UPT metric, it was clear that the average items bought in a single purchase on desktop was more in the control variant (without toast) than the enhancement variant. This pointed to an alternative hypothesis that perhaps the toast was too frequent on desktop, and maybe it interrupted customers as they were building a high-value cart. This premature interruption likely pushed customers to check out early, with fewer items in their cart. We decided to validate this hypothesis in another round of testing.
Since the toast performed well on mobile, we released the enhancement variant to 100% of the traffic. I took away the findings of the test and the alternative hypotheses back to the drawing board. My main goal now was to make small tweaks to the implementation of the toast and tailor the experience for each platform (desktop, tablet, mobile). These changes would be put through an A/B test again to enable us to learn and incrementally inch towards delivering the best experience. As of today, I am currently working on designing the next set of improvements for the toast functionality.