Notifications

Intro

Push notifications bombard our strained attention and clutter our phone screens. Notifications about movies and TV, typically tools for unwinding and escaping, are hardly at the top of anyone’s list of “Things that Require Immediate Attention”. The content of notifications may provide some value and the people who used Hulu’s streaming service very much wanted a better way to find out about the things they watch and wanted to watch. So how could we do that in a respectful way? This case study covers several features and initiatives that l launched as part of a multi-year strategy to understand and shape how we presented information to customers on their mobile devices.

Context

Let’s roll back to 2017. People’s phones lit up daily with Instagram and Facebook notifications, “fake news” and protests dominated news alerts, and late prestige-era shows like The Handmaid’s Tale, Twin Peaks: The Return, and Riverdale first hit screens. Beloved streaming service and future property of The Walt Disney Company Hulu also launched a massive redesign of its interface along with a brand new service: Live TV.

Product leaders at the company scrapped notifications for the launch of the new service, but because we also released a commercial prominently featuring notifications while running on a beach, we knew we had to get to them eventually. We decided to use this opportunity to create a new strategy around notifications. I facilitated conversations with cross-functional teams, including user and market research, content marketing, customer relationship management, product, and engineering. We shared a goal: to develop a new understanding of our viewers’ needs, habits, and other factors in their lives.

For user-generated content platforms like YouTube and Twitter, notifications center around subscriptions to specific creator feeds. Traditional media platforms like CBS and Fox center their notifications around new offerings and time-based announcements. Making viewers aware of newly available content helps these businesses by re-engaging users and bringing additional engagement to the platform. Our challenge was balancing viewers’ desire to keep up with shows they liked (”When is the new season of my show out?”), with the business goal to introduce an endless stream of new content to viewers to ensure they remained subscribed to the service, or even better, added additional content streams to their plan.

Process

I worked closely with a product manager to decide what we would build. I pushed to do more research into Hulu’s customers and to understand how entertainment notification should be different. I performed a literature review of external research on push notifications across a variety of devices and platforms, working with our Head of UX Research to navigate the space. I synthesized this research to build a baseline of knowledge. A study the Android team at Google conducted on push notifications served as my north star: I wanted to advocate for the humans whose legs, wrists, and bags we might be vibrating with our updates. However, we could only get so much information from qualitative research. The PM and I decided: we would focus on building a backend service for internal roles (marketing, editorial) to generate and target push notifications while I would lead research efforts and UI explorations for future phases.

I would work with 2 different UX researchers over the course of this multi-year project. Together we developed goals and priorities based on our understanding of existing research uncovered in the literature review and business needs. We understood what questions we had about people who used our product and their notification reactions and priorities, but also understood that the corporation’s desire for engagement, growth, and the looming core metric: number of total minutes watched, would ultimately compete with those needs.

I anticipated an adversarial relationship with Hulu’s CRM team; a team who created, sent, and analyzed communications to viewers through a variety of channels. I had a background in marketing, designing and coding the same sort of communications they would send. Since push and in-app notifications come from marketing with the goal of increasing engagement and other metrics I wanted to find the intersection between the business’ goals and the user needs. This would allow us to work together with full transparency toward the same goal. I setup conversations with members of this team and included them in some design ceremonies so that they felt involved and empowered in the process that ultimately might limit the scope of their communications. Ultimately, our partnership turned out to be positive.

While the CRM team began to send push notifications using the newly created tooling, I wanted to get qualitative feedback to get deeper insight into the metrics we were getting. Working with user researchers I designed a series of internal qualitative studies that would explore how mobile push notifications and in-app notifications fit into our viewers’ lives. Here I developed three core insights:

• Viewers need to feel notifications are relevant and controllable
• There is general agreement among viewers on notification categorization
• Make it easy to turn notifications on and off

By now, it was clear that notifications are incredibly personal and inherently fraught with risk.

The first study simulated personalized push notifications by pre-surveying participants about TV preferences. We then took sentiments from this study into an open and closed card sorting study to understand grouping logic. Finally, we talked to our viewers about notifications in general, how they fit into their life, and how they thought about newness, security and entertainment.

We found that viewers wanted control, the ability to know when things were new, and important security updates. I took this knowledge coupled with our extensive research on push notifications and selected a few key human values our team used in the design process: autonomy and honesty. We used these principles in the design and prioritization process for next steps.

You’ll see a common event in projects I worked on at Hulu: The Changeover. This change was a series of executive changeover in Engineering, Content, and Product that along with a new CEO, change the direction of the company. Many projects, including notifications, saw the end of the their progress - shelved potentially indefinitely. Our notifications initiative found itself on this shelf. It would be two years before I would be able to enter the cupboard and dust off the project.

At this point a new PM expressed interest in improving notifications. This PM was tasked with improving mobile as a platform and increasing engagement with existing app users and bringing on new ones. Together we dug through the previous research efforts and agreed on a vision that would drive the team’s metrics while respecting user autonomy.

We now had two years worth of direct feedback from customers and data showing engagement trends with notifications. Having a centralized location for information inside the app would give value regardless of if people used push notifications while providing granular notification controls for in-app and push notification would give the customization our customers craved.

I conducted an extensive competitive analysis of over 75 apps and to understand how they used in-app notification centers and performed permission priming for pushes. I wanted to provide a balance of familiarity by including common patterns familiar to mobile users while ensuring the IA made sense to Hulu specifically. I identified the need for a number of new patterns in our mobile design system.

While leading design for the project, I partnered with an associate designer on developing the visual language of notifications. We kept our north store in mind when iterating and collaborated with engineering regularly to understand native patterns and the engineering load to build new features into our backend service.

Once we had designs in a good place from a design and engineering perspective, I worked with a user researcher to conduct a usability test. I developed an urgency/pleasure matrix to help us pinpoint visceral reactions to notification types. This study informed where viewers would enter notifications, and what information we included for the initial release.

Outcome

As it turned out, despite an abundance of evidence on the human side that we should offer user-facing controls for push and in-app notifications, the designs for these controls did not reach the initial release. Our team made a time/cost decision and ultimately found that the number of notifications sent was so small that it didn’t warrant the engineering cost to implement granular controls. Instead, we set guardrail limits on the number of notifications a user could receive at once (initially to prevent overwhelming people who had subscribed to a high number of sport teams). We also appended a requirement to each new type of notification that we would send either manually or automatically: the development work for new types must also include a user-facing control to toggle it on/off.

My team successfully shipped for changes to the layout of the profile section in our mobile apps, using native split layouts on iOS devices to take advantage of screen real estate and make the destination more useful. We also launched a notification inbox, where app users could view a history of account, content, and Live TV notifications, personalized to the last logged-in profile. These profile changes included as new profile text icon that would eventually make its way to other platforms. As part of personalization and individualization efforts, I envisioned user-customizable avatars. Due to time constraints and resourcing limits, I opted for two-letter avatars, generated from profile names using the first two letters for clarity and allowing users to enter single emoji as de facto avatars. This avatar and profile structure would eventually roll out to other platforms.

Communications performed between 16-24% better in an A/B test of viewers with the notification inbox and profile updates. Presence of the in-app notification destination and profile changes also resulted in 1.5-2% sustained visit days per visitor and visit days per subscriber. These metrics were considered quite successful. I’m excited to see the future changes my team designed eventually roll out to Hulu customers.