Explicit user feedback on a content reco engine

Summary

Recommendation engines require lots of data to provide better suggestions to viewers. We developed tools to help viewers generate better recommendations and hide specific content.

Role & Activities

Growing from a team of 3 to a team of 7 during the course of this project I co-led early explorations across platforms and was responsible for developing workshops, company-wide events, and research studies to develop a series of features to personalize recommendations for our viewers.

Facilitating workshops
Strategy
User testing
Visual Design
Wireframing

Background

What background could there possibly be, didn’t you just copy Netflix? It’s easy sometimes to see two companies come out with very similar features and assume that the one that launched its feature second copied the work of the first. My team having launched a new interface and platform to support Live TV in 2017 worked closely with our international team building Hulu’s recommendation engine to understand the factors that went into presenting a piece of content we thought might interest a viewer. We also worked closely with our internal customer support organization, Viewer Experience, to understand the perception of Hulu’s content suggestions.

Goal

Hulu tasked my team with figuring out how to improve the recommendations Hulu was making, but also improve viewer confidence in the system itself.

Research

My initial partner on this project, Jessica Baluyot and I began working with the team designing Hulu’s recommendation engine and that team’s product manager to understand how it worked. We learned how traits of TV and movie content related to one another. We learned what implicit signals the engine took as inputs. We learned from research conducted in interviews with viewers previously and analysis of customer contacts that viewers did not feel recommendations were accurate and they felt powerless to improve them. We identified the need for tools to put viewers in control.

Being avid users of other streaming and entertainment apps, Jessica and I conducted a competitive audit of feedback tools across these systems and unrelated apps like social media. We wanted to understand the types of feedback consumers used so the ones we designed felt intuitive.

We developed a series of proposals to our project stakeholders during this exploratory phase ranging from rating systems, to in-app customization and AI generated loglines. We worked with a UX researcher to test out some of these concepts with actual viewers. We worked with the researcher to develop a base understanding about early iconography, language, and general opportunities to provide feedback in our apps on web, mobile, and living room devices.

We gathered data from internal analysts on paths people took to playback and potential influences. In parallel to our exploratory design work, we needed a qualitative understanding of how people made choices when it came to long form entertainment. By this time we had additional members of our design team so we worked together to design a company-wide pop-up with three activities aimed at understanding some of the decision-making acts people engaged in when choosing their content. This pop-up design fair helped us understand the relationship between artwork, genre, and content as well as the role high level grouping like seasonal or thematic collections play in viewer perceptions of content prior to watching it.

Process

Using the knowledge gathered from our exploratory activities, we broke the tools for user feedback into three categories: explicit user feedback, recommendation presentation, and preference management. The core areas would touch on all the pain points viewers felt, give our recommendation engine the fuel it needed to improve it’s output, and fall in-line with our team’s goal of empowering viewers to control their viewing experience.

We held workshops and design sessions to map out our plans. We found that browsing was the most important area for viewers to give feedback. Working with a copywriter we tested different strings that best communicated what the action would do and the signal it would give Hulu. We created prototypes using Framer and Sketch to test our concepts on viewers and simulate the response from the recommendation engine.

In coordination with our UX research team we conducted usability tests in our Santa Monica research lab with a variety of Hulu and non-Hulu viewers.

Internal decisions guided our feature to appear in more areas of the app.

Outcome

Since the launch in 2019 we hope our early explorations coupled with learning of the impact will help develop better tools for viewers.