Summative Usability Test
Why ESPN? Business Snapshot
Owned by Disney since 1995 (acquired with the purchase of ABC).
ESPN TV subscriptions recently are declining steadily (along with overall cable tv)
ESPN+ digital subscription service started in 2018 to help build digital subscribers.
ESPN+ has approximately 11.5 million subscribers.
$5.99 monthly subscription (now often offered with Disney + and Hulu bundle $13.99).
ESPN+ is currently not a profitable product.
Web Site Vitals (ESPN homepage)
*According to Google Pagespeed Insights
- 11.1 seconds to interactive (recommended under 3 seconds)
- 6,135 elements on the homepage (recommended under 1,500)
- 3.145 seconds page is blocked (recommended under 3)
- 7 elements to load over 200kb (recommended to be 0)
- 408 server requests (recommended under 50)
- 14/100 desktop speed score (good score is over 90)
Research Plan: Objectives
GOAL
Test ESPN.com based on the core aspects of usability.
- Effectiveness
- Efficiency
- Learnability
- Memorability
- Satisfaction
Test Plan
- Screen and select participants 1
- Write a script, test tasks, and user consent forms 2, 3
- Conduct usability interviews
- Collect and analyze user data 4, 5
Research Plan: Methodology
Preparation
- Screened potential participants to select 5 relevant users. Offer a $10 incentive for completing the interview.
- Plan tasks and measurements for determining site usability.
Participants
- Select 5 users with good technology and high domain (sports) interest and knowledge.
- Schedule zoom Interviews, with one interviewer and two notetakers, recorded interviews with permission.
User Testing
- 5 tasks based on effectiveness, efficiency, learnability, memorability, and satisfaction of ESPN.com
- Post-Test Interview. 10 questions to measure users’ self-reported information.
Analysis & Finding Report
- Review data, categorize all interviews, and analyze responses and observations.
Research Plan: Qualitative Measurements
Post Test Interview Topics (User Reported)
- Happiness
- Overall Experience
- Expectations
- Likely to Recommend
- Site Expertise (self-reported)
- ESPN Brand Loyalty
Participant Data
Key Screener Data
- Similar Demographics (Male, ages 26 – 45)
- Similar Domain Knowledge (High Sports)
- All watch ESPN and attend live sports
- Core ESPN Consumer Group
- Familiarity with ESPN
User Testing: TASKS
1. Please find which two teams played in the most recent NBA game.
2. Can you please show us how you would subscribe to ESPN+?
3. Attempt to contact ESPN.com customer service.
4. Please find who is 3rd in the 2021 Nascar Cup Series Standings
5. Please find the rules for ESPN Fantasy Streak games.
6. Post-test Interview
User Testing: TASK 1
Please find which two teams played in the most recent NBA game.
- 3/5 participants navigated to the NBA Scoreboard via the
“NBA” header menu.
- Although navigating to games by date is possible, one pain point for all 5 participants was being able to distinguish game time and thus the “most recent” game.
- 4/5 of the participants did not engage with the scores banner at the top of the homepage.
- Because clicking the “…” more header link, 1 user automatically opened the dropdown and had difficulty finding NBA due to it.
User Testing: TASK 2
Can you please show us how you would subscribe to ESPN+?
- 4/5 participants navigated to ESPN+ via the top-right header link and missed the prominent ESPN+ “Subscribe Now” button on the homepage, inevitably requiring additional clicks to get to the signup page.
- One challenge that at least 2/5 participants faced was that the ESPN+ subscription pages didn’t include a header that allowed them to return to ESPN.com.
User Testing: TASK 3
Attempt to contact ESPN.com customer service.
- All 5 participants scrolled to the bottom of the page in search of the “Contact Us” footer link, a demonstration of Jakob’s law of familiarity.
- None of the participants were able to locate the customer service email address address, which was located in an email icon under the social icons, and most of the users expressed frustration or confusion during the task.
- After browsing the available FAQs and Help Center, all of the participants said they would have preferred the option to chat or call a customer service representative.
User Testing: TASK 4
Please find who is 3rd in the 2021 Nascar Cup Series Standings.
- All 5 participants were able to navigate to the Nascar Standings page via the “…” more header link, an action some of the users became familiar with in the first task, thus demonstrating the site’s learnability.
- Task 4 seemed to highlight the site’s strengths. The “menu” mental model was shown to be strong on ESPN.com with users able to complete tasks quickly and efficiently with ease just by accessing the main menu systems.
User Testing: TASK 5
Please find the rules for ESPN Fantasy
Streak game.
- Most participants (3/5) were able to quickly navigate to Fantasy Streaks via the header.
- The users’ familiarity with Fantasy sports and the ESPN site influenced their success during this task, and 1 participant who rarely visits ESPN faced significant difficulty.
Analysis: Task Completion Time (TCT)
Tasks 1 & 2 were simple tasks and were completed relatively quickly.
Task 3 caused users to struggle and fail while taking a long time to do so.
Task 4 was the quickest completed task. It followed a simple menu model approach that users were comfortable with.
Task 5 was quickly completed by everyone but AM. He struggled with technical difficulties and distractions during this task which may have influenced this time.
Analysis: Post-Test Questionnaire
4/5 of our test subjects reported experience on ESPN.com, their likeliness to refer, and their average task completion time-correlated with their self-reported expertise score.
Analysis: Interview Responses
In Task 1, users who visit ESPN often to look at scoreboards were able to complete the task without errors.
In Task 2, the inconsistency between pages led to errors among some users.
In Task 3, some users were so frustrated they almost bounced out to a search engine to complete the task.
In Task 4, the learnability from Task 2 proved to reduce overall errors.
In Task 5, users who were familiar with Fantasy sports made significantly fewer errors than those who were not.
Our Team:
Ali Hoss • Daniel Gulino • Hannah Mussi
My Role:
– UX Researcher