I chose to evaluate the group session feature because I had a disappointing experience of using the feature. My disappointment stemmed from three key areas:
Before honing in on a focus for the evaluation and designing the approach, I first embarked on understanding the problem area more deeply.
I created a survey aimed at gathering insight into the problem areas I had outlined.
Focusing on the refined audience, of the twenty-two participants that use Spotify premium, there were three key insights I took from the survey.
Goals and success
With the survey insights gathered and problems validated. I now had two key issues to build my tasks around and key questions to answer:
5x remote moderated task-based observational evaluation including:
With my tasks refined and improvements made to the facilitation of the test, I completed the evaluation with five participants. I made use of Microsoft forms to record all data and excel to compile insights.
The post-test questionnaire section of my evaluation was the key stage for collecting results and insights. The questionnaire included the SUS statements and responses as well as a section qualitative feedback in the form of three open questions.
This process gave me the results in the standardised format and made further analysis and comparison easier. Some key insights I found:
I cross-referenced these scores using details in the same guide used to normalise the scores. This guide details how scores can be graded upon a curve using a more familiar alphabetical system, from A-F (high to low).
I deduced that participants rated the experience with Spotify's group sessions in the task a a D-. A highly unsatisfactory score considering Spotfy’s prowess. But, as the SUS system is not diagnostic, further analysis is required to diagnose why the feature scored so low. This is where I looked at the qualitative feedback.
I asked each participant the three questions and recorded their answers as notes. I then compiled the responses into the same spreadsheet as the SUS questionnaire.
To convert the qualitative feedback into more manageable and useful insight, I extracted the key points from each response. I then tallied up the mentions of each of these points across all five evaluations. I focused on the negatives outlined by looking at the questions, ‘what was your biggest pain point in the tasks?’ and ‘what would you change about the feature?’. This gave me insights on participant issues.
Looking at the table there are two leading pain points and issues both mentioned by 4 out of 5 participants. Given the scope of the study I decided to initially focus on these two:
With the most important problem areas identified, I set about analysing the interface where the issues occurred.
The above screen outlines what users are presented with when starting a session. This screen has the function of displaying who is currently listening as well as the ability to invite friends.
The invite functionality comes in the form of a button. The ‘invite friends’ button is what all of my participants used to complete task 2. Interestingly, the pink waveform image at the bottom of the screen is a scannable code that allows other uses to join. However, none of my participants used the scannable code. And when following up in the qualitative feedback 4 out 5 did not know that it was a scannable code.
This is interesting because the pink code is afforded a proportionally more significant amount of space compared to the ‘invite friends’ button. Though, it is clearly being missed by users. I have deduced three reasons for why it may be being overlooked:
“Users will transfer expectations they have built around one familiar product to another that appears similar”.
Spotify break Jakob’s law by implementing their own take of a quick response (QR) code. According to a survey by Statista an estimated 11 million households scanned a QR code in the US in 2020. Of an estimated 330 million people in America that is a small figure, even if we are generous towards our estimated amount of people per household. So we’re in a position where QR codes are still growing in use and becoming more ubiquitous, however, this data implies they are still not used by the majority
What’s more is Spotify’s version of the scannable code actually has closer similarities to a ‘waveform’ (a visual representation of music being played). This further plays into users expectations of what this scannable code is. It also justifies why, when tasked with ‘inviting a friend to the session, users do not notice the waveform, and instead click straight on the ‘invite friends’ button; despite the scanning option being a more direct path.
“People will perceive and interpret ambiguous or complex images as the simplest form possible, because it is the interpretation that requires the least cognitive effort of us.”
The scannable waveform Spotify has implemented follows this law by paying close resemblance to a waveform. There is nothing in the shape to indicate that the code is scannable, therefore users recognise this as its most closely related shape and not as a scannable object. This causes a detrimental effect on the user experience as the functionality of the code is hidden.
Lastly, the aforementioned factors make it hard to distinguish the code as a scannable. However, aesthetically, and considering beyond usability factors, Spotify’s code fits well into their visual design patterns. The justification for not including a traditional QR code is somewhat understood. What is less justified I feel, is the lack of labelling that, if implemented, would likely mitigate risks of overlooking the code.
With the analysis in mind I have made a number of recommendations to improve the QR code scanning visibility and ease of use. To convey the ideas of these recommendations I have taken a screenshot of the interface and redesigned it to include my suggested improvements.
The joining screen was a pain point for 4 out of 5 of my participants, with one citing “changing the wording of the joining screen would provide the most immediate improvement”. To understand where the issue lies with the joining screen I sought to understand the two options that you are presented with.
Both options allowed guests to share control of the queue and the functionality was identical when going down either route. However, by clicking 'Join on the same device as Host’ would continue playing audio from only the host’s device. By clicking ‘On my own device’ the music the host was listening to begins to play on the guests device. Essentially, the user is being asked ‘do you want the music to be played on your device or not?’.
To elaborate further on these differences I have outlined some scenarios for each option below:
The main issues I feel are present here are the fact that Spotify does not provide much detail on the scenarios or use cases for each configuration, and the group sessions feature in general. The vagueness and vast potential use cases make it harder to understand the two options you are presented with. But, because the main difference is where the audio is played, it makes sense for Spotify to remove the use case or scenario from the options and instead focus on the device where the music is played. However, the wording of these options does not provide clarity on what will happen and breaks some heuristics of UX writing.
Begin with the objective - when a sentence describes an objective and the action needed to achieve it, start the sentence with the objective
Nick Babich — 16 Rules of Effective UX Writing
Additionally, the modal that pops up lacks depth of information and detail. The space could be used more effectively to include diagrams or further description of what will happen. I found most of my participants paused for a long time when landing on the screen and the minimal detail likely made the experience slower due to trying to process the outcomes.
To convey the ideas of these recommendations I have redesigned the interface to include rough drafts of my recommendations implemented.
Given a wider scope for improvements and analysis, I would look towards the next two items on the prioritised list of issues and pain points that participants felt:
Going forward if I were to continue from where I left off my main focus would be to repeat the evaluation, using the prototypes I designed. I would explore A/B testing and comparative methods to monitor any impact of my recommendations. This would help to validate the quality of the recommendations and help inform the decision on whether they should be implemented.
Upon reflection of my usability evaluation, there have been some key lessons I have taken from the process.
In planning the evaluation I chose to use the think aloud methodology while my participants carried out tasks. However, I found it difficult to write notes and at the same time as observing and the need for notes was reduced by the fact that I included some open-ended qualitative questions in my post-test questionnaire.
If I were to carry out the evaluation again, I would remove think aloud and instead create a more defined usability specification table with metrics for success rate and completion of the tasks. This would have provided more quantitative data which would have been helpful to cross-reference with findings on the participants' user groups and SUS scores.
A challenge I found in the process of the assignment was connecting the user grouping dataI gathered to the rest of the evaluation insights. I feel this was down to the mostly qualitative analysis making it laborious to find insights in any of the data, let alone linking it with previous data. The vision for the user grouping questions (user's experience with Spotify and position on the diffusion of innovation model) was to test whether past experience and attitude to innovation played a part in the usability of the feature. Despite only having five participants I would have liked to have explored this hypothesis further.
Thanks to the CI604 Usability Evaluation module leader, Sanaz Fallahkhair, as well as the student cohort for participating in the evaluation.