User testing process
Information on how to conduct user testing for Ushahidi.
User testing process
When a design piece’s impact on users is unknown we should be committing to at least 1 round of user testing on a minimum of 5 testers. These pieces can be new features, significant changes to an existing journey or significant changes to UI visuals.
We should only ever test prototypes and never the live/test deployment sites as users tend to get distracted by other functionality of the site.
Aim for a 45 min session but understand they often run over. Try to end early if possible.
Encourage other team members to join in on test sessions.
We should not test on the same people/users month to month.
The notetaker does not talk in the session except to introduce themselves as the notetaker.
1 - Produce testable design work prototype example.
2 - Identify user testing needs, hypothesis and reasons for testing.
3 - Write a test plan example test plan
4 - Other design team member inputs on the content of the test plan. It may also be relevant to include programs/implementation staff and Product managers in the test plan draft.
5 - Finalise plan and save a document in a global/shareable folder and/or an open issue referencing the issue/feature/epic being user-tested.
6 - User testers should be recruited/engaged with constantly so we have a pool of relevant and regular testers to call upon.
7 - Organise the available slot for the user tester. Be mindful of the timezones of other people involved.
8 - The leader of the test sessions should not be the designer who did a majority of the prototype/design thinking/visuals in order to remain as impartial as possible. There should be a test leader and a note-taker on each test session. Test sessions can be in person or remote. Our preference is for remote due to the location of staff.
9 - Conduct test. Record audio and/or video if possible.
10 - Debrief between those present during the test.
11- Data process right after the session using a document template.
12 - Formulate comms to Ushahidi team & key contributors.
13 - Share findings with Ushahidi team & key contributors.
Recruitment
Finding people to user test with is one of the most difficult and time consuming parts of user testing.
You need to look for a certain set of criteria that you want to test for. Typically this includes:
1 - The kind of user that would use the Ushahidi tool and/or the specific feature that is being designed. Works or spends a majority of their time working with what the design is trying to solve e.g. Better data export = A user that exports data at least once per day from any software.
2 - One that doesn't have bias e.g. has neither a good nor bad view of Ushahidi products, company or services
3 - Inclusive of multiple diversity elements like; technical competency, socio-economically diverse, literacy competency, language competencies/not English as a first language, ethnically diverse, Impairment diverse, neurotypically diverse, gender identification diverse, age-diverse, geographically diverse.
4 - Ushahidi cannot allocate finance budget to Usertesters. As such we request that you find people who are willing to offer their time and insight for free. In very rare occasions Ushahidi may be able to offer incentives like vouchers, free merchandise and services in kind.
5 - Happy to be tested with, notes be taken regarding comments and participate. Preferably also sound and/or video recorded.
Set up
If possible, let the participant use their own machine for the test.
Take notes in a way that will jog your memory after the test is over and be non-intrusive. Try not to position a notebook or laptop between you and the participant because it makes it much less conversational.
Position yourselves in a way where you are both looking at the screen and you can observe the participant’s expressions, body language, keyboard and mouse behavior, etc.
Double-check any recording equipment you might be using.
Have a copy of your test plan handy for reference.
Preamble
After introducing yourself and the reasons for the session, there are a few things that you must always say before a test begins.
Introduce yourself and why you are there (build a relationship)
Be friendly and conversational. Ideally, the participant will feel like they are having a relaxed conversation with you.
Set expectations for the interview length (time, format, relaxed nature)
Tell them that they are using a prototype, and it won’t have all the functionality they might expect Tell the participant not to worry about making mistakes. You are testing the designs and the prototype, not them.
Let them know they are in control. Tell the participant that they can stop at any time.
Ask them to “think out loud” as much as possible. You can’t read their mind, so you need them to describe what they are thinking/doing.
If you plan on recording video, or audio, or taking notes, you must obtain the participant’s consent BEFORE you start recording.
End early if you can.
Interviewing Basics
Be friendly, open, personable, and easy to talk to. This should feel like a casual conversation as much as possible. Be curious, ask politely to explain any behaviour they are doing naturally.
Ask open-ended questions. The goal of the session is to find out things you don’t know yet, so never frame questions as a yes/no, this will limit the responses you get. Generally, you are in the clear if your question begins with “How...”, “Why...”, or “Tell me about...”. Open-ended questions can make people nervous. Be flexible with the format.
Don’t ask leading questions. Leading questions frame the response before it’s ever given. For example, don’t ask “How helpful is Ushahidi?” because it assumes Ushahidi is helpful at all. Instead ask: “How have you found Ushahidi so far?” or “What has your experience with Ushahidi been like so far?”
Avoid asking what people want in a feature “What would you like/do/want in a feature” People are often unreliable self-reporters in hypothetical situations - More useful to ask them to describe a behaviour they do.
Asking people to describe how frequently they do something is unreliable too “How often do you do/use…” People will over/underestimate. Better to give a specific time frame - “How often did you use feature this week”
You don’t have to follow the script exactly you can double back, ask questions out of ‘order’ and build a more natural conversation.
Role-play the scenario “You are a data worker who has just finished finding the data that you want to analyse further...” “You now need to find a way of completing the next steps…” Pick the relevant parts of the role-play e.g. If they are already a ‘data worker’ stress the ‘You have just got to the point where you know what information you need to get”
Replay responses and dive deeper - if the participant says something interesting and you want them to expand more, re-play back to them what they said and ask them to elaborate. For example, “I noticed that you mentioned _____, can you tell me more about that?”
Let silences play out. Often users can be quiet for several seconds while they think about something. Letting that silence hang between you almost always results in them giving you more information. If you jump into the silence immediately, it shuts them down.
User testers will ask you questions. “Is it working right?” “Is it supposed to do that?” “What does that mean?” Avoid answering these, instead turn the question back: “How would you expect it to work?” “What do you think it means?”
Ask for critiques, not compliments. Give permission for them to be harsh. Tell them you have no connection to the piece of work you’re testing if that helps (even if it’s a bit of a lie)
Laddering Conversation Technique
Laddering makes explicit connections between an experience and the value (ex. self-esteem, accomplishment, belonging, self-fulfillment, family, security)
Person A: Why do you like your computer?
Person B: Because I like to read the news online.
Person A: Why do you want to read the news online?
Person B: Because I want to keep up with industry activity.
Person A: Why is it important to keep up with the industry?
Person B: Because it helps me to be better at my job.
Person A: And being better at your job means?
Person B: I am able to support my family.
What to look for
Body language - are they hunched up in an agitated way? Leaning into the screen? Arms crossed defensively? Etc.
Extra clicks, mouseovers - if they click on something that isn’t intended to be clicked on, you can ask them what they expected to happen?
If you are testing remotely, it'll be harder to or you won’t be able to catch any of this.
There is a tendency for people to tell you what you want to hear. If you think this is happening, try to probe deeper on a line of questioning.
Data collection
Never collect data without a participant’s consent (e.g. photos, video, voice recordings, written notes, etc.). Ask them if you may collect the data, and be VERY clear and honest about who will see that data in the future.
As a rule of thumb, don’t collect data that you don’t intend to use. For example, don’t take voice recordings unless you’re prepared to make transcripts and do analysis on those transcripts. Don’t take photos that you don’t intend to analyse.
After the test
Always ask “Is there anything else we haven’t covered you want to raise?” This is often when you’ll get the most valuable information. Embrace silence here. Testers usually say the most important thing just as your packing up or even when you’re about to walk out the door.
Don’t forget to thank them for their time :) Leave a good impression so we can test with them again.
After your participant has left, immediately spend minimum 15-30 minutes re-reading your notes, and making longer, more detailed notes. Researchers call this “coding”.
Try to remain as detailed as possible during this phase. For example, instead of noting “participant was well dressed” note, “shiny shoes even though it’s muddy outside, pressed shirt, new manicure, etc.”
When reviewing your notes think about:
What was the ‘Typical behavior’ - What was easy to complete and was self-explanatory
What was the ‘Variant behavior’ - What seemed to be harder for the user than other areas? Why?
Did they voice any ‘Untapped desires/needs’ - “I’m worried about the price...how can I send it to review?”
What people wish they could do with ‘the feature’ - “I want a quicker way to do X” “I want to use a CSV”
Problems they have - confusions in the results or presentation - Look for frowns, pauses or sighs and ‘grrs’
A lot of user testing interviewing requires ‘reading between the lines’ as objectively as possible and without calling explicit attention to the fact that’s what you’re looking for too early.
Every second you wait after a session is over, the data in your memory decays.
User testing recommended reading
Don't Make Me Think, Revisited: A Common Sense Approach to Web Usability
Interviewing Users: How to Uncover Compelling Insights