Avoiding testing bias with a small pool of users

As a user experience designer user testing is a key component of the UXD process, offering the opportunity to test if your designs will work for your users and give you a greater overall understanding about the product you are building. 


If you were offered the chance to have access to 100% of your users I guess most designers would snap your hand off, user recruitment for testing is never easy so it’s a no brainer right?

So you have access to all the users who will use the product you’re building and available to test throughout the project. The only caveat is that your total number of users is 21. Now that’s not very many and you can start to imagine how during testing this might actually be counter intuitive. 

With the very real possibility of testing bias from your own users this can potentially pose some risks to the project. So how with such a small pool of users do you avoid testing bias?

We had to think creatively to overcome this challenge and used some clever techniques including rotating users when testing for certain story epics/product features, subbing in users for different iterations of testing and being flexible by expanding the testing pool using users with a similar background.

Rotate who you are testing with

As part of our discovery once we realised we only had 21 users total we made an effort to try and speak to as many as possible. As we felt if we could speak to nearly everyone it would help us in terms of defining different roles and we were going to get to know them very well through various user testing sessions we had planned to complete as part of the project.

Once in delivery we always took note who had tested certain features throughout so we were able to look at our log and made sure we rotated users accordingly so it had been a good amount of time since they last had a prototype in front of them. We worked closely with our product owner on this and she would help us select the most suitable users for testing certain features based on their role within the team. As we were showing different parts of the product with users it allowed us to still gain insights without them being influenced by what they had seen previously.


Only get feedback once per user on any given feature

As happens working on a certain feature or part of the product we didn’t always get it right the first time. Sometimes would need to complete several iterations of user testing to iron out some of the complexities to make it work for our users. 

So naturally like how we rotated users for testing different features we also implemented another tactic to avoid bias. We decided not to have the same users participate over several iterations of the same feature we were working on as a principle. This meant when we did the test whether it was first iteration or third/fourth we knew the feedback we were receiving would be valuable as it was the first time that particular user had seen this feature as part of the product. 

Expand the pool with similar users

Our final tactic to ensure we avoided bias was at times because our user base was so small even with the techniques mentioned earlier we would fall into the situation where because of managing diaries users who were available had only just tested a previous feature or previous iteration we had just run through. 

So our next best bet was to source users who were very close to our own, often they sat within our same client organisation as it was key they had context of the product and where it would be used to make testing with them a useful exercise. Not only did this help when we were short on our original users but it also expanded the number of users we could actually test with. With the unexpected bonus giving us confidence that if someone slightly removed from the team who would eventually be using our product could also understand the scenarios and complete journeys in user testing easily then our actual users should be able to as well. 


Wrap up

To have the opportunity to have access to all users who will eventually use your product is very unique indeed. Something which doesn’t happen often and to the point of being on first term names and knowing about their hobbies meant it made us feel even more than usual we had to be sure what we were creating was going to work and didn’t want any politeness to get in the way of this. 

Even though our user pool was small we were able to manage it in order to keep our testing sessions valid and valuable. A great experience and probably something which I won’t get the opportunity to go through again, but if I do come across a similar situation in future I’ll be ready to go through the motions all over again!

Sign up to Badger News