Qualitative Testing: the Ultimate Guide

With one out of the three customers willing to leave a brand they love after just one bad experience, it is clear that collecting customer data to improve the user experience will be a never-ending facet of your business. Qualitative testing provides you with feedback on how your product, website, or app makes your customer feel.

Although we focus on qualitative usability and user testing in this guide, your business needs to combine quantitative and qualitative data to sustain growth. For now, we will focus on demonstrating what qualitative testing is and the types of research methods associated with it.

What is qualitative testing?

Qualitative testing methods assess how well its intended audience can use a product or service. In a qualitative study, you can give participants tasks to complete so that your business can assess how useful their current design of a product or service is. In addition to this, participants can give their feedback on any problems they encountered when using your product or service.

For instance, if they had trouble completing a task, they could detail why and how a business could make it easier, qualitative testing can be conducted at the initial, first redesign and the evaluation of the redesign stage as it is an interactive process. At the end of qualitative tests, a business can identify the strengths and weaknesses of design and will involve using a smaller number of users (5-8) to channel the product or service in the right direction.

Is qualitative research useful?

Your customers’ frustrations, joys, and satisfaction of completing a goal will come to light when they give you their opinion on how useful your product or service is. At the same time, qualitative feedback provides you with the ‘why’ while quantitative usability testing provides you with the objective reasoning behind their emotional user journey. It tells you the ‘how’ as it can inform you of the pages that your target audience visited and how long they stayed, as well as where they came from and what they clicked on.

Quantitative testing methods justify the data received from qualitative tests as it can demonstrate what works and what is not. For instance, quantitative tests like the system usability score (SUS) will indicate which target users had the worse or the best overall experience on your website or app. In line with this data, you could focus on watching the videos from these types of videos all the way through as they would prove to be more useful for improvements. In other words, quantitative data can help you focus on the qualitative data that is the most valuable.

But without qualitative testing, you don’t know why a user stayed on your homepage for a long time or why they left your landing page. In addition, there is no other way to discover the story of their user experience journey on your website or app. In the next section, we will discuss a few of the best qualitative testing methods you can adopt to improve your user experience.

Qualitative testing methods

Unmoderated interview

In this type of study, a participant will speak aloud when interacting with your product or service. You can record their screen and voice by using online remote user testing software. If you choose PlaybookUX as your choice of software, we can also offer annotations and video transcripts. However, an unmoderated study does not have a moderator.

This facet means that while participants can complete a study at any time and any location, they cannot ask anyone questions if they get stuck. So, it is best to use this type of research method if you don’t need to test anything complicated and want a quicker result at a lower cost. When conducting unmoderated qualitative testing, use our free user testing templates to help you do a first impressions test or to assess the usefulness of your website or app, among others.

Moderated interview

A moderated study can take place online or in person. Unlike an unmoderated test, it has a moderator to work directly with your participants to guide them through any challenges during the test. This type of qualitative usability testing is best if you want to test a complicated process or need to conduct a session on your initial prototype or wireframe.

As well as answering a participant’s questions, the moderator can also observe the body language that participants will display when frustrated with an issue. Moderated qualitative testing methods also allow you to modify your script if any areas require a more detailed examination. Utilize moderated and unmoderated interviews for qualitative user testing and usability testing.

While both our moderated and unmoderated tests offer access to our vetted participant pool with individuals from over 40 different countries, our AI-based software will also provide you with analytics such as time on task and sentiment analysis.

A/B Testing

This qualitative or quantitative testing method allows you to compare two different versions of the same design to see which one your target audience prefers. For example, if you want to test your call to action button, you could have one version with the words’ add to basket’ while the other version could have a ‘buy now’ wording. You will then need to test which button has the most significant number of people clicking it.

An A/B test would involve asking questions like:

1. What are your initial thoughts of this <page, website, prototype>?
2. Looking at this image, could you explain what you would do next?
3. What do you like about this design?
4. What would you change about this design? Why?
5. Overall, do you prefer version A or version B? Why do you prefer this version?

Click Testing

If you are looking for a quick way to test static images on your website or app, use click testing. This type of qualitative testing could show you where participants would click to access something they need.

Concept or Prototype Testing

Get early feedback from users on your idea or prototype before investing in lousy design for your product or service. To get the best out of this qualitative user testing method, you should conduct a moderated interview and let participants know that this idea or prototype is still a work in progress. This instruction will encourage them to avoid looking at any facets that are missing or incomplete.

If you have a proposed solution to a problem, you could ask participants questions like:

1. If this solution were available to use, how often would you use it?
2. Does this solution solve all of your frustrations and problems? If not, why?
3. Is there any reason why you would not use this solution?

Voice of the Customer/Intercept Survey

This type of qualitative testing aims to collect ongoing feedback from customers who visit your site or app. Using a pop-up form on your website or app, or a feedback form through email, customers are asked qualitative questions such as:

1. Did you accomplish your goals?
2. Are you happy with your experience on this app/website?
3. How likely are you to recommend this site/app to others?
4. How can we make this site better?

Brand Perception study

Ask participants what they think of your brand using this qualitative user testing study. You could show participants images of your live product or wireframe and then ask them to describe what they see. You could give them a list of words and ask them to choose the words that they feel resemble the image. Alternatively, you could pose questions to your participants, without a list of words:

1. What would you improve about <your brand>? Why?
2. Could you say that you trust <your brand>? If not, why?
3. What three words would you use to describe <your brand>?

Diary/Camera study

A diary study is one of the qualitative testing methods that help you collect information from participants over some time. Ask participants to record their experiences of using a product or service in a diary. They can also use photos to document their experience. Once this period is over, the researcher will analyze the findings.

These findings could tell a brand more about the time of the day that a user usually interacts with the product, what drives them to complete specific tasks and what their typical customer journey is when using a product or service.

Diary studies can also take some time to plan, consider preparing a pilot study first. This type of test will allow you to test the design of your study and relevant materials first.

Ethnographic field study

This type of qualitative testing involves observing how participants carry out tasks in their natural environments. It will give businesses an idea of how a participant’s location and environment affect carrying out specific actions in their daily lives.

Benchmark study

Longitudinal qualitative tests like these involve participants carrying out identical tasks on your products or services over time. It could also include measuring your failures, determining how your site performs when compared to your competitors, or assess best-in-industry examples to mimic.

If conducting competitor research over time, you could ask questions like these:

1. What is your impression of <brand one>?
2. What do you like about <brand two>?
3. Which company has provided you with the best experience, <brand one or brand two>?
4. Out of the two, which company are you most likely to use next?

While PlaybookUX cannot be used to conduct quantitative tests like card sorting, tree testing, or eye-tracking, it can be followed up by qualitative testing like moderated and unmoderated interviews to get further feedback from your target audience.

Wrap-up

Qualitative testing helps you discover your target audience’s experience when they use your product or service. The results from this study will allow you to get inside the head of your target user as they share their thoughts and feelings that will tell you how and why you need to fix an issue.

While qualitative testing methods can help you answer why a participant feels a certain way about your product or service, quantitative research can help you analyze the qualitative outcomes. It can also help a business calculate ROI for improvement or change to a product or service. Want to see how PlaybookUX can help you source better qualitative insights? Sign up to get started or let us organize a test for you.