Many organizations on their journey to make digital products more accessible start with a focus on testing against standards such as the Web Content Accessibility Guidelines (WCAG). This helps ensure that basic accessibility requirements are met. But how do we know whether people with disabilities can use a product for its intended purpose?
The best way to do this is to include people with disabilities in the product development process – providing input to what should be built, and testing that what has been built is usable and useful.
In reality, many organizations may lack the time, budget, and expertise to regularly involve people with disabilities in design, development and testing phases. One way to help address this challenge is to engage people with disabilities in rapid, focused evaluations of specific functionality. At TPGi, we refer to this activity as Assistive Technology (AT) user flow testing.
Why Is Assistive Technology User Flow Testing important?
Conformance testing focuses on accessibility specialists testing against standards using defined test procedures, not on observing users with disabilities interacting with the resource being tested. This means that relying on standards-based testing can only provide limited insights into the quality of the user experience (UX) for disabled people.
By contrast, AT user flow testing is a way to gather feedback directly from people with disabilities on the usability of performing high-priority tasks when using a digital product. The focus is on understanding the location and impact of accessibility barriers that limit or prevent successful task completion.
This way, we identify barriers that will have the most significant consequences on user experience – barriers that if removed, will increase the chances that affected users will successfully be able to use the product to complete tasks and achieve their objectives.
Compared to traditional usability testing, AT user flow testing can save time and money by reducing effort to recruit testers, by asking people with disabilities to follow a very precisely defined user journey or task flow through the product being tested. That way, testers can conduct testing independently in their own time and report results to the product team.
How Does Assistive Technology User Flow Testing Work?
AT user flow testing involves engaging one or more people with disabilities to step through a defined task flow and report any issues they encounter. Ideally, recruitment should be quick and easy, so participants could be coworkers from your organization or people you know from outside who are available at short notice to test.
The task flow can be something that is considered business-critical, for example a product selection and purchase process or a sign-up flow. The flow could also include triggering error conditions and other important edge cases. The key is to define the flow in very specific steps, so you can be sure the tester is interacting with the product in exactly the way you define. To avoid the testing becoming too time-consuming, we suggest limiting the number of flows to no more than five in a session.
The testers go through each step of each flow and document any issues encountered during the testing process, including location, impact and information that might indicate the source of the problem. The testing process can be further enhanced by video-recording the testing experience – this way, members of the product team can directly see and hear the user experience from the tester’s perspective, and particularly the impact of barriers on user experience.
Benefits and Limitations of Assistive Technology User Flow Testing
AT user flow testing has many benefits:
- It’s an efficient way for product teams to include people with disabilities in the product development process.
- It helps product teams identify high-impact accessibility barriers that need to be fixed as a priority for improving user experience for disabled people.
- The focus on specific flows and functionality and the quick turnaround makes the method ideal for integrating accessible UX evaluation into agile development processes.
- It can be used to verify remediation efforts after an accessibility conformance review, to confirm that the fixes made enable people with disabilities to complete tasks.
- Regularly retesting the same flows over time helps identify and address any regressions in accessibility or user experience over time.
There are also some limitations with the method. The basic assumption behind the method is that if a tester encounters a barrier, then the barrier is likely to be experienced by other members of that disability group.
However, like any user research method that relies on feedback from a small number of users, there is a risk that changes are made to a product based on user expertise or behavior that may not be representative of a wider population of that disability group. This is especially the case if testers are power-users or are very familiar with the product being tested.
So be cautious when acting on feedback that may require significant changes to the product or appear to require deviating from known best practice in accessible user experience design. If in doubt, conduct further user research to understand the nature of the barrier, how likely others are to encounter it, and the best way to address it.
AT user flow testing is a method for quickly gathering targeted feedback at various stages of your development process from people with disabilities on the user experience of a digital product, without requiring significant resources. In combination with conformance testing against accessibility standards and comprehensive usability evaluation, AT user flow testing can be a very effective part of your digital accessibility strategy.
