We’ve all purchased new technologies, trinkets, and toys that were so counterintuitive that it was surprising how the product ever made it to market. While a poor gadget may present little more than inconvenience or frustration, in the medical device community, user confusion creates real danger, placing patients and doctors at risk.

That is why, in June 2011, the U.S. Food and Drug Administration (FDA) issued draft guidance, available here , outlining how the agency would look at human factors and usability testing for medical devices. Though the FDA notes that the guidance contains “nonbinding recommendations,” the agency’s message was clear: if medical device manufacturers want FDA approval for new devices—or approval for devices entering new markets or with new packaging— manufacturers must validate that those products can be used safely and effectively by the intended users in the intended use environments.

Human factors validation results must be included in the manufacturer’s 510(k) application, and conclude with the following statement—“The has been found to be reasonably safe and effective for the intended users, uses, and use environments.” All the investment, development, and testing boils down into that one simple statement.

No doubt human factors research requires additional investment in the already-costly product development process. However, when conducted appropriately this research can inform the development of features and design controls that not only make devices safer, but also easier to use, leading to competitive advantages and longer shelf life.

Formative Evaluations

Formative evaluations help manufacturers identify potential changes early in product development, at a stage where it is less costly to make changes. In formative evaluations, device manufacturers are not only working to optimize the design of a device, they are also laying the groundwork to answer the questions that are fundamental to a successful validation study – the “who,” “what,” “where,” and “how:” Who are the users; what will the device be used for; where will they use it; and how will it be used?

In their draft guidance, the FDA advises manufacturers to ensure their product is “reasonably” safe and effective— leaving it to the manufacturer to make the case for what users, environments, and uses are reasonable.

These may seem like easy questions with straightforward answers, but nuances of specific user groups, use cases, or use environments are sometimes overlooked. Users can be doctors, nurses, home care professionals, and even patients. Each user group brings varying levels of experience and sophistication, and may use the device in a different location and, potentially, for different purposes or in different ways. For example, functions that doctors or nurses have access to might not be necessary for home caregivers or, in a more extreme case, may be inappropriate or even dangerous for home caregivers to use. All of these variables must be thought out and studied, with the conclusions informing the implementation of design controls as well as the parameters of the validation tests to come.

Validation Testing

Where the formative evaluations are used to find the answers to the questions above, the sole purpose of the validation testing is to demonstrate that the device can be used safely and effectively by the intended users, for the intended uses, in the intended use environments.

Rigid scientific protocols are followed in a validation test, focusing on asking users to perform tasks in simulated environments that mirror real-life situations. While the patients are not real, the look and feel of the rooms are designed to mirror rooms where they will be used, simulating hospital rooms, intensive care units, or even a typical living room. Administrators of a validation test should be skilled researchers who are ever cognizant of the impact they can have on the disruption of a simulated environment.

With respect to the number of users who must be included in a validation test, the short answer is “it depends.” For many devices, the FDA advises 15 users per user group be included in a validation test to demonstrate that a device can be used safely and effectively by that user group. For infusion pumps, that number is 25. Questions typically arise when deciding what constitutes a user group. The answer to these questions should be uncovered during the formative evaluations and analysis of post-market surveillance data from predicate or similar devices. While there are some guidelines as to which populations tend to warrant their own user group (e.g., those under the age of 21 or over the age of 65), data-driven decisions should be made when deciding on the number and composition of user groups for a validation study.

In terms of the outcome required to “pass” a validation test, the FDA is really looking to ensure that any critical task failures observed during the validation test cannot be attributed to or remedied by the design of the interface. A root cause analysis of all task failures must be conducted, and if there are any failures that are not solely attributable to artifacts of the test environment, there may be cause for additional risk mitigation (e.g., implementation and validation of additional design controls).

Validation testing can also include training on the device when all users would receive the same training. In the formative evaluation stage, manufacturers investigate similar products and analyze predicate devices for the product in development, learning what may be different about this specific model when it comes to training.

In many cases, when training is administered to actual users in the field, there is a period of non-use that immediately follows the training. In these cases, users can suffer from training decay. This effect should be incorporated into the simulated use environment constructed for the validation test. The decay might be an afternoon, a day, or potentially even a week or longer. The point is that whatever decay duration is observed in the field during formative evaluations should be simulated to the extent reasonably possible during the validation test.

The actual tasks to be assessed during the validation test based two main factors: (1) potential for risk to the patient, and (2) the frequency with which the task is performed. Validation tests do not necessarily need to be comprehensive in assessing user performance on every single task that users might carry out with a device. However, any tasks that have the potential to cause harm to the patient or that are necessary to ensure the device can be used effectively should have been identified as critical tasks during the formative evaluations. It is imperative that the validation test critical includes these critical tasks.

Conclusion

By incorporating user feedback early and often, medical device manufacturers can increase the probability of a successful validation test. Ultimately, that leads to products that are safer and easier to use, that will produce fewer use errors, and that are better positioned to incorporate new technology and features in the future.

This article was written by Korey Johnson, Associate Director, GfK User Centric, Inc., Oakbrook Terrace, IL, a global user experience research and design firm. For more information, visit http:/info.hotims.com/45599-161.