An Introduction to Measuring Reliability
Measuring reliability can improve the quality and value of our health care systems and quality improvement projects. Reliability measures how consistent the quality and safety of health care systems or processes perform over a required period of time. A highly reliable system has a lower risk of errors and process failures that can cause patients harm. Routine anesthesia, for example, is considered very reliable. It’s associated with very few errors and process failures, and there are seldom deaths resulting from those errors or failures.
Reliability matters in population health initiatives too. For example, a reliable developmental screening system would result in a consistent process for providing developmental screenings to young children and then, when necessary, referring them to interventions.
Ensuring reliable systems, whether at the hospital, state, or community level, requires measuring it. Measuring reliability enables us to understand and learn from variability in our systems and discover whether our improvements result in higher reliability.
Evaluating current processes
In order to measure reliability, teams must first observe and evaluate the system processes. In a developmental screening system, one process may be training health care professionals to conduct screenings during well-visits, and ensuring those screenings align with the American Academy of Pediatrics (APP) developmental milestones.
The Institute of Medicine (IOM) identified six dimensions for health care improvement. Use these aims to evaluate the effectiveness and reliability of each of your processes:
- Safety: Patients should not be harmed by the care that is intended to help them.
- Effective: Care should be based on scientific knowledge and offered to all who could benefit, and not to those not likely to benefit.
- Patient-Centered: Care should be respectful of and responsive to individual patient preferences, needs, and values.
- Timely: Waits and sometimes-harmful delays in care should be reduced both for those who receive care and those who give care.
- Efficient: Care should be given without wasting equipment, supplies, ideas, and energy.
- Equitable: Care should not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socio-economic status.
Focusing on processes rather than people, is important, says NICHQ Quality Improvement Advisor, Jane Taylor, EdD, MBA, MHA, “By focusing on processes, we avoid assigning blame. We shouldn't be looking for who's at fault for a failure because from the patient's perspective, it doesn't matter who's at fault. All that matters is what happened or didn’t happen. We must look at systems and process design and their reliability.”
How is reliability measured
The effectiveness of these processes and occurrence of failures is the key to measuring the reliability of your system. One way of measuring reliability involves comparing the number of actions that achieved the intended results to the total number of actions taken.
One example of a failure may be a roadblock in your processes that then prevents systems from achieving their intended outcomes. Referring to developmental screenings, you might discover that the current practices for screening children does not follow AAP recommendations on how often screening should take place. If the screenings are not done, then the referral isn’t made. The result of fewer screenings and referrals than recommended is that children miss opportunity for early interventions and optimal outcomes. Every missed screening would be considered a failure. In the developmental screening example, you would track how many screenings and referrals were made. . To measure reliability, you would compare the number of infants who received a screening against the number of infants due for a screening who did not receive a screening. This helps you determine your failure rate, and the higher the failure rate, the less reliable your system. For example, if 80% of infants are receiving the screening, then 20 out of 100 times a failure occurs, and the process is considered unreliable and chaotic.
Steps to increase reliability
To increase the reliability, it’s helpful to backtrack by identifying where a failure occurred, then analyzing each process and outcome that led to that failure. The first step is to collect data over time and display it in a run or control chart to understand the process or system performance, signaling when a system moves from stability to instability or vice versa. During this process, it’s important to pay attention to human factors and poor system and process design. The IOM dimensions may be useful in understanding the type of failure, such as a failure in safety, timeliness, etc.
Then the second step is to prevent future failure with tools like standard work, checklist, prompts, or memory aides. Develop a way to identify failures in real time and use error-proofing strategies to mitigate harm from the failure.
The third step is for critical process that cause serious harm or injury. It requires redesign of the process using human factors engineering and to design robust systems and processes that rarely fail.
You may find you need a project team to support improving reliability. In doing so, you can begin by identifying causes for failures and developing options to improve these processes, test the ideas, and track data to determine whether your changes are improving reliability. This strategy will help you create stronger processes, result in fewer failures, and create lasting change.
In this article, learn more about developing a successful measurement strategy.
NICHQ Employee Spotlight: Stacey C. Penny
With NICHQ's Rare As One Network Workstream Facilitation Initiative at a halfway point, Senior Project Director Stacey C. Penny, MSW, MPH shares an inside look at one of NICHQ's most collaborative projects.
Are Screens Making our Children’s Eyes Worse?
Even before the COVID-19 pandemic, children of all ages were spending more screen time than ever before on cellphones, tablets, and laptops. Prolonged periods of time staring at a screen that may be too big, too bright, or too close to our eyes can cause eye fatigue or even lead to dry eye, a chronic eye condition common in older adults. With eye conditions becoming more prominent in children, it's important for health professionals to encourage healthy screen viewing habits and support children’s eye health in the digital age.
NICHQ Employee Spotlight: Olivia Giordano
Olivia Giordano, MPH, Project Manager shares how her work with NICHQ’s Supporting Healthy Start Performance Project (SHSPP) is supporting 101 Healthy Start community sites to harness lessons learned, implement innovative approaches to improvement, and ultimately start to close the disparity gap in maternal and child health.
It Starts with Us and It Starts Now: Healing for Moms and Babies Begins with Ourselves and Our Systems
NICHQ CEO Scott D. Berns, MD, MPH, FAAP shares a message on healing and the ongoing need for equity-designed systems in 2021 and beyond.
Safe Sleep and Breastfeeding Initiative Invites Advocates to Join Communities of Practice
A multi-year initiative to improve infant safe sleep and breastfeeding is launching sector-specific Communities of Practice in 2021 to address policies, improve skills, and learn from other advocates’ experiences.
Top Equity Resources for Pursuing Change in 2021
Throughout this year, it has become clearer to many that the hard truth of the racism that shaped our nation requires us all—now, today—to acknowledge and address its impact in our health care systems and in ourselves. That's why we're sharing a collection of NICHQ articles and webinars that your community found most valuable in their equity journeys.