An Introduction to Measuring Reliability
Measuring reliability can improve the quality and value of our health care systems and quality improvement projects. Reliability measures how consistent the quality and safety of health care systems or processes perform over a required period of time. A highly reliable system has a lower risk of errors and process failures that can cause patients harm. Routine anesthesia, for example, is considered very reliable. It’s associated with very few errors and process failures, and there are seldom deaths resulting from those errors or failures.
Reliability matters in population health initiatives too. For example, a reliable developmental screening system would result in a consistent process for providing developmental screenings to young children and then, when necessary, referring them to interventions.
Ensuring reliable systems, whether at the hospital, state, or community level, requires measuring it. Measuring reliability enables us to understand and learn from variability in our systems and discover whether our improvements result in higher reliability.
Evaluating current processes
In order to measure reliability, teams must first observe and evaluate the system processes. In a developmental screening system, one process may be training health care professionals to conduct screenings during well-visits, and ensuring those screenings align with the American Academy of Pediatrics (APP) developmental milestones.
The Institute of Medicine (IOM) identified six dimensions for health care improvement. Use these aims to evaluate the effectiveness and reliability of each of your processes:
- Safety: Patients should not be harmed by the care that is intended to help them.
- Effective: Care should be based on scientific knowledge and offered to all who could benefit, and not to those not likely to benefit.
- Patient-Centered: Care should be respectful of and responsive to individual patient preferences, needs, and values.
- Timely: Waits and sometimes-harmful delays in care should be reduced both for those who receive care and those who give care.
- Efficient: Care should be given without wasting equipment, supplies, ideas, and energy.
- Equitable: Care should not vary in quality because of personal characteristics such as gender, ethnicity, geographic location, and socio-economic status.
Focusing on processes rather than people, is important, says NICHQ Quality Improvement Advisor, Jane Taylor, EdD, MBA, MHA, “By focusing on processes, we avoid assigning blame. We shouldn't be looking for who's at fault for a failure because from the patient's perspective, it doesn't matter who's at fault. All that matters is what happened or didn’t happen. We must look at systems and process design and their reliability.”
How is reliability measured
The effectiveness of these processes and occurrence of failures is the key to measuring the reliability of your system. One way of measuring reliability involves comparing the number of actions that achieved the intended results to the total number of actions taken.
One example of a failure may be a roadblock in your processes that then prevents systems from achieving their intended outcomes. Referring to developmental screenings, you might discover that the current practices for screening children does not follow AAP recommendations on how often screening should take place. If the screenings are not done, then the referral isn’t made. The result of fewer screenings and referrals than recommended is that children miss opportunity for early interventions and optimal outcomes. Every missed screening would be considered a failure. In the developmental screening example, you would track how many screenings and referrals were made. . To measure reliability, you would compare the number of infants who received a screening against the number of infants due for a screening who did not receive a screening. This helps you determine your failure rate, and the higher the failure rate, the less reliable your system. For example, if 80% of infants are receiving the screening, then 20 out of 100 times a failure occurs, and the process is considered unreliable and chaotic.
Steps to increase reliability
To increase the reliability, it’s helpful to backtrack by identifying where a failure occurred, then analyzing each process and outcome that led to that failure. The first step is to collect data over time and display it in a run or control chart to understand the process or system performance, signaling when a system moves from stability to instability or vice versa. During this process, it’s important to pay attention to human factors and poor system and process design. The IOM dimensions may be useful in understanding the type of failure, such as a failure in safety, timeliness, etc.
Then the second step is to prevent future failure with tools like standard work, checklist, prompts, or memory aides. Develop a way to identify failures in real time and use error-proofing strategies to mitigate harm from the failure.
The third step is for critical process that cause serious harm or injury. It requires redesign of the process using human factors engineering and to design robust systems and processes that rarely fail.
You may find you need a project team to support improving reliability. In doing so, you can begin by identifying causes for failures and developing options to improve these processes, test the ideas, and track data to determine whether your changes are improving reliability. This strategy will help you create stronger processes, result in fewer failures, and create lasting change.
In this article, learn more about developing a successful measurement strategy.
Continuing Sickle Cell Disease Care During the COVID-19 Pandemic
People who have sickle cell disease and very vulnerable to COVID-19 impact. As we enter the fourth month of the pandemic in the U.S., we’re sharing how providers and advocates have remained nimble to ensure that comprehensive and consistent care is maintained throughout the pandemic and beyond.
Making Fathers Visible in Maternal and Child Health
From cognitive and social emotional development to education and accomplishments, children with involved fathers achieve better health outcomes. Yet despite fathers’ positive impact on maternal and child health, many of the systems intended to serve women and children were not designed with fathers in mind. That’s why we’re sharing strategies to increasing father involvement in early childhood programs.
NICHQ Employee Spotlight: Barbara Lambiaso
Each month, we’re shining a spotlight on a NICHQ employee, asking them to share their memories, advice, and goals. Project Manager Barbara Lambiaso shares her commitment to various NICHQ projects, including an initiative that serves children and adults with sickle cell disease.
Ensuring Flexible Evaluation as Project Needs Change
Have you ever had to update your program implementation plan part way through? We’re sharing four strategies on creating flexible evaluation processes for greater program success.
Communities in Ohio Address Rising Inequalities in Response to COVID-19
A serious compounding problem of COVID-19 is how it is intensifying inequalities across the country, including in Ohio where significant disparities in maternal and child health persist. Learn how Mahoning County is proactively identifying how the pandemic will affect at-risk populations and then developing a comprehensive plan that bring together partners from across the health system.
NICHQ Employee Spotlight: Judith Gooding
Each month, we’re shining a spotlight on a NICHQ employee, asking them to share their memories, advice, and goals. Here, Chief Operating Officer Judith Gooding shares her pride in NICHQ and her appreciation for her colleagues’ nimbleness in the face of COVID-19