Marianne, What Are You Doing Out Here?
A Message from Marianne McPherson, PhD, MS
NICHQ’s Director of Applied Research and Evaluation
At one point in Lewis Carroll’s Alice’s Adventures in Wonderland, the White Rabbit confuses Alice with his maid, Mary Ann, and asks her: “Why, Mary Ann, what are you doing out here?” My friends and colleagues sometimes ask me a similar question: “Why, Marianne, what are you doing out here?” What does it mean to be Director of Applied Research and Evaluation at NICHQ?
Here’s the official answer, per my bio on the NICHQ website: It means that I integrate quality improvement science and social science research methodologies—including program evaluation, health services research, and both qualitative and quantitative methods—to understand the implementation and effectiveness of NICHQ’s projects.
OK, and what does that mean? For help, I return to Lewis Caroll:
Alice to the Cheshire Cat: “Would you tell me, please, which way I ought to go from here?”
The Cat to Alice: "That depends a good deal on where you want to get to."
Where do we at NICHQ want to go? We are driven toward our mission: improve child health and well being by improving the systems responsible for the delivery of children’s healthcare. That’s bold. That’s big. So, how do we know if we’re getting there? It is my privilege and my challenge to help us figure that out.
An important guide for my work is not Lewis Caroll, but the Model for Improvement, which prompts us to ask three simple, but often elusive, questions:
1. What are we trying to accomplish?
2. How will we know if a change is an improvement?
3. What changes can we make that will result in improvement?
Here’s how I answer these questions and, hopefully, bring some clarity to the discipline of NICHQ’s applied research and evaluation efforts:
What are we trying to accomplish? What are our aims? We aim to build a system of learning both within and across NICHQ projects. We measure processes to improve how we implement our projects, and we measure outcomes to assess how well our projects are achieving their intended aims. We then continuously apply what we learn within a project (e.g., lessons from Phase 1 applied to Phase 2), as well as across projects (e.g., transferring learnings about optimizing the role of parent partners from one project to another).
How will we know if a change is an improvement? What measures should we establish? Our evaluation plans serve as the roadmap for our work. At NICHQ, evaluation is embedded within our projects. Our evaluation team applies both qualitative and quantitative methods, as we believe that understanding the “how” and “why” behind what we see (that qualitative methods can uncover) is as important as understanding “what” we see unfolding in projects. Understanding the unique strengths and techniques of improvement science—frequent measurement, small sample sizes—allows us to tailor our evaluation measures accordingly. There certainly are both strengths and challenges to having an internal project evaluation team. Some key strengths include our ability to provide nearly real-time feedback to project teams, to be sensitive to improvement science in our evaluation design, and to promote an “evaluation with” approach rather than an “evaluation of” approach.
What changes can we make that will result in improvement? How do we select changes to test? We work with project teams as they apply recommendations from evaluation to improve project implementation and progress towards intended aims. Program evaluation aims to inform decision making. That sets it apart from other types of research. As we design our research questions, our measures, and even the way we deliver our reports, we start by asking project teams about the decisions they are hoping to make based on what they learn. What does their project aim to accomplish, and what data will help them accomplish it? Along the way, what data can we gather that will help improvement teams, faculty, program staff, and project participants understand which way they ought to go from here, wherever “here” is in the life of the project?
“So, Marianne, what are you doing out here?” I am here to help our dedicated project teams determine where they ought to go from here. NICHQ’s work is improving the lives of providers and children and families across the country, and I’m honored to be part of this meaningful endeavor.