Measuring Quality - The Art and the Science
In a previous post I clarified the meaning of quality as being more than clincally competent care, but being much broader
True quality is the difference between being grateful for something and begrudging acceptance.1
Now this definitely covers the qualitative aspect of quality, but in order to satisfy the clinicians among us there is a quantitative aspect that must be considered.
What to measure?
Let's start with a very simple position that has lasted for centuries:
Measure what is measurable, and make measurable what is not so. Galileo Galilei
This is possibly one of the best quotes on the subject, simple and elegant, masking the depth of thinking!
In healthcare one of our greatest challenges when considering the measurement of quality is to create meaningful measure of incredibly complex systems, process and interactions.
Consider the example of surgical time out in the context of the WHO Surgical Safety Checklist. 2 How should we measure the effectiveness of this practice? Should we look at:
- The number of wrong site surgeries?
- The number of procedures where the process has been followed?
- The existence of a policy?
Each of these measure has its merits, but which would be most meaninful to the users of the data?
Checking for the existence of such a policy isn't going to tell us anything about its effectiveness. Wrong site surgery is already incredibly rare, so how long would we need to measure before we encountered a wrong site surgery?
It becomes clear then, in this circumstance that the most beneficial measure is one of compliance with the checking process. This is, of course, based on the premise that we believe that the evidence from the study proves the benefit.
Structure, process or outcome?
Donebedian considered that quality could be measured in terms of structure, process and outcome.3
Structure - This realm denotes the attributes of the setting in which care occurs and includes material and human resources as well as organisational structure
Process - Process refers to what is actually done in giving and receiving care
Outcome - The effect of care on the health status of the patient or population
Donabedian (1997)
When deciding what is the best way to measure quality and safety, allocating a realm can help ensure the measure is a valid and useful one.
Who is the data for?
Would you present surgical site infection rates the same way when presenting to clinicians as you would patients or communities? Does your board or governance committee want the data in the same format?
Pronovost argues that data presented for the wrong audience is of little value and in the worst cases, damages credibility and participation.
Early, robust discussion with the consumers of collected data will ensure positive engagement with the quality activity.
Imagine you are trying to measure on time starts in the operating theatre, what do we consider to be the actual start time?
- When the patient arrives in the theatre?
- When anaesthesia commences?
- When the patient is anaesthetised and ready for surgey?
- When the drapes go on?
- When the incision is made?
The answer is, in fact, it doesn't matter! What matters is that the consumers of the data agree. This avoids the semantic debate later if a team or individual needs to be challenged on their performance.
Presenting the data
I will spend some time in future posts looking at how the data should be presented, but as a simple set of priciples consider the following
Enough already with the 3D effects! Making your bars 3D with shadows and shading contributes nothing to understandability and in many cases makes the results more difficult to interpret
If it doesn't aid understandability, get rid of it! Do you need all those gridlines?
Present the raw data wherever possible
Ensure you share numerator and demoninator when presenting percentages. Without context, impressive numbers may be misleading
Be cautious with interpretation, correlation and causation are not the same thing - as shown below in a classic piece of misdirection showing (tongue in cheek) that global warming is caused by the decline in pirate numbers!
If you want to gain a better understanding of statistics I highly recommend this free online course https://www.udacity.com/course/st101
1 Feckie, M The modern quality movement https://mfeckie.dev/modern-quality-movement 2013
2 Haynes A, Weiser T, Berry W, Lipsitz S, Breizat A, Dellinger E, et al. A surgical safety checklist to reduce morbidity and mortality in a global population. New England Journal of Medicine. 2009;360(5):491-9.
3 Donabedian A. The quality of care. How can it be assessed? Archives of pathology & laboratory medicine. 1997;121(11):1145-50.
4 Pronovost P, Nolan T, Zeger S, Miller M, Rubin H. How can clinicians measure safety and quality in acute care? Lancet. 2004;363(9414):1061-7.