Quality of care

Quality of health services is variable everywhere, but is especially low in lower- and middle-income countries [8,10-15]. Through the World Bank’s Service Delivery Indicator (SDI) Surveys carried out in various Sub-Saharan African countries, the poor state of delivering quality health services is apparent (Table 1).

An early established global framework for understanding quality of health services is that of Donabedian in which a distinction is made between structural, process and outcome measures of quality [16]. Structural measures are the inputs necessary for providing quality services such as equipment, drugs and trained providers; process measures are the actions and activities of health providers, and outcomes are the results of medical action such as patient satisfaction, improved health, disability or death [16]. Process measures of quality, ie, what happens between the patient and the provider also referred to as provider effort are commonly associated more closely with health outcomes [8,9,17]. Frequently, structural and process measures of quality are present conjointly to lead to desired outcomes [18-20]. As an example, in the case of treating pulmonary tuberculosis: tuberculosis drug availability (structural element) is closely linked to making the right diagnosis, the right explanations to the patient, and the delivery of effective direct observed treatment (process elements) and should all be present to yield the highest possible chance for successful cure (outcome).

The strength of this framework, for example in the case of moderately severe diarrhea in a young child, is that appropriate care can be effectively “provided under a palm tree” by a “knowledgeable person” (structural) if that provider makes the right diagnosis, and provides correct advice using integrated management of childhood illnesses (process). A unilateral focus on structural quality measures leads to measuring the wrong things [8,17,21]. Instead, we will consider the systemic nature of health systems and by extension health service delivery when considering interventions that influence the delivery of quality service. PBF is an illustration of a systemic intervention designed to work towards Universal Health Coverage, not merely by addressing volume, but influencing the structures of quality health service delivery [22-26].

Performance-based financing and strategic purchasing

PBF is a health financing approach that originated in Cambodia in the late 1990s [27-32] and evolved to its current form through various iterations in countries like Democratic Republic of the Congo (DRC), Rwanda and Burundi [24]. In Rwanda, after a series of successful PBF pilot programs [33-37], PBF was scaled up through a mix of government and development partner financing and operational efforts. A rigorous impact evaluation [38] showed significant results on volume and quality of maternal and child health services [39] and HIV services [40]. The PBF approach which evolved between 2005-2009 in Rwanda was further developed in Burundi [41-44] and other countries such as DRC [45], Cameroon and Nigeria [24].

Key features of these PBF approaches are (a) defining basic and complementary health packages for strategic purchasing; (b) strategic purchasing adjusted through measures of service quality; (c) purchasing from public, private and quasi-public providers; (d) involvement of national and sub-national health administration in stewardship, supervision, quality measurements and other technical and managerial supportive tasks; (e) community client satisfaction surveys to strengthen community voice; (f) rigorous internal and external verification mechanisms; (g) creation of fiscal space through linking to national health financing strategies, since 2015 also assisted through the Global Financing Facility [46].

PBF approaches are systemic approaches [22], and can best be conceptualized as a constellation of leveraging, health system strengthening and accountability mechanisms. These approaches work best when embedded in overall health reforms in which various system pillars are simultaneously addressed such as human resources, health information systems, the pharmaceutical sector and fiscal space for health (Table 2) [23,25,26,47,48].

PBF approaches have expanded and reached scale in many countries (eg, in Sub-Saharan Africa (SSA), Central and South-East Asia and the Caribbean) making it possible to look at the impact of strategic purchasing for UHC and any novel or promising quality interventions. The map below (Figure 1) shows the rapid expansion of PBF approaches in SSA. Some programs such as the Nigerian State Health Investment Project or the Health System Development Project in DRC, for example, cover significant populations; 33 million and 22 million respectively. Cameroon is slated to scale up PBF nationwide during 2019 to cover a population of 24 million. A project in Kyrgyzstan covers all 63 Rayon (district) hospitals, and a project in Cambodia, a country with 16 million, will during 2018 cover all public health centers and hospitals, including the sub-national health administration.

Figure 1.  Rapid expansion of performance-based financing programs (PBF) in SSA 2006-2015.

PBF and quality of care

Quality adjusted strategic purchasing of essential health services is an essential aspect of PBF approaches. Since the start, PBF approaches have included the use of a quantified quality checklist applied quarterly to each contracted health facility [24]. Such checklists differ depending on level (health center/community vs hospital) and context, and typically involve over 120 data elements [21]. They have proved popular both because it is easier to administer and verify data related to infrastructure and supplies and as the initial startup funds are often tied to making improvements in these structural elements.

Indeed, health facilities in poor countries, due to chronic underfunding, lack many structural quality features that are routine for health facilities in countries with more resources. For instance, if a service availability and readiness assessment in DRC shows that only 13% of public and private health facilities have magnesium sulphate (an essential drug for managing morbidity and mortality due to pregnancy related eclampsia) or it shows that less than 12% of health facilities can provide basic emergency obstetric care, we clearly have a problem [49]. However, this problem becomes more complex when next to service availability and readiness we consider provider knowledge and competency. In SSA these are measured through SDI surveys, which are done in SSA (Table 1). In addition to significant gaps in structural quality (as an extreme example, 46.7% of providers are absent in Uganda), such surveys show a huge variation in provider knowledge and competency. For instance, on average in nine SSA countries only 27.4% of providers adhere to guidelines for neonatal and maternal health complications, 35.9% adhere to clinical guidelines for diarrhea, pneumonia and malaria while diagnostic accuracy for these conditions was only 50%. Studies in India show a similarly dire picture [8,13,15]. Here is a general recognition that most checklists are focused upon structural factors at the expense of process measures and accordingly, the various quality checklists have increased weights for process measures of quality. In Nigeria, for example, process elements form 30% of the weight (103 out of 345 points available) at health center level and 41% of the weight at hospital level (240 out of 587 points available) of the overall quality checklist. These process measures are mostly based on file or record reviews and sometimes on observations. They are known to be difficult to ascertain or to counter-verify. For instance, in maternal medicine, a commonly used process measure, “well-filled partogram”, is frequently filled after delivery when missing partogram elements can be filled by knowledgeable providers before a verification would take place [50]. In a similar vein, criterion-based medical audits - although perhaps working in different contexts [51] - also break down when used in PBF approaches due to a similar propensity for gaming the right answer instead of recording the actual practice.

Integrating further quality measures into PBF

Below, we reflect on a selection within the armamentarium of methods to measure and document quality processes and outcomes and options to integrate these methods in such a manner that they are effective along three dimensions: (a) efficient; can be added to an existing PBF program at minor incremental cost; (b) responsive; able to document an improvement or lead to an advancement on the previous state of affairs, and (c) scalable; can be done at the population level.

The operationalization of these schemes needs to be undertaken in such a way that the schemes can really strengthen indices for health care quality and work in the environment of a PBF operation, and validated by regular and rigorous assessments (backed by third-party verifications), performance feedback, elements of supportive supervision and public benchmarking. We postulate that PBF environments offer a fertile context for behavior change as they mix various behavioral triggers to practice improvement [8,14,52,53].

We looked at five methods used for assessing medical process quality: (1) chart review, (2) vignettes, (3) direct observations, (4) mystery patients, and (5) exit interviews [8]. Chart reviews abstract practice documentation from the patient’s chart using explicit evidence based criteria. Vignettes are standardized medical cases, where providers care for the same patients to facilitate comparison of practices; they are done in various ways from full cases akin to role-play, or as a short answer, effecting a written exam. Mystery patients are trained actors who present unannounced with certain symptoms and are trained to observe and report on medical actions performed on them. Finally, patients can be interviewed on what has been done to them when leaving a consultation room. These methods are variably used for research and for audits, for accreditation or teaching. Considering a pre-existing PBF program, efficiency, responsiveness and scalability, the following options to measure process and outcome quality are available in Table 3.

The methods started to be used in the four countries of DRC, ROC, Kyrgyzstan, and Cambodia are: (i) Vignettes; (ii) Direct Observation and (iii) Client Satisfaction Surveys. Below, we describe how these methods are being applied in these four country contexts.

Direct observation in Cambodia and Kyrgyzstan

In Cambodia and Kyrgyzstan, direct observation, modified to a competency test, using the MamaNatalie and NeoNatalie tools, is also used. Evaluation of provider performance on these tools is performed by District and Provincial supervisors who have been trained and certified to carry out these assessments. MamaNatalie is used as a competency test for post-partum hemorrhage, one of the leading causes of maternal mortality, and the NeoNatalie is used for a competency test for neonatal resuscitation, which has been shown to decrease neonatal mortality by 30% [63]. Both competency tests have been quantified and weighted to score 100 points when 100% correctly executed. Both vignettes and competency tests are weighted a combined 30% in the hospital balanced score card (plus a 30% valuation on file reviews) and 60% in the health center balanced score card.

In Table 4, adjustments to these methods are described to make the methods dovetail with PBF approaches:

Patient satisfaction surveys

Patient satisfaction surveys are being conducted in Kyrgyzstan. A systematic random sampling of the inpatient register of women who delivered at the hospital is used. Ten randomly selected patients from each of the 63 district hospitals are surveyed once per quarter. The patients are contacted through mobile phone by trained supervisors, and after obtaining consent are asked five questions. Four of the five questions have responses on a five-point Likert scale, while one final question is binary seeking to clarify whether the patient paid providers informally. If the last question is answered ‘yes’, then it voids the points for the entire interview. The ten interviews are weighted 10% of the total quarterly performance score for each hospital.

In Cambodia, DRC and ROC, the interview tool contains 15 questions, all quantified and on various scales, and weighted differentially. The tool contains content-of-care tracers such as questions related to taking blood pressure or temperature and whether the provider touched the belly of the patient or listened to the lungs. In Cambodia, patients are sampled from registers and called directly by the supervisors; while in DRC and ROC, larger community client satisfaction surveys are carried out by grassroots organizations, which validate the patient-provider contact (to deter phantom patients), in addition to running the patient experience questionnaire. In all three contexts, these surveys are weighted 10% of the total quality score.

Box 1, Box 2, Box 3 and Box 4 give short descriptions of the PBF programs in Kyrgyzstan, Cambodia and the two Congo’s.

Figure 2.  Kyrgyzstan 3-year balanced score card (BSC) results in the three impact evaluation groups.

Figure 3.  Republic of Congo: Quality Index in Health Centers increased from 43% to 62% over a one-year period.

Figure 4.  : Cambodia results from the first two rounds of ex-ante verification Health Center, Hospital and Health Administration performance (quarter 2, 2017; first wave of scaling up).

Figure 5.  Democratic Republic of Congo. Lowest, Average and Maximum Quality Index Scores at Baseline for health centers in the six health districts of Kwango Province, Q3, 2016.

Micro level

At the micro level, there is the conceptual challenge of the know-do gap, with design and implementation of these novel quality tools in general, and design and implementation of PBF approaches specifically. The know-do gap [8,38], is an observed gap between what providers know and what they do in practice with that knowledge [38. This know-do gap is a problem in knowledge-based tests. For example, providers with a high score on the vignettes and competency tests might systematically skimp on actual care. We know very little about this phenomenon in PBF contexts. However, a rigorous impact evaluation in Rwanda showed that under PBF knowledgeable providers did more of what they knew [39,68]. A study in China on tuberculosis diagnoses found that providers knew much more as tested through vignettes, than they did as tested through mystery patients. Although this study was not in a PBF context, incentives linked to prescribing antibiotics and other drugs might have had an influence on the discrepancies [61,62]. A well-designed randomized controlled study in the Philippines showed that doctors who did better on a comprehensive set of vignettes designed so that they were difficult to game, improved their care and these improvements were reflected in better health outcomes [14].

There are a few possible ways to mitigate this challenge of the know-do gap with vignettes. First, direct feedback to the providers on their vignette results (individual learning element), and provider effort is helpful for learning. This is enhanced by structured formative supervision, a key component of PBF approaches. Second, the change in provider payment mechanism (moving to a case-based payment for curative care instead of an itemized billing) aligns incentives to prescribe drugs or initiate treatment more rationally, ie, adhere to protocols (taught through the vignettes). Third, perceived enhanced quality by patients (observed structural elements; more time spent in consultation) leads to a positive virtuous cycle [69]. Fourth, observation with positive feedback, and benchmarking of health facility performance (individual health worker performance is not made public) are all strong enablers for enhancing performance. And finally, linking health facility performance payments to these novel quality measures are intuitively appealing as they counter some of the negative effects of routine vignette applications [61,70,71]. Ultimately, the know-do gap must be measured not by gaps in performance but by whether serial measurement and feedback leads to improvement as measured by health outcomes.

While scalable in practice, the design and implementation of these novel quality tools holds another set of challenges [24,26,47,72]. Field observations from Kyrgyzstan and Cambodia have taught us that continued high quality mentoring and support are necessary pre-conditions to measurement implementation. Investments need to be made in creating capacity of a group of national trainers who will be able to continue implementing and developing these measurement approaches. With training and measurement capacity, a necessary ‘bottom-up’ design and approach can be used to strengthen local acceptance, gather inputs, and ensure adaptation of quality measurement tools. A period of instrument validation may be required to ensure validity within the local context [18]. A further implementation challenge for these approaches is the use of local administrative structures for the assessments. Whereas engaging the local health administration is a powerful strategy for strengthening local health systems, using PBF approaches necessitates very good counter-verification measures to make all adhere to the new rules of the game.

Macro level

At the macro level, there are political challenges. Countries that do not invest in PBF or do not create a line item for PBF and quality measurements in the Ministry of Finance budget, pose a sustainability risk to these reforms. Successful reforms such as in Rwanda and Burundi were brought about by Governments that planned for a budget to finance PBF. Leadership changes pose further challenges to sustainability. A new team might have different priorities and could derail these reforms. Also, development partner buy-in is crucial in many contexts. Other macro challenges to PBF implementation come from those that stand to lose out due to these reforms. For instance, decentralizing drug procurement and decentralizing budgets to health facilities all lead to lesser income for those that previously used to manage these resources at higher levels.

Finally, in many countries weak clinical training is the root cause of poor provider performance. Implementing novel quality measurement and enhancement methods will certainly be beneficial, however, it will not be the lone solution. Deeper reforms of the pre-service medical training curricula will be required. For example, once the habit of measurement and feedback has been established, these could also be applied to the pre-service curricula.