Program description

cStock is an SMS and web–based reporting and resupply system that is used by HSAs to report stock data via SMS through their personal mobile phones. cStock calculates HSA resupply quantities and sends this information to HF staff to use to pick and pack products for HSAs and notify them about a collection time.

cStock was designed using a consultative, user–centered and iterative process. Potential users at all levels of the health system provided inputs based on their experience with the existing manual reporting and resupply system. Information on the existing flow of data across levels of the health system was combined with inputs from supply chain specialists to ensure the system was based on supply chain best practices. In designing the workflows and dashboard for cStock, an important criterion was to ensure the health care workers and managers at each level would have access to data most relevant to them, at the right time and in a format that could be easily accessed, interpreted and used for decision making. HSAs and HF staff would interact with cStock using SMS messages on their own phones, while district and central level staff would receive alert messages from cStock on their own phones. District and central level staff would use computers to access the web–based dashboard for reports. The dashboard was redesigned six months after implementation so that district and central level users could incorporate their experience interacting with the system into the redesign and prioritize metrics and visuals most useful for their day–to–day operations.

cStock is a key component of both the EM and EPT intervention packages. The EM intervention addresses challenges related to data availability and visibility, as well as low motivation among HSAs while the EPT intervention addresses challenges of transport in addition to data visibility. The additional component of the EM intervention was District Product Availability Teams (DPATs). These are multilevel quality improvement teams that use data supplied by cStock to monitor performance of the supply chain and make informed supply chain decisions. In contrast, the additional component of the EPT intervention consisted of training all HSAs on bicycle maintenance, provision of a basic tool kit, and the use of a continuous review inventory control system.

HSAs and HF staff in six districts where the project was piloted were trained on the use of cStock for reporting and resupply and used their own phones to register with cStock. Training was conducted using a cascade approach. The project trained a group of trainers consisting of central and district level staff from pilot districts as TOTs and they in turn trained the HSAs, HF and district staff. Both intervention groups were trained over a 2–day period, with one day dedicated to training users on how to register and use cStock. HSAs in the EM group were trained to send reports to cStock using a fixed monthly reporting schedule between the last day of the month and second day of the next month, while those in the EPT group were trained to send reports to cStock at any time during the month when they planned to travel to the HF giving them the flexibility to report and receive product. Once trained, HSAs were provided with job aids to carry back to their facilities and expected to start using cStock immediately. Implementation support for cStock was provided through group messages sent by the system administrator to users to correct common errors, automated messages sent directly by the system to users in response to formatting errors, and field visits made by Ministry and project staff to monitor and reinforce good practices.

In designing the interventions, the project identified feasibility and acceptability of cStock as critical components to enhancing uptake and laying the foundation for country ownership and scalability. See Figure 1 for a visual representation of the intervention design.

Figure 1.  Components of the two intervention groups (EM – Enhanced Management and EPT – Efficient Product Transport). DPAT – district product availability teams.

Study site

Malawi is a landlocked country that covers an area of 118 484 km² and shares boundaries with Zambia in the west, Mozambique in the east, south and southwest, and Tanzania in the north. The country is divided into three administrative regions, namely the northern, central and southern regions with 28 districts in total [8]. The study purposefully selected 10 out of Malawi’s 28 districts for the 2010 baseline assessment, in consultation with stakeholders. Selection criteria for the districts included the existence of a functioning iCCM program, a balance of iCCM partner support, and a relatively balanced geographical coverage across the three administrative regions of the country.

Evaluation framework

To evaluate the effectiveness of the interventions, the project designed a feasibility and acceptability framework ( Figure 2 ) that identifies key domain areas as important measures for predicting the likelihood of long–term effectiveness, scalability and ownership of an intervention.

Figure 2.  Feasibility, acceptability and effectiveness framework.

In this framework a feasible intervention is one that seeks to address a defined problem and that provides a viable solution or platform for solving the problems within existing health system structures and staff capacities in a country. The solution should be easy to learn, quick and easy to use, with few or no difficulties experienced by the user. For this evaluation the measures of feasibility are:

• Improved reporting rates, and

• HSAs and HFs have the necessary skills and ability to use cStock.

An acceptable intervention is one that users perceive as valid, reliable, and beneficial; users are satisfied that the intervention meets their needs by helping them adequately solve a problem or make improvements in their daily work. Acceptability is achieved when the majority of users use the intervention on a regular basis for supporting routine tasks. For this evaluation the measures of acceptability are:

• cStock has become the primary means for HSAs to order or request health products from their resupply point,

• cStock has become the primary tool for HF staff to use for resupplying HSAs, and

• Teams use data from cStock to measure, monitor, and improve supply chain performance.

An effective intervention is one that achieves the intended or desired outcome. In this study the desired outcomes across the two intervention groups are improvements in data visibility, responsiveness and supply reliability. For this evaluation effectiveness is measured and compared across the EM and EPT groups as:

• Improved data visibility measured as improved reporting rates and complete reporting rates,

• Improved responsiveness measured as improved lead times, and

• Improved supply reliability measured as reductions in stock out rates.

Assessment tools and study groups

The project employed mixed methods to conduct baseline and follow up assessments in May 2010 and February 2013, respectively. Permissions for the assessments were obtained from the MOH. The quantitative survey tool was adapted from the Logistics Indicator Assessment Tool (LIAT) originally developed by the USAID | DELIVER PROJECT [9], including questionnaires, inventory assessment forms, and key informant interview guides. The survey was tailored to the district, HF and community level, to capture processes, behaviors, and product availability data along each step in the supply chain and to measure indicators of intervention implementation.

A local evaluation partner, the Malaria Alert Center (MAC), was selected through a competitive process to lead data collection activities. For quantitative data collection, enumerators were trained to interview HSAs and other staff managing supplies of medicines. Data collectors used Nokia e71 and e63 smart phones loaded with Data Dyne’s Magpi application, which allowed for streamlined data entry and immediate review of data after uploading records to a web–based system. Data were saved on a daily basis to external memory cards to prevent loss. Frequencies were carried out using STATA version 11.

In addition to the cross–sectional data collection at baseline and follow up, focus group discussions (FGDs) were carried out in February 2013 (at follow up) by a team of two researchers from MAC, with HSAs and health facility staff in all six pilot districts. Two FGDs per district, with 6–10 participants each, were held with staff from several health facilities. In each district, one FGD included only HSAs (one male and one female per HF), and the other included only HF staff handling CCM products (HSA Supervisors, Drug Store In–Charge / HF In–Charge). A thematic guide was used to collect qualitative information around community supply chain data visibility and use, with particular focus on contributions made by the cStock system. For the EM group, FGDs also gathered views on the EM intervention and achievements made by the DPATs, as well as lessons learned about the intervention. The data collection team transcribed notes immediately after discussions, and qualitative results were thematically synthesized using a notes–based analysis.

Routine monitoring data on HSA drug stock level reports, submitted using cStock, were utilized to study supply chain (SC) performance trends over time between the EM and EPT groups. These data were retrieved for the 18–month period from January 2012 to June 2013. The web–based cStock dashboard provided reports showing monthly stock reporting rates, average time taken between sending an order request and receiving health products (lead time), product availability, and stock–outs for these time periods for the six intervention districts (three for EM and three for EPT). The average values of these key indicators were calculated for the 18–month period and paired Student’s t–tests were conducted to compare the trends between the EM and EPT groups. Table 1 presents an overview of the various data sources and frequency of collection.

HSAs were the primary unit of survey analysis. This paper presents two sets of samples, one from the baseline and follow up surveys, the second from backend data collected via cStock where the sample is comprised of all HSAs who ever registered on cStock at any point in the program cycle. 56 HSAs from the EM districts, and 44 HSAs from the EPT districts were sampled at baseline, 81 HSAs from EM districts and 78 HSAs from the EPT districts were sampled at follow up. The cStock sample of HSAs registered in each group as of June 2014, is n = 393 HSAs in EM, and n = 253 HSAs in EPT (total n = 646).

Sampling strategy

After the formative assessment, the project formed three groups of three, three, and four districts from the ten districts by matching geographical and demographic characteristics, and other external dimensions including iCCM partner coverage, prevalence of diarrhea, malaria, and cough, as well as baseline HSA iCCM product availability, to create comparable groups. The three groups were randomly assigned to receive the EPT intervention (three districts), the EM intervention (three districts) and no intervention (four districts). This paper presents results from the six intervention districts where the project implemented EM and EPT, and thus where cStock was tested.

As the project chose 10 districts purposefully for evaluation, a representative sample of HFs was chosen from within the 10 districts at baseline and follow up. Probability proportional to size sampling was used to randomly select health centers based on number of associated village clinics (sites where HSAs work), and approximately three HSAs who manage health products were randomly selected per health center and visited at their village clinic.

Feasibility of cStock for users

The feasibility of cStock was evaluated by looking at staff capacity to use cStock, practicability, and relevance. Follow up findings (February 2013) demonstrated an improvement in reporting of community–level stock data using the cStock system, with 85% of HSAs in six districts sending reports to cStock. In comparison, community data was much less visible at baseline, with only 61% of HF staff across six districts (n = 51) reporting HSA supply chain data up to district level, and only 29% of those HFs reporting this data disaggregated from HF data. Table 2 presents a summary of feasibility findings from the follow up survey.

A majority of users at all levels of the system found the cStock technology easy to learn, with 80% of HSAs and 92% of HF staff in the EM group, and 71% of HSAs and 91% of HF staff in EPT group able to send and receive messages without any challenges. HSAs also noted that cStock is easy to use, with the majority able to “type in the SMS” without any challenges. District staff in all three EM districts and two of three in EPT districts reported feeling comfortable accessing and navigating the dashboard. Minor use challenges reported by HSAs and HF staff were mainly infrastructural in nature, related to poor network penetration and lack of charger or a place to charge phones.

FGD findings highlight the practicability of cStock as shown by the convenience of the system in enabling HSAs to send data on stock levels at any time, and corroborate the survey findings where majority (86%) of HSAs were able to compile and send stock data within an hour. One HSA from the EM group explains: “… [cStock] is also good because we are able to send the messages at any time even in the night and we get the response in time.”

FGD findings related to the theme of relevance to users include HSAs description of cStock and its components as a fast system for sending messages and receiving feedback. An HSA Supervisor in the EPT group offered: “Yes, [cStock is] very useful because communication is very fast between the HSA and the In–Charge, whenever they want emergency order they receive feedback right away from the health facility.”

HSAs and HF staff also perceived cStock as a linkage that facilitates communication between levels from the villages to the HSAs, HSAs to the HF. One HSA Supervisor in the EPT group offered, “… [cStock] is a linkage between the HSAs and the supervisors, even the In–Charge because when the products are ready we just send the message to let them know that the drugs are ready. While before it was hard to reach every HSA.” Users also noted that the cStock system helps users remember to take necessary actions, for example one HSA in the EPT group stated, “cStock is good because it reminds you what to do, like if you have one day left to send the report it always reminds you that you have one day left.”

Acceptability of cStock

The acceptability of cStock was evaluated by looking at its level of routine use, its effect on users’ daily work, and perceived benefits identified by the user. Data for the six intervention districts, analyzed over 18 months from Jan 2102 – June 2013, showed a steady increase (79% to 99% in EM and 71% to 90% in EPT) in routine use of cStock to report on stock levels averaging at 94% for EM, 79% for EPT. Follow up findings also showed that a large majority (97% in EM and 91% in EPT) of HSAs reported that cStock had become their primary means for ordering or requesting health products from their resupply point. Additionally, 92% of health facility Drug Store In–Charges in EM and 91% in EPT reported using cStock to determine the quantities to resupply to HSAs. Table 3 presents a summary of acceptability findings from the follow up survey.

In addition to routine orders, HSAs also reported using cStock as the primary means for submitting emergency orders. While at baseline, 52% of HSAs across six districts (n = 98) reported travelling to the HFs to submit emergency orders, this percentage dropped to 28% in EM and 21% in EPT at follow up, with the majority (94% in EM, 88% in EPT) of HSAs now using cStock to send emergency orders. All HSAs (100% in EM, 99% in EPT) also reported using cStock routinely to send receipt messages upon collecting their products. Most (96% in EM and 87% in EPT) of the HF staff interviewed reported always using cStock to inform HSAs of orders ready for collection; 44% of HF staff in EM and 30% in EPT used it for follow–up on non– and incomplete reporting, both critical SC performance indicators, particularly for the DPATs.

FGD findings show that cStock contributed to improved accountability for managing products, as HSAs, HFs, and district staff were all responsible for and played roles in reporting and communicating stock–related issues. One positive result of this is reduced product wastage; an HSA Supervisor from the EM group explained “There is more transparency now as drugs are not given on a friendly basis, when you find products are ready, we count with the specific HSA for accountability...”

According to FGD participants, cStock is an important source of feedback and information. HSAs perceive monthly cStock reminders about reporting as useful, and HF staff noted that cStock alerts are beneficial in notifying them of late reporting from HSAs. An HSA in the EPT group shared “l feel very happy whenever l am sending SMS to cStock because l receive feedback right away and because of this I am motivated.” An HSA Supervisor from EPT adds, “Yes, it’s very useful because communication is very fast between the HSA and the In–Charge, whenever they want emergency order they receive feedback right away from the health facility.” cStock provides useful information for monitoring SC performance. HF staff reported that cStock enables supervisors to monitor HSA performance and supervisors’ own performance through self–assessment. In the EM group, a HSA Supervisor offered “…we look at their reporting rate, the time they sent their report, the completeness of the report and supervisor notes that, on cStock helps us to monitor performance as the information is easily available.”

FGD participants felt that cStock increases the overall efficiency of the community supply chain system. HSAs reported that cStock has resulted in a marked reduction in the significant effort, time, and/or money they had previously spent in collecting products from the HF. One HSA in the EM group shared, “as my friends have said, transport costs have been reduced significantly and that is good for us.” HF staff reported being able to manage inflow of HSAs to the HF to collect products by picking and packing in advance as well as sending notifications to HSAs that their order is ready thus serving them better. cStock also increases staff ability to monitor and communicate information on product usage. HSAs reported that cStock allows tracking product distribution and use, both at the HSA Supervisor level and higher. An HSA from the EPT group added, “… there is transparency on cStock that your boss is able to know if the products are distributed.”

Acceptability of EM (added effect of DPAT)

Acceptability of the DPAT component of the EM intervention was evaluated by looking at how districts, HF staff and HSAs used information supplied by cStock for routine monitoring and managing of community SC performance, improving on their DPAT targets and rewarding well performing HSAs. All district staff in EM districts reported visiting the dashboard at least once a week to obtain data from cStock for planning, coordination, and supervision. District staff also printed cStock reports for use by HF staff to monitor HF and HSA performance. HF staff reported using cStock information obtained either through the resupply worksheets or cStock reports to discuss reporting (79%) and stock management (53%) during DPAT meetings. Seventy–eight percent of the HSAs said their supervisors referred to the performance targets such as reporting during the DPAT meetings and 80% knew they would receive rewards if they performed well in stock management using cStock. Table 3 presents summary of acceptability findings for DPATs from the follow up survey.

One aspect that made EM acceptable to users was the DPAT role in monitoring SC performance. HSAs reported that DPAT meetings were useful for discussing SC issues, such as HSA performance, reporting rates, and performance targets. One example from an HSA in the EM group is “sometimes it happens that you receive less products when you see more cases, we discuss how best to cater for these cases.” HSA Supervisors reported discussing similar topics, such as HSA performance, supply management, stock outs, reporting rates, and recognition of good performance among HSAs. And HSAs described being motivated by the performance monitoring, positive feedback, and performance targets used as part of the DPAT design. Another HSA in the EM group stated, “for us we share ideas during the meeting, as a single person you cannot build a house, we are motivated.” HF staff also cited DPAT meetings as useful in providing a forum for reviewing SC issues and performance.

HSAs and HF staff said that EM helped link people and activities across the supply chain, aligning the flow of information and clarity of processes. Challenges to DPATs included HSA attendance, HSA hunger when no food or refreshments were offered, lack of follow up on issues discussed at meetings, lack of needed materials, and poor coordination/communication by HF In–Charge.

In terms of comparison with the EPT group, performance on most acceptability indicators in Table 3 was slightly higher in the EM group. For example, district staff in the EM group used cStock more than their EPT counterparts, with 2 of 3 (67%) EM district staff using cStock to target health facilities for supervision compared to 0 of 3 in EPT.

Effectiveness: supply chain performance of EM vs EPT groups

Findings on effectiveness are presented as a comparison of supply chain performance between the EM and EPT groups according to these four indicators: reporting, complete reporting, lead time, and stock out rates over time, using data from cStock dashboard reports.

Reporting rates

Analysis of cStock results for January 2012 – June 2013 showed higher mean reporting rates in EM (94%) compared to the EPT (79%) group. There was a slow increase in time for both groups, and greater variation seen within the EPT group than EM. This study found that the EM group had significantly higher reporting rates (94.0 ± 10.7%) compared to the EPT group (79.1 ± 11.0%), t₍₁₀₆₎ = 6.9766, P < 0.001 ( Figure 3 ).

Figure 3.  Mean reporting rates to cStock by HSAs, on all commodities in EM (Enhanced Management, n = 393) and EPT (Efficient Product Transport, n = 253) districts, January 2012 – June 2013.

Complete reporting

Completeness in reporting measures the extent to which HSAs send in stock on hand messages to cStock for all the products they manage. Mean completeness rates were found to be higher among EM (85%) than EPT districts (65%). Figure 4 shows the trends in complete reporting between the two groups over 18 months. This study found statistically significantly higher completeness rates (85.0 ± 9.7%) in the EM group compared to the EPT group (65.2 ± 11.1%), t₍₁₀₆₎ = 9.8953, P < 0.001.

Figure 4.  Mean reporting completeness by HSAs (health surveillance assistants), on all commodities, in EM (Enhanced Management, n = 393) and EPT (Efficient Product Transport, n = 253) districts, January 2012 – June 2013. Asterisk – findings from a two sample t–test with equal variances suggest that the differences in means between the EM and EPT districts are statistically significant (P < 0.001).

Lead times. On average, lead times, a measure of responsiveness in the supply chain, were significantly (P < 0.05) lower/better in EM compared to EPT districts with HFs in the EM group taking on average 12.8 days to fulfill an HSA order compared to 26 days in the EPT group. The difference in the two groups was small (four percentage points) in Jan 2012, increasing (to 24 percentage points) in June 2013. This study found statistically significantly lower lead times (12.8 ± 6.2) in the EM group compared to the EPT group (26.4 ± 11.3), t₍₁₀₆₎ = –7.75, P < 0.001 ( Figure 5 ).

Figure 5.  Lead time for HSA (health surveillance assistant) drug resupply calculated by cStock, in number of days, in EM (Enhanced Management, n = 393) and EPT (Efficient Product Transport, n = 253) districts, January 2012 – June 2013. Asterisk – findings from a two sample t–test with equal variances suggest that the differences in means between the EM and EPT districts are statistically significant (P < 0.001).

Supply reliability

Supply reliability can be defined as consistent availability of health products required in the community to treat sick children [10]. Reliability results are presented here as a measure of mean percent HSA stock out rates by product. cStock data analyzed for six iCCM products over an 18-month period ranged between 5–7% stocked out for all products in the EM group and between 10–21% in the EPT group. These differences were statistically significant at the P < 0.001 level for all products. The difference in stockout rates was largest in the case of Paracetamol with mean stockout rates of 5% in the EM group districts and 21% in the EPT group districts ( Figure 6 ).

Figure 6.  Mean percentage stockout rate over 18 months, by product, for EM (Enhanced Management) vs EPT (Efficient Product Transport) districts, (January 2012 – June 2013). Asterisk – P < 0.001.

Generalizability of findings

Although there is a growing evidence base for implementation of mHealth systems to solve public health problems, only a limited number of solutions aim to improve supply chain effectiveness [11]. To the best of our knowledge, our study is the first to address feasibility, acceptability and effectiveness of a mHealth solution that targets CHWs at the last mile of supply chains. Our findings on the use of mHealth to improve data visibility and use are consistent with the studies conducted by SMS for Life in Kenya and Tanzania and an SMS-based malaria reporting system supported by Rapid SMS tested in Uganda at peripheral health facilities (one step above the last mile), that demonstrated that mobile technology can improve reporting rates and lead to a reduction in stock outs if managers respond to the timely data [12–14].

The importance of exploring ways to promote routine system use and enhance acceptability is demonstrated by the SMS for Life study in Kenya and Tanzania, where financial incentives were provided for reporting stock information, resulting in higher reporting rates than in the Uganda study, where incentives were not used. The Uganda study however recognizes that better impact related to stock management was likely if district teams were fully engaged, emphasizing the importance of such reporting systems being nested within organization support structures. Similarly in Malawi, the implementation of cStock was integrated with additional components in the two intervention groups designed to enhance the overall performance of the supply chain; the findings from the EM group suggest that the team approach had an additive and significant effect in further improving SC performance for reporting rates, reporting completeness, lead times (responsiveness of the supply chain) and stock out rates (supply reliability). This finding is in agreement with the broader discussion by experts in the field of mHealth who have suggested that integration of mobile–based tools with existing health management systems and structures will be critical to national level adoption and scaling up of such programs [15]. We believe this is one of the first papers that has tested the hypothesis that while a viable mobile–based tool like cStock is important, the integration of information flow systems with management structures such as DPAT is critical to the tools’ adoption and ability to achieve scale and long term effectiveness.

Limitations

One of the greatest challenges in implementation and evaluation of cStock was related to the complex supply chain and product management environment in Malawi. Due to the economic crisis in the country iCCM implementing partners were provided funding by their donors to supplement the national supply of iCCM medicines and distribute products directly to HSAs. These parallel donor–funded distribution systems bypassed the government–funded public supply chain and therefore masked true product availability throughout the supply chain. Thus attribution of improvements in product availability as a result of the EM and EPT interventions was not possible to determine and supply reliability was used as an indicator instead. Additionally, the scope of this research was limited to analysis of effectiveness of cStock and DPATs (EM) in terms of supply chain outcomes, and did not delve into the additional cost and cost saving related to either component of this integrated approach.