A framework for Made in Africa Evaluation and influence of the African Peer Review Mechanism on policy outcomes

The last three decades have seen a rise in political recognition of the significance of evaluation in shaping evidence-based policies for effective governance, resulting in the institutionalisation of evaluation systems across Africa (Porter and Goldman, 2013; Searle et al., 2024). In 2003, the African Peer Review Mechanism (APRM) was established as a home-grown governance evaluation tool to promote a self-sustaining continent capable of generating its own solutions (African Union Commission, 2015). In 2017, efforts to enhance APRM’s evaluation influence led to an African Union (AU) resolution “to reposition the APRM to play a Monitoring and Evaluation (M&E) role for the AU Agenda 2063 and the United Nations 2030 Sustainable Development Goals” (APRM, 2017, p.1). The mechanism is described as “a post-colonial indigenous evaluation paradigm intended to address the limitations of traditional western paradigms in response to Africa’s problems” (Banda, 2022, p.33), yet assessments of its evaluative impact reveals underuse, misuse, and non-use. While this downward trend undermines APRM’s strategic goal as a catalyst for realising the AU Agenda 2063, it has received limited research attention. This development coincides with scholarly efforts to establish a Made in Africa (MAE) perspective, which has largely remained theoretical, with minimal practical application supported by empirical evidence (Fish, 2022; Siar, 2023).

This paper presents an empirical case analysis of the implementation of the APRM as a MAE approach, responding to calls for evidence of the practical application of a MAE theory of change (Chilisa and Mertens, 2021). The objective is to generate a framework to enhance the APRM evaluation process (inputs, activities, outputs) and, in turn, operationalise it to influence policy outcomes among AU member states. The research questions are:

• (a)

  How does the nature of evaluation inputs affect the type of use or non-use of the APRM evaluation data in public policy?

• (b)

  What is the effect of the APRM activities on the type of use or non-use of APRM data in public policy?

• (c)

  How does the quality of review reports affect the type of use or non-use of APRM data in public policy?

The paper contributes to policy literature by proposing a framework to enhance the APRM evaluation process (inputs, activities, outputs) and operationalise it to positively influence policy outcomes among AU member states. Furthermore, it builds on the theoretical foundation of MAE, drawing on the works of Cloete (2016) and Gaotlhobogwe et al. (2018). They advocate for promoting customised African evaluation frameworks that incorporate inputs from indigenous knowledge systems. The paper aims to fill the gap in empirical application by examining the case of APRM implementation in Uganda. With the proposed framework, it is hoped that APRM stakeholders can be guided in selecting appropriate inputs, activities, and outputs to enhance the realisation of the desired policy influence.

The rest of the paper is structured as follows: first, it presents the conceptualisation and background of APRM as a MAE approach; next, it discusses the theoretical foundation developed from the literature; followed by the methodology, presentation of the research findings, and discussion of the results. Finally, the last section outlines the conclusion and policy recommendations, proposing an operational framework.

In the 1990s, evaluation was emphasised as part of the New Public Management (NPM) reforms for accountability to donors and to inform efficient policies. In the 2000s, research shifted from a narrower concept of evaluation use to a broader notion of evaluation influence, with calls for empirical evaluation field studies to map a theory of change that influences policy outcomes (Holvoet et al., 2012; Herbert, 2014). Recent studies present two major approaches to assessing the influence of evaluation on public policy: assessment by external oversight actors or internal peers (Streicher, 2017; Basheka and Byamugisha, 2015).

In Africa, the APRM, an Afro-centric accountability and learning tool mandated by Article 5(2) of the African Union (AU) Constitutive Act of 2000, marked a paradigm shift from reliance on western methodologies to peer pressure approaches that are culturally responsive to generate the continent’s own solutions to its challenges (APRM, 2016; Tshiyoyo, 2017; Auriacombe and Cloete, 2019). In assessing APRM’s influence in 38 countries between 2008 and 2019, the African Governance Report (APRM, 2019), reflects mixed findings from 22 base reviews and three second-generation reviews. The reported downward trend in the APRM influence raises significant concern, which may result in what Højlund (2014, p. 29) refers to as an “evaluation crisis”. Different from previous outcome-oriented APRM studies, this paper addresses this gap by proposing a process-based framework to enhance the APRM’s influence towards the realisation of the AU Agenda 2063.

From the literature, the concept of evaluation refers to the systematic collection of data to make recommendations concerning improvements, while evaluation influence is defined by Kirkhart (2000, p. 7) as “the capacity or power of an evaluation to cause a spiral effect”. Influence goes beyond the use of data to include the sources of influence (process and results), intention (intended and unintended), time and the levels of feedback generated by evaluation (Herbert, 2014; Searle et al., 2024). Amid concerns about the influence of Western evaluation approaches to drive change in Africa, the APRM was introduced in 2003 as a context-sensitive evaluation tool, emphasising multiple stakeholder engagement, collegial assessment of governance progress, and collective learning for improvement among member states. In 2007, increased calls for decolonisation of evaluation led to the introduction of MAE as an indigenous evaluation perspective defined by the common African relational philosophy of “Ubuntu”, the African world views, and values that meet the needs of the African people (Chilisa and Mertens, 2021). Unlike donor agencies which emphasised external pressure (Basheka and Byamugisha, 2015), the APRM is presented as an adaptive MAE approach informed by the indigenous knowledge and inclusive African communal ideology (Gaothobogwe et al., 2018). Despite two decades of implementation and a downward trend in influence, limited empirical studies have been conducted to establish a framework to enhance its evaluative impact (Bosman and Clifford, 2021).

To link the key elements in the study objectives, the paper draws on Mark and Henry’s (2004) schematic theory of evaluation influence, which focuses on the major change processes triggered by evaluation. The theory emphasises the fact that attention should be given to the process of evaluation beyond the direct use of findings (Herbert, 2014). A major strength in this mechanism-based model lies in its adaptability and flexibility, enabling researchers to develop logical explanations and predictions for context-specific policy frameworks (Højlund, 2014). Mark and Henry (2004) identify evaluation influence process into three elements: firstly, inputs (decision setting and context), which, if favourably supportive, lead to responsive activities. Secondly, evaluation activities (planning, selection of stakeholders, data collection and analysis, reporting and dissemination), which, if objectively conducted, lead to credible outputs. Thirdly, evaluation of outputs in the form of reports whose influence depends on credibility, sophistication, responsiveness, and timeliness to affect outcomes. The outputs are mediated by a set of mechanisms, namely, general influence, motivational, behavioural, cognitive, and affective to translate into policy outcomes (Streicher, 2017).

This research adopted a case study design, where qualitative data were collected through interviews and FGDs from a sample of 35 stakeholders involved in the APRM implementation decisions between 2008 and 2021 in Uganda. Other researchers have similarly adopted and recommended this design to investigate the effect of evaluation influence on policy outcomes (Gildemyn, 2014; Streicher, 2017). The country case was selected based on Uganda’s position as a pioneer member of the APRM, the second country to voluntarily undergo a second review, and one of the first three African countries in which Monitoring and Evaluation received government political backing as a development intervention (Chirayu and Ramasobana, 2022). Selection of participating stakeholders followed three criteria. First, stakeholders with experience of participating in APRM implementation during the first and second review. Secondly, willingness to share experiences through interviews or FGDs. Thirdly, their positions offer them senior decision-making experience on data use.

Table 1 shows categories of participants in the in-depth interviews and FGD as drawn from the APRM Secretariat units.

As reflected in Table 1, the unit of analysis was the individual stakeholders, purposively selected based on their day-to-day involvement in the APRM implementation.

Data were collected between January and May 2023 in Uganda as part of the PhD research, using interviews and FGDs with open-ended questions administered to participating stakeholders via an interview guide to share their experiences (Tracy, 2010). A list of research questions informed by the research objectives were selected based on literature review of evaluation influence and the APRM studies to address the research gap. Firstly, the questions were refined through field pre-tests for pilot data collection involving four audio-recorded interviews with stakeholders. Secondly, the validation of the questions was achieved through review and approval of tools by experts, to generate a list of final interview questions. Research ethical principles were observed from expert review of questions to confidential data management. The researchers secured ethical review and approval of data collection tools by the University of Pretoria (Protocol number EMS 108/21), and the Uganda National Council of Science and Technology (project registration number SS1568ES). The questions covered the understanding of evaluation influence, stakeholder perceptions, participation, and contextual factors shaping APRM’s influence, contribution of APRM actors, and strategies for improved utilisation of data.

The ethical principles of obtaining consent and ensuring confidentiality were adhered to before the researcher utilised audio recording to capture the stakeholders’ perceptions and experiences. Validity was ensured through audiotaped, verbatim, and carefully studied interview transcripts from carefully chosen information-rich participants for quality and legitimacy. Thirdly, internal and external validities were enhanced through the triangulation of data generated from multiple sources and techniques, collaboration with expert views and field notes for dependability. Fourthly, for communicative validity, member checks were conducted with participating stakeholders to guarantee accurate interpretations of participant’s responses with sub-substantial backing of quotations. The FGD was moderated with eight core MDA focal point persons drawn from the Office of the Prime Minister, the Ministry of Justice and Constitutional Affairs, the National Planning Authority, and the National Information Technology Authority.

Data analysis followed an iterative process, starting with a preliminary exploration of each data transcript to identify emerging findings that align with the evaluation influence. Based on the findings generated from the research questions, qualitative data analysis was adopted inductively through thematic and content analysis. The transcripts in MS-word text were collaborated with field notes, cleaned, and loaded into ATLAS ti Version 08 software, which can make the text analysis simpler, efficient, and more structured (Friese, 2019). Using ATLAS ti, linkages between the research questions and the derived second order themes were established (Table 2).

Open coding was adopted to condense large volumes of in-depth data into manageable sizes and comparison of responses, data categorisation based on common content, ideas, characteristics, and meanings (Bernard, 2013). From data, keywords were identified by first order codes, with code names from which themes were developed.

To illustrate this process of coding and grouping codes into themes. Table 2 presents the aggregate response to question 2 by participant 1. “Based on your experience, what policy framework informs the use of APRM evaluation findings and recommendations in Uganda’s policy process?”.

As reflected in Table 2, an analysis of the participants’ responses yielded six most frequent keywords coded as: “Institutional Structure” (IS), “Resource Capacity” (RC), “Stakeholder Participation” (SP), “Incentives” (IU), and “Reporting” (RI). The codes were used to identify general ideas from the data, with related codes grouped under same second-order themes, and supported by specific stakeholder quotes to show the participant’s voices. For example, phrases like “absence of a separate budget for the APRM secretariat, dependency on unreliable donor funding, lack of direct appropriation of funds by parliament”, were coded as “Resource Capacity” (RC) and aggregated under the “inputs” theme. Thematic analysis identified three major factors influencing evaluation use: inputs, activities, and outputs, supported by quoted evidence. Finally, the major findings based on the themes were discussed in relation to literature to contribute to evaluation influence knowledge in public policy.

The findings on the use of APRM data indicate that all 35 participants (Figure 1) perceived that the nature of evaluation inputs, activities, and outputs as directly affected the degree of APRM’s influence, leading to different types of uses.

Figure 1 shows that 24 out of 35 participants affirmed that the APRM evaluations were directly used in Uganda’s National Development Plan. In addition, seven participants indicated selective use, and four participants answered “non-use”. While this finding demonstrates Government of Uganda’s commitment to ensuring instrumental use of the APRM recommendations into policies, the trend is said to have declined in the second review. The following sections present the findings based on the themes with corresponding quotations to ensure logical flow.

Inputs into the APRM were resources identified which include plans, tools, human resources, and finances.

• i.

  Plans and guidelines

Findings show that, in the first peer review, a communication and advocacy plan was the major input in the APRM process. To show how this affected the degree of evaluation influence, Participant 11 (National Secretariat) explained:

Therefore, absence of a robust strategy hindered shared understanding of the APRM objectives, thus negatively affected stakeholder roles and use. However, while guiding documents from the continental secretariat supported the planning, Participant 20 (NGC) affirmed the different forms of use and argued:

Therefore, above statement suggests that plans are not sufficient to guarantee use and influence which necessitates the need for the proposed framework.

• ii.

  Financial input in the APRM evaluation system

Participants revealed that while the first peer review attracted a lot of donor funding, the lack of adequate budgetary resources in the second review denied the country an opportunity to run a consultative review process. Participant 20 (NGC) reported on the effect of limited financial input from the government:

Therefore, the inadequate domestic sources negatively affect the inclusive nature of the consultative process, commitment of the NGCs, and the quality of the data collected, leading to rampant symbolic and non-use use of APRM findings.

• iii.

  The human resources inputs

Any credible evaluation is judged based on the evaluation capacity of the TPI consultants and the NGC members (APRM, 2003, p. 3). The findings indicate the existence of declining relevant human capacity. Participant 21 (Focal Point) emphasised:

The above statement implies that the initial selection criteria of NGC members focused on representation and capacity to influence credible reporting to ensure influence.

However, some participants emphasise that capacity sometimes falls short of expectation especially concerning the second review. To explain this concern, Participant 14 (NGC) reported:

The above situation of declining capacity has worsened. UNECA (2011) attributes this to the challenge of bureaucratic and political capture by the ruling regime, leading to failure to include politically sensitive views.

The findings indicate that even with contextually relevant questions as recommended by the APRM Base Document, the adapted questionnaire captures more elite-centric data. This defies the principles of stakeholder consultation and ownership, making it less relevant to influence instrumental use in public policy. For example, participant 3 (National Secretariat) argued:

The above response suggests that the relevance of the data depends on the customised questionnaire’s ability to capture inclusive views of the stakeholders within the country context.

The findings show that Uganda follows the activity schedule provided in the APRM (2016) from stakeholder selection, data collection, and reporting. Most participants indicated that evaluation activities are effective. Respondent 7 (Secretariat) said:

Therefore, the above submission implies that the participatory nature of the activities in the APRM process which is technically managed are a great incentive towards influencing the use of the data.

However, some respondents also expressed optimism about the elite and technical nature of the activities. Participant 9 (key informant) stated:

Thus, the APRM evaluation activities are conducted mainly within the dockets of the elites in CSOs and government technocrats - a challenge addressed through the proposed framework, which facilitates the intended philosophy of multiple stakeholder participation, ownership, and consequently the use of data by different stakeholders.

The findings indicate that outputs in the form of progress reports are submitted to the Government and the APR Forum, but raised concerns about quality, alignment, and politics. With reference to the first review, participants perceived that wide stakeholder consultation contributed to a more responsive report that partly informed the NDP1. This is well illustrated by what Participant 9 (APRM Secretariat) stated:

This evidence points towards the instrumental use of reports as credible sources for government planning, CSO advocacy and lobbying activities. Such insights indicate that reports backed with quality evidence collectively influenced the government’s policy positions. It was reported that in second-generation reporting, politically sensitive content makes some politicians question its methodology, consequently leading to a more symbolic or non-use of APRM data.

Regarding how the nature of inputs influences use, the findings revealed limited participation in plans, resource constraints, and inappropriate questions, all of which negatively affected data utilisation.

• i.

  Limited participation in the development of peer review plans

For the first review, major concerns about the inputs centred on limited non-state stakeholder involvement in preparing peer review plans. This finding aligns with Banda (2022), who revealed that, in Africa, the change processes are weak partly because of limited stakeholder participation and ownership of evaluation input. The absence of a robust awareness plan in the second-generation review exacerbated this issue, leading to ad hoc stakeholder participation in the peer review activities. This collaborates Goldman and Pibari (2020), who point out the challenges of using and influencing arising from the undeveloped or poorly or non-communicated plans to the stakeholders’ awareness.

• ii.

  Technical and financial resource capacity towards effective review

While the APRM (2016) emphasises the African ownership of the peer review through internally generated resources, the findings revealed increasing challenges associated with inadequate domestic financial and human resource capacity to effectively conduct objective peer reviews (KII:20). The finding is in line with Goldman and Pabari (2020) who assert that Africa has pressing social problems but limited resources hinder evidence use in policy process. The implementation of evaluation recommendations remains elusive and challenging, despite the rhetoric associated with its importance. This inadequacy negatively affects quality-reporting stakeholders’ confidence in the use of data in policy decisions.

• iii.

  Relevance of the questionnaire

Most participants pointed out that the APRM data collection tool has inappropriate evaluation questions which are overly technical, repetitive, and elite-centric, administered in English and excluding the views of the majority grass root citizens (KII:11). This finding aligns with Cousins and Leithwood (1986), who identify poor-quality questions as a barrier, leading to non-responsive and irrelevant recommendations.

• i.

  Lack of a clear theory of change informing peer review activities

The major issue identified by participants that affect use is the lack of a theory of change to guide the peer review process. Participants observed that despite the availability of the APRM (2003; 2016) informing the APRM implementation activities, the data management process remains complex and unclear about who participates from the stakeholder categories and at what stage (KII-4; 5). This finding collaborates with Banda (2022), who similarly attribute methodological limitations to evaluation use and influence. Therefore, non-compliance to guidelines leads to questionable output that affects the degree of influence.

• ii.

  An elite-based research process

The findings indicate that the technical nature of the APRM implementation methodology limits inclusive participation of the grass root people. The participants are selectively drawn from the top government MDAs, the private sector and civil society elites influenced by top local government officials who are loyal to the ruling party with a focus on government supporters, which reduced a sense of ownership, and interest in utilisation of data (KII-9). This finding concurs with Goldman and Pabari (2021) who identify inadequate incentives for accurate data production as a major obstacle to collection and use of data in Africa. In addition, Streicher (2017) attributes this situation to the under-developed evaluation culture. The evidence also presents an unfavourable political influence especially in the appointment of the NGC and TPI Consultants by the executive, based on one’s loyalty as opposed to competence which affects accurate data production (KII-14). This concurs with Alkin and King (2017), who explain that politicians may intentionally subvert the evaluation process by either preventing access to the necessary data or cutting funding, to drive evaluation activities in their direction of interest.

For the issue of “How APRM output influences use”, the major factors identified include credibility of findings and alignment with national plans.

• i.

  The credibility of the reports

The findings reflect mixed perceptions, with most participants defending the quality of reports produced through a technical process. Regarding the first review, they argue that the credible evidence led to the integration of the POA into the 5-year NDP 2010/2011-2014/2015. This finding aligns with Mark and Henry (2004, p.37), who argue that relevant evaluation findings translate immediately into “specific actions leading to continuation, revision or even termination of policy implementation”. The misuse of APRM reports was also cited as a major challenge, where data were used to justify pre-determined political decisions, especially in sensitive thematic area of democracy and political governance (FGD-3). This revelation concurs with Alkin and King’s (2017) assertion that reporting quality is compromised by misrepresentation or omission of findings for self-interest, biasing users.

• ii.

  Non-alignment of report content with national plans and budgets

Empirical evidence from participants indicates that the production of APRM reports is insufficient, and the alignment of implementable policy recommendations with national development priorities in the second review affected the use of data in policy processes (FGD-5). In line with the above finding, Johnson et al. (2009) attributes the non-use of some recommendations to a lack of proper alignment with the government plans, implementation difficulty, irrelevant or excessive recommendations, and poor timing of evaluation reporting. Banda (2022) attributes the non-alignment or misalignment of evaluation to the non-participation of the strategic planning unit in the evaluation process, resulting in reports lacking evidence of implemented action points.

The paper analysed qualitative data from stakeholders involved in implementing the APRM as MAE in Uganda to propose a framework for enhancing its evaluation influence on public policy outcomes. This study contributes to empirical application of MAE through a proposed framework informed by Mark and Henry’s (2004) schematic theory of evaluation influence.

The findings reflect that systematic observation of a five-stage process involving inputs (participatory planning, robust legal framework), activities (objective participant selection, data collection, and credible reporting), and outputs (aligned, actionable recommendations) that can build an operational framework for the APRM’s enhancement as an effective MAE approach. This operational framework is presented in Figure 2.

Figure 2 illustrates the stages linking inputs, activities, and outputs for an effective peer review process to translate into the use of data. The stages inform a framework that suggests gradual improvement in evaluation influence across stages 1-5. The highest level of influence manifests in instrumental use, achievable when inputs include a well-established legal framework and plans, activities involve objective selection of participants, data collection and reporting, and processes remain free of political manipulation. The second component is stakeholder participation. Unlike Mark and Henry’s (2004) schematic theory of evaluation influence, which treats stakeholder identification and participation as an activity, the adapted framework integrates stakeholder analysis from Yu et al.’s (2011) public policy analytical model for systematic stakeholder identification, management, and analysis.

The third stage involves quality reporting with responsive, credible, timely, and effectively disseminated outputs to stimulate the different types of mechanisms in stage four. This progression is likely to lead to gradual improvement in influence from non-use to instrumental use and integration of APRM recommendations in national policies for social betterment, as reflected in the fifth stage.

The fourth stage focuses on the process of generating reports and dissemination of the findings. The details of the participants and their roles in validation activities, compilation, editing, and dissemination strategy are analysed, as well as their implications for the use and influence of the APRM on national policies. The quality of the reports is assessed against attributes such as responsiveness, credibility, sophistication, communication, and timeliness.

The fifth component adopts the general mechanism through which the outputs translate into outcomes with attributes such as justification, persuasion, salience, skill acquisition, policy deliberation, coalition formation, and standard setting. These core change processes and linkages constitute evaluation influence. Therefore, the environmental factors, mediated by the mechanism, determine the type of use, which may be instrumental, symbolic, conceptual, or non-use.

Upon structurally creating the framework, its testing for operationalisation began when it was discussed and enriched with input from APRM stakeholders from Kenya, Côte d’Ivoire and Egypt. Following the structural creation of the framework, testing for its operationalisation commenced after discussions and input from APRM stakeholders from Kenya, Côte d’Ivoire, and Egypt during a technical working group workshop dedicated to building the APRM governance index, held from 28 to 30 November 2023 in Dar es Salaam, Tanzania.

The paper offers practical implications. For practitioners involved in evaluation, it emphasises that adequate attention should be given to the process elements – inputs, activities, and outputs - rather than the input/output approach, which sustains the “missing middle” challenge in understanding how and why of use and non-use. For researchers, focusing on empirical case studies provides valuable insights into the limitations of undertaking evaluations within African contexts, which is critical for advancing the operationalisation of MAE.

Finally, the paper is not without limitations. The researchers acknowledge that the study’s qualitative design and single case focus on Uganda (2005 to 2018) pose risks of researcher bias. Consequently, the findings may not be generalisable across different member states on the continent. Future studies should adopt a comparative and/or mixed method approaches of APRM implementation across multiple countries to test the proposed framework to evaluate its effectiveness and enhance the generalisability of findings.