Individual evaluator and average scoring of PDAP for all QAF criteria, with justifications for the score allocated
| Principle | Criteria | E1 | E2 | E3 | E4 | Avg. | Justification/Comments |
|---|---|---|---|---|---|---|---|
| Relevance | Clearly defined socio-ecological context | 2 | 2 | 2 | 1 | 1.75 | Proposal and dissertation provide clear and full description of the problem context from a global to local level (Moshi); dissertation outlines the Tanzanian private development aid context; clear identification of entry points (literature/knowledge gap); the decision-making context of the organizations investigated and how recipients perceive them were missing (justification for why the research is needed in the socio-ecological context) |
| Socially relevant research problem | 2 | 2 | 2 | 2 | 2 | Research problem is well-aligned with the Tanzanian private aid context; informants reflect on relevance and value of study; dissertation identifies the value of the findings to diverse audiences (e.g. academics, policy-makers, non-profit practitioners, philanthropists/donors) to make better informed funding decisions | |
| Engagement with problem context | 2 | 2 | 2 | 2 | 2 | Researcher interacted substantially with the problem context (via personal experience, engagement with literature, and in fieldwork during the research process); researcher’s engagement was appropriate to objectives of the research; researcher was well positioned to define the research problem and influence the context (extensive international experience within the development and philanthropic-giving sector, interned during research with DAC, collaborations and consulting work aligned with research interests, access to extensive networks) | |
| Explicit theory of change | 1 | 2 | 1 | 1 | 1.25 | Dissertation identifies the main intended outcomes and how they would be realized, though brief and not actor-specific; supervisor notes that change was intended and built into the research (had a ToC, but this document was not shared); other informants remarked on their recognition of the intentionality behind the PI’s project | |
| Relevant research objective and design | 2 | 2 | 1 | 2 | 1.75 | Objectives are relevant and stakeholder needs/values are considered; decisions around research design are relevant, appropriate, and transparent; informants reflected positively on the design of the study; informants discussed the relevance of the research and its design | |
| Appropriate project implementation | 2 | 2 | 2 | 2 | 2 | Informants (participants, advisory committee, and practitioners) noted relevance of study’s focus on private aid in Moshi, Tanzania; extensive fieldwork supported deep engagement with the country and problem context; second round of interviews was regarded positively by participants and advisory committee; informants believed the array of organizations was appropriate, as was the extent of stakeholders representative of those organizations (e.g. end-use recipients, aid implementers, donors, partners, etc.); reflection and adaptation are noted transparently; adaptations made to project implementation to accommodate context | |
| Effective communication | 1 | 2 | 1 | 2 | 1.5 | Researcher planned and achieved extensive communication during (multiple interview sessions, frequent e-mail correspondence) and post-project (sharing findings with participants, dissemination and presentation of findings); researcher maintained contact with participants post-project (achieved more than typical); some informants noted communication on topic could have been clearer or done with more background/notice (one instance depends on a gatekeeper, however); multiple media were used to communicate the findings (dissertation, summary of findings, e-mails, conferences, practitioner magazine, book chapter, website/blog, SSHRC Storytelling video, etc.); several informants appreciated the clear and accessible writing style of the dissertation/outputs; some informants believed the PI should have written academic articles or a book targeted to practitioners | |
| Credibility | Broad preparation | 2 | 2 | 2 | 2 | 2 | Researcher conducted comprehensive literature review (private development aid, philanthropy, grant-making, non-profit management, social venture/entrepreneurship, outcomes based and impact giving, development evaluation, etc.); dissertation demonstrates integration of disciplines and academic and grey literature |
| Clear research problem definition | 2 | 2 | 2 | 2 | 2 | Research problem is stated in the dissertation, and value of understanding the problem from recipient perspectives is justified and grounded within the literature and Tanzanian context | |
| Clear research question | 2 | 2 | 2 | 2 | 2 | Research questions are clearly stated in the dissertation; justification and relevance of research questions to context are provided | |
| Objectives stated and met | 1 | 1 | 1 | 1 | 1 | Objectives are scattered throughout the research proposal and final dissertation, and are logically connected to the problem context identified; more explicit and clear documentation of objectives would have aided the structure and clarity of the project; objectives appear to have been met | |
| Feasible research project | 2 | 2 | 2 | 1 | 1.75 | Project supported by sufficient funding (SSHRC Fellowship, Mitacs [Accelerate Fellowship], IDRC), which enabled extended time for fieldwork and second round of interviews; advisory committee felt funds were appropriately allocated; does not explain how the sub-question on theory of change will be addressed or answered; research design was flexible and enabled adaptation (expanded from 12 focal organizations to 33 projects, accommodating group interviews, discarding of photo-voice method, appropriate use of a recorder); some informants noted researcher accommodated unexpected invitations to various events or meetings to meet stakeholders/end-use recipients; most informants did not have suggestions for what could have been done differently – only suggested more time in the field or facilitating a forum to bring organizations together, which are outside the bounds of a doctoral project | |
| Adequate competencies | 2 | 2 | 2 | 2 | 2 | Researcher had extensive prior experience in the international development and philanthropic sector; dissertation reflects on methods’ implications for bias and researcher’s ‘outsider’ positionality; translators were hired to support data collection from end-use recipients; research proposal, dissertation, and researcher interview briefly reflected on bias/implications of translators’ involvement (e.g. withholding information, political agenda) | |
| Research approach fits purpose | 1 | 1 | 1 | 1 | 1 | Approach is justified in relation to the context and research objectives (e.g. case study approach, second round of interviews); rationale for different methodologies is explicit, though lacking in terms of discussion on disciplines and epistemologies (researcher interview conveyed appreciation of diverse perspectives gained through the DSocSci cohort) and interrogation of methods and what they brought together was insufficient; discussion on conflicts (but reflecting more on adaptation) is brief | |
| Appropriate method | 2 | 2 | 2 | 1 | 1.75 | Methods are fit to purpose, clearly described, and transparent; methods were systematic yet adaptable to meet the needs of participants or accommodate cultural sensitivity; methods and adaptations, including reflections on the implications of each, are transparent (e.g. abandonment of photo-voice, group interviews); informants reflected positively on the methods and their rigour | |
| Clearly presented argument | 1 | 1 | 1 | 1 | 1 | Results are clearly presented; analyses are comprehensive and logically connected to the conclusions and recommendations; alternative explanations are not explored | |
| Transferability and generalizability of the findings | 1 | 2 | 2 | 1 | 1.5 | A statement is included in the dissertation recognizing the possible limit to generalizability of the findings of a single case study; transferability is not discussed in sufficient detail (what, how, and extent of transferability); transferability of findings to other contexts is evident from various outputs of the research; findings have been transferred and applied in other contexts by the PI, participants, and practitioners; informants believed the findings to be widely transferable | |
| Limitations stated | 1 | 1 | 1 | 1 | 1 | The limitations section in the dissertation is underdeveloped (one sentence is given to exploratory nature of the project and how findings are not generalizable, but this could have been explained in more detail); strengths and disadvantages (shortcomings) of methods selected are discussed in the methodology section; the research proposal includes brief reflection on limitations | |
| Ongoing reflexivity and monitoring | 2 | 2 | 1 | 2 | 1.75 | Researcher self-reflection is mentioned as part of the research process, but not discussed in detail in the dissertation (some indications from various e-mail correspondence to the advisory committee and mention of how interviews stimulated additional literature review); changes and adaptations made to methods/approach are transparent and justified (e.g. photo-voice method) | |
| Legitimacy | Disclosure of perspective | 2 | 1 | 1 | 1 | 1.25 | Potential bias is briefly mentioned in the dissertation, but the PI is clearly aware of their positionality based on other documentation (admissions proposal, research proposal, book chapter); sources of support (funding, collaboration with DAC) are noted; however, bias and positionality are not discussed in terms of the effect on the project results (only in terms of how funding supported deeper inquiry) |
| Effective collaboration | 1 | 1 | 1 | 1 | 1 | Interviews conveyed that the internship with DAC for the Mitacs funding and collaboration to funnel/apply findings to the organization was appropriate and clearly agreed upon (however, this was not documented within available project documentation); researcher interview relayed collaborative efforts with participating organizations to provide pro bono services/support; there was potential for more intentional collaboration with participating organizations that would have mutual benefits, but this was not capitalized based on the exploratory nature of the project; collaboration between the PI and the advisory committee was discussed positively | |
| Genuine and explicit inclusion | 1 | 2 | 1 | 1 | 1.25 | Project involved multiple participants from 33 projects, representing over 20 organizations in Moshi – dissertation describes the different characteristics of these organizations (donor country of origin, project focus, size, age); project’s focus on voice of recipients of private development aid clearly distinguishes between end-use recipients and aid implementers; dissertation also distinguishes between the cultural background of local Tanzanian, ex-patriate, and foreign implementers; project also collected data from private donors; second round of interviews functioned to validate accurate representation of the findings; roles of advisory committee, translators, and other support (e.g. transcription, draft editors, editing for SSHRC Storytelling video) are briefly mentioned in the acknowledgements; interview approach had greater emphasis on collecting information than co-generation of knowledge | |
| Research is ethical | 2 | 2 | 2 | 2 | 2 | Project received ethical approval by RRU Research Ethics Board; project sought and received approval from the Tanzanian Commission for Science and Technology to conduct the research; dissertation outlines in detail the steps taken to obtain consent and ensure anonymity of participants; informants reflected on the PI’s commitment to confidentiality and anonymity, which is maintained during this evaluation; dissertation outlines instances of ethical challenges and how they were solved (e.g. use of a recorder, redacted information/turned off the recorder, group interviews, photo-voice method, multinational merger, cultural appropriateness, etc.) | |
| Effectiveness | Research builds social capacity | 2 | 2 | 1 | 1 | 1.5 | Some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; several informants noted that the findings confirmed/reinforced that they were on the right path; practitioner noted that the research “opened up a dialogue within the NGO world in the organization of accountability”; capacity-building was not explicitly built into the project design, though there is potential that pro bono services supported organizational capacity-building; participant, advisory committee, and practitioner informants believed the doctoral experience and project supported the PI’s knowledge and capacity-building in the sector; some participants/organizations have changed their practices as a result of the research findings |
| Contribution to knowledge | 2 | 2 | 2 | 2 | 2 | Informants noted that the research uncovered relevant practice gaps they were facing and contributed to the academic literature; some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; participants, advisory committee, and practitioners described the findings as “useful” and “lessons to be learned”; researcher disseminated findings to both academic and practitioner target audiences (via conferences, magazine publications, website, SSHRC video, researcher’s consulting, etc.); participants and practitioners shared the findings through their networks | |
| Practical application | 2 | 2 | 2 | 2 | 2 | Intentionality for practical application is discussed by informants; findings have been applied by some participants/organizations (e.g. boards have adjusted the focus of their discussions, added question on unintended consequences into project proposal and monitoring processes); researcher has applied findings in own work and consultancies (e.g. Transform International, Lalafofofo, CAGP, SOVI-BCCIC); findings applied by practitioners (e.g. DAC, Lalafofofo, Transform International) | |
| Significant outcome | 1 | 2 | 1 | 2 | 1.5 | All ten end-of-project outcomes were fully or partially realized with clear project contributions; some organizations have changed their practices as a result of the research (with intent to become more effective); organizational changes are likely to support positive social changes through their work |
| Principle | Criteria | E1 | E2 | E3 | E4 | Avg. | Justification/Comments |
|---|---|---|---|---|---|---|---|
| Relevance | Clearly defined socio-ecological context | 2 | 2 | 2 | 1 | 1.75 | Proposal and dissertation provide clear and full description of the problem context from a global to local level (Moshi); dissertation outlines the Tanzanian private development aid context; clear identification of entry points (literature/knowledge gap); the decision-making context of the organizations investigated and how recipients perceive them were missing (justification for why the research is needed in the socio-ecological context) |
| Socially relevant research problem | 2 | 2 | 2 | 2 | 2 | Research problem is well-aligned with the Tanzanian private aid context; informants reflect on relevance and value of study; dissertation identifies the value of the findings to diverse audiences (e.g. academics, policy-makers, non-profit practitioners, philanthropists/donors) to make better informed funding decisions | |
| Engagement with problem context | 2 | 2 | 2 | 2 | 2 | Researcher interacted substantially with the problem context (via personal experience, engagement with literature, and in fieldwork during the research process); researcher’s engagement was appropriate to objectives of the research; researcher was well positioned to define the research problem and influence the context (extensive international experience within the development and philanthropic-giving sector, interned during research with DAC, collaborations and consulting work aligned with research interests, access to extensive networks) | |
| Explicit theory of change | 1 | 2 | 1 | 1 | 1.25 | Dissertation identifies the main intended outcomes and how they would be realized, though brief and not actor-specific; supervisor notes that change was intended and built into the research (had a ToC, but this document was not shared); other informants remarked on their recognition of the intentionality behind the PI’s project | |
| Relevant research objective and design | 2 | 2 | 1 | 2 | 1.75 | Objectives are relevant and stakeholder needs/values are considered; decisions around research design are relevant, appropriate, and transparent; informants reflected positively on the design of the study; informants discussed the relevance of the research and its design | |
| Appropriate project implementation | 2 | 2 | 2 | 2 | 2 | Informants (participants, advisory committee, and practitioners) noted relevance of study’s focus on private aid in Moshi, Tanzania; extensive fieldwork supported deep engagement with the country and problem context; second round of interviews was regarded positively by participants and advisory committee; informants believed the array of organizations was appropriate, as was the extent of stakeholders representative of those organizations (e.g. end-use recipients, aid implementers, donors, partners, etc.); reflection and adaptation are noted transparently; adaptations made to project implementation to accommodate context | |
| Effective communication | 1 | 2 | 1 | 2 | 1.5 | Researcher planned and achieved extensive communication during (multiple interview sessions, frequent e-mail correspondence) and post-project (sharing findings with participants, dissemination and presentation of findings); researcher maintained contact with participants post-project (achieved more than typical); some informants noted communication on topic could have been clearer or done with more background/notice (one instance depends on a gatekeeper, however); multiple media were used to communicate the findings (dissertation, summary of findings, e-mails, conferences, practitioner magazine, book chapter, website/blog, SSHRC Storytelling video, etc.); several informants appreciated the clear and accessible writing style of the dissertation/outputs; some informants believed the PI should have written academic articles or a book targeted to practitioners | |
| Credibility | Broad preparation | 2 | 2 | 2 | 2 | 2 | Researcher conducted comprehensive literature review (private development aid, philanthropy, grant-making, non-profit management, social venture/entrepreneurship, outcomes based and impact giving, development evaluation, etc.); dissertation demonstrates integration of disciplines and academic and grey literature |
| Clear research problem definition | 2 | 2 | 2 | 2 | 2 | Research problem is stated in the dissertation, and value of understanding the problem from recipient perspectives is justified and grounded within the literature and Tanzanian context | |
| Clear research question | 2 | 2 | 2 | 2 | 2 | Research questions are clearly stated in the dissertation; justification and relevance of research questions to context are provided | |
| Objectives stated and met | 1 | 1 | 1 | 1 | 1 | Objectives are scattered throughout the research proposal and final dissertation, and are logically connected to the problem context identified; more explicit and clear documentation of objectives would have aided the structure and clarity of the project; objectives appear to have been met | |
| Feasible research project | 2 | 2 | 2 | 1 | 1.75 | Project supported by sufficient funding (SSHRC Fellowship, Mitacs [Accelerate Fellowship], IDRC), which enabled extended time for fieldwork and second round of interviews; advisory committee felt funds were appropriately allocated; does not explain how the sub-question on theory of change will be addressed or answered; research design was flexible and enabled adaptation (expanded from 12 focal organizations to 33 projects, accommodating group interviews, discarding of photo-voice method, appropriate use of a recorder); some informants noted researcher accommodated unexpected invitations to various events or meetings to meet stakeholders/end-use recipients; most informants did not have suggestions for what could have been done differently – only suggested more time in the field or facilitating a forum to bring organizations together, which are outside the bounds of a doctoral project | |
| Adequate competencies | 2 | 2 | 2 | 2 | 2 | Researcher had extensive prior experience in the international development and philanthropic sector; dissertation reflects on methods’ implications for bias and researcher’s ‘outsider’ positionality; translators were hired to support data collection from end-use recipients; research proposal, dissertation, and researcher interview briefly reflected on bias/implications of translators’ involvement (e.g. withholding information, political agenda) | |
| Research approach fits purpose | 1 | 1 | 1 | 1 | 1 | Approach is justified in relation to the context and research objectives (e.g. case study approach, second round of interviews); rationale for different methodologies is explicit, though lacking in terms of discussion on disciplines and epistemologies (researcher interview conveyed appreciation of diverse perspectives gained through the DSocSci cohort) and interrogation of methods and what they brought together was insufficient; discussion on conflicts (but reflecting more on adaptation) is brief | |
| Appropriate method | 2 | 2 | 2 | 1 | 1.75 | Methods are fit to purpose, clearly described, and transparent; methods were systematic yet adaptable to meet the needs of participants or accommodate cultural sensitivity; methods and adaptations, including reflections on the implications of each, are transparent (e.g. abandonment of photo-voice, group interviews); informants reflected positively on the methods and their rigour | |
| Clearly presented argument | 1 | 1 | 1 | 1 | 1 | Results are clearly presented; analyses are comprehensive and logically connected to the conclusions and recommendations; alternative explanations are not explored | |
| Transferability and generalizability of the findings | 1 | 2 | 2 | 1 | 1.5 | A statement is included in the dissertation recognizing the possible limit to generalizability of the findings of a single case study; transferability is not discussed in sufficient detail (what, how, and extent of transferability); transferability of findings to other contexts is evident from various outputs of the research; findings have been transferred and applied in other contexts by the PI, participants, and practitioners; informants believed the findings to be widely transferable | |
| Limitations stated | 1 | 1 | 1 | 1 | 1 | The limitations section in the dissertation is underdeveloped (one sentence is given to exploratory nature of the project and how findings are not generalizable, but this could have been explained in more detail); strengths and disadvantages (shortcomings) of methods selected are discussed in the methodology section; the research proposal includes brief reflection on limitations | |
| Ongoing reflexivity and monitoring | 2 | 2 | 1 | 2 | 1.75 | Researcher self-reflection is mentioned as part of the research process, but not discussed in detail in the dissertation (some indications from various e-mail correspondence to the advisory committee and mention of how interviews stimulated additional literature review); changes and adaptations made to methods/approach are transparent and justified (e.g. photo-voice method) | |
| Legitimacy | Disclosure of perspective | 2 | 1 | 1 | 1 | 1.25 | Potential bias is briefly mentioned in the dissertation, but the PI is clearly aware of their positionality based on other documentation (admissions proposal, research proposal, book chapter); sources of support (funding, collaboration with DAC) are noted; however, bias and positionality are not discussed in terms of the effect on the project results (only in terms of how funding supported deeper inquiry) |
| Effective collaboration | 1 | 1 | 1 | 1 | 1 | Interviews conveyed that the internship with DAC for the Mitacs funding and collaboration to funnel/apply findings to the organization was appropriate and clearly agreed upon (however, this was not documented within available project documentation); researcher interview relayed collaborative efforts with participating organizations to provide | |
| Genuine and explicit inclusion | 1 | 2 | 1 | 1 | 1.25 | Project involved multiple participants from 33 projects, representing over 20 organizations in Moshi – dissertation describes the different characteristics of these organizations (donor country of origin, project focus, size, age); project’s focus on voice of recipients of private development aid clearly distinguishes between end-use recipients and aid implementers; dissertation also distinguishes between the cultural background of local Tanzanian, ex-patriate, and foreign implementers; project also collected data from private donors; second round of interviews functioned to validate accurate representation of the findings; roles of advisory committee, translators, and other support (e.g. transcription, draft editors, editing for SSHRC Storytelling video) are briefly mentioned in the acknowledgements; interview approach had greater emphasis on collecting information than co-generation of knowledge | |
| Research is ethical | 2 | 2 | 2 | 2 | 2 | Project received ethical approval by RRU Research Ethics Board; project sought and received approval from the Tanzanian Commission for Science and Technology to conduct the research; dissertation outlines in detail the steps taken to obtain consent and ensure anonymity of participants; informants reflected on the PI’s commitment to confidentiality and anonymity, which is maintained during this evaluation; dissertation outlines instances of ethical challenges and how they were solved (e.g. use of a recorder, redacted information/turned off the recorder, group interviews, photo-voice method, multinational merger, cultural appropriateness, etc.) | |
| Effectiveness | Research builds social capacity | 2 | 2 | 1 | 1 | 1.5 | Some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; several informants noted that the findings confirmed/reinforced that they were on the right path; practitioner noted that the research “opened up a dialogue within the NGO world in the organization of accountability”; capacity-building was not explicitly built into the project design, though there is potential that |
| Contribution to knowledge | 2 | 2 | 2 | 2 | 2 | Informants noted that the research uncovered relevant practice gaps they were facing and contributed to the academic literature; some participants noted they learned from the interview discussions and findings (e.g. unintended consequences), while others did not; participants, advisory committee, and practitioners described the findings as “useful” and “lessons to be learned”; researcher disseminated findings to both academic and practitioner target audiences (via conferences, magazine publications, website, SSHRC video, researcher’s consulting, etc.); participants and practitioners shared the findings through their networks | |
| Practical application | 2 | 2 | 2 | 2 | 2 | Intentionality for practical application is discussed by informants; findings have been applied by some participants/organizations (e.g. boards have adjusted the focus of their discussions, added question on unintended consequences into project proposal and monitoring processes); researcher has applied findings in own work and consultancies (e.g. Transform International, Lalafofofo, CAGP, SOVI-BCCIC); findings applied by practitioners (e.g. DAC, Lalafofofo, Transform International) | |
| Significant outcome | 1 | 2 | 1 | 2 | 1.5 | All ten end-of-project outcomes were fully or partially realized with clear project contributions; some organizations have changed their practices as a result of the research (with intent to become more effective); organizational changes are likely to support positive social changes through their work |
Sharing content requires targeting cookies to be enabled. Please update your cookie preferences to use this feature.