Request for Proposal: Supporting Good Governance in Healthcare Administration (SAHA) Midterm Evaluation -CIPE Retour vers les opportunités

•  Abstract

The Center for International Private Enterprise (CIPE) is seeking a locally based evaluation firm in Tunisia to conduct an external independent midterm evaluation for a project funded by the Bureau of Near Eastern Affairs’ Office of Assistance Coordination (NEA/AC) and the United States Embassy in Tunisia.

• About CIPE

The Center for International Private Enterprise (CIPE) is one of the four core institutes of the National Endowment for Democracy.  CIPE has over 30 years of experience working with chambers of commerce, business associations, think tanks, and other civil society organizations (CSOs) to strengthen democracy around the globe through private enterprise and market-oriented reform.  

CIPE’s approach to programming is guided by our organizational mission to promote democracy based upon three principles: respect, integrity, and excellence.  Our programs are developed based upon local ownership for sustainability, inclusion of all stakeholders, learning and innovation based on lessons learned, and accountability to the communities where we work, partners and donors.

CIPE is continually updating our strategies and approaches so that we can deliver the most effective programming possible.  In 2019, CIPE adopted an evidence-based and data-driven approach to how evaluation supports programming.  This approach is intended to go beyond project-based donor reporting and accountability to developing better understanding of the outcomes and impact of distinct approaches to demonstrate innovation and “what works.” 

1. Project Background

CIPE has been contracted to work with the Tunisian Ministry of Health (MoH) to strengthen good governance in the Tunisian health sector by improving transparency and management.

Through partnership with Tunisian civil society organizations, I WATCH and BEDER along with private sector firms, CIPE will tackle the many dimensions of corruption in Tunisia’s healthcare system by: 

1. Working with the Tunisian Directorate of Pharmacies and Medicine (DPM) to digitize its processes and reduce arbitrary discretion in decision-making and increase transparency in distribution of medicines.
2. Developing a more effective regulatory framework for the MoH to promote good governance and transparency in the health sector; and
3. Building the capacity of the MoH’s network of Regional Training Centers to improve the implementation of good governance in public hospitals. 

• Scope of the Evaluation:

The purpose of this midterm performance evaluation is to make a systematic and unbiased assessment of the degree of fulfillment of project objectives and the effectiveness of the current program implementation process. The evaluation should provide information that is credible and useful, enabling the incorporation of lessons learned into the decision-making process of CIPE, its partners, and the State Department to inform the program implementation process for the remainder of the project.

Principles underpinning the approach to the evaluation are:

• Impartiality and independence of the evaluation process from the programming and implementation functions;
• Credibility of the evaluation, through use of appropriately skilled and independent experts and the transparency of the evaluation process and the presented results;
• Participation of stakeholders in the evaluation process, to ensure different perspectives and views are taken into account;
• Usefulness of the evaluation findings and recommendations, through timely presentation of relevant, clear, and concise information to decision makers.

CIPE would like the midterm performance evaluation to answer the following evaluation questions (EQs):

1. What has been the progress made so far towards meeting the objectives of the project’s three working groups? What changes need to be made to improve their progress and efficiency to ensure successful completion by the end of the project? 
2. How effective have project efforts associated with updating the legal framework to monitor Tunisia’s healthcare system been? What strategies or resources (if any) are needed to move this process along? 
   1. What specific actions has the Legislative and Regulatory Reform Working Group done to further these efforts? 
   2. How effective have these actions been to reach the project end goal?
   3. What changes need to be made to improve the progress and efficiency of the Legislative and Regulatory Reform Working Group? 
3. What are the successes and challenges affecting the development of a new digital system for the DPM? 
   1. What specific tools, strategies, resources or techniques are needed to fully operationalize the new digital system? 
   2. To what degree has the Working Group moved this objective forward? 
4. What are the most significant training curriculum updates which have been made for Ministry of Health staff and doctors? 

1. Are Ministry of Health staff and doctors satisfied with the revised training curriculum? If not, what additional changes do they recommend? 
2. To what extent can appropriate use of the revised training curriculum be evidenced?

1. How well is the SAHA project engaging with all relevant stakeholders (i.e. doctors, government officials, civil society organizations and implementing partners) involved in the project? 
   1. Are there ways that communication and engagement across stakeholders could be improved? If so, how? 
   2. What is the administrative and organizational perspective of all relevant stakeholders involved regarding the implementation of the project thus far? 
   3. Are there additional stakeholders or populations with whom project implementers should be working? If so, how can the implementation team better involve those stakeholders and/or populations moving forward? 

CIPE is requesting a mixed methods performance evaluation using quantitative and qualitative data and drawing on both primary and secondary data sources. Secondary sources feeding into a desk review may include reliable and relevant sources such as government and ministry statistics, program documents, and other relevant background information. Primary sources may include focus group discussions, key informant interviews, structured observations, and stakeholder surveys. Participants should be those who have been a part of the SAHA project, such as government partners (Tunisian Ministry of Health) and implementing partners (I Watch & BEDER), as well as public hospital doctors and civil society transparency organizations. CIPE, partners, and the selected contractor will agree upon a list of stakeholders. The selected evaluation firm will use this list as a starting point for recruitment, and the selected contractor may use snowball sampling to increase the sample size and generate a list of alternate respondents. The selected contractor should also conduct a background document review and a review of monitoring data that has already been collected and cleaned by CIPE. This information will be provided to the selected evaluation firm for analysis and triangulation purposes upon contract award.

The selected contractor will prepare a technical and financial proposal that addresses all aspects of the data collection and final report. The final evaluation report should be no longer than 30 pages (excluding annexes) and should be structured as follows:

• Acronym list
• Executive summary
• Main section
  • Introduction (which should include the following):
    • Project description
    • Project context
    • Evaluation objectives and methodology

• Analysis of the findings based on the evaluation criteria
• Conclusions and recommendations
• Lessons learned
• Annexes (which could include the following as applicable):
  • Map of the project area
  • List of actors consulted
  • Literature and documentation consulted
  • Other technical annexes

The conclusions and recommendations section of the final report should detail forward-looking recommendations on any suggested improvements to the project implementation processes to ensure the program meets all its planned objectives by the end of the project in an efficient, sustainable, and impactful manner. 

The estimated level of effort to potentially include the following anticipated tasks and assignment phases:

Please note: Due to the coronavirus pandemic, CIPE is anticipating that some data collection for this evaluation will be conducted remotely. All proposals submitted should detail the capacity of the evaluation firm to perform data collection virtually and what local resources and staff are available for the proposed data collection slated to take place in Tunisia. 

Estimated Level of Effort Anticipated Tasks and Assignment Phases

Note: The evaluation firm should consider what efficiencies can be proposed within this anticipated 

scope of effort.  The number of days can be adjusted (either increased or decreased) within reason depending on the proposed technical approach and case made by the selected contractor.

Background Documents will be provided to the selected evaluation firm such as: 

• Project workplans
• M&E logic model and workplan 
• Project progress reports
• Monitoring data 
• Contextually focused background documents 

General Focus Group/Key Informant Interview/Semi-Structured Interview Approach 

If included as a part of the technical proposal design methodology, the evaluation firm shall be responsible for the design and execution of the focus group discussions/key informant interviews/semi-structured interviews proposed, including the following tasks:

• Recruitment.  The evaluation firm will recruit all participants according to the screening criteria provided by CIPE. If discussion groups are conducted, the consultant should recruit 8-10 participants for each group as well as 2-3 alternates in case any original participants are unable to participate. For key informant interviews, the evaluation firm will recruit enough alternates. All participants will be required to give oral or written informed consent to participate in this evaluation. 
• Discussion Protocol Guide.  The evaluation firm will produce the initial discussion protocol guide for focus group discussions/key informant interviews/semi-structured interviews within four weeks of the beginning of the contract award.  CIPE will review and provide suggestions for improvement to the draft discussion guide(s). It is expected that the evaluation firm will format and translate the discussion guide into the local language of the participants. CIPE reserves the right to review the translation(s). Draft informed consent language should be included which can be adjusted by the evaluation firm to conform to legal requirements in the research country. The discussion guide will be piloted with a group of 8-10 pilot participants for focus groups or 3 key informants who fit the recruitment criteria. If a focus group discussion is conducted, this pilot will be moderated by the same moderator who will moderate the actual groups. If a key informant interview is conducted, the pilot key informant interviews (KIIs) will be conducted by the same qualitative interviewers who will conduct the actual KIIs. The pilot session(s) will be audio recorded, and the recording will be provided to CIPE. The consultant will prepare a 1-2-page pilot report based on a template provided by CIPE. Based on this pilot report, CIPE will suggest any additional changes needed to the discussion protocol guide and the evaluation firm will address those changes before the discussion guide being approved for data collection purposes.  
• Focus Group Moderator and/or Qualitative Interviewer.  The consultant will provide trained and experienced moderator(s) and/or qualitative interviewer(s).  The moderator(s) and/or qualitative interviewer(s) should familiarize him/herself with the topic of discussion prior to the discussion groups or interviews.
• Recording.  Depending on the context, the evaluation firm will audio record all aspects of research necessary for full and accurate data collection, transcript generation and inclusion of illustrative samples in the final analytical report. The consultant will audio record focus groups or interviews and the recording should be in high quality with clear sound. 
• Summary, Transcripts, and Analytical Report.  The selected contractor will provide CIPE with full verbatim transcripts in English of each focus group discussion, semi-structured interview or key informant interview that will identify each participant speaker by number to link each comment to the participant’s gender, exact age, education level, city and occupation or other demographic category determined by CIPE to be relevant to the study. The selected contractor will analyze all discussions or interviews and write and edit a final analytic report (see Deliverables) for review, editing and acceptance by CIPE within four weeks of the final focus group session or interviews. CIPE will provide a report template. The report is expected to synthesize findings across all participants and all discussion sessions or interviews, while noting any significant differences of opinions between participants and sessions. The report must present actual analysis and illustrative quotes from participants.

1. Deliverables and Timeline 

EVALUATION DELIVERABLES

1. Inception report, including research methodology
2. Draft final report with recommendations
3. Revisions following two feedback rounds
4. Final report
5. Learning/sensemaking presentation highlighting major findings and recommendations

FOCUS GROUP / KEY INFORMANT INTERVIEW / SEMI-STRCUTURED INTERVIEW DELIVERABLES (if offeror includes in technical proposal design)

1. Draft discussion guide(s)
2. Final discussion guide(s)
3. A 1-2-page pilot report per each discussion guide
4. Proposed participant recruitment plan
5. Full transcripts of all focus group discussions/key informant interviews/semi-structured interviews as described above
6. Video/audio recordings of the discussions and interviews, with clear audio. Recording should be delivered to CIPE electronically (for example via a file transfer service) or delivered to the CIPE office.

SURVEY DELIVERABLES (if offeror includes in technical proposal design)

1. Sampling plan (revised after CIPE feedback)
2. Pretest report and revisions of survey protocol with CIPE feedback; details of data entry plan, pilot test, action plan for data collection, check back plan, recruitment, training, and contracting of experienced field staff
3. Copies of the final questionnaire, as fielded, in English and local translations
4. A complete data set and codebook formatted as Excel or SPSS file
5. A topline report with data frequencies for all questions and crosstabs by key demographics of interest (as agreed with CIPE)

The final report will be expected to be completed by January 31, 2023.  The duration of the consultancy will begin in October 2022 (depending on selection time and length of contracting process for selected contractor).

While the dates of the consultancy may be adjusted based upon the evaluation firm proposal, to be most effective, the completed evaluation needs to be timely to be useful for performance improvement. 

• Evaluation Standards

The consulting firm should adopt a consultative, inclusive, and participatory approach while maintaining an independent perspective consistent with OECD DAC and American Evaluation Association standards.   CIPE expects all monitoring and evaluation work and products to be based on standards aligned with the American Evaluation Association Guiding Principles for Evaluators including:  systematic inquiry; competence; integrity and honesty; respect for people; and responsibilities for general and public welfare as outlined on their website:  http://www.eval.org/p/cm/ld/fid=51 (and attached to this document). As relevant to the specific consultancy, CIPE also expects that the consultant will comply with global human subjects research standards and respect vulnerable and marginalized populations, and as appropriate the following standards:  American Association for Public Opinion Research (AAPOR)   (2015)  The Code of Professional Ethics and Practices (available online at:  https://www.aapor.org/Standards-Ethics/AAPOR-Code-of-Ethics/AAPOR_Code_Accepted_Version_11302015.aspx) and the Gender Practitioners Collaborative  (2016), Minimum Standards for Mainstreaming Gender Equality (available online at: http://genderstandards.org/).

The evaluation firm affirms that they have read these principles, that they have no roles or relationships that might pose a real or apparent conflict of interest and agrees to follow them in conducting this evaluation.  

• Evaluation Accountability

To ensure compliance of technical evaluation deliverables with CIPE evaluation standards and other expectations set forth in section VI, the CIPE Evaluation Department, which is independent from the CIPE program team managing this contract, will review, assess, and approve the contractors’ deliverables before payments are made to the contractor according to the payment schedule.

• Remuneration

Remuneration will be negotiated based on the professional qualifications and relevant experience. All applicants are instructed to submit a detailed budget indicating the costs associated with the approach aligned in their scope of work. 

• Team Composition and Evaluator Competencies

The consultancy firm needs to be a legally licensed international/national organization with a commendable track record and at least 10 years of progressive experience from which at least 5 years’ experience in providing consultancies with evaluation skills and expertise in and understanding of the theory and practice of anti-corruption programs, health, and gender programs in developing problem-solving skills, evaluation of training;  Contractors who have successfully conducted independent evaluations for international development organizations similar to CIPE will be given preference.  Key staff should have the appropriate experience, expertise, and credentials for their role. Resumes should be submitted for all key staff mentioned in the staffing section of the proposal. 

• Technical Evaluation Criteria Scoring

CIPE will base the selection of the evaluation firm on the following criteria:

• Strength of evaluation design and methodology detailed in technical proposal (30%)
• Competence of proposed personnel leading the evaluation (20%)
• Relevance of experience/ past performance (15%)
• Quality of data processing, data quality control, and data analysis procedures (10%)
• Cost (25%)