Leave No One Behind

AIRFE App is an innovative tool to ensure inclusive evaluation in times where adaptation is needed

⛶ Full screen

Who We Are:"Leave No One Behind"

Our team includes;

Ariella Carolino, Freelancer Monitoring and Evaluation Consultant (Brazil),

Claudia Olavarría, Sociologist, University of Chile (Chile),

Sylvie Margat, Evaluator, French Agency for Development (France) and

Sharon Ndandula, Results Measurement Consultant, International Finance Corporation (Zambia).

What We Are Doing

In the context of the COVID-19 pandemic and other crises, data collection within remote communities and with hard-to-reach end beneficiaries faces challenges due to travel restrictions for international consultants, limited availability of national evaluators due to growing demand, and shifting priorities of field office staff and counterparts due to the COVID-19 response. The limited access to field sites can lead to a heavy reliance on remote data collection methods, which, in combination with convenience sampling, entails intrinsic potential biases against under-represented groups in the selection of respondents.

Our team is working on developing an inclusive- oriented tool, that is adaptable for times of crises, that evaluators can use to get in touch with the most vulnerable and hard-to-reach persons that evaluators can use in order to solve the problem of adequate data collection (and collaborative evaluation management) during moments of crisis, like COVID-19.

10.07.2020 15:13

Daily Progress and Outcomes

*The team has been meeting daily to brainstorm and has come up with some interesting ideas. We are collaborating via Google Docs, too, to share our ideas and comments.

*The group worked well as a team once it started the brainstorming process through video calls, bringing common ideas/problems during the meetings and developing further from there. There was no need to vote on a better idea because the solution was gradually developed together.

*Due to the hackathon format of delivering a 2-3 minute pitch video, the group decided that an innovative tool would be the best idea to follow, instead of a checklist of standards and procedures to ensure equity and comprehensive representation in data collection in moments where adaptation is needed. That research effort, although extremely necessary, is more easily found across the globe, in diverse evaluation think tanks/institutions. In that case, the innovation itself would be more of a systematization type, which is an ongoing work already, and would demand an immense time that is unavailable during a hackathon.

*The team conducted some interviews with some stakeholders working in crisis areas to find out what tools they are using but also across the data collection community of evaluators. The interviews helped the team understand better the tools for data collection used and how these could be improved.

*The innovative tool idea, on the other hand, would be greatly useful from a hands-on perspective. In face of an immediate crisis (whether it be a climate change-related disaster, a pandemic such as Covid-19, or the outbreak of war), evaluators would benefit from a tool both accessible and easy to tailor to their own needs.

*There was a consensus around the scope of the challenge and the multiple issues/concerns involved, which also led the team to keep a broad frame while thinking about the "ideal persona". Like the team name itself, our goal is to “leave no one behind” during evaluations undertaken in times of crises. This meant trying to develop a tool flexible enough to adapt and to use in various types of scenario and data collection necessities.

Main issues that our challenge should tackle and address

1. Illiteracy

2. Digital inaccessibility (no access to the internet, Wi-Fi or equipment, such as computers, tablets and smartphones)

3. High Vulnerability (marginalization, domestic violence, poverty, drug disorders)

4. Intense spatial mobility during work hours (street vendors, for example)

5. Homelessness or frequent housing mobility

6. Hard-to-reach communities, such as indigenous people in the Amazon region

7. Refugees and recent immigrants without proper documentation

=> The issues, above, also describe some of our stakeholder’s persona most challenging social and economic characteristics. They are at the core of challenge 3’s concern, which is how to ensure that evaluation being undertaken in times of crisis (such as Covid-19) can be successfully inclusive and leave no one behind.

The Adaptive and Inclusive Remote Field Evaluation App

View post on
--- `10.07.2020 15:16` The most effective way that hard to reach populations can be reached through an innovative app, with a set of features and toolkit, to ensure inclusive evaluation in times where adaptation is needed. The goal of the app is to facilitate reliable, comprehensive and effective data collection and collaborative evaluation management in vulnerable, distant and other hard-to-reach populations or communities during a crisis (whether it be a climate change-related disaster, a pandemic such as Covid-19, or the outbreak of war). Further, the app intends to facilitate high ethical and technical standards for the design and implementation of data collection in evaluation processes and enable a step-by-step and self-tailored evaluation process design, by allowing evaluators/Project designers to answer a detailed questionnaire about his/her data collection targets, goals, focal points, limitations and other relevant information for designing his/her tailored process in the app (following best practices in evaluation: culturally responsive, do no harm, equity, gender-sensitive and free consent, among others). --- `11.07.2020 12:56` The team has discussed and come up with initial designs of what the app should contain and how it should be work. The app features are applicable for data collection according to the specific needs in the evaluation design phase. --- `11.07.2020 13:05` In order to utilize the app, a set of different users have to be predefined for each evaluation project. It has tailored tools for working with vulnerable and hard-to-reach populations. The “project owner” should select vulnerability conditions and accessibility barriers to ensure pertinence of features (for example, physical disabilities, illiteracy, differences in language, age, gender or violence-related problems, digital limitations etc). The app also comes with the following; - Research Design Tools such as question banks and question builders - Management tools such as real-time reports on progress made in data collection and an alert system - Collaborative Mapping Tools (with public and private user interfaces) - Qualitative Data Collection Tool with various user-friendly interfaces for video, audio, images on the specific requirement by the evaluation team. - Dissemination tools in order to share findings with all stakeholders --- `11.07.2020 13:11`

Prototyping and Testing (requirements to enhance app’s usability)

  • Engagement of grassroot community organizations (GRO);
  • Identification and engagement of focal points/facilitators in communities;
  • Training GRO and community focal points to contribute with data collection in the app;
  • A methodological design that incorporates a set of complementary tools;
  • An “evaluation design” feature in the app, making it possible to tailor the process according to program/project, territory, target-population, evaluation goals and limitations (in terms of language, culture, digital inaccessibility, illiteracy, physical disabilities etc);
  • A representative (intentioned) sample design;
  • Budget to distribute devices to use the app, in case it is needed;
  • Provide internet access for grassroot community organization, if needed and feasible.

12.07.2020 13:24

After a lot of discussions and iterations, we have our prototype!

12.07.2020 20:22

WE DID IT! Our video has finally been posted. After many hours of discussions, google doc editions and takes, we finally got it right on our 4th take :) This has been a great opportunity for collaboration. We have all learnt so much and built a great network! Thankyou IPDET for the opportunity to participate in the Evaluation Hackathon!

12.07.2020 20:59

Launched at Evaluation Hackathon by

ariella_kreitlon_carolino sylvie_margat sharon_ndandula claudiaolavarriam felix_stips

Maintainer hackathons-ftw

Updated 13.07.2020 09:35

  • sharon_ndandula / update / 13.07.2020 09:35
  • sharon_ndandula / update / 13.07.2020 09:32
  • sharon_ndandula / update / 13.07.2020 09:28
  • sharon_ndandula / update / 13.07.2020 09:26
  • sharon_ndandula / update / 13.07.2020 09:23

Adaptive Evaluation

Inclusive and adaptive evaluations in times of Covid-19

During crises such as COVID-19, evaluation teams need to rely on remote data collection methods. This entails intrinsic potential biases against hard-to-reach populations that need to be mitigated through innovative methods and tools. What methods and tools can evaluators use to ensure that hard-to-reach populations are not left behind in evaluations undertaken during crises? Covid 19 unveiled an invisible thread linking methodological challenges, need for equity, and “do no harm”. To be effective, evaluations have to adapt and respond to each of these challenges. What concrete and practical tools can we develop by looking at the intersection of methodological challenges, “do no harm”, equity, and innovation? What can we learn and immediately apply from evaluation methods designed to operate in fluid and uncertain conditions and with imperfect information?

All attendees, sponsors, partners, volunteers and staff at our hackathon are required to agree with our Code of Conduct. Organisers will enforce this code throughout the event. We expect cooperation from all participants to ensure a safe environment for everybody.

Creative Commons LicenceThe contents of this website, unless otherwise stated, are licensed under a Creative Commons Attribution 4.0 International License.