EXPERT VIEW

Experts have a number of weaknesses that have been demonstrated to negatively affect their judgment, and studies have shown in some areas, such as predicting the future, that “many of the experts did worse than random chance, and all of them did worse than simple algorithms.” These concerns limit experts’ usefulness and make us confident that they should not be the only perspective used. Many of our biggest concerns with experts are cognitive biases that cross-apply to the vast majority of human judgments. Not all the following concerns apply to every expert, but they are generalized concerns that will apply to a large number of experts. 

​

• Unequal application of rigor

• Inconsistent and unclear epistemology

• Cognitive bias

  • Anchoring 

  • Groupthink

  • Illusion of control (weaknesses with randomness) 

  • Confirmation bias

  • Many other human biases

• Lack of transparency in argument generation 

• Memory concerns

• Limited specificity

• Lack of decisiveness 

Unequal application of rigor: A major concern with experts is equal application of rigor. Given all the information currently available, a motivated actor can find evidence supporting almost any viewpoint. Thus a fairly weak argument could hold a lot of weight in an expert’s view if he or she has not considered it skeptically or if it fits a prior worldview the expert holds. Similarly, if an expert does not like an idea, he or she finds it easy to be significantly more critical of it than would be justified compared to other ideas or viewpoints he or she holds. This rigor concern makes it highly challenging to take expert conclusions without a deeper sense of how they, for example, react to any new idea.

Inconsistent and unclear epistemology: Another factor that makes expert judgment less strong a source of evidence is the relative rarity of a formal or consistent epistemic system. Experts often have views about how to weigh different types of evidence but few have thought about this problem explicitly, and very few have publicly laid out how they would compare and integrate different pieces of evidence into their endline viewpoints. 

Cognitive bias: There are a number of cognitive biases that affect humans. Experts are fundamentally just more informed humans and thus generally suffer from the same biases. Some evidence suggests that experts can be affected even more strongly by some biases than the general population. One mitigating factor is that if multiple experts are spoken to, their biases will not necessarily overlap, and their average quality of judgment tends to do better than that of a single expert. There are hundreds of biases that can affect judgment and decision-making, but some that seem particularly relevant to experts when considering charity ideas are: 

Anchoring: Is when an individual depends too heavily on an initial piece of information offered (considered to be the “anchor”) when making decisions (1). Experts can often anchor on a specific idea for a charity early in a conversation or before the conversation has even started. Many experts will have projects they have already supported or invested time into, and these existing projects will generally be compared to any new idea with a high level of comparative skepticism regarding competing or different ideas. 

Groupthink: Is a psychological phenomenon that occurs within a group of people in which the desire for harmony or conformity in the group results in an irrational or dysfunctional decision-making outcome. Group members try to minimize conflict and reach a consensus decision without critical evaluation of alternative viewpoints by actively suppressing dissenting viewpoints and by isolating themselves from outside influences (2). In the case of charity ideas, if an idea has not been previously tested or considered by experts in the field, often they will be more inclined to dismiss the idea than they would if the same concept was presented by someone connected to their ingroup. Although this is a useful heuristic for experts to use, it can make them underweight new ideas relative to more established ones, particularly if the new ideas are generated using an intelligent process (e.g., CEAs). 

Illusion of control: Is the tendency for people to overestimate their ability to control events; for example, it occurs when someone feels a sense of control over outcomes that he or she demonstrably does not influence (3). This connects closely to experts having difficulty discerning effects from randomness or noise. The way this connects to charity ideas is that often experts will put more weight on personal experiences they have had; for example, if idea A has worked in the past, idea A will always work, and if idea B failed in the past, idea B is likely to fail. These are often assumptions held without careful consideration of the environmental factors that are different or the nonresults-focused factors; for example, a higher chance of failure might be worth it if the win is several times larger.  

Lack of transparency in argument generation: Experts have formed their views using a considerable number of sources and experiences. A byproduct of this is the great difficulty of tracking down the basis for a given viewpoint. This can make it very challenging to confirm or disprove a given idea or even know how much weight it should be given. This is not the fault of the expert but is a flaw inherent in expert-based information. 

Memory concerns: Evidence has demonstrated that memory is a fallible tool, but generally when speaking to an expert there is a high level of reliance on the expert’s memory. A remembered version of a study or conversation could be significantly different than the original. It is also hard or impossible to detect these memory effects given the lack of transparency.  

Limited specificity: Many experts are unwilling to give specific estimates such as a percentage-based chance of success. Experts are often unwilling to make claims that could be used in other methodologies such as CEAs, particularly if those claims cannot be anonymized. 

Lack of decisiveness: Similar to the specificity concerns, experts are often unwilling to make decisive claims even when taking a neutral or unsure stance would have its own ramifications. This is often taught practice in academia and can be a good habit when it comes to truth-seeking, although it can impair comparison among different options.  

 

4. HOW MUCH WEIGHT WE GIVE EXPERTS

 

Experts are quite an important source of information, and a lot of fundamental information comes from EpV as interpreted by others. However, experts suffer from many human biases that more algorithmic systems are less affected by. Experts are one of our five perspectives, the others being: WFM, team intuition, CEA, and our prior view. We expect our experts to generally hold more than one-fifth weight but likely not more than one-third of the weight of the model. Given the considerable variation depending on the specific charity idea and cause area, we expect experts to be stronger in areas where it is harder to get solid numbers and commonsense intuitions can be used as an effective guide. ​

 

5. OUR PROCESS FOR SPEAKING TO EXPERTS

 

5.1. SELECTING

 

Who do we count as an expert, and how do we speak to experts? We have two main processes to generate expert names. Our list of experts is generally created opportunistically instead of systematically.

 

1. Finding via research. As we are conducting directed or undirected research, names often come up of key people in the area. These names are then noted down and later contacted as experts. 

2. By recommendation. Often when we speak to experts we ask for others who would be helpful to talk to. Thus many of our experts come from peer recommendations.  

The types of experts we end up speaking to in practice fall into the following categories. 

​

1. Broad experts: Generally broad experts are the smallest group we speak to. These are people involved in comparing a range of interventions within a given area. Often their knowledge will be more generalized when compared to domain or specialist experts. An example of a broad expert might be a researcher at an evaluation organization such as GiveWell or Copenhagen Consensus or an author on the Disease Control Priority reports. However, they could also be a large funder or cross-area researcher who has a good sense of the space overall. These experts would be considered “broad experts” in a cause area such as global poverty.  

2. Domain experts: We generally speak to many domain experts, and they often consist of the largest group of experts we speak to. Often such an individual could be the author of multiple studies in a given area, someone who worked on a synthesis of the intervention or domain being considered, or someone who conducted an evaluation or meta-analysis of the area or an adjacent area. These experts would be considered “domain experts” within an intervention area such as vaccination reminders.

3. Specialist experts: We often speak to a small number of specialist experts. These are individuals who are highly informed but in a small niche within a given charity idea or intervention we are considering. This might be a biology specialist who has deep information about food fortification uptake rates. It could be a person experienced in an element of running an intervention on the ground, such as country-level experience, or an individual who has written a study related to the execution of a given intervention. We consider these experts specialists who could have insight into an element of, but not the whole, intervention. ​

 

5.2. CONTACTING

 

When contacting an expert for the first time, we generally use an email similar to the following. 

Dear _____

_____ person recommended that I speak to you because of your background in _____ OR I am researching ______and I read your paper on _____ (which was very _____! (compliment, like interesting or well done). Based on this, I thought you might know the answers to a few questions about the topic. 

I am a research associate at an organization that researches and funds new nonprofits which put that research into action (our website here). Previously we worked mainly in global poverty and animal welfare, but we have expanded to mental health so we can make this issue a higher priority in the global sphere. We are based out of London and so far have work underway in Oxford, London, the US, India, and Asia. 

Some questions I have are: ____ and  ____. Would you happen to have the time to jot down some quick answers to the above, or maybe discussing it via Skype might be easier? We’d really love to have your input and research inform our funding decisions and what charities to work on in the sphere. 

Best regards,
______

 

5.3. HOW TO SPEAK TO EXPERTS

 

Experts are ultimately just people like anyone else, so most standard conversational rules apply to them. A few elements to highlight are:

• Be humble – At the point you are talking to experts, you are new to the field, so you should try to come across as someone who is surprisingly informed for a nonexpert. Ask for only a little bit of time, such as one Skype instead of longer commitments. Say you are happy to keep any comment anonymous if they review the conversation notes, or say it should be so during the conversation. Try to take a broad interest in the topic matter as a whole even if it is not directly tied to the question you are asking.  

• Be prepared – Being thoughtful with an expert’s time is important. If they have written a whole book on a given topic, for example, you should at the least review a summary before talking to them about it. The same goes for website content they have created. As well as reading content beforehand, you should think about the most important questions and which ones could get cut if you run out of time. Have a backup system for contacting if the first one does not work (e.g., Google phone credit you can use to call someone if Skype is not working). 

• Frame opposing views using a citation – If you want to push on a point or perspective that an expert has claimed, do not describe it as “you are skeptical of point A”; instead tie it directly to your research (“A different expert I spoke to was skeptical of point A”). 

• Go deeper – Try to cover the key questions on your agenda, but if something comes up that seems important you can ask more questions relating to that area. Ask follow-up questions, such as “You said that . . . Why do you think that is?” 

• Ask comparative questions – Few experts will have a great sense of what the percentage chance of something happening is or a clear expected value for a given intervention, but they often give excellent answers to more comparative questions. “Does X seem like it would cost more per person than Y?” is easier than giving an exact number for either. 

• Ensure that they have answered your question – If you ask questions such as “What are the main strengths and weaknesses of x intervention?” it is quite easy for them to forget the initial question once they have been talking about the strengths of the intervention for a few minutes. Follow up on this with something like “Thanks for outlining the strengths of x intervention; what do you think are the main weaknesses?”

• Give them space to think – Don’t move onto the next question immediately after they stop answering the previous one. Ensure that there is a small pause so that they can add something else if they think of it.

 

5.4. RECORDING

 

At the start of the interview, we ask the expert if they are comfortable having the interview recorded. This means that we don’t have to take notes during the interview so we can entirely focus on the questions and their answers to them, and it also makes summarizing the interview easier because we can go back and listen to their answer to x question again, for example.

To be safe, we usually record in two different ways.

1. Recording the interview through Skype. More information on this can be found here.

   1. When recording through Skype, remember to click “Stop Recording” at the top of the screen before ending the call, otherwise there is a risk that the recording will not be saved.

   2. Skype only keeps the recording for thirty days, so you should save the recording to your computer immediately. We also store these recordings in our Research Agenda -> <cause area> -> Expert Interviews folder. 

2. Recording the audio of the interview using an external program.

   1. We recommend MP3 Skype Recorder for Windows users and Ecamm Call Recorder for Mac users. Both can record automatically.

 

5.5. REVIEWING AS POTENTIAL MENTORS

​

It is critical to use the research team’s extensive interactions with researchers to identify additional mentors. At the same time, the additional effort for a CE researcher should be limited. Hence, the review of researchers as potential mentors should be quick and fully integrated into the existing process that outlines conversations with researchers. Otherwise, the likelihood of poor retention or poor data quality is high.

 

5.6. PROCESS

​

1. Brief survey: Each research conversation is reviewed with a very brief survey/review. This survey is fully integrated into the normal conversation-tracking process. 

2. One-time assessment: In March/April 2020, or whenever suitable, the CE research and curriculum teams will reevaluate the initial reviews. Researchers with potential will be contacted appropriately and eventually sent the Mentor Application Form.

 

 

5.7. BRIEF SURVEY

 

CE Expert Assessment

The survey is kept intentionally short to ensure consistent application. It will be automatically graded and return a suitability score.

Questions to ask 

• 35 percent general cause neutral questions

• 45 percent customized cause (e.g. mental health)

• 20 percent customized person (e.g. expert X)

 

5.7.1 CAUSE-NEUTRAL QUESTIONS

 

A little bit about the project: I am surveying <cause area> experts to get a sense of what would be the best areas to research and launch charity startups in. 
 

1. Could I record our interview for the purpose of making more accurate conversation notes later?

2. Give an outline of the interview (types of questions, length, etc.) 

   1. Do you have any questions about the interview?

3. What got you interested in <cause area>?

4. How long have you been working in <cause area>?

5. All things considered (cost effectiveness, execution difficulty, what existing organizations are already doing, etc.) what specific organizations would you like to see founded in the next five years?

6. All things considered, what intervention or organization do most people think is effective but in your opinion is not? Why?

7. Are there any areas you think are neglected by current actors in the field? Why do you think these areas are neglected?

8. Are there any areas that seem unusually cost effective and evidence based relative to others in the sphere? 

At the end:

1. What are good resources to read (blogs, books, podcasts)?

2. Do you know anyone who would be interested in talking to, mentoring, or supporting a new charity founded in this space?

3. Who else in the movement do you think would give valuable information about these sorts of questions?

4. Would it be possible for you to introduce me to them?

5. Do you have any questions for me?

6. This conversation was really helpful; would it be possible for me to write up a summary of some of the points we talked about and send you a copy for review? Some experts are happy for us to put up a published set of conversation notes on the topic. But we can also anonymize the conversation or combine it with points from other experts into a more overall view (eg., five out of twelve experts think that this is the most promising intervention) depending on what you are most comfortable with. ​

 

5.7.2. MENTAL HEALTH QUESTIONS

 

Customized cause area questions:
There are some broad areas of mental health we are considering. For each area, we consider two questions.
1) How effective the area seems generally (for starting new charities) (below average - average - above average - the best intervention) 
2) What might be the most promising specific things to do within an area. For example, one of the broad areas is therapy, and one specific area that some people think is promising is online apps for lower-income countries. 

• Areas

  • Task Shifting 

  • Peer Support

  • App-Based Therapy

  • Skype-Based Therapy

  • Direct Therapy

  • Therapy

  • Social Change 

  • Lifestyle Change 

  • Screening 

  • Research

  • Medication

    • Diet Supplementation

  • Medical Procedures 

  • Government Lobbying

    • Do you know of any existing organizations that are lobbying for subjective well-being/mental health?

  • Corporate Campaigns

  • Lifestyle Improvements (e.g. weighted blankets)

• Do you think there are important broad areas that are not covered under one of these headings?

• Do you know of any broad resources that compare different global mental health interventions?

• How do you think different metrics of subjective wellbeing compare to DALYs? Which do you think is the strongest measure in the mental health space? 

Customized person questions:
These will be customized questions to ask a specific person. 
For example, a question to ask someone who mainly works in global health:

• What is your perception of the funding landscape for mental health compared to global poverty? Is a new organization likely to be constrained by funding, or would possible sources of funding have good counterfactuals? 

Or asking someone with more knowledge about specific types of mental health interventions:

• How do you think task shifting compares to computer-based therapies in terms of cost effectiveness? ​

 

5.7.3. ANIMALS QUESTIONS

 

Customized cause area questions:

There are some broad areas of animal advocacy we are considering. For each area we have two questions we are considering:
1) How effective does an area seem (for starting new charities)?
2) What might be the most promising interventions within an area? For example, within the area of food technology, one intervention that may be promising is lobbying governments to ensure fair labeling of plant- and cell-based products?

• What is the effectiveness of starting a new charity in the following areas, and what are the best and worst interventions?

  • Corporate outreach    

    • Meat reduction campaigns (eg. meatless Mondays in universities)

    • Chronic welfare improvements campaigns (eg. environmental conditions)

    • Acute welfare improvements (eg. slaughter)

  • Governmental outreach

    • Welfare improvements lobbying (eg. environmental conditions); follow-up questions for welfare improvements:

      • Acute (eg. slaughter, handling, transport) vs chronic (eg. environmental conditions)

      • Increasing follow-through rates of existing campaigns vs new campaigns

      • Positive welfare interventions (eg. environmental enrichment) vs negative welfare interventions (environmental conditions)

    • Meat reduction lobbying (ban the advertising of meat, meat tax, ban imports from lower welfare countries etc.)

  • Wild animal suffering (WAS)

    • Research 

    • Targeted interventions

  • Veg outreach

  • Neglected areas e.g. crustacean interventions, farmed rodents, bugs for human use (entomophagy, snails, silkworms, insects used in research etc.)

  • Research

    • Building the evidence base 

    • Targeted research (eg. institutional ask)

  • Food technology

• Do you think there are important broad areas that are not covered under one of these headings?

• Do you know of any broad resources that compare different animal interventions?

Customized person questions:

These will be customized questions to ask a specific person. 
For example, a question to ask someone who works with funders/works for a fund:

• Is there an intervention that the funders you work with are particularly excited about funding?

  • Is there some metric that these funders are always looking for in the interventions they fund (e.g., high cost effectiveness vs. neglected areas)?

Or asking someone with knowledge of our priority countries:

• What are the top considerations and challenges we should keep in mind when working in <country>?

 

5.7.4. FAMILY PLANNING QUESTIONS

 

​Broad questions

1. How long have you been working in/researching family planning and what got you interested in it?

2. All things considered (cost-effectiveness, execution difficulty, what existing organizations are already doing, etc.), if a nonprofit was starting tomorrow, what specific program would you like to see implemented?

3. All things considered, what intervention or organization do most people think is effective but in your opinion is not? Why?

Comparative questions

1. There are some broad areas of family planning we are considering. For each area, we would like to get a sense of

   1. How effective the area seems generally (for starting new nonprofits) (below average - average - above average - the best) 

   2. What might be the most and least promising specific things to do within an area? For example, you might think contraceptives supply and distribution is average compared to other broad areas, and within that community, health workers are most promising and social franchising is least promising.

   3. Contraceptives provision/distribution/supply 

   4. Incentives 

   5. Information/education

   6. Policy/advocacy 

   7. Research 

   8. Services delivery/ quality 

   9. Social and behavioral change

   10. Women’s and girls’ empowerment

2. Which contraceptive is the most/least important to promote among, say, IUDs, pills, emergency contraception, condoms, implants, injectables, sterilization, and natural/traditional methods?

3. Do you know of any broad resources that compare different family planning interventions?

Funding

1. What kind of programs do you think are particularly easy to get funded, and which ones aren’t?

2. Do you think there is funding available for new organizations to scale evidence-based programs?

 

5.8. NOTE-TAKING AND SUMMARIZING

​

For every conversation, we ask the expert if we can take notes and share them, either named or anonymized as input from one of several experts we have interviewed on the topic. We offer to send them a copy of any notes we take so they can comment if they feel we misunderstood anything. We also offer to send them a copy of our full report if they are interested in seeing other experts’ views or other synthesized research we conduct on this topic. If they are interested, we would love their feedback on the full report. 
​
Conversation notes are summarized into an easily readable document and then sent back to the expert for confirmation that we did not misunderstand or misrepresent anything, similar to the GiveWell conversation notes. 
 
In an experiment using Otter (an automatic transcription service) to record the conversation vs. manual conversation notes (making notes whilst listening to the audio), we found that the time taken for the automatic transcription to be edited into a readable form is slightly longer than the time taken to manually write up conversation notes.

 

5.9. SYNTHESIS

 

Although the bulk of the expert report will be the conversation notes, the project lead will have to synthesize these thoughts into a one-page easy-to-read summary. This can include both narrative explanation of concepts that came up multiple times in conversation notes or table-based data with rough quantification on what experts thought. An example of this can be seen here. The section above the “additional details” is more reflective of the detail that would be helpful to have for a specific idea expert synthesis.

 

5.10. EXPERT REVIEW

​

Our final utilization of experts is when our full report is nearing conclusion. We send out our full report to any expert that indicated an interest in seeing the endline results and ask for any feedback they have.  ​

 

6. DIFFERENT LEVELS OF EXPERT DEPTH

 

6.1. FIVE MINUTES

 

Process 
Speaking to an expert for five minutes is not possible because even finding and contacting them would take close to this amount of time. However, one broad expert conversation can cover a lot of ground. If there are 300 interventions in a given cause area and each is given five minutes, this adds up to twenty-five hours of total time. Of this, seven hours would need to be used to find and contact experts, two hours to prepare the most important questions that would give helpful information across an intervention area, ten hours to interview the top five who respond, and finally eight hours to synthesize the notes from speaking to them and scoring interventions based on the responses. This results in around five minutes per intervention in a given area or about five experts for the area.

Interviews at this level of depth will be used in our research agenda phase. We will contact five broad experts to help us narrow down our long list of ~300 <cause area> intervention ideas. For this level of depth, we will use an automatic transcription service (Otter seems best). We will send this transcription to the experts for review following our Skype call, explaining that this is an automatic transcription that has not been edited and will not be published. 

Expected outcome

• Have spoken to several broad experts to get their sense of the area

• Summarized conversation and how it affects the intervention (one paragraph)

• Synthesized information from the interviews that will enable rating of each idea 

Advice

• Contact at least three times the number of experts you want to talk to. 

• Reach out to them before you start any of the methodologies. Even if EpV turns out to be the third or last methodology, they will take time to respond, so you can pursue the others in the meantime.

• Send your questions to them before your Skype call so they are aware of what kind of questions you will be asking and can prepare if they feel they need to.

Lessons learned
The categorization of interventions into a broad category (which you will ask experts to rate) is a highly important part of this process that we should have put more thought into for three main reasons.

1. It was hard to come up with broad categories that all interventions could fall under, but this was necessary because most experts would not have enough time to rate each individual intervention idea, so we had to get them to rate broad category areas instead.

2. Because all interventions did not fit neatly into the broad category areas, we sometimes had to make inferences about how they would score an intervention based on specific comments made during the interview.

3. In some cases, it was difficult to score individual interventions because they could fall under multiple broad category areas. For example, in animals, we had a lot of wild animal suffering interventions, so we came up with the broad category of “Wild Animal Suffering” (WAS) and got experts to rate this category. We also had a lot of interventions to prevent acute suffering, so we came up with the broad category of “Acute Suffering Interventions” and got experts to rate this category. In cases such as this, it was difficult to tell, for example, what category the intervention “using snap traps over glue traps for wild rodents” would best fall under: wild animal suffering, or acute suffering? There were a few instances like this that made some interventions difficult to score.

Potential solutions

1. Send experts the whole list of interventions and get them to rate each one individually, but this would be very time consuming for them.

2. Add subcategories (e.g., from the example above “Acute WAS Interventions” and “Chronic WAS Interventions”).

3. Send them a Google form survey that better illustrates the type of interventions that would fall under each category and get them to input their ratings there.

4. Ask experts about one representative intervention and infer from this the score for the rest of the interventions in that category/subcategory.

5. If we were to publish the results from the five-minute expert surveys, experts might be more interested in rating all of the individual interventions.

6. Remove the expert survey from the five-minute process as a standalone factor, but integrate it with informed considerations.

What we think would be best to do in the future: Incorporate both a Google form survey and the Skype interview into the EpV (e.g., Skype with experts and ask open-ended questions to get a sense of their values etc., and after the Skype send them a Google form survey and get them to rank subcategories of interventions on there, explaining what sort of intervention would fall into these categories). 

 

6.2. TWO HOURS

 

Process 
Given the generalized information that has been gained from the five-minute process, the most helpful expert to speak to next would be a single domain expert. This interview would take one hour including prep, contacting the expert, and summarizing the notes afterward, as well as one hour speaking directly to the expert about the key questions that would be hard for a broader expert to answer. If an expert can cover more than one cause area, more time can be used to prep for his or her interview. Over a given cause area this would result in one expert per domain or five to thirty experts in total. This stage will only happen for family planning this year. In other cause areas, we will use a different two-hour methodology. 

Interviews at this level of depth will be used to help us understand a more specific charity idea such as what a promising country or approach might look like. These interviews, which will often be with domain experts, will help us determine, for example, what country would be most promising to run a conditional cash transfer (CCT) program for intrauterine devices (IUDs) in. For this level of depth, we will manually write up conversation notes (because these notes will be published, they need to be higher quality than for the other levels of depth). We will send this conversation summary to the experts for review before publishing. We will also invite these specialist experts to review the report as a whole. 

Expected outcome

• Have spoken to at least one domain expert to get their sense of the area

• Summarized conversation and how it affects the intervention (one paragraph)

• Synthesized information from the interviews that will enable rating of each idea 

Advice

• At this stage, you could offer the following options to the experts you are interviewing:

  • You can send your questions in a Google document and they can write their answers on there, with a quick follow-up Skype to ask any further questions if necessary

  • One-hour Skype interview 

    • If they choose this, you should send your questions to them before your call so they are aware of what kind of questions you will be asking and can prepare if they feel they need to.

 

6.3. TWENTY HOURS

 

​Process
At the twenty-hour level, roughly the same process would be used as for the five-minute level. Of this time, two hours would need to be used to find and contact experts, as well as two hours preparing the most important questions that would give helpful information across an intervention area, six hours interviewing the top five that respond, four hours synthesizing the notes from speaking to them and scoring interventions based on the responses, and six hours sending the results to the experts and getting their feedback on both their personal notes and the report as a whole. This would result in five experts spoken to about a single intervention. 

These expert interviews will often be with specialist experts, such as speaking to an animal advocate in Taiwan or speaking to a fish disease specialist to help us determine whether paying farmers to use vaccines to treat diseased tilapia in Taiwan seems like a promising intervention for farmed fish. One of the experts contacted will be a person leading an organization implementing similar interventions in the same region as the recommended charity. The goal of this interview is to find out if they could be influenced to change their program to a more cost-effective one (recommended by CE). Such a change would alter where we plan to allocate resources. For example, influencing a charity fortifying flour with iron to add folic acid would alter the score of a folic acid fortification intervention, and would lead to starting a tobacco taxation charity instead.

For this level of depth, we will manually write up conversation notes (because these notes will be published, they need to be higher quality than for the other levels of depth). We will send this conversation summary to the experts for review before publishing. We will also invite these specialist experts to review the report as a whole. 

Timeline relative to other methods (eighty-hour report) 

• Ten hours – Broad undirected reading and crucial consideration, CC (IC)

• Sixteen hours – Directed research (WFM)

• Ten hours – Finding and talking to experts (EpV)

• Twenty hours – CEA creation (CEA) 

• Four hours – Directed research (WFM) 

• Ten hours – Summary writing and internal contemplation (IC) 

• Ten hours – Showing endline report to experts (EpV)

Expected outcome

• One-page summary of the synthesis expert conversion 

• Conversion notes for all experts interviewed

• Expert feedback given on the final report

• An expert or two highlighted who would be willing to mentor or speak to a new charity

Advice

• At this stage, you could offer the following options to the experts you are interviewing:

  • You can send your questions in a Google document and they can write their answers on there, with a quick follow-up Skype to ask any further questions if necessary.

  • One-hour Skype interview 

    • If they choose this, you should send your questions to them before your Skype so they are aware of what kind of questions you will be asking and can prepare if they feel they need to.

 

6.4. EXTERNAL EXPERT DATA

​

Experts we interview are not the only source of expert data we use. If there are previously written interviews, conversation notes, or other direct sources of expert data, we also include these in our expert report. Data like these would be searched for in the directed research phase of the project but would be included in the evidence section of data. ​

 

7. SUMMARY

​

By the time a charity is recommended in an area, we will have spoken to five broad experts in the cause area, as well as six domain experts or specialist experts. Some of these experts will also have reviewed the overall report and given comments or suggested improvements. We also will have taken into account any publicly available expert surveys or summaries of other related conversation notes. 

These experts are spoken to using a consistent methodology. Our conversation notes, as well as our summarized interpretation of the conversations, are published in a single section but clearly differentiated from each other. 

 

8. DEEPER READING

​

1) External resources on how to be a good interviewer
2) Evidence on good forecasting practices from the Good Judgment Project: An accompanying blog post
3) The Black Swan: The Impact of the Highly Improbable
4) Expert Political Judgment: How Good Is It? How Can We Know?
5) Future Babble: Why Expert Predictions Fail and Why We Believe Them Anyway
6) Online Bettors Can Sniff out Weak Psychology Studies.