When you are wheeled into a hospital with a broken arm, you place your trust in multiple people and establishments. You trust that the nurses are giving you the right medication to prepare you, and that the doctor will make the right call on how to fix your arm. And yet you personally know relatively little about the specific treatments that are going to be applied. We defer to people all the time on different issues, whether it’s the doctor at a hospital, the weatherman for the forecast, or the baker who tells us the bread is fresh. Even in our domains of expertise many judgment calls are made by others, and we have to trust or distrust their data. Knowing who to trust is both a difficult and important skill. Trust the wrong person, and they can fill your head with wrong information. But trust no one, and you have to fix every broken bone yourself. So how can we determine who to trust – who is credible and who is not? There are four key ways to determine whether a source or person is worth putting your trust in. In descending order of how good an indicator it is, you can:
Check against reality
Check against further research/reasoning
Check the source’s reliability in other areas
Check for signs of credibility signaling
Each of these is more of a spot check than perfectly predictive, and not all can be done in every case.
1. Check against reality
The best way to test if you can trust a person or source is to check their statements against reality. Say there are two weathermen and you are unsure who to trust. In this case, a reality check is easy. You could compare each of their historical predictions with the historical weather to see which has been accurate more often. This does not guarantee who will be a better source in the future, but it’s strongly suggestive. Similarly if a source predicts a certain reality, particularly in a manner that is easily falsifiable, this evidence can be used to support or create skepticism for the sources. Reality checks can also be used for groups of sources. For example, lots of people go to the hospital with broken arms and generally come out with a cast and an improved state. Thus I might generally trust hospitals to fix broken arms, even if I have not checked my specific doctor. Reality is the ultimate arbiter. It does not matter if one weatherman speaks more confidently, wears a better suit, or has a PhD – the one whose predictions more closely correlate with reality is the better source.
2. Check against further research/reasoning
Not all claims can be checked against reality, but a large number can be checked with further research. For example, the first time I heard from GiveWell that global poverty was reducing over time I was surprised. However, when I checked multiple other sources it indeed looked like this was the case. This made me trust GiveWell’s research more. If it had turned out that global poverty was in fact on the rise, I would have been more skeptical of their research. Enough spot checks and slowly a source as a whole can become trustworthy. For example, I have checked over a dozen different sections of GiveWell’s work, often putting in several dozen hours of research into a specific claim. Again and again, from my best assessment it looks like they are correct. Over time this builds trust, so that now I can use GiveWell as a reliable source to check other claims against.
3. Check the source’s reliability in other areas
Trustworthy in one domain does not always mean trustworthy in another. Despite the hospital fixing my arm, I would be wary of their ability to predict the weather. However, sources can often make claims in some areas that are testable and others that are not. In this case it can be useful to look into a source’s reliability in a different area. For example, imagine I have a friend who for fun memorizes facts that my research shows consistently to be correct. Were this friend to share a new fact in a hard-to-check area, I would be likely to believe him. Likewise if a source is highly trustworthy in one area, it’s more likely to be trustworthy in others. If GiveWell started recommending animal charities, I would be inclined to trust these recommendations even if I had not yet checked them against reality or further research.
4. Check for signs of credibility signaling
The last way you can try to get a sense of who to trust is by looking at generally accepted forms of credibility signaling (e.g. legitimacy of the source, or an individual’s qualifications). This has the advantage of being quick, but it's also fairly unreliable compared to the other methodologies. A nice website signals that a source has funding, but is a pretty weak signal in terms of them being trustworthy. Credibility signaling is often where people go wrong with trusting a source – by giving a certain signal far too much weight compared to its actual correlation with reality.
Summary
When assessing a person or piece of evidence, we often look at multiple sources. If I am assessing the credibility of a study, for example, I might first look for replication studies (checking against reality). If those cannot be found, I would check the methodology of the original study (checking against reasoning/process) to see if it had been well run and seems internally valid. If that doesn’t work I would look for other studies and sources that point in the same or opposite directions (checking against further research). If I find some but not enough to be convinced, I might look at the authors’ other studies or the work they have done in an area I know quite well (checking against other areas). Finally, I might look at what school the authors were from or what journal they published in (check for credibility signaling). When any of these sources conflict, it’s easy to see which one wins. I care far more about the source that is consistently in line with reality than I do about the source with the most PhDs. Over time as sources become highly reliable, you can use them to check other sources. For example, if you check Cochrane a dozen times and it dovetails with further research, you can eventually start to use it as a reliable source to check others against. Over time your trust can become more and more refined. For example, you might only trust certain sources in some areas and not others.
Comments