“Without reflection, we go blindly on our way, creating more unintended consequences, and failing to achieve anything useful.”
- Margaret J. Wheatley
WHAT ARE FLOW-THROUGH EFFECTS?
Your charity will have a direct impact, but it will also have many indirect flow-through effects, also known as second-order effects. For example, an organization distributing bednets might cause a reduction in malaria, but its second-order effect might be increased economic growth due to less loss of productivity from disease. One direct effect can set off an infinite chain of second-, third- and fourth-order effects, which might even surpass the direct impact of a charity.
However, flow-through effects aren’t always positive. That’s why it’s so important to consider them beforehand, to minimise your chances of encountering a nasty surprise further down the line. For example, while a charity that gives out bednets to prevent malaria likely has very valuable short-term effects, could it be creating harmful overpopulation in the medium-term? Or could it be creating a net population loss, because richer people have fewer children?
HOW TO ACCOUNT FOR FLOW-THROUGH EFFECTS
What cause area and intervention you pick will determine how much research into flow-through effects will be necessary. We strongly recommend trying to identify and understand as many short-, medium- and long-term effects as you can before implementing that intervention. Calculating direct effects is hard enough without taking second order effects into account. Good giving opportunities are scarce, good science is difficult, errors can go uncorrected for long periods of time, and it’s hard to even figure out which metric to focus on. It’s even more challenging to predict medium- and long-term effects, which might not emerge for another 50 or 100 years. While flow-through effects will still require significant analysis, it is important to make sure they do not paralyze your decisions. We think weighted quantitative modeling is usually the strongest strategy for incorporating flow-through effects into impact predictions because it strikes the right balance between time and the ability to make progress in a generally intractable area. However, there is large disagreement in this area and many ways to deal with the issue. Below is a list of the different methods along with their advantages and disadvantages. Ignore flow-through effects The simplest strategy, and possibly the most common one in the charity sector, is to ignore potential flow-through effects entirely. Some charities believe flow-through effects are too complicated to include in their impact calculations. Some take positive short-term effects as a proxy for good medium- and long-term effects. Some underestimate the magnitude, importance or likelihood of flow-through effects. We don’t think burying your head in the sand is a productive solution, however. If you neglect to consider indirect impact, you are very likely to miss large, clear, and important effects that will change your expected cumulative impact.
Make an educated guess
In the absence of certainty, you might do well to incorporate a best ‘guestimate’ of a particular flow-through effect in your impact calculation. You can adjust your impact estimate accordingly, perhaps usinga “many weak arguments” framework. Even if each individual argument is weak, if there are a sufficient number of them all supporting the same conclusion, that can be more reliable than one very strong argument supporting a different one. Making an educated guess, based on the limited information you have available, will at least give you a ballpark figure to work with.
Create a weighted quantitative model
You could create a weighted model based on your confidence in the direction (positive vs negative) and magnitude of each potential flow-through effect. For example, second-order effects (which are harder to measure) would usually have less weight compared to first-order effects (which are easier to measure), because they are less predictable. This is similar to having a bayesian prior which is described elsewhere.
If you were only 60% sure of the size of an intervention’s first order effects, you would give the figure a 60% weighting in your calculations, to account for that uncertainty. Similarly, if you were even less confident about the size of the second-order effects, with only 30% confidence in your guess, you would give the second-order effects a weighting of just 30% within your calculations. This model prevents very speculative flow-through effects from impacting your calculations too much; the lower your confidence in a figure, the weaker its effect on your end result. Thus, fourth-order effects would have a far greater impact on standard expected value calculations than in a weighted quantitative model.
Generally, for expected value calculations (if the uncertainty is equal on both sides which is often assumed) there is no discount applied for weakness of evidence or uncertainty of the flow through effect. Instead, the probability you put on a value is taken at face value. This is strange, because putting a 50% chance on flipping a coin and getting heads is very different from putting a 50% chance that your partner will say yes if you propose to them. In the former case, if you flip the coin a thousand times, you can be very confident you will get heads approximately five hundred times. However, you will (hopefully) not propose to your partner one thousand times. You might say that you put it in the same category as a multitude of other things that you put a 50% chance on, of which 50% will be accurate. That does not solve the problem though, because it is well established that people are very poorly calibrated, and if they put a 50% chance on something, that is usually not the actual probability that it will happen. You can improve your calibration skills by practicing, which we recommend, yet there’s quite high odds that calibration is domain specific. Practicing estimating distances will not cross-apply to estimating the release of a new technology. Additionally, some areas might be impossible to calibrate on because the feedback loops are too slow, such as making predictions one hundred years in the future.
A weighted quantitative model could be described as an expected value calculation that takes into account more factors than naive ones normally do, specifically strength of evidence or your meta-confidence in the estimate. In other words, the weighted quantitative model is more weighted towards ‘no effect’ than are typical models. As a result, the more uncertain you are in an effect (even if your best guess is it's the same effect), the more you regress the effect towards ‘no effect’.
Conduct in-depth research
You could list all the potential flow-through effects and try to research them as much as is feasible. Some questions may be much harder to research than others. You may, however, be able to examine historical analyses of similar situations, or meta-research into the ways others have dealt with flow-through effects in the past. Some researchers have offered proxy variables you could attempt to measure and connect causally with various interventions.
Choose robust interventions
Choosing an intervention that you’re confident has net positive consequences under many different conditions would minimize the margin for negative flow-through effects. For example, it seems very unlikely that improving education in developed countries, promoting international cooperation, or advocating for philosophical thoughtfulness will be net negative, even if they do have a sizable risk of being net neutral.
9 E.g. many developed world education initiatives have reduced risk of negative flow-through effects, yet end up producing little to no results.
AREAS WITH LARGE POTENTIAL FLOW-THROUGH EFFECTS
This is by no means an exhaustive list. Rather, this is a list of the indirect effects that have been discussed most commonly in the effective charity community.
Flow-through effects and compounding
It has been argued thatthe effects of doing good now compound. For instance, if you inspire one person to donating to effective charities, that person will continue inspiring new people, and those people will inspire more people in a chain reaction, thus “earning interest on your interest”. We believe this is an oversimplification.
For one, say you start a direct poverty charity and that inspires approximately one new charity per year, and those charities have the same “inspiration rate”. This won’t go on forever until everybody in the world is starting direct poverty charities. It’s not an exponential curve, but rather an s-shaped one. There are the initial low hanging fruit, exponential growth for a period of time, then a diminishing amount. However, this isn’t the end of the story. After all of these charities start, they don’t last forever. People retire, charities shut down, problems are solved, etc. So really after the tapering, there is a probably relatively linear comedown.
Additionally, compounding benefits apply to doing good later as well. It’s not like if you start a charity 10 years from now, nobody will care anymore. However, there is still a penalty for starting later. If you spent 39 years researching, then spent 1 year doing, you’d only have 1 year of inspiring, so only one extra charity started because of you, rather than if you had spent 1 year researching and 39 years doing, at which point you’d have far more charities inspired by you.
Furthermore, this is probably an overly optimistic scenario. There are many more ways to deviate from doing good through doing rather than giving. If you are earning to give and you inspire another person to earn to give, if they donate to the same charity or set of charities as you, it’s easy to see how much good they are doing by your standards. Starting a charity or working for another is much more complicated because of the diversity of options. Depending on how pluralistic your values and epistemics are, inspiring others is more or less good.
How to deal with low probability but highly harmful flow-through effects
This lesson will discuss what to do if there is a chance that your intervention may have a very harmful flow-through effect, but your research suggest that this is unlikely. Perhaps there have been studies rejecting the hypothesis, but the studies still have some room for error.
To illustrate our point, we will use the example of iron supplementation, an intervention that may possibly increase the risk of contracting malaria. Lisa wants to start an iron supplementation charity and her research found that meta-analyses drawing on over 20 RCTs (Shankar, 2000; Gera, 2002) found no significant effect on malaria. Therefore, while Lisa should not ignore this potential flow-through effect, she should discount its weighting in her calculations according to the strength of the evidence. There is also strong evidence suggesting that iron and folic acid fortification has some positive flow-through effects, such as depression reduction (Beard, et. al., 2008; Ballin, et. al., 1992; Rahn, et. al., 2008; Luca, et. al., 2008).
It is generally a good idea to err on the side of caution, and use pessimistic figures in cost-effectiveness estimates. If, after comparing the cost-effectiveness of implementing the intervention in different countries, Lisa finds that the top few locations happen to be malaria-free zones, that’s fantastic. If there are several malaria-free zones and several malaria-prone zones with similar cost-effectiveness estimates, then it makes sense to avoid high-malaria areas, in order to minimize the risk of negative flow-through effects.
If, however, the locations in which the program would be most cost-effective happen to be malaria-prone, and implementing in malaria-free locations would be much less cost-effective, then she will have to think seriously about whether this tradeoff might be worth the risk.
Thinking deeper about this topic Lisa would have to consider that among countries that are malaria-prone, some have better measures in place to control the disease than others (e.g. a more thorough bednet distribution program.) Having said that, no country’s preventative measures have 100% coverage. After comparing the best options that are both malaria risks and malaria free she would have to estimate which one would have a better net effect on the world.
Comments