Re: [givewell] Disease research update
- We haven't found and aren't sure how to find information that would shed light on these questions.I agree they're important factors to consider.-Elie
On Tue, May 18, 2010 at 12:48 PM, Nick Beckstead <nbeckstead@...> wrote:
Very helpful, Elie.
What about the question of whether things like (i), (iii), and (iv) are highly significant? Is it within the realm of reasonable estimates, for instance, that they account for more than 25% of the benefit?
NickOn Tue, May 18, 2010 at 12:34 PM, Elie Hassenfeld <elie@...> wrote:
Nick,The estimates are not meant to be precise -- they're a preliminary impression and we're planning to look into these studies more.The point we want to make with the estimates is that it appears to us that (a) on a per-life-saved or per-DALY basis, disease research will likely not be competitive with developing-world direct aid (http://www.givewell.net/international/technical/programs) but (b) are competitive with estimates of the value of a statistical life in the United States (~$10,000-100,000 per life year saved -- DCP Pg 158-9).It's also important to note that the average past cost-effectiveness of disease research is not necessarily a good indication of either (a) the average future cost-effectiveness for research or (b) the cost-effectiveness of the marginal dollar donated to disease research.To answer some of your specific questions:The estimates I've considered include (ii) but not your (i), (iii), or (iv). The lower end of the estimates are from the "best cases" -- the lower end are attempts to look at "averages."Some of the papers I've looked at are:
- The Return to Biomedical Research: Treatment and Behavioral Effects -- David Cutler and Srikanth Kadiyala
- The effect of new drug approvals on HIV mortality in the US, 1987–1998 -- Frank Lichtenberg
- The effect of new drugs on mortality from rare diseases and HIV -- Frank R. Lichtenberg
- Effect of a US National Institutes of Health programme of clinical trials on public health and costs -- S Claiborne Johnston, John D Rootenberg, Shereen Katrak, Wade S Smith, Jacob S Elkins
On Tue, May 18, 2010 at 11:54 AM, Nick Beckstead <nbeckstead@...> wrote:
Forgive me, for asking such a basic questions, but how are you or other researchers weighing benefits from medical research that are highly indirect or occurring in the future? When you say that this research has an impact in the ballpark of $1000/DALY, does this include benefits from the following sorts of things?
(i) Indirect current/past benefits: Benefits from research that was made possible by doing the research that was originally funded.
(ii) Future direct benefits: Benefits to people who will receive treatment within the next 10 years, where the treatment is directly made possible by the research.
(iii) Distant future direct benefits: Benefits to people who will receive treatment within the next 10-50 years, or later, etc, where the treatment is directly made possible by the research.
(iv) Indirect future benefits: Benefits from research that will be made possible by doing the research that was originally funded, even if these benefits haven't materialized yet.
If these future benefits are included, how long into the future are we thinking about them? If indirect benefits are included, how do we weigh cases where future research is made possible by some past research, but a particular bit of past research plays a relatively small role in the future research?
I'm imagining that the future/indirect benefits are nearly impossible to quantify, and so are ignored in this kind of cost-benefit analysis. Is that true? Is it likely that they are highly significant? Maybe the thing to do here is just to point me at some paper/introductory text, which would be fine.
On an unrelated note, is $1000/DALY an average figure, or an estimate of some of the best instances of medical research you've seen/are likely to find?
NickOn Tue, May 18, 2010 at 11:27 AM, Elie Hassenfeld <elie@...> wrote:This is an update on our progress on the cause of disease research. (Previous update and related messages at http://groups.yahoo.com/group/givewell/message/163.)At the top level, the questions we're trying to answer are:
We've found a decent amount of literature focused on question 1 -- largely case studies of the cost and impact of drugs to fight heart disease, cancer, HIV, orphan diseases, and Polio. We've also found research that tries to evaluate the cost effectiveness of research in general focusing on all newly approved drugs or selected sets of clinical trials.The researchers whose work we've looked at most are David Cutler and Frank Lichtenberg.From what we've read, we'd ballpark that the cost-effectiveness of past disease research in the range of $1,000/DALY or $10,000-$100,000 per life saved.For question 4, we've found a good deal of literature on the interplay between public and private funding (including a literature review at http://elsa.berkeley.edu/~bhhall/papers/DavidHallToole%20RP00.pdf by Bronwyn Hall, Paul David and Andrew Toole), though for a variety of reasons we don't think this literature can shed much light on the specific question we're asking.We've found practically nothing on questions 2 and 3. The ideal for us would be a group of experts (along the lines of the Disease Control Priorities Report or Copenhagen Consensus) laying out their case for top priroties for research.We're still not sure about question 5, a lot will depend on what we end up finding for questions 2 and 3. One factor we've considered so far is along the lines of today's blog post: http://blog.givewell.net/2010/05/18/how-the-american-cancer-society-and-susan-g-komen-for-the-cure-spend-their-money/At this point we're planning to focus more on conversations with relevant people, including charities; we usually try to have a basic grounding in the academic literature first to make these conversations more efficient, but at this point it seems like we've found what we're going to find (which is relatively little).
- Has medical research been a cause of improved health? What has the cost-effectiveness been?
- Is there a consensus on what type of research *has been* most successful?
- Is there a consensus on the best type of research *to fund*?
- Is there room for more funding for research in general?
- What factors would we use to choose between charities?
- In discussing different models of research funding (NIH vs. corporations vs. small startups vs. funding through charities) I thought it might be useful to point out the partnership between GlaxoSmithKline and the Gates Foundation at Tres Cantos:
Basically, it's my understanding that the Gates Foundation has helped fund this GSK lab (Tres Cantos) in Spain for the purpose of malaria research, but in exchange has demanded that the lab make its findings public. This appears to be a new idea; researchers tend to be very secretive about their work, just like charities!
Sounds like a step in the right direction.
This is an update on what we've learned since our previous update (http://groups.yahoo.com/group/givewell/message/182) and what our plans are going forward.
In the last month or so we've made progress to answering the following questions, primarily through conversations with FasterCures (http://fastercures.org/) about their Philanthropy Advisory Service (http://www.philanthropyadvisoryservice.org/), Chris Lipinski (http://myprofile.cos.com/lipinskica) and John Overington (http://www.nature.com/nrd/journal/v5/n12/authors/nrd2199.html). Notes from our conversations with Drs. Lipinski and Overington are available on the GiveWell website at http://www.givewell.org/node/1339
- What can we know about total funding allocated to each disease? The NIH publishes funding by disease, but we don't have a way of getting funds invested by other parties (i.e., for-profit industry). We've found a PLoS paper (http://www.plosone.org/article/info:doi/10.1371/journal.pone.0007015) that shows total (industry + government + non-profit) funding by broad area (e.g., neuroscience, oncology, cardiovascular) but not by specific disease. John Overington (see below) told us that he does not believe that industry funding by disease is available.
- What are the possibilities we have available for assessing an organization's track record? Both John Overington and Chris Lipinski thought that assessing an organization's publication/citation record would be reasonable. Dr. Lipinski suggested that instead of doing citation/publication analysis for all of an organization's papers, we instead identify the most important/influentual papers in each disease (by seeing which papers are cited in major literature reviews) and see who funded them. Dr. Overington suggested we also look at patents an organization had received (on the notion that a patent implies that the organization created something describable and worth protecting).
- What are the possibilities available for assessing the likelihood of a particular organization developing a new drug or treatment? Dr. Overington's company "scores" company's/organization's pipelines on the likelihood that they will produce effective treatments. He said that his company works on a case by case basis and that a general assessment of "would this approach work" does not exist.
- Are there particular approaches to research that have been successful/not successful? Chris Lipinski said that there's a consensus that academic biology is good for basic science but doesn't translate well at all into outcomes for patients. Until recently NIH had a poor track record in the translation of basic research into real drug discovery. Recently, he says that the NIH appreciates the problem of the translational medicine gap and has for a number of years made a good faith effort to address this gap. Dr. Lipinski mentioned the approach of drug repurposing/repositioning, taking existing drugs and testing whether they can successfully treat other conditions.
- Are there existing reports on disease research organizations that will point us to potentially outstanding charities? The FasterCures PAS reports have a lot of useful information (and are among the meatiest sources of information on charities I've seen), but they don't answer all of our questions. (We have asked FasterCures to refer us to the advisors they consulted in creating these reports, so we could ask them our remaining questions.) The MS Society (the organization we use for the examples below) declined to speak with us and told us to rely on the FasterCures report. Examples of FasterCures' report statements and our remaining questions:
- According to the FasterCures report, a key accomplishment for the National MS Society is "Contributing to research that resulted in the six disease-modifying drugs approved by the U.S. Food and Drug Administration (FDA) and research that contributed to the development of the current therapeutic pipeline." We still would ask, in what way, specifically, did the MS Society contribute to the development of these drugs? What other organizations/entities contributed to the development of these drugs and what role did they play?
- FasterCures states, "PAS’s scientific advisors point out that it has been the single organization serving as a catalyst for new research in MS." We ask, what do the advisors base this conclusion on? What does it mean to be "the catalyst" and what would have happened in the Society's absence?
- FasterCures states, "The Society is a key source of new knowledge about MS. Internal tracking of publications of its grantees indicates that a typical grant results in 4-6 publications, and that in 2008, research grants resulted in 240 publications." We still would ask, what was the impact of their publications? Were any publications extremely influential? Were they published in top journals? What was their citation record?
Our plan going forward
Learning the answers to the above questions have caused us to re-evaluate our approach. We think it's unlikely that independent research / scanning charities' websites will enable us to find organizations we can recommend.
Our plan now is to:
- Narrow our scope and focus primarily (for now) on two diseases covered by FasterCures Philanthropy Advisory Service (discussed above): malaria and multiple sclerosis. The goal is to get a sense for what is possible by focusing on diseases for which a lot of information is likely to be available.
- Identify relevant people with knowledge about ongoing research in those areas and speak with them about what future research might accomplish and what benefit additional funding would yield. Also, look at papers on malaria and MS in Nature Reviews Drug Discovery to see what the most important papers have been and who has funded them. The idea is to produce some content on (a) the relative merits of funding work in the two different areas; (b) the relative merits of different organizations with each area. If this approach goes well we can potentially apply it to other diseases as well.
- Talk to funders (e.g., Gates Foundation, Wellcome Turst) about how they choose which organizations to support.
- Try to determine which researchers in different fields are "top" vs "average" vs "below average" and talk to them about their funding needs, with the aim of determine whether (a) "top" have all the funding they need and marginal funding goes to "below average" researchers or (b) "top" researchers can use additional funds.