Loading ...
Sorry, an error occurred while loading the content.

2008-2009 international aid recommendations published

Expand Messages
  • Holden Karnofsky
    See http://blog.givewell.net/?p=396
    Message 1 of 6 , Jul 1, 2009
    • 0 Attachment
    • jonbehar
      After reading through your charity reviews, I put together some initial thoughts on your rankings. Here s how I d think about ranking the charities, coming
      Message 2 of 6 , Jul 5, 2009
      • 0 Attachment

        After reading through your charity reviews, I put together some initial thoughts on your rankings.  Here's how I'd think about ranking the charities, coming from the perspective of a donor:

        I want my donation to save/improve lives in a cost effective way.  To have confidence that it will, I want to see empirical data and strong intuitive logic to support a given program/charity.  I know there's so much measurement error around any metrics and data we see that that I'm mostly concerned about big downside risk, i.e. that there's a material risk my money is simply wasted; it's just not worth taking that risk when there's other charities available that don't have it.

        With that in mind, some of the charities have enough of that downside risk to warrant being in the second tier.  It's certainly possible that further research could assuage these concerns.

        Global Fund:  There are enough red flags that it's tough to get excited about the Global Fund.  There's the lack of clarity about where the marginal dollar goes, the reliance on procedure-less independent auditors, no indication of that ineffective programs (of which there are undoubtedly many) are being shut down, and the significant resources dedicated to relatively cost-ineffective programs (anti retroviral treatment.)

        GAVI: Without a good explanation for the outside grants and a list of unfunded projects, I see no reason to risk a donation to a charity that might can't provide a good reason why they need the funds. 

        AMF: The lack of utilization data is precisely the type of thing I'm worried about.  Given the other alternatives, I don't see any reason to incur the risk that the nets aren't being used or maintained.  While this concern also applies to PSI, I think PSI has a few advantages: they are collecting data on net usage (though the monitoring leaves much to be desired), they explicitly focus on marketing, and on the margins I think selling rather than freely distributing nets is more likely to lead to sustained usage.

        Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such large magnitudes.  Among these three, I feel confident that a donation would meet my goal of saving/improving lives in a cost-effective manner.  In thinking through how to choose between them, I'd be weighing off fairly subjective concerns like whether it's better to focus on the most cost-effective interventions (which would favor PSI or Stop TB) or whether the more wide-ranging treatments PIH offers is actually a better way to improve lives. 

        That leaves Village Reach, which I'm kind of torn about.  Their model is simple and logical, and their commitment to monitoring, evaluation, and reflection seems fantastic.  My intuition is also that a small, focused organization like Village Reach is more likely to be able to effectively use the results of monitoring and evaluation to make necessary adjustments than a larger organization would be.  The methodology in their monitoring is about as sound as we're likely to see, and the data looks great.  But… there are really large risks, larger than some of the risks of the charities I see as second tier.  I think you nailed them in your summary- it's a young charity, trying to massively expand in scope, that hasn't proven it can realize a substantial part of the benefit of its activities yet (via the transfer to governments).  That's a really scary combo.  Since there are alternatives that I don't think have risks of these magnitudes, I'd grudgingly put Village Reach in the second tier.  A couple of things might change my mind.  It would be great to get an assessment of how well the pilot program went from a project management perspective (did it stay on budget, were timelines met, etc.).  If it went well, I'd be less concerned about the expansion in scope.  Some recovery in the data from the pilot program would be nice too.     


         

         

        --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
        >
        > See http://blog.givewell.net/?p=396
        >

      • Elie Hassenfeld
        Thanks for sending these comments. Here are some initial thoughts; we ll plan to send more later. There s a big difference in the confidence I have in our
        Message 3 of 6 , Jul 6, 2009
        • 0 Attachment
          Thanks for sending these comments. Here are some initial thoughts; we'll plan to send more later.

          There's a big difference in the confidence I have in our "top-rated" vs merely"recommended" charities, so I disagree with some of the conclusions you reach below. I wanted to discuss two points in particular:
          1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such large magnitudes.  Among these three, I feel confident that a donation would meet my goal of saving/improving lives in a cost-effective manner."
          2. "[VillageReach] is a young charity, trying to massively expand in scope, that hasn't proven it can realize a substantial part of the benefit of its activities yet (via the transfer to governments).  That's a really scary combo.  Since there are alternatives that I don't think have risks of these magnitudes, I'd grudgingly put Village Reach in the second tier."
          PIH and PSI vs Stop TB

          I think there are strong reasons to support both PSI and PIH, but at the same time, both have significant weaknesses that would lead me to support Stop TB before either of them.

          PIH: PIH has a common-sense model, but I think there's a big risk that the effect visible to a donor (a fully-functioning health facility in a location where one did not previously exist) is largely the result of shifting resources from one location in a country to another. For example, because they don't train doctors, for PIH to staff its Rwandan facility with Rwandan doctors, it relies on relocating doctors from one location in Rwanda to another. To the extent that PIH is doing that, we think there's reason to significantly discount the overall impact their programs have.  We discussed this issue in our review at http://givewell.net/pih#Possiblenegativeoffsettingimpact

          Even though we this issue leads me to significantly discount PIH's apparent impact, I think they are still have some impact by (a) providing trained doctors with the facilities they need to provide top-notch healthcare. In addition, in some of their facilities, many of the clinicians are developed-world doctors who travel abroad to staff part/much of the clinic.

          PSI: PSI does have a stronger commitment to monitoring and evaluation than almost any other charity, but it's the evidence provided in that documentation provides a mixed case for the impact that PSI's programs have. PSI supports bednet provision, and monitors bednet use, but it's unclear that PSI's activities have increased the number of people that are sleeping under nets. And, bednets are a relatively easy case because ITN-distribution and promotion is a program with extremely strong evidence behind it (see givewell.net/node/329. Condom promotion and distribution is much trickier. If you can get people to consistently use condoms, you'll likely reduce HIV/AIDS transmission but there's no "proven" approach for accomplishing that (for more see givewell.net/node/375). PSI's approach seems as good as any, but their relatively inconsistent monitoring and evaluation that relies solely on self-reported accounts of behavior is somewhat questionable. It's far less compelling to me than either Stop TB's data on completed TB treatments and patient outcomes or VillageReach's data on children vaccinated.

          I don't think Stop TB has these same kinds of weakness, and therefore, among these three, I'd support them.

          VillageReach

          I think the evidence that VillageReach provides for the impact of their program is unmatched among any charity I've seen. VillageReach came to Cabo Delgado; they reorganized and supplemented the vaccine-delivery system; they measured (a) changes in drug availability in clinics and (b) changes in children immunized, a life-saving intervention. They compared Cabo Delgado's success to that of a nearby district. On all measures, VillageReach's programs appears a success. No other charity I've looked at can offer a case for impact as compeling as that.

          I am not terribly concerned about the question of whether VillageReach can successfully pass off its activities to the government, because we evaluated them mostly under the assumption that they can't do so *at all*, and even with this assumption still consider them to be as proven and cost-effective as any of the other charities we've seen (see the VillageReach review for details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for details).

          I'd guess that the risk of VillageReach scaling up is somewhat low, though there is clearly some risk. VillageReach is currently seeking $750,000 for 2009 which would lead to (approximately) a scale up of 2-3x the size of their current projects. That seems like an appropriate increase given the strong success of their current project.

          -Elie

          On Mon, Jul 6, 2009 at 12:06 AM, jonbehar <jonbehar@...> wrote:


          After reading through your charity reviews, I put together some initial thoughts on your rankings.  Here's how I'd think about ranking the charities, coming from the perspective of a donor:

          I want my donation to save/improve lives in a cost effective way.  To have confidence that it will, I want to see empirical data and strong intuitive logic to support a given program/charity.  I know there's so much measurement error around any metrics and data we see that that I'm mostly concerned about big downside risk, i.e. that there's a material risk my money is simply wasted; it's just not worth taking that risk when there's other charities available that don't have it.

          With that in mind, some of the charities have enough of that downside risk to warrant being in the second tier.  It's certainly possible that further research could assuage these concerns.

          Global Fund:  There are enough red flags that it's tough to get excited about the Global Fund.  There's the lack of clarity about where the marginal dollar goes, the reliance on procedure-less independent auditors, no indication of that ineffective programs (of which there are undoubtedly many) are being shut down, and the significant resources dedicated to relatively cost-ineffective programs (anti retroviral treatment.)

          GAVI: Without a good explanation for the outside grants and a list of unfunded projects, I see no reason to risk a donation to a charity that might can't provide a good reason why they need the funds. 

          AMF: The lack of utilization data is precisely the type of thing I'm worried about.  Given the other alternatives, I don't see any reason to incur the risk that the nets aren't being used or maintained.  While this concern also applies to PSI, I think PSI has a few advantages: they are collecting data on net usage (though the monitoring leaves much to be desired), they explicitly focus on marketing, and on the margins I think selling rather than freely distributing nets is more likely to lead to sustained usage.

          Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such large magnitudes.  Among these three, I feel confident that a donation would meet my goal of saving/improving lives in a cost-effective manner.  In thinking through how to choose between them, I'd be weighing off fairly subjective concerns like whether it's better to focus on the most cost-effective interventions (which would favor PSI or Stop TB) or whether the more wide-ranging treatments PIH offers is actually a better way to improve lives. 

          That leaves Village Reach, which I'm kind of torn about.  Their model is simple and logical, and their commitment to monitoring, evaluation, and reflection seems fantastic.  My intuition is also that a small, focused organization like Village Reach is more likely to be able to effectively use the results of monitoring and evaluation to make necessary adjustments than a larger organization would be.  The methodology in their monitoring is about as sound as we're likely to see, and the data looks great.  But… there are really large risks, larger than some of the risks of the charities I see as second tier.  I think you nailed them in your summary- it's a young charity, trying to massively expand in scope, that hasn't proven it can realize a substantial part of the benefit of its activities yet (via the transfer to governments).  That's a really scary combo.  Since there are alternatives that I don't think have risks of these magnitudes, I'd grudgingly put Village Reach in the second tier.  A couple of things might change my mind.  It would be great to get an assessment of how well the pilot program went from a project management perspective (did it stay on budget, were timelines met, etc.).  If it went well, I'd be less concerned about the expansion in scope.  Some recovery in the data from the pilot program would be nice too.     


           

           

          --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
          >
          > See http://blog.givewell.net/?p=396
          >


        • Holden Karnofsky
          A couple things to add. For me, the distinction between *** and ** is a very important one. Basically, I feel that for a *** charity, we have reasonable
          Message 4 of 6 , Jul 10, 2009
          • 0 Attachment

            A couple things to add.

            For me, the distinction between *** and ** is a very important one.  Basically, I feel that for a *** charity, we have reasonable confidence in the full set of its activities, and feel that the cost-effectiveness estimate provided is a reasonable (if very rough) approximation to its overall impact.  By contrast, for a ** charity there is a crucial piece of the puzzle missing – missing data on a highly questionable link in the chain, questions about how representative the data we have is, etc. – and while we feel the charity is a much better bet than lower-rated charities, and is likely doing a substantial amount of good, we don't have a good sense for how often its activities are going as hoped.

            To me, the two charities Jon mentioned as being in his "top tier" are in the ** tier:

            PSI.  PSI stands above non-recommended charities because it is systematically asking the questions we feel need to be asked to give confidence in their activities, and a significant amount of data appears to be actually collected and available.  That said:

            1.      We've only seen a sample of their data, and while we don't feel the sample was "cherry-picked," we do feel that better-run projects may be more likely to get their data in to the central office.  So we're concerned about representativeness.

            2.      The data do not point strongly to impact.  *Changes* in reported behavior do not particularly suggest impact; the high *levels* of reported condom/ITN use, combined with PSI's role as a dominant supplier, make it seem likely that PSI is getting these materials to people who are using them.  It may simply be substituting for for-profit suppliers, though a limited set of studies (see http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests that subsidized/free distribution has benefits.

            PIH.  Elie has addressed a couple of the concerns with PIH.  I would add that

            1.      PIH does appear to conduct a variety of programs whose impact would be harder to assess than that of direct medical care (microloans, school scholarships, population-based health initiatives).  In the Rwanda budget (the only budget we have that can give a sense for the relative size of these programs), these programs appear to consume about 7% of the funds.  We aren't sure whether we're under-allocating administrative expenses to these programs, whether they've grown since we last looked, whether they're larger at other locations, etc.

            2.      We have next to no actual data on health outcomes; as our report states, we're basing our recommendation on the feeling that their model has a lower burden of proof and a high profile.

            Our top-rated charities. 

            With Stop TB, because of the consistency of its programming and its auditing, we can see a summary of how things are going in every country, and a sample of the details that go into this summary data.  With VillageReach, we are looking at a charity that has had one pilot project we feel is successful and is looking to scale up the same model to more areas at a pace we feel is reasonable (it is not looking to drastically expand its funding or diversifying its activities).  The case for these charities isn't airtight or close to it, but in both cases I feel I can look across the complete set of the organization's activities, know what information is available on the biggest questions, and feel that the organization as a whole is a good bet.  With the ** charities, there are huge advantages over "typical" charities and reason to believe that they're having positive impact in many cases, but the "missing pieces" are qualitatively bigger and the sense of what you get for a donation to the organization as a whole is much weaker.

            On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld <ehassenfeld@...> wrote:


            Thanks for sending these comments. Here are some initial thoughts; we'll plan to send more later.

            There's a big difference in the confidence I have in our "top-rated" vs merely"recommended" charities, so I disagree with some of the conclusions you reach below. I wanted to discuss two points in particular:

            1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such large magnitudes.  Among these three, I feel confident that a donation would meet my goal of saving/improving lives in a cost-effective manner."
            2. "[VillageReach] is a young charity, trying to massively expand in scope, that hasn't proven it can realize a substantial part of the benefit of its activities yet (via the transfer to governments).  That's a really scary combo.  Since there are alternatives that I don't think have risks of these magnitudes, I'd grudgingly put Village Reach in the second tier."
            PIH and PSI vs Stop TB

            I think there are strong reasons to support both PSI and PIH, but at the same time, both have significant weaknesses that would lead me to support Stop TB before either of them.

            PIH: PIH has a common-sense model, but I think there's a big risk that the effect visible to a donor (a fully-functioning health facility in a location where one did not previously exist) is largely the result of shifting resources from one location in a country to another. For example, because they don't train doctors, for PIH to staff its Rwandan facility with Rwandan doctors, it relies on relocating doctors from one location in Rwanda to another. To the extent that PIH is doing that, we think there's reason to significantly discount the overall impact their programs have.  We discussed this issue in our review at http://givewell.net/pih#Possiblenegativeoffsettingimpact

            Even though we this issue leads me to significantly discount PIH's apparent impact, I think they are still have some impact by (a) providing trained doctors with the facilities they need to provide top-notch healthcare. In addition, in some of their facilities, many of the clinicians are developed-world doctors who travel abroad to staff part/much of the clinic.

            PSI: PSI does have a stronger commitment to monitoring and evaluation than almost any other charity, but it's the evidence provided in that documentation provides a mixed case for the impact that PSI's programs have. PSI supports bednet provision, and monitors bednet use, but it's unclear that PSI's activities have increased the number of people that are sleeping under nets. And, bednets are a relatively easy case because ITN-distribution and promotion is a program with extremely strong evidence behind it (see givewell.net/node/329. Condom promotion and distribution is much trickier. If you can get people to consistently use condoms, you'll likely reduce HIV/AIDS transmission but there's no "proven" approach for accomplishing that (for more see givewell.net/node/375). PSI's approach seems as good as any, but their relatively inconsistent monitoring and evaluation that relies solely on self-reported accounts of behavior is somewhat questionable. It's far less compelling to me than either Stop TB's data on completed TB treatments and patient outcomes or VillageReach's data on children vaccinated.


            I don't think Stop TB has these same kinds of weakness, and therefore, among these three, I'd support them.

            VillageReach


            I think the evidence that VillageReach provides for the impact of their program is unmatched among any charity I've seen. VillageReach came to Cabo Delgado; they reorganized and supplemented the vaccine-delivery system; they measured (a) changes in drug availability in clinics and (b) changes in children immunized, a life-saving intervention. They compared Cabo Delgado's success to that of a nearby district. On all measures, VillageReach's programs appears a success. No other charity I've looked at can offer a case for impact as compeling as that.

            I am not terribly concerned about the question of whether VillageReach can successfully pass off its activities to the government, because we evaluated them mostly under the assumption that they can't do so *at all*, and even with this assumption still consider them to be as proven and cost-effective as any of the other charities we've seen (see the VillageReach review for details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for details).


            I'd guess that the risk of VillageReach scaling up is somewhat low, though there is clearly some risk. VillageReach is currently seeking $750,000 for 2009 which would lead to (approximately) a scale up of 2-3x the size of their current projects. That seems like an appropriate increase given the strong success of their current project.

            -Elie

            On Mon, Jul 6, 2009 at 12:06 AM, jonbehar <jonbehar@...> wrote:


            After reading through your charity reviews, I put together some initial thoughts on your rankings.  Here's how I'd think about ranking the charities, coming from the perspective of a donor:

            I want my donation to save/improve lives in a cost effective way.  To have confidence that it will, I want to see empirical data and strong intuitive logic to support a given program/charity.  I know there's so much measurement error around any metrics and data we see that that I'm mostly concerned about big downside risk, i.e. that there's a material risk my money is simply wasted; it's just not worth taking that risk when there's other charities available that don't have it.

            With that in mind, some of the charities have enough of that downside risk to warrant being in the second tier.  It's certainly possible that further research could assuage these concerns.

            Global Fund:  There are enough red flags that it's tough to get excited about the Global Fund.  There's the lack of clarity about where the marginal dollar goes, the reliance on procedure-less independent auditors, no indication of that ineffective programs (of which there are undoubtedly many) are being shut down, and the significant resources dedicated to relatively cost-ineffective programs (anti retroviral treatment.)

            GAVI: Without a good explanation for the outside grants and a list of unfunded projects, I see no reason to risk a donation to a charity that might can't provide a good reason why they need the funds. 

            AMF: The lack of utilization data is precisely the type of thing I'm worried about.  Given the other alternatives, I don't see any reason to incur the risk that the nets aren't being used or maintained.  While this concern also applies to PSI, I think PSI has a few advantages: they are collecting data on net usage (though the monitoring leaves much to be desired), they explicitly focus on marketing, and on the margins I think selling rather than freely distributing nets is more likely to lead to sustained usage.

            Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such large magnitudes.  Among these three, I feel confident that a donation would meet my goal of saving/improving lives in a cost-effective manner.  In thinking through how to choose between them, I'd be weighing off fairly subjective concerns like whether it's better to focus on the most cost-effective interventions (which would favor PSI or Stop TB) or whether the more wide-ranging treatments PIH offers is actually a better way to improve lives. 

            That leaves Village Reach, which I'm kind of torn about.  Their model is simple and logical, and their commitment to monitoring, evaluation, and reflection seems fantastic.  My intuition is also that a small, focused organization like Village Reach is more likely to be able to effectively use the results of monitoring and evaluation to make necessary adjustments than a larger organization would be.  The methodology in their monitoring is about as sound as we're likely to see, and the data looks great.  But… there are really large risks, larger than some of the risks of the charities I see as second tier.  I think you nailed them in your summary- it's a young charity, trying to massively expand in scope, that hasn't proven it can realize a substantial part of the benefit of its activities yet (via the transfer to governments).  That's a really scary combo.  Since there are alternatives that I don't think have risks of these magnitudes, I'd grudgingly put Village Reach in the second tier.  A couple of things might change my mind.  It would be great to get an assessment of how well the pilot program went from a project management perspective (did it stay on budget, were timelines met, etc.).  If it went well, I'd be less concerned about the expansion in scope.  Some recovery in the data from the pilot program would be nice too.     


             

             

            --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
            >
            > See http://blog.givewell.net/?p=396
            >



          • jonbehar
            I think you ve done a good job clarifying the advantages of Stop TB relative to PIH and PSI. I m now in agreement that Stop TB is a notch above. However, I
            Message 5 of 6 , Jul 12, 2009
            • 0 Attachment

               

              I think you've done a good job clarifying the advantages of Stop TB relative to PIH and PSI.  I'm now in agreement that Stop TB is a notch above.  However, I still have some questions about Village Reach.

               

              The evidence that Village Reach had a major impact in its pilot program in Cabo Delgado is pretty compelling.  So, I think the key questions are how closely the proposed expanded program resembles the pilot and how likely it is that the expanded program can be executed.  Specific questions I have along those line are:

               

              ·         In the broader program, who will be responsible for distributing the vaccines to the hospital?  The effectiveness of the pilot program suffered when field coordinators were no longer responsible for this delivery (see pages 20-21 of

              http://www.villagereach.org/PDF%20Documents/VillageReach%20Evaluation%20of%20the%20Project%20to%20Support%20PAV%20-%20Executive%20Summary%20and%20Report.pdf)

              ·         What is the extent of Village Reach's scaling up?  Elie put out an estimate of 2-3x current activities.  How was this arrived at?  Scanning Village Reach's past expenditures, it looks like they totaled ~4mm over the life of pilot program.  My understanding is that the bulk of that was spent on the pilot itself.  Yet the expansion of the program is expected to cost 3.5mm (including government funds).  It doesn't make sense to me that the expanded program would cost less than the pilot (though the pilot ran for longer than the expanded program is due to run, so the relative annual costs make a little more sense).  Do you guys have a sense of why the budgeted cost of the expanded program seems so low relative to the pilot?  Will the expanded program have the same high degree of monitoring that the pilot had?

              ·         Was the pilot program roughly on budget?  If not, were overruns diagnosed such that we can have confidence that the budget for the broader program is reasonable?

              ·         How secure is the Mozambique government's commitment to fund ~1/3 of the cost of the expanded program? 

               


              --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
              >
              > A couple things to add.
              >
              > For me, the distinction between *** and ** is a very important one. Basically,
              > I feel that for a *** charity, we have reasonable confidence in the full set
              > of its activities, and feel that the cost-effectiveness estimate provided is
              > a reasonable (if very rough) approximation to its overall impact. By
              > contrast, for a ** charity there is a crucial piece of the puzzle missing –
              > missing data on a highly questionable link in the chain, questions about how
              > representative the data we have is, etc. – and while we feel the charity is
              > a much better bet than lower-rated charities, and is likely doing a
              > substantial amount of good, we don't have a good sense for how often its
              > activities are going as hoped.
              >
              > To me, the two charities Jon mentioned as being in his "top tier" are in the
              > ** tier:
              >
              > *PSI. *PSI stands above non-recommended charities because it is
              > systematically asking the questions we feel need to be asked to give
              > confidence in their activities, and a significant amount of data appears to
              > be actually collected and available. That said:
              >
              > 1. We've only seen a sample of their data, and while we don't feel the
              > sample was "cherry-picked," we do feel that better-run projects may be more
              > likely to get their data in to the central office. So we're concerned about
              > representativeness.
              >
              > 2. The data do not point strongly to impact. *Changes* in reported
              > behavior do not particularly suggest impact; the high *levels* of reported
              > condom/ITN use, combined with PSI's role as a dominant supplier, make it
              > seem likely that PSI is getting these materials to people who are using
              > them. It may simply be substituting for for-profit suppliers, though a
              > limited set of studies (see
              > http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests that
              > subsidized/free distribution has benefits.
              >
              > *PIH. *Elie has addressed a couple of the concerns with PIH. I would add
              > that
              >
              > 1. PIH does appear to conduct a variety of programs whose impact would
              > be harder to assess than that of direct medical care (microloans, school
              > scholarships, population-based health initiatives). In the Rwanda budget
              > (the only budget we have that can give a sense for the relative size of
              > these programs), these programs appear to consume about 7% of the funds. We
              > aren't sure whether we're under-allocating administrative expenses to these
              > programs, whether they've grown since we last looked, whether they're larger
              > at other locations, etc.
              >
              > 2. We have next to no actual data on health outcomes; as our report
              > states, we're basing our recommendation on the feeling that their model has
              > a lower burden of proof and a high profile.
              >
              > *Our top-rated charities. *
              > With Stop TB, because of the consistency of its programming and its
              > auditing, we can see a summary of how things are going in every country, and
              > a sample of the details that go into this summary data. With VillageReach,
              > we are looking at a charity that has had one pilot project we feel is
              > successful and is looking to scale up the same model to more areas at a pace
              > we feel is reasonable (it is not looking to drastically expand its funding
              > or diversifying its activities). The case for these charities isn't
              > airtight or close to it, but in both cases I feel I can look across the
              > complete set of the organization's activities, know what information is
              > available on the biggest questions, and feel that the organization as a
              > whole is a good bet. With the ** charities, there are huge advantages over
              > "typical" charities and reason to believe that they're having positive
              > impact in many cases, but the "missing pieces" are qualitatively bigger and
              > the sense of what you get for a donation to the organization as a whole is
              > much weaker.
              >
              > On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld ehassenfeld@...wrote:
              >
              > >
              > >
              > > Thanks for sending these comments. Here are some initial thoughts; we'll
              > > plan to send more later.
              > >
              > > There's a big difference in the confidence I have in our "top-rated" vs
              > > merely"recommended" charities, so I disagree with some of the conclusions
              > > you reach below. I wanted to discuss two points in particular:
              > >
              > > 1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of
              > > such large magnitudes. Among these three, I feel confident that a donation
              > > would meet my goal of saving/improving lives in a cost-effective manner."
              > > 2. "[VillageReach] is a young charity, trying to massively expand in
              > > scope, that hasn't proven it can realize a substantial part of the benefit
              > > of its activities yet (via the transfer to governments). That's a really
              > > scary combo. Since there are alternatives that I don't think have risks of
              > > these magnitudes, I'd grudgingly put Village Reach in the second tier."
              > >
              > > *PIH and PSI vs Stop TB
              > >
              > > *I think there are strong reasons to support both PSI and PIH, but at the
              > > same time, both have significant weaknesses that would lead me to support
              > > Stop TB before either of them.
              > >
              > > *PIH:* PIH has a common-sense model, but I think there's a big risk that
              > > the effect visible to a donor (a fully-functioning health facility in a
              > > location where one did not previously exist) is largely the result *of
              > > shifting resources* *from one location in a country to another.* For
              > > example, because they don't train doctors, for PIH to staff its Rwandan
              > > facility with Rwandan doctors, it relies on relocating doctors from one
              > > location in Rwanda to another. To the extent that PIH is doing that, we
              > > think there's reason to significantly discount the overall impact their
              > > programs have. We discussed this issue in our review at
              > > http://givewell.net/pih#Possiblenegativeoffsettingimpact
              > >
              > > Even though we this issue leads me to significantly discount PIH's *
              > > apparent* impact, I think they are still have some impact by (a) providing
              > > trained doctors with the facilities they need to provide top-notch
              > > healthcare. In addition, in some of their facilities, many of the clinicians
              > > are developed-world doctors who travel abroad to staff part/much of the
              > > clinic.
              > >
              > > *PSI: *PSI does have a stronger commitment to monitoring and evaluation
              > > than almost any other charity, but it's the evidence provided in that
              > > documentation provides a mixed case for the *impact *that PSI's programs
              > > have. PSI supports bednet provision, and monitors bednet use, but it's
              > > unclear that PSI's activities have increased the number of people that are
              > > sleeping under nets. And, bednets are a relatively easy case because
              > > ITN-distribution and promotion is a program with extremely strong evidence
              > > behind it (see givewell.net/node/329. Condom promotion and distribution is
              > > much trickier. If you can get people to consistently use condoms, you'll
              > > likely reduce HIV/AIDS transmission but there's no "proven" approach for
              > > accomplishing that (for more see givewell.net/node/375). PSI's approach
              > > seems as good as any, but their relatively inconsistent monitoring and
              > > evaluation that relies solely on self-reported accounts of behavior is
              > > somewhat questionable. It's far less compelling to me than either Stop TB's
              > > data on completed TB treatments and patient outcomes or VillageReach's data
              > > on children vaccinated.
              > >
              > > I don't think Stop TB has these same kinds of weakness, and therefore,
              > > among these three, I'd support them.
              > >
              > > *VillageReach*
              > >
              > > I think the evidence that VillageReach provides for the impact of their
              > > program is unmatched among any charity I've seen. VillageReach came to Cabo
              > > Delgado; they reorganized and supplemented the vaccine-delivery system; they
              > > measured (a) changes in drug availability in clinics and (b) changes in
              > > children immunized, a life-saving intervention. They compared Cabo Delgado's
              > > success to that of a nearby district. On all measures, VillageReach's
              > > programs appears a success. No other charity I've looked at can offer a case
              > > for impact as compeling as that.
              > >
              > > I am not terribly concerned about the question of whether VillageReach can
              > > successfully pass off its activities to the government, because we evaluated
              > > them mostly under the assumption that they can't do so *at all*, and even
              > > with this assumption still consider them to be as proven and cost-effective
              > > as any of the other charities we've seen (see the VillageReach review for
              > > details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for
              > > details).
              > >
              > > I'd guess that the risk of VillageReach scaling up is somewhat low, though
              > > there is clearly some risk. VillageReach is currently seeking $750,000 for
              > > 2009 which would lead to (approximately) a scale up of 2-3x the size of
              > > their current projects. That seems like an appropriate increase given the
              > > strong success of their current project.
              > >
              > > -Elie
              > >
              > > On Mon, Jul 6, 2009 at 12:06 AM, jonbehar jonbehar@... wrote:
              > >
              > >>
              > >>
              > >> After reading through your charity reviews, I put together some initial
              > >> thoughts on your rankings. Here's how I'd think about ranking the
              > >> charities, coming from the perspective of a donor:
              > >>
              > >> I want my donation to save/improve lives in a cost effective way. To have
              > >> confidence that it will, I want to see empirical data and strong intuitive
              > >> logic to support a given program/charity. I know there's so much
              > >> measurement error around any metrics and data we see that that I'm mostly
              > >> concerned about big downside risk, i.e. that there's a material risk my
              > >> money is simply wasted; it's just not worth taking that risk when there's
              > >> other charities available that don't have it.
              > >>
              > >> With that in mind, some of the charities have enough of that downside risk
              > >> to warrant being in the second tier. It's certainly possible that further
              > >> research could assuage these concerns.
              > >>
              > >> Global Fund: There are enough red flags that it's tough to get excited
              > >> about the Global Fund. There's the lack of clarity about where the marginal
              > >> dollar goes, the reliance on procedure-less independent auditors, no
              > >> indication of that ineffective programs (of which there are undoubtedly
              > >> many) are being shut down, and the significant resources dedicated to
              > >> relatively cost-ineffective programs (anti retroviral treatment.)
              > >>
              > >> GAVI: Without a good explanation for the outside grants and a list of
              > >> unfunded projects, I see no reason to risk a donation to a charity that
              > >> might can't provide a good reason why they need the funds.
              > >>
              > >> AMF: The lack of utilization data is precisely the type of thing I'm
              > >> worried about. Given the other alternatives, I don't see any reason to
              > >> incur the risk that the nets aren't being used or maintained. While this
              > >> concern also applies to PSI, I think PSI has a few advantages: they are
              > >> collecting data on net usage (though the monitoring leaves much to be
              > >> desired), they explicitly focus on marketing, and on the margins I think
              > >> selling rather than freely distributing nets is more likely to lead to
              > >> sustained usage.
              > >>
              > >> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such
              > >> large magnitudes. Among these three, I feel confident that a donation would
              > >> meet my goal of saving/improving lives in a cost-effective manner. In
              > >> thinking through how to choose between them, I'd be weighing off fairly
              > >> subjective concerns like whether it's better to focus on the most
              > >> cost-effective interventions (which would favor PSI or Stop TB) or whether
              > >> the more wide-ranging treatments PIH offers is actually a better way to
              > >> improve lives.
              > >>
              > >> That leaves Village Reach, which I'm kind of torn about. Their model is
              > >> simple and logical, and their commitment to monitoring, evaluation, and
              > >> reflection seems fantastic. My intuition is also that a small, focused
              > >> organization like Village Reach is more likely to be able to effectively use
              > >> the results of monitoring and evaluation to make necessary adjustments than
              > >> a larger organization would be. The methodology in their monitoring is
              > >> about as sound as we're likely to see, and the data looks great. But… there
              > >> are really large risks, larger than some of the risks of the charities I see
              > >> as second tier. I think you nailed them in your summary- it's a young
              > >> charity, trying to massively expand in scope, that hasn't proven it can
              > >> realize a substantial part of the benefit of its activities yet (via the
              > >> transfer to governments). That's a really scary combo. Since there are
              > >> alternatives that I don't think have risks of these magnitudes, I'd
              > >> grudgingly put Village Reach in the second tier. A couple of things might
              > >> change my mind. It would be great to get an assessment of how well the
              > >> pilot program went from a project management perspective (did it stay on
              > >> budget, were timelines met, etc.). If it went well, I'd be less concerned
              > >> about the expansion in scope. Some recovery in the data from the pilot
              > >> program would be nice too.
              > >>
              > >>
              > >>
              > >>
              > >>
              > >>
              > >> --- In givewell@yahoogroups.com, Holden Karnofsky Holden@ wrote:
              > >> >
              > >> > See http://blog.givewell.net/?p=396
              > >> >
              > >>
              > >
              > >
              > >
              >

               

            • Holden Karnofsky
              I think these are good questions for VillageReach, and more generally, I think you re right to express skepticism about giving to a charity based on a single
              Message 6 of 6 , Jul 15, 2009
              • 0 Attachment

                I think these are good questions for VillageReach, and more generally, I think you're right to express skepticism about giving to a charity based on a single pilot project.

                I also concede that we are much lighter on details than we'd like to be about VillageReach's future projects.  We understand that they will be taking the same basic approach, and planning the same level of rigorous monitoring and evaluation, but we are still waiting on details about their projected expenses as well as about the specifics of the regions they'll be expanding into.  (Note that we are still communicating with both VillageReach and Stop TB, asking for more information about a variety of things.)

                Still, I would personally be at least as confident giving to VillageReach as to StopTB, and Elie feels the same way.  I'll try to explain why:

                VillageReach's small size carries additional benefits as well as additional risks.  It's true that with Stop TB, we have much more "sample size" and the case is stronger that it has a pattern of success, rather than a single success.  On the other hand, Stop TB is so huge and working in so many different places that there's a limit to how confident I can be, even with the exceptional amount of information it provides.  With VillageReach, I feel that I will be able to track and stay up to date on every project they carry out (for the foreseeable future), and furthermore that every project they carry out will have the direct involvement of people who were instrumental in the first success.

                I believe that uncertainties come up both with extrapolating from a single project to future success, and with extrapolating from audits to a general picture of the impact of a large bureaucracy.

                The "riskiness" of VillageReach cuts both ways as well.  VillageReach is still largely trying to establish an approach to improving health systems; the approach may turn out not to work in new settings, but if it does work repeatedly, it may eventually influence the work of other charities and other governments, attract large amounts of government aid, etc. 

                Generally, we stay away from "innovative" or "pilot" programs because we feel that individual donors (including us) are not well positioned to understand their likelihood of success; but having actual past results from a past project puts VillageReach in a different category.  I recognize a substantial risk of failure for VillageReach's expansion, but feel confident that the risk is worth funding (I am more confident that this is a "good bet" than I am that PSI or PIH is a "good bet").

                Bottom line: for any charity, we can make a long list of things that still might go wrong (and for both of the charities in question, we're still working on getting more info).  But both VillageReach and Stop TB have provided fairly compelling, systematic, empirical answers to the biggest questions that jump to mind about what they're trying to do (and, what they're trying to do is very cost-effective in changing lives if successful).  That puts them in the same basic category to me, even though the specifics of their advantages and disadvantages are very different, and individual donors will probably differ quite a bit on whether they are more comfortable with an already large and established charity or with a small but promising charity.

                On Sun, Jul 12, 2009 at 11:00 PM, jonbehar <jonbehar@...> wrote:


                 

                I think you've done a good job clarifying the advantages of Stop TB relative to PIH and PSI.  I'm now in agreement that Stop TB is a notch above.  However, I still have some questions about Village Reach.

                 

                The evidence that Village Reach had a major impact in its pilot program in Cabo Delgado is pretty compelling.  So, I think the key questions are how closely the proposed expanded program resembles the pilot and how likely it is that the expanded program can be executed.  Specific questions I have along those line are:

                 

                ·         In the broader program, who will be responsible for distributing the vaccines to the hospital?  The effectiveness of the pilot program suffered when field coordinators were no longer responsible for this delivery (see pages 20-21 of

                http://www.villagereach.org/PDF%20Documents/VillageReach%20Evaluation%20of%20the%20Project%20to%20Support%20PAV%20-%20Executive%20Summary%20and%20Report.pdf)

                ·         What is the extent of Village Reach's scaling up?  Elie put out an estimate of 2-3x current activities.  How was this arrived at?  Scanning Village Reach's past expenditures, it looks like they totaled ~4mm over the life of pilot program.  My understanding is that the bulk of that was spent on the pilot itself.  Yet the expansion of the program is expected to cost 3.5mm (including government funds).  It doesn't make sense to me that the expanded program would cost less than the pilot (though the pilot ran for longer than the expanded program is due to run, so the relative annual costs make a little more sense).  Do you guys have a sense of why the budgeted cost of the expanded program seems so low relative to the pilot?  Will the expanded program have the same high degree of monitoring that the pilot had?

                ·         Was the pilot program roughly on budget?  If not, were overruns diagnosed such that we can have confidence that the budget for the broader program is reasonable?

                ·         How secure is the Mozambique government's commitment to fund ~1/3 of the cost of the expanded program? 

                 


                --- In givewell@yahoogroups.com, Holden Karnofsky <Holden@...> wrote:
                >
                > A couple things to add.
                >
                > For me, the distinction between *** and ** is a very important one. Basically,
                > I feel that for a *** charity, we have reasonable confidence in the full set
                > of its activities, and feel that the cost-effectiveness estimate provided is
                > a reasonable (if very rough) approximation to its overall impact. By
                > contrast, for a ** charity there is a crucial piece of the puzzle missing –
                > missing data on a highly questionable link in the chain, questions about how
                > representative the data we have is, etc. – and while we feel the charity is
                > a much better bet than lower-rated charities, and is likely doing a
                > substantial amount of good, we don't have a good sense for how often its
                > activities are going as hoped.
                >
                > To me, the two charities Jon mentioned as being in his "top tier" are in the
                > ** tier:
                >
                > *PSI. *PSI stands above non-recommended charities because it is
                > systematically asking the questions we feel need to be asked to give
                > confidence in their activities, and a significant amount of data appears to
                > be actually collected and available. That said:
                >
                > 1. We've only seen a sample of their data, and while we don't feel the
                > sample was "cherry-picked," we do feel that better-run projects may be more
                > likely to get their data in to the central office. So we're concerned about
                > representativeness.
                >
                > 2. The data do not point strongly to impact. *Changes* in reported
                > behavior do not particularly suggest impact; the high *levels* of reported
                > condom/ITN use, combined with PSI's role as a dominant supplier, make it
                > seem likely that PSI is getting these materials to people who are using
                > them. It may simply be substituting for for-profit suppliers, though a
                > limited set of studies (see
                > http://givewell.net/node/329#Freenetsvssellingnetsforafee) suggests that
                > subsidized/free distribution has benefits.
                >
                > *PIH. *Elie has addressed a couple of the concerns with PIH. I would add
                > that
                >
                > 1. PIH does appear to conduct a variety of programs whose impact would
                > be harder to assess than that of direct medical care (microloans, school
                > scholarships, population-based health initiatives). In the Rwanda budget
                > (the only budget we have that can give a sense for the relative size of
                > these programs), these programs appear to consume about 7% of the funds. We
                > aren't sure whether we're under-allocating administrative expenses to these
                > programs, whether they've grown since we last looked, whether they're larger
                > at other locations, etc.
                >
                > 2. We have next to no actual data on health outcomes; as our report
                > states, we're basing our recommendation on the feeling that their model has
                > a lower burden of proof and a high profile.
                >
                > *Our top-rated charities. *
                > With Stop TB, because of the consistency of its programming and its
                > auditing, we can see a summary of how things are going in every country, and
                > a sample of the details that go into this summary data. With VillageReach,
                > we are looking at a charity that has had one pilot project we feel is
                > successful and is looking to scale up the same model to more areas at a pace
                > we feel is reasonable (it is not looking to drastically expand its funding
                > or diversifying its activities). The case for these charities isn't
                > airtight or close to it, but in both cases I feel I can look across the
                > complete set of the organization's activities, know what information is
                > available on the biggest questions, and feel that the organization as a
                > whole is a good bet. With the ** charities, there are huge advantages over
                > "typical" charities and reason to believe that they're having positive
                > impact in many cases, but the "missing pieces" are qualitatively bigger and
                > the sense of what you get for a donation to the organization as a whole is
                > much weaker.
                >
                > On Mon, Jul 6, 2009 at 6:30 PM, Elie Hassenfeld ehassenfeld@...wrote:

                >
                > >
                > >
                > > Thanks for sending these comments. Here are some initial thoughts; we'll
                > > plan to send more later.
                > >
                > > There's a big difference in the confidence I have in our "top-rated" vs
                > > merely"recommended" charities, so I disagree with some of the conclusions
                > > you reach below. I wanted to discuss two points in particular:
                > >
                > > 1. "Among PIH, Stop TB Partnership, and PSI, I don't have concerns of

                > > such large magnitudes. Among these three, I feel confident that a donation
                > > would meet my goal of saving/improving lives in a cost-effective manner."
                > > 2. "[VillageReach] is a young charity, trying to massively expand in

                > > scope, that hasn't proven it can realize a substantial part of the benefit
                > > of its activities yet (via the transfer to governments). That's a really
                > > scary combo. Since there are alternatives that I don't think have risks of
                > > these magnitudes, I'd grudgingly put Village Reach in the second tier."
                > >
                > > *PIH and PSI vs Stop TB
                > >
                > > *I think there are strong reasons to support both PSI and PIH, but at the
                > > same time, both have significant weaknesses that would lead me to support
                > > Stop TB before either of them.
                > >
                > > *PIH:* PIH has a common-sense model, but I think there's a big risk that
                > > the effect visible to a donor (a fully-functioning health facility in a
                > > location where one did not previously exist) is largely the result *of
                > > shifting resources* *from one location in a country to another.* For
                > > example, because they don't train doctors, for PIH to staff its Rwandan
                > > facility with Rwandan doctors, it relies on relocating doctors from one
                > > location in Rwanda to another. To the extent that PIH is doing that, we
                > > think there's reason to significantly discount the overall impact their
                > > programs have. We discussed this issue in our review at
                > > http://givewell.net/pih#Possiblenegativeoffsettingimpact
                > >
                > > Even though we this issue leads me to significantly discount PIH's *
                > > apparent* impact, I think they are still have some impact by (a) providing
                > > trained doctors with the facilities they need to provide top-notch
                > > healthcare. In addition, in some of their facilities, many of the clinicians
                > > are developed-world doctors who travel abroad to staff part/much of the
                > > clinic.
                > >
                > > *PSI: *PSI does have a stronger commitment to monitoring and evaluation
                > > than almost any other charity, but it's the evidence provided in that
                > > documentation provides a mixed case for the *impact *that PSI's programs
                > > have. PSI supports bednet provision, and monitors bednet use, but it's
                > > unclear that PSI's activities have increased the number of people that are
                > > sleeping under nets. And, bednets are a relatively easy case because
                > > ITN-distribution and promotion is a program with extremely strong evidence
                > > behind it (see givewell.net/node/329. Condom promotion and distribution is
                > > much trickier. If you can get people to consistently use condoms, you'll
                > > likely reduce HIV/AIDS transmission but there's no "proven" approach for
                > > accomplishing that (for more see givewell.net/node/375). PSI's approach
                > > seems as good as any, but their relatively inconsistent monitoring and
                > > evaluation that relies solely on self-reported accounts of behavior is
                > > somewhat questionable. It's far less compelling to me than either Stop TB's
                > > data on completed TB treatments and patient outcomes or VillageReach's data
                > > on children vaccinated.
                > >
                > > I don't think Stop TB has these same kinds of weakness, and therefore,
                > > among these three, I'd support them.
                > >
                > > *VillageReach*
                > >
                > > I think the evidence that VillageReach provides for the impact of their
                > > program is unmatched among any charity I've seen. VillageReach came to Cabo
                > > Delgado; they reorganized and supplemented the vaccine-delivery system; they
                > > measured (a) changes in drug availability in clinics and (b) changes in
                > > children immunized, a life-saving intervention. They compared Cabo Delgado's
                > > success to that of a nearby district. On all measures, VillageReach's
                > > programs appears a success. No other charity I've looked at can offer a case
                > > for impact as compeling as that.
                > >
                > > I am not terribly concerned about the question of whether VillageReach can
                > > successfully pass off its activities to the government, because we evaluated
                > > them mostly under the assumption that they can't do so *at all*, and even
                > > with this assumption still consider them to be as proven and cost-effective
                > > as any of the other charities we've seen (see the VillageReach review for
                > > details) (see http://givewell.net/node/370#Whatdoyougetforyourdollar for
                > > details).
                > >
                > > I'd guess that the risk of VillageReach scaling up is somewhat low, though
                > > there is clearly some risk. VillageReach is currently seeking $750,000 for
                > > 2009 which would lead to (approximately) a scale up of 2-3x the size of
                > > their current projects. That seems like an appropriate increase given the
                > > strong success of their current project.
                > >
                > > -Elie
                > >
                > > On Mon, Jul 6, 2009 at 12:06 AM, jonbehar jonbehar@... wrote:
                > >
                > >>
                > >>
                > >> After reading through your charity reviews, I put together some initial
                > >> thoughts on your rankings. Here's how I'd think about ranking the
                > >> charities, coming from the perspective of a donor:
                > >>
                > >> I want my donation to save/improve lives in a cost effective way. To have
                > >> confidence that it will, I want to see empirical data and strong intuitive
                > >> logic to support a given program/charity. I know there's so much
                > >> measurement error around any metrics and data we see that that I'm mostly
                > >> concerned about big downside risk, i.e. that there's a material risk my
                > >> money is simply wasted; it's just not worth taking that risk when there's
                > >> other charities available that don't have it.
                > >>
                > >> With that in mind, some of the charities have enough of that downside risk
                > >> to warrant being in the second tier. It's certainly possible that further
                > >> research could assuage these concerns.
                > >>
                > >> Global Fund: There are enough red flags that it's tough to get excited
                > >> about the Global Fund. There's the lack of clarity about where the marginal
                > >> dollar goes, the reliance on procedure-less independent auditors, no
                > >> indication of that ineffective programs (of which there are undoubtedly
                > >> many) are being shut down, and the significant resources dedicated to
                > >> relatively cost-ineffective programs (anti retroviral treatment.)
                > >>
                > >> GAVI: Without a good explanation for the outside grants and a list of
                > >> unfunded projects, I see no reason to risk a donation to a charity that
                > >> might can't provide a good reason why they need the funds.
                > >>
                > >> AMF: The lack of utilization data is precisely the type of thing I'm
                > >> worried about. Given the other alternatives, I don't see any reason to
                > >> incur the risk that the nets aren't being used or maintained. While this
                > >> concern also applies to PSI, I think PSI has a few advantages: they are
                > >> collecting data on net usage (though the monitoring leaves much to be
                > >> desired), they explicitly focus on marketing, and on the margins I think
                > >> selling rather than freely distributing nets is more likely to lead to
                > >> sustained usage.
                > >>
                > >> Among PIH, Stop TB Partnership, and PSI, I don't have concerns of such
                > >> large magnitudes. Among these three, I feel confident that a donation would
                > >> meet my goal of saving/improving lives in a cost-effective manner. In
                > >> thinking through how to choose between them, I'd be weighing off fairly
                > >> subjective concerns like whether it's better to focus on the most
                > >> cost-effective interventions (which would favor PSI or Stop TB) or whether
                > >> the more wide-ranging treatments PIH offers is actually a better way to
                > >> improve lives.
                > >>
                > >> That leaves Village Reach, which I'm kind of torn about. Their model is
                > >> simple and logical, and their commitment to monitoring, evaluation, and
                > >> reflection seems fantastic. My intuition is also that a small, focused
                > >> organization like Village Reach is more likely to be able to effectively use
                > >> the results of monitoring and evaluation to make necessary adjustments than
                > >> a larger organization would be. The methodology in their monitoring is
                > >> about as sound as we're likely to see, and the data looks great. But… there
                > >> are really large risks, larger than some of the risks of the charities I see
                > >> as second tier. I think you nailed them in your summary- it's a young
                > >> charity, trying to massively expand in scope, that hasn't proven it can
                > >> realize a substantial part of the benefit of its activities yet (via the
                > >> transfer to governments). That's a really scary combo. Since there are
                > >> alternatives that I don't think have risks of these magnitudes, I'd
                > >> grudgingly put Village Reach in the second tier. A couple of things might
                > >> change my mind. It would be great to get an assessment of how well the
                > >> pilot program went from a project management perspective (did it stay on
                > >> budget, were timelines met, etc.). If it went well, I'd be less concerned
                > >> about the expansion in scope. Some recovery in the data from the pilot
                > >> program would be nice too.
                > >>
                > >>
                > >>
                > >>
                > >>
                > >>
                > >> --- In givewell@yahoogroups.com, Holden Karnofsky Holden@ wrote:
                > >> >
                > >> > See http://blog.givewell.net/?p=396
                > >> >
                > >>
                > >
                > >
                > >
                >

                 


              Your message has been successfully submitted and would be delivered to recipients shortly.