Loading ...
Sorry, an error occurred while loading the content.

CoP Evaluation

Expand Messages
  • Jon Ford
    I m lucky enough to be able to say that our organization has taken on faith that CoPs are a good idea and that we have pursued them in as free-form a fashion
    Message 1 of 17 , Jul 3, 2008
    • 0 Attachment
      I'm lucky enough to be able to say that our organization has "taken on
      faith" that CoPs are a good idea and that we have pursued them in as
      free-form a fashion as one can actually "pursue" something free-form."
      (My head hurts just thinking about the paradox of that one.) I'm also
      lucky enough that we had a great CoP initiator in our own backyard -
      that being Hildy Gottlieb, who is a member of this group.



      Except now, we're being given the push to put an evaluation "structure"
      around our groups.



      No doubt many of you have resources on this. Fred Nickols and Hildy have
      already responded to a colleague of mine on a different list (I believe
      it was the American Evaluation group list). I'm quite certain I have
      some memory of this topic coming up before on this list also. In fact
      Fred may have had the last word on this already, noting that each CoP is
      unique and therefore requires a customized evaluation approach. (Sorry
      Fred, if my paraphrase is not quite accurate.)



      All that said, in what direction should we turn to learn more about
      evaluating CoPs with some rigor?



      Jon



      ---------------

      Jon Ford

      Associate Director, Communications

      St. Luke's Health Initiatives

      2929 North Central Avenue, Suite 1550

      Phoenix, AZ 85012

      office: 602-385-6508

      cell: 480-650-8997

      fax: 602-385-6510

      e-mail: jon.ford@...

      web: www.slhi.org





      [Non-text portions of this message have been removed]
    • Hildy Gottlieb
      ... *************************** First, to clarify, my advice on the EvalTalk list was, I believe, simply to suggest that the request be posted here. So much
      Message 2 of 17 , Jul 5, 2008
      • 0 Attachment
        Jon Ford wrote:
        >
        > All that said, in what direction should we turn to learn more about
        > evaluating CoPs with some rigor?
        >



        ***************************

        First, to clarify, my advice on the EvalTalk list was, I believe, simply
        to suggest that the request be posted here. So much for my seeming font
        of wisdom, but thanks for the kind words, Jon!
        :-)

        Since that initial post, I have spent quite a bit of time thinking about
        "evaluation," as it was one of the last chapters to be written in the
        book I just finally completed writing last week. My focus on evaluation
        in that context, as it might be applied to CoPs, would sound something
        like this:
        Why evaluate the CoP?
        To what end?
        To measure what?
        And once you measure, then what? What decisions will ride on the
        evaluation?


        If we have some sense of why the CoP was created in the first place,
        then we might measure to determine if the CoP is, in fact, accomplishing
        what the sponsors wanted it to accomplish. From there, an appropriate
        question might be, "If it is not accomplishing what we had hoped, then
        what do we need to do to get closer to those goals? What can we learn
        from this analysis, that can be applied to our future actions?"


        I have been frustrated over time with the emphasis on evaluation as
        somehow - well, punitive. It is evaluation based on self-justification,
        proving oneself. That sense has always nagged at me, especially in
        something like a CoP, where, in truth, I cannot find a down-side to
        people engaging about a practice.


        And then I came upon this marvelous quote, from the Buddhist monk and
        thinker, Chogyam Trungpa, from Cutting through Spiritual Materialism:


        "The basic problem we seem to be facing is that we are too involved with
        trying to prove something, which is connected with paranoia and a
        feeling of poverty. When you are trying to prove or get something, you
        are not open anymore, you have to check everything, and you have to
        arrange it "correctly." It is such a paranoid way to live and it really
        does not prove anything..."


        He was not talking about Communities of Practice, obviously, and not
        talking about evaluation of any organization-sponsored activity. And
        yet my main complaint about standard "evaluation" tools was crystalized
        in that paragraph. Standard evaluation does not err on the side of
        affirming; it errs on the side of "ok, kid - prove yourself!"


        Very long and rambling, I know - sorry to Jon and to the list. But I
        guess, short of asking, "Is the CoP doing what you had hoped? And if
        not, what can we do to get there?" then I'm not sure what other purpose
        or scientifically rigorous methodology is needed for measurement. No,
        that does not address the perceived need to "evaluate with some degree
        of rigor" but again, I am not sure what purpose such evaluation would
        serve. In a strength-based, vision-based effort (which I know Jon's
        is), if the group is accomplishing what the sponsors had wanted to
        accomplish by establishing the group in the first place, what else is
        left to measure?


        IMNSHO (as always)
        :-)

        Hildy

        Hildy Gottlieb
        *The Community-Driven Institute **at Help 4 NonProfits
        *Making Visionary Change Practical
        _http://www.Help4NonProfits.com <http://www.help4nonprofits.com/>
        _/*BLOG:*/ http://www.HildyGottlieb.com/
        Hildy@...
        520-321-4433
        or 1-888-787-4433 *
        *





        [Non-text portions of this message have been removed]
      • John D. Smith
        Good question, Jon! ... One thing to be careful about is, as Hildy mentions, what exactly is to be evaluated? Is it the community or some of the resources
        Message 3 of 17 , Jul 6, 2008
        • 0 Attachment
          Good question, Jon!

          > All that said, in what direction should we turn to learn more about
          > evaluating CoPs with some rigor?

          One thing to be careful about is, as Hildy mentions, what exactly is
          to be evaluated? Is it the community or some of the resources that
          support it? As Etienne repeats tirelessly, "it's a matter of
          *discovering* the communities of practice that are in some sense
          already THERE, rather than creating or building something from scratch."

          Often what's to be evaluated is us -- the folks that are interested in
          or possibly being payed for supporting and stimulating the
          communities. Here are a couple of ideas I've used when I've been
          brought in as an outsider to help with the evaluation work:

          * Use the community, practice, and domain model to identify different
          directions or aspects of the evaluation. For example, regarding
          community, you can ask whether idiots are highly respected or sages
          are ignored (although that may be a question that only competent
          members can answer).

          * What communities DO is evaluate, so any evaluation effort needs to
          be constructed to actually support ongoing evaluation. Often the
          folksy evaluation in a community will be more productive than a
          professionalized version.

          * Finally (this is actually a big topic, so I have to stop for the
          moment, or I won't be ready for a meeting in 20 minutes!) there's a
          question of boundaries. It seems to me that what we expect from our
          communities is that they change the world. So we need to ask that.
          What's impressive about Michael Fung Kee Fung's work is that it's
          actually changed the evaluation framework around the community. Have
          a look at this:

          http://www.implementationscience.com/imedia/1788022150101911_article.pdf

          * I notice that this page deserves some attention:
          http://cofpractice-biblio.wikispaces.com/Measurement+and+assessment

          John
          *
          * John D. Smith ~ Voice: 503.963.8229 ~ Skype: smithjd ~
          http://www.learningAlliances.net
          * "Consequently, I rejoice, having to construct something upon which
          to rejoice." - T.S. Eliot
        • John D. Smith
          Wonderful quote, Hildy. Very relevant to the topic of evaluating communities of practice. I think you make it clear that we have to admit that prove to
          Message 4 of 17 , Jul 6, 2008
          • 0 Attachment
            Wonderful quote, Hildy. Very relevant to the topic of evaluating
            communities of practice.

            I think you make it clear that we have to admit that "prove to whom?" is a
            very tricky question. It's worth pointing out that when Trungpa Rinpoche
            went about presenting Buddhism in the West, he didn't just give a lecture,
            translate a text or do a flower arrangement. He actually set up many, many
            communities of practice (as described by Sorbonne professor Fabrice Midal in
            his book on "Chogyam Trungpa"), ranging from a translation group that still
            is extant to the Dorje Kasung, etc., etc.

            One of the projects in CPsquare that I have in mind for the Fall is to
            organize a largish conference on how religious organizations and communities
            are using technology and a community of practice framework (consciously or
            not) to be relevant and productive. It's kind of a big project and has been
            brewing for several months. I'll recruit for help and participation on this
            list, so please be thinking about it. (At the moment we've got our heads
            buried in Kimble, Hildreth and Bourdon's "Communities of Practice: Creating
            Learning Environments for Educators," so this future project needs to keep a
            lower profile.)

            Cheers!

            John
            *
            * John D. Smith ~ Voice: 503.963.8229 ~ Skype: smithjd ~
            http://www.learningAlliances.net
            * "Consequently, I rejoice, having to construct something upon which to
            rejoice." - T.S. Eliot
          • Rosanna Tarsiero
            Hildy, ... Just few words. This topic always resurfaces. There mainly are two sides to the argument: the more rigor one and the accountability to whom
            Message 5 of 17 , Jul 7, 2008
            • 0 Attachment
              Hildy,

              > Very long and rambling, I know - sorry to Jon and to the list. But
              > I guess, short of asking, "Is the CoP doing what you had hoped? And
              > if not, what can we do to get there?" then I'm not sure what other
              > purpose or scientifically rigorous methodology is needed for
              > measurement.

              Just few words.

              This topic always resurfaces. There mainly are two sides to the
              argument: the "more rigor" one and the "accountability to whom" side.
              If much of the former has to do with an overreliance on the
              engineer/scientific method, much of the latter has to do with
              deconstructing everything, so a la page today.

              Basically, we (the practitioners!) need to decide whether CoP and
              CoP-related expertise can be sold or not. If it can, we MUST provide
              our clients with some form of measurement which goes beyond "my very
              wise opinion" kind of circular arguments. If it can not be sold, we
              MUST be ok with being constantly questioned, and paid peanuts (and
              rightly so, if the expert can't explain his/her knowledge and convince
              people to "buy in", why should they hire him/her to help with a CoP?).

              Of course the truth is somewhere in the middle... some things can be
              measured, some can't. Some results can be predicted, some can't. Some
              CoPs can be evaluated, some can't. Some experts are worth the money,
              some can't.

              How does the user (ie the client) tell who's the charlatan and who's
              not, in this maze of "everything depends"? Nobody would want a field
              such as law, medicine, engineering as downregulated as CoP-ology is.
              And, that's a problem too, because the user has no guarantee and no
              idea on what to base his/her assessment of a professional and his/her
              competency.

              I think CoP-ology will, eventually, become a real profession, pretty
              much like coaching has done, but it will take some dinosaur to either
              die or change profession.

              Rosanna Tarsiero
            • Stephen Denning
              John writes: As Etienne repeats tirelessly, it s a matter of *discovering* the communities of practice that are in some sense already THERE, rather than
              Message 6 of 17 , Jul 7, 2008
              • 0 Attachment
                John writes: "As Etienne repeats tirelessly, "it's a matter of *discovering*
                the communities of practice that are in some sense already THERE, rather
                than creating or building something from scratch."

                While I agree that the easy thing, the obvious thing, is to identify
                communities that already exist, communities must have come into being
                somehow. How? My current work is focused on understanding how this happens -
                how communities come into existence -- with the object of understanding how
                we could facilitate and accelerate their creation, as opposed to passively
                waiting for lightning to strike.

                I'm particularly interested in high-performance communities i.e. communities
                that are exceptional in performance and deeply meaningful for the
                participants. Is it possible to take active steps to generate such
                communities?

                I know that there is a school of thought to the effect that communities
                can't be created -- they emerge; attempts to create them will be
                counterproductive. And if the repertoire of options available were simply
                the clumsy tools of traditional comand-and-control management, I would
                agree. But there is a wider set of options, particularly narrative modes of
                communicating, which I believe create different possibilities.

                If anyone has experiences or ideas in this area, I would love to hear from
                you.

                Steve Denning
                steve@...
                www.stevedening.com

                PS: As to rigorously evaluating communities and understanding what is
                exceptional performance, one does have to take into account the different
                purposes for which communities may exist. John writes: "It seems to me that
                what we expect from our communities is that they change the world." A CFO
                might have a very different perspective ("Are they helping the bottom line
                of this organization?"). A participant might have still another view: ("Are
                they helping me in my work?) Not all communities are aimed at changing the
                world. Rigor may be in the eye of the beholder.


                [Non-text portions of this message have been removed]
              • Hildy Gottlieb
                ... ************************ Rosanna: I m so glad you ve jumped in! I guess I don t see this as a can t measure vs. can measure issue. I see it as a
                Message 7 of 17 , Jul 7, 2008
                • 0 Attachment
                  Rosanna Tarsiero wrote:
                  >
                  > Of course the truth is somewhere in the middle... some things can be
                  > measured, some can't. Some results can be predicted, some can't. Some
                  > CoPs can be evaluated, some can't. Some experts are worth the money,
                  > some can't.
                  >





                  ************************

                  Rosanna:
                  I'm so glad you've jumped in! I guess I don't see this as a "can't
                  measure" vs. "can measure" issue. I see it as a "what to measure"
                  issue. And that is where I think the only thing that matters is the
                  outcomes / results the client wanted from having the CoP in the first place.


                  As for how we get hired as professionals to do that work, I think it is
                  no different than any other consultant who focuses on creating
                  organizational change (rather than consultants who are doing more
                  technical contracting than pure "consulting"). It is indeed why the
                  consulting trade overall (not just CoP consultants) has the reputation
                  as charlatans - if the only measure of success is "I'll know it when I
                  see it," that ambiguity allows for both the good and the bad to thrive.


                  That said, I have come to the place in my own practice - whether
                  facilitating or doing a conference keynote address - where all my work
                  is focused on outcomes and results, predefined by the client before we
                  even get to the proposal stage. I am not being hired to "facilitate a
                  CoP." I am being hired to accomplish X result for the client. From
                  there, the scope of work, the fee - and the measurement falls. The CoP
                  is one tool among many that I might use to get the client where he/she
                  wants to be.


                  My questions to Jon were what I would ask any client, before, during and
                  after the work. What will success look like, and how will we know if we
                  got there? The problem with such questions is that while they
                  absolutely separate the charlatan from the professional, they also
                  frequently cannot be measured with what some might consider "rigor."
                  Which again adds to my questions to Jon - who wants to measure, and
                  why? What do they want to do with that measurement?


                  I am therefore not suggesting that measuring is bad or impossible - I am
                  instead suggesting it is counterproductive when we are measuring against
                  what someone else thinks is important, and not against what we wanted to
                  achieve in the first place.


                  I hope that helps to clarify. And again, good to hear from you. (I
                  actually acknowledged you in the book I just finished writing last week,
                  as having encouraged my thinking, years back, on the question of how
                  nonprofit boards of directors can function as Communities of Practice.
                  Thank you once again for that encouragement!)

                  Hildy

                  Hildy Gottlieb
                  *The Community-Driven Institute **at Help 4 NonProfits
                  *Making Visionary Change Practical
                  _http://www.Help4NonProfits.com <http://www.help4nonprofits.com/>
                  _/*BLOG:*/ http://www.HildyGottlieb.com/
                  Hildy@...
                  520-321-4433
                  or 1-888-787-4433



                  [Non-text portions of this message have been removed]
                • Rosanna Tarsiero
                  Hildy, ... I don t see it as a can t vs can either, but I know there are people in this field who would disagree with my stand :) ... Right! But organizational
                  Message 8 of 17 , Jul 8, 2008
                  • 0 Attachment
                    Hildy,

                    > Rosanna:
                    > I'm so glad you've jumped in! I guess I don't see this as a "can't
                    > measure" vs. "can measure" issue. I see it as a "what to measure"
                    > issue. And that is where I think the only thing that matters is the
                    > outcomes / results the client wanted from having the CoP in the
                    > first place.

                    I don't see it as a can't vs can either, but I know there are people
                    in this field who would disagree with my stand :)


                    > As for how we get hired as professionals to do that work, I think it
                    > is no different than any other consultant who focuses on creating
                    > organizational change (rather than consultants who are doing more
                    > technical contracting than pure "consulting").

                    Right! But organizational change is a tad more measurable than CoPs!
                    This difference is due, in part, to the very essence of the CoP
                    concept/model and, in part, to a certain mindset of CoP practitioners :)


                    > even get to the proposal stage. I am not being hired to "facilitate
                    > a CoP." I am being hired to accomplish X result for the client.
                    > From there, the scope of work, the fee - and the measurement falls.
                    > The CoP is one tool among many that I might use to get the client
                    > where he/she wants to be.

                    Fascinating! I know that there are other CoP practitioners who clearly
                    market themselves as "CoP facilitator" (one being me *grin), as well
                    as I know of some who would argue that CoPs can only be discovered but
                    not facilitated (or "nucleated" as I love to put it). This last take
                    sounds bogus to me but I know it is practiced.


                    > My questions to Jon were what I would ask any client, before, during
                    and
                    > after the work. What will success look like, and how will we know
                    if we
                    > got there? The problem with such questions is that while they
                    > absolutely separate the charlatan from the professional, they also
                    > frequently cannot be measured with what some might consider "rigor."
                    > Which again adds to my questions to Jon - who wants to measure, and
                    > why? What do they want to do with that measurement?

                    I have many of the same question whirling in my head. To some, I have
                    some provisional answers, to some I don't. However, I got blasted in
                    this very list *precisely* because I advocated more rigor in teh CoP
                    field (that is to say, as simple as rather than starting from the
                    "CoPs can't be measured" one should try to measure them first and then
                    resort not to only if s/he doesn't find a suitable way of measuring them).

                    Similarly, I prefer to work with virtual CoPs because I found them and
                    their indicators far easier to quantify. Most of the face-to-face
                    peripheral dialogue is ephemeral, can't even be witnessed (sometimes
                    it happens in elevators, phone calls, etc), and therefore one can have
                    the impression that a given intervention is the cause of success
                    because s/he doesn't know that - for example - 10 members went to a
                    cinema together and ended up discussing practice-related stuff.

                    Do let me know when your book will get published. I'll be happy to
                    purchase it. I've always admired your sense of wonder and acute
                    intelligence. Maybe you're too much for the field (and it's NOT a diss
                    at you -- please understand me here -- it's more discouragement about
                    how the CoP "practice" is done in practice... excellent theory, but
                    the practice --- ahem --- there's not enough practical stuff YET for
                    it to be practiced!).

                    Rosanna Tarsiero, CKM
                  • pete bond
                    Hi Stephen, I think it is possible to create the conditions in which a CoP would crystallize from a wider system, but in my experience it a takes a while. An
                    Message 9 of 17 , Jul 8, 2008
                    • 0 Attachment
                      Hi Stephen, I think it is possible to create the conditions in which a
                      CoP would crystallize from a wider system, but in my experience it a
                      takes a while. An example which started from nothing. I organise a
                      chapter of Dave Gurteen's knowledge cafe for the North West UK. My
                      vision is that a CoP would form around the practice of KM to become an
                      asset for the regional economic development agencies who might then
                      refer to it when developing, for example, SME networks or industrial
                      clusters. (latest event at knowledgecafe0151.eventbrite.com). The
                      mechanism through which relations are formed is conversation, so the key
                      is to intensify and extend a network of conversations, or to precipitate
                      a new conversations through, for example, face to face events (the
                      cafes) and enabling follow up conversations on-line through one of the
                      new collaborative spaces (ning.com or collectiveX). After a year I feel
                      something is happening, but it was not what I expected. Instead of KM
                      practitioners, I've attracted a lot of business and personal coaches.

                      I've been utilising a methodology for accelerating the development of
                      learning communities called KALiF, which was developed by my colleagues
                      as part of an EU research programme.

                      peter


                      Stephen Denning wrote:
                      >
                      > John writes: "As Etienne repeats tirelessly, "it's a matter of
                      > *discovering*
                      > the communities of practice that are in some sense already THERE, rather
                      > than creating or building something from scratch."
                      >
                      > While I agree that the easy thing, the obvious thing, is to identify
                      > communities that already exist, communities must have come into being
                      > somehow. How? My current work is focused on understanding how this
                      > happens -
                      > how communities come into existence -- with the object of
                      > understanding how
                      > we could facilitate and accelerate their creation, as opposed to passively
                      > waiting for lightning to strike.
                      >
                      > I'm particularly interested in high-performance communities i.e.
                      > communities
                      > that are exceptional in performance and deeply meaningful for the
                      > participants. Is it possible to take active steps to generate such
                      > communities?
                      >
                      > I know that there is a school of thought to the effect that communities
                      > can't be created -- they emerge; attempts to create them will be
                      > counterproductive. And if the repertoire of options available were simply
                      > the clumsy tools of traditional comand-and-control management, I would
                      > agree. But there is a wider set of options, particularly narrative
                      > modes of
                      > communicating, which I believe create different possibilities.
                      >
                      > If anyone has experiences or ideas in this area, I would love to hear from
                      > you.
                      >
                      > Steve Denning
                      > steve@... <mailto:steve%40stevedenning.com>
                      > www.stevedening.com
                      >
                      > PS: As to rigorously evaluating communities and understanding what is
                      > exceptional performance, one does have to take into account the different
                      > purposes for which communities may exist. John writes: "It seems to me
                      > that
                      > what we expect from our communities is that they change the world." A CFO
                      > might have a very different perspective ("Are they helping the bottom line
                      > of this organization?"). A participant might have still another view:
                      > ("Are
                      > they helping me in my work?) Not all communities are aimed at changing the
                      > world. Rigor may be in the eye of the beholder.
                      >
                      > [Non-text portions of this message have been removed]
                      >
                      >
                    • Stan Garfield
                      ... with some rigor? These blog entries may be of interest: * CoP Strategy and Measurement
                      Message 10 of 17 , Jul 8, 2008
                      • 0 Attachment
                        --- In com-prac@yahoogroups.com, "Jon Ford" <jon.ford@...> wrote:
                        > in what direction should we turn to learn more about evaluating CoPs
                        with some rigor?

                        These blog entries may be of interest:

                        * CoP Strategy and Measurement
                        http://www.communities.hp.com/online/blogs/garfield/archive/2007/05/02/H\
                        PPost3267.aspx
                        <http://www.communities.hp.com/online/blogs/garfield/archive/2007/05/02/\
                        HPPost3267.aspx>
                        * Community Metrics
                        http://www.communities.hp.com/online/blogs/garfield/archive/2007/08/22/H\
                        PPost4233.aspx
                        <http://www.communities.hp.com/online/blogs/garfield/archive/2007/08/22/\
                        HPPost4233.aspx>

                        Regards,
                        Stan
                        http://stangarfield.googlepages.com/
                        <http://stangarfield.googlepages.com/>




                        [Non-text portions of this message have been removed]
                      • Stan Garfield
                        ... understanding how ... passively ... in this area, ... Steve, I had positive experience with this at HP. Feel free to contact me directly to discuss
                        Message 11 of 17 , Jul 8, 2008
                        • 0 Attachment
                          --- In com-prac@yahoogroups.com, "Stephen Denning" <steve@...> wrote:
                          > My current work is focused on understanding how this happens -
                          > how communities come into existence -- with the object of
                          understanding how
                          > we could facilitate and accelerate their creation, as opposed to
                          passively
                          > waiting for lightning to strike. If anyone has experiences or ideas
                          in this area,
                          > would love to hear from you.


                          Steve,

                          I had positive experience with this at HP. Feel free to contact me
                          directly to discuss further.

                          Regards,
                          Stan
                          http://stangarfield.googlepages.com/
                          <http://stangarfield.googlepages.com/>




                          [Non-text portions of this message have been removed]
                        • fnickols
                          Hmm. I was browsing the messages this morning and came across this one from Jon Ford. I subsequently read the rest of the thread so I hope my response adds
                          Message 12 of 17 , Jul 9, 2008
                          • 0 Attachment
                            Hmm. I was browsing the messages this morning and came across this
                            one from Jon Ford. I subsequently read the rest of the thread so I
                            hope my response adds to and reinforces what others have had to say.
                            I don't think I'm contradicting anyone.

                            --- In com-prac@yahoogroups.com, "Jon Ford" <jon.ford@...> wrote:

                            <snip>

                            > ...we're being given the push to put an
                            > evaluation "structure" around our groups.
                            >
                            > No doubt many of you have resources on this. Fred Nickols
                            > and Hildy have already responded to a colleague of mine
                            > on a different list (I believe it was the American
                            > Evaluation group list). I'm quite certain I have some
                            > memory of this topic coming up before on this list also.
                            > In fact Fred may have had the last word on this already,
                            > noting that each CoP is unique and therefore requires a
                            > customized evaluation approach. (Sorry Fred, if my
                            > paraphrase is not quite accurate.)
                            >
                            > All that said, in what direction should we turn to learn
                            > more about evaluating CoPs with some rigor?

                            Allow me some assumptions. First, someone at your organization touted
                            CoPs as a good thing to do, either for a specific purpose or as a
                            good thing in general. Second, someone at your organization decided
                            to authorize, support, fund or in some other way commit organizational
                            resources to some kind of CoP effort. Third, someone at your
                            organization now wants to know what it got in return for those
                            resources. Hence the "push" for some kind of evaluation structure.

                            If I have these right, much of the counsel you've received seems solid
                            enough. Start with the basic questions:

                            First, Who wants to know what about CoPs? Are they popular? Well
                            received? Contributing to the organization's goals? Improving CoP
                            member performance?

                            Second, What will be done with that information? Are there decisions
                            waiting to be made? Will the resources allocated to CoPs change?
                            Will someone try to shut them down if the evaluation results don't
                            meet with approval? Will the organization's strategy toward CoPs
                            change? Will the person who originally touted CoPs to the
                            organization be hanged, drawn and quartered?

                            Third, take stock of what it will take to conduct the kind of
                            evaluation indicated and what it will cost to do it. It is entirely
                            conceivable, if not downright likely, that it will cost more to carry
                            out a "rigorous" evaluation than the results of the evaluation will be
                            worth. That's not a particularly good use of scarce organizational
                            resources.

                            Fourth, assuming the person(s) wanting the evaluation are reasonable
                            and rational, they'll likely see that if the costs of a rigorous
                            evaluation outweigh the benefits, there's not much point in doing it.
                            However, it doesn't end there. They still want to know some things
                            and now they're frustrated because their initial thrust has been
                            thwarted. So, find some "less rigorous" ways of getting some answers
                            to their questions. Start by setting clear priorities on just what it
                            is they want to know and why. In the end, maybe all they really want
                            is simply to be confirmed in their original decision to support, fund
                            or authorize CoPs. Or, perhaps they're on a cost-cutting spree and
                            nothing less than axing some things will satisfy them. I don't know
                            and I can't say but I do know that every time I hear "rigorous
                            evaluation" (and not just in relation to CoPs), my "Oh Oh" alarm goes
                            off, warning of a chunk of money headed down the drain.

                            Now, back to your closing question, Jon, the one about where you can
                            "turn to learn more about evaluating CoPs with some rigor." This list
                            is obviously a good place. The EvalTalk list, which you mentioned, is
                            doubtless an excellent resource for learning about evaluation with a
                            focus on programs and interventions. I'm no expert in evaluation but
                            I do know this: To evaluate something is to determine its worth.
                            That entails measurements, standards and judgments. The basic task at
                            hand is finding or establishing those in relation to CoPs. I doubt
                            there is a ready-made solution lying around waiting to be found; it
                            must, as you attributed to me earlier, be crafted to fit the situation
                            at hand.

                            Finally, in your shoes, I'd go to the CoPs in your organization and
                            pose the matter to them. I'd ask them what they think about the
                            issue. On what basis should they be evaluated? Should they be
                            evaluated at all? If they were to be evaluated, how would they stack
                            up at the present time? What do they see as their worth or value to
                            the organization? How do they gauge that? Do they even know what
                            kind of organizational resources they consume or what those cost? How
                            would they determine their worth?

                            P.S. I note you're in Phoenix. My wife and I winter in Tucson and
                            will be there by early October. I have friends and colleagues in
                            Phoenix so maybe we might meet and chat about this later this year.

                            Regards,

                            Fred Nickols
                            Managing Partner
                            Distance Consulting, LLC
                            nickols@...
                            www.nickols.us
                          • Nancy White
                            This has been a really great thread. I hope the originator will consider creating a summary resource. That would be great community building. ! :-) John, there
                            Message 13 of 17 , Jul 9, 2008
                            • 0 Attachment
                              This has been a really great thread. I hope the originator will
                              consider creating a summary resource. That would be great community
                              building. ! :-) John, there is a wiki someplace, right?

                              Nancy

                              Nancy White | Full Circle Associates | Connecting communities online
                              nancyw@... | +1 206 517 4754 | GMT - 8 |skype - choconancy |
                              Twitter NancyWhite
                              http://www.fullcirc.com/


                              [Non-text portions of this message have been removed]
                            • Jon Ford
                              Nancy and All, This HAS been an excellent thread. Many thanks to all the respondents. I have to confess that I haven t fully digested it all due to a death in
                              Message 14 of 17 , Jul 9, 2008
                              • 0 Attachment
                                Nancy and All,



                                This HAS been an excellent thread. Many thanks to all the respondents. I
                                have to confess that I haven't fully digested it all due to a death in
                                the family this week, but I do nonetheless have the goal of getting back
                                on the ball here and producing some type of summary resource. I'm happy
                                to share that back to the thread or post it to a wiki - or both.



                                It seems too that some folks are just getting to replying (life does
                                that), so I'm also keeping my eyes open for other feedback/input.



                                Again, many thanks to all. Keep 'em coming!



                                Jon





                                ---------------

                                Jon Ford

                                Associate Director, Communications

                                St. Luke's Health Initiatives

                                2929 North Central Avenue, Suite 1550

                                Phoenix, AZ 85012

                                office: 602-385-6508

                                cell: 480-650-8997

                                fax: 602-385-6510

                                e-mail: jon.ford@...

                                web: www.slhi.org



                                From: com-prac@yahoogroups.com [mailto:com-prac@yahoogroups.com] On
                                Behalf Of Nancy White
                                Sent: Wednesday, July 09, 2008 10:57 AM
                                To: com-prac@yahoogroups.com
                                Subject: Re: [cp] Re: CoP Evaluation



                                This has been a really great thread. I hope the originator will
                                consider creating a summary resource. That would be great community
                                building. ! :-) John, there is a wiki someplace, right?

                                Nancy

                                Nancy White | Full Circle Associates | Connecting communities online
                                nancyw@... <mailto:nancyw%40fullcirc.com> | +1 206 517 4754 |
                                GMT - 8 |skype - choconancy |
                                Twitter NancyWhite
                                http://www.fullcirc.com/

                                [Non-text portions of this message have been removed]





                                [Non-text portions of this message have been removed]
                              • John D. Smith
                                I just bumped into a blog post that proposes a schedule -- a kind of sufficient unto the time are the cop metrics therof approach: * Phase 1 (0-12 months):
                                Message 15 of 17 , Jul 11, 2008
                                • 0 Attachment
                                  I just bumped into a blog post that proposes a schedule -- a kind of
                                  "sufficient unto the time are the cop metrics therof" approach:

                                  * Phase 1 (0-12 months): New members, pageviews, ratings, comments

                                  * Phase 2 (12-24 months): UGC, posts/user, visits/user/month, % of active
                                  members

                                  * Phase 3 (24 months +): Activity of community leaders, initiatives/ideas
                                  generated, ROI/value measures

                                  http://www.thesocialorganization.com/2008/06/do-enterprises-have-the-patienc
                                  e-to-develop-communities.html (which shortens to: http://bit.ly/cpeval)

                                  Not as deep as Fred's summary, but an important idea.

                                  John
                                  *
                                  * John D. Smith ~ Voice: 503.963.8229 ~ Skype: smithjd
                                  * Portland, Oregon, USA http://www.learningAlliances.net
                                  * see http://technologyforcommunities.com/tools/
                                  * “You can't improvise on nothing, man,
                                  you've gotta improvise on something.” -- Charles Mingus
                                • Hildy Gottlieb
                                  ... *************************** And then there are those who have already weighed in and cannot help but return to the scene of the crime! ... I came across
                                  Message 16 of 17 , Jul 11, 2008
                                  • 0 Attachment
                                    Jon Ford wrote:
                                    >
                                    > It seems too that some folks are just getting to replying (life does
                                    > that), so I'm also keeping my eyes open for other feedback/input.
                                    >



                                    ***************************

                                    And then there are those who have already weighed in and cannot help but
                                    return to the scene of the crime!
                                    :-)

                                    I came across this blog post, and while it is specifically about the
                                    desire to measure the effectiveness of social media, I think there is
                                    much here that relates, including the list the group came up with (near
                                    the bottom of the post).
                                    http://thebrandbox.blogspot.com/2008/07/plurkshop-6-measuring-social-media.html

                                    I hope it's helpful. Have a great weekend, all!

                                    Hildy

                                    Hildy Gottlieb
                                    *The Community-Driven Institute **at Help 4 NonProfits
                                    *Making Visionary Change Practical
                                    _http://www.Help4NonProfits.com <http://www.help4nonprofits.com/>
                                    _/*BLOG:*/ http://www.HildyGottlieb.com/
                                    Hildy@...
                                    520-321-4433
                                    or 1-888-787-4433




                                    [Non-text portions of this message have been removed]
                                  • Cornejo Castro, Miguel
                                    Hi all, great thread indeed, a nice surprise after holidays... My three cents :-): - I agree (strongly) that a goals-based evaluation is the only sensible way
                                    Message 17 of 17 , Jul 14, 2008
                                    • 0 Attachment
                                      Hi all,

                                      great thread indeed, a nice surprise after holidays...

                                      My three cents :-):

                                      - I agree (strongly) that a goals-based evaluation is the only sensible way to evaluate any activity, whatever the nature.
                                      - Finding indicators for those goals (i.e. objective things that give you a clue about how things are really going) is a bit more difficult than it sounds.
                                      - Measurements without a baseline are rather meaningless. Do you know where you were when you started? Can you track the effect of other initiatives?

                                      Back in '06 I wrote this paper for the Knowledgeboard. It's a sort of measurement guidebook, mostly for online communities, based on the methods used to fine-tune Macuarium along the years:

                                      http://www.macuarium.com/cms/index.php?option=com_remository&Itemid=169&func=fileinfo&id=447

                                      It's not complete (lots of ground not covered), and it's not perfect, but it might be useful as a guide into the whys and why-nots of using certain indicators to deduce some things about a CoP. In other words, the measurements may be useful, but the logic will probably be.

                                      The paper is still at the KBoard, but that requires registration.

                                      Hope it's useful. Best regards,

                                      Miguel


                                      ________________________________
                                      De: com-prac@yahoogroups.com [mailto:com-prac@yahoogroups.com] En nombre de Hildy Gottlieb
                                      Enviado el: sábado, 12 de julio de 2008 0:22
                                      Para: com-prac@yahoogroups.com
                                      Asunto: Re: [cp] Re: CoP Evaluation


                                      Jon Ford wrote:
                                      >
                                      > It seems too that some folks are just getting to replying (life does
                                      > that), so I'm also keeping my eyes open for other feedback/input.
                                      >

                                      ***************************

                                      And then there are those who have already weighed in and cannot help but
                                      return to the scene of the crime!
                                      :-)

                                      I came across this blog post, and while it is specifically about the
                                      desire to measure the effectiveness of social media, I think there is
                                      much here that relates, including the list the group came up with (near
                                      the bottom of the post).
                                      http://thebrandbox.blogspot.com/2008/07/plurkshop-6-measuring-social-media.html

                                      I hope it's helpful. Have a great weekend, all!

                                      Hildy

                                      Hildy Gottlieb
                                      *The Community-Driven Institute **at Help 4 NonProfits
                                      *Making Visionary Change Practical
                                      _http://www.Help4NonProfits.com <http://www.help4nonprofits.com/>
                                      _/*BLOG:*/ http://www.HildyGottlieb.com/
                                      Hildy@...<mailto:Hildy%40Help4NonProfits.com>
                                      520-321-4433
                                      or 1-888-787-4433

                                      [Non-text portions of this message have been removed]





                                      [Non-text portions of this message have been removed]
                                    Your message has been successfully submitted and would be delivered to recipients shortly.