Loading ...
Sorry, an error occurred while loading the content.

Agile development with many projects and one team

Expand Messages
  • flyerguybham
    Hi all, We are considering moving more towards agile development practices in our development group. We are a group of about 12 developers. Right now we re
    Message 1 of 9 , Aug 21, 2013
    • 0 Attachment
      Hi all,

      We are considering moving more towards agile development practices in our development group. We are a group of about 12 developers. Right now we're split into teams of 1-3 members, each servicing one or more projects (typically, one main project and some smaller ancillary projects for each team).

      This has led to unnecessary duplication of effort across the projects, which are all somewhat similar in regards to the requirements and challenges that need to be addressed, but each serves a different group of customers. Every time we spin up another team of 1-3 folks to service a new project, they're not benefitting from the collective experience and wisdom of the staff as a whole, and are prone to either 1. repeat past mistakes, or 2. make technology and design decisions based on their individual/small-team biases and lead to yet another silo of slightly different/diverging technology stack. We've tried informal means of cross-project pollination (monthly team presentations, assembling "tiger teams" around a single topic like status monitoring or JavaScript libraries, etc...) and while these generate a lot of enthusiasm at the moment, they rarely lead to lasting effects or significant convergence in the individual projects.

      So - we want to operate more as a larger, unified team. One path to this is to take an agile approach and assemble 1 or maybe 2 cross-functional agile teams (with the idea that we don't want team sizes greater than about 7 or 8, we're probably looking at two teams). However, each team would have, in Scrum terms, a significant number of unrelated (or, at best, slightly related) product backlogs feeding into the team, one from each of the projects/customer groups that the team is servicing.

      The goal is that over time this structure would bring the individual project codebases closer together, lead to more reusable components, and help us converge on technology best practices and solving problems the same way across all projects. But, especially at first, these projects are pretty independent of one another.

      How have other organizations dealt with this issue? What techniques do you use to ensure that each project gets its fair share of attention based on how much they're contributing to the funding (or whatever other metric is important in your org for this)? How did the developers react to this - did they start working on each other's projects, or did they all just continue to focus on the stories from the project(s) they used to be servicing when there were separate small project teams? How did the customers react - was it a hard sell to them (and to management)?

      Thoughts and opinions from others who have already tackled this would be greatly appreciated. I haven't come across a lot of material that addresses agile in multi-project environments - much more material assumes that there is a single large project and/or customer being serviced by the team.

      Thanks,
      Fran
    • Adam Sroka
      ... Are you familiar with Conway s Law? I wonder if the alignment of your projects more closely resembles your internal organizational structure rather than
      Message 2 of 9 , Aug 21, 2013
      • 0 Attachment
        On Wed, Aug 21, 2013 at 12:57 PM, flyerguybham <fran@...>wrote:

        > **
        >
        >
        > Hi all,
        >
        > We are considering moving more towards agile development practices in our
        > development group. We are a group of about 12 developers. Right now we're
        > split into teams of 1-3 members, each servicing one or more projects
        > (typically, one main project and some smaller ancillary projects for each
        > team).
        >
        >
        Are you familiar with Conway's Law? I wonder if the alignment of your
        projects more closely resembles your internal organizational structure
        rather than the view from an outside customer's perspective. Maybe if it
        were more aligned to the latter the problem would be easier to solve.


        > This has led to unnecessary duplication of effort across the projects,
        > which are all somewhat similar in regards to the requirements and
        > challenges that need to be addressed, but each serves a different group of
        > customers. Every time we spin up another team of 1-3 folks to service a new
        > project, they're not benefitting from the collective experience and wisdom
        > of the staff as a whole, and are prone to either 1. repeat past mistakes,
        > or 2. make technology and design decisions based on their
        > individual/small-team biases and lead to yet another silo of slightly
        > different/diverging technology stack. We've tried informal means of
        > cross-project pollination (monthly team presentations, assembling "tiger
        > teams" around a single topic like status monitoring or JavaScript
        > libraries, etc...) and while these generate a lot of enthusiasm at the
        > moment, they rarely lead to lasting effects or significant convergence in
        > the individual projects.
        >
        >
        I'm not sure that this is actually a problem. Repeating past mistakes is
        part of the learning process. It is likely that a team that is doing
        something now that you did before is learning something that you needed to
        learn to get where you are. In nature, DNA is not constrained from making
        the same mutations over and over again. Perhaps this works because what was
        an unsuccessful adaptation before becomes beneficial in a new context.

        Individual/small-team biases are worth listening to, because those are the
        folks who have to live in the context in which the problem came up. So,
        perhaps their insights are valid there even if they seem contradictory to
        something in some different context.


        > So - we want to operate more as a larger, unified team. One path to this
        > is to take an agile approach and assemble 1 or maybe 2 cross-functional
        > agile teams (with the idea that we don't want team sizes greater than about
        > 7 or 8, we're probably looking at two teams). However, each team would
        > have, in Scrum terms, a significant number of unrelated (or, at best,
        > slightly related) product backlogs feeding into the team, one from each of
        > the projects/customer groups that the team is servicing.
        >
        >
        Sounds like it's worth a shot. My own personal experience suggests that
        smaller teams focused on individual products sometimes get better results
        sooner. However, it sounds like that isn't working for you. The worst case
        scenario is that changing it will give you worse results. If that happens
        you will have learned something.


        > The goal is that over time this structure would bring the individual
        > project codebases closer together, lead to more reusable components, and
        > help us converge on technology best practices and solving problems the same
        > way across all projects. But, especially at first, these projects are
        > pretty independent of one another.
        >
        >
        Again, I am not convinced, based on my own experience, that convergence of
        codebases or practices are beneficial. I certainly don't believe that they
        are worthy business goals in and of themselves. If each team is serving its
        customers effectively those customers aren't likely to care if they are
        doing a variety of things to accomplish it. It is certainly possible that
        teams can learn valuable lessons from one another and the broader community
        but from a business perspective it is at most a secondary goal.


        [Non-text portions of this message have been removed]
      • Steven Gordon
        Fran, This smells more like a business problem that a methodology or software development problem. Is it really best to fragment the company s capabilities on
        Message 3 of 9 , Aug 21, 2013
        • 0 Attachment
          Fran,

          This smells more like a business problem that a methodology or software
          development problem.

          Is it really best to fragment the company's capabilities on so many
          parallel projects? Wouldn't concentrating the capabilities on getting
          fewer projects completely done and earning revenue, and then coming back to
          the other projects be more profitable?

          Even if these projects are for clients, you will not get them done faster
          by spreading your capabilities so thinly. Prioritizing gets some projects
          done sooner and none any later than doing them all at the same time. Draw
          them out on graph paper, and you will see this is the case. When you
          factor in context-switching time, even the last projects done in priority
          order get done earlier than when they are all worked on in parallel.

          Probably not your decision, but it is the crux of the issue. If you can
          get to concentrating on finishing N projects at a time, I would recommend N
          separate teams.

          I would ask how many of these projects are new development vs. mostly
          maintenance? This is a factor in how you might go about organizing.




          On Wed, Aug 21, 2013 at 9:57 AM, flyerguybham <fran@...> wrote:

          > **
          >
          >
          > Hi all,
          >
          > We are considering moving more towards agile development practices in our
          > development group. We are a group of about 12 developers. Right now we're
          > split into teams of 1-3 members, each servicing one or more projects
          > (typically, one main project and some smaller ancillary projects for each
          > team).
          >
          > This has led to unnecessary duplication of effort across the projects,
          > which are all somewhat similar in regards to the requirements and
          > challenges that need to be addressed, but each serves a different group of
          > customers. Every time we spin up another team of 1-3 folks to service a new
          > project, they're not benefitting from the collective experience and wisdom
          > of the staff as a whole, and are prone to either 1. repeat past mistakes,
          > or 2. make technology and design decisions based on their
          > individual/small-team biases and lead to yet another silo of slightly
          > different/diverging technology stack. We've tried informal means of
          > cross-project pollination (monthly team presentations, assembling "tiger
          > teams" around a single topic like status monitoring or JavaScript
          > libraries, etc...) and while these generate a lot of enthusiasm at the
          > moment, they rarely lead to lasting effects or significant convergence in
          > the individual projects.
          >
          > So - we want to operate more as a larger, unified team. One path to this
          > is to take an agile approach and assemble 1 or maybe 2 cross-functional
          > agile teams (with the idea that we don't want team sizes greater than about
          > 7 or 8, we're probably looking at two teams). However, each team would
          > have, in Scrum terms, a significant number of unrelated (or, at best,
          > slightly related) product backlogs feeding into the team, one from each of
          > the projects/customer groups that the team is servicing.
          >
          > The goal is that over time this structure would bring the individual
          > project codebases closer together, lead to more reusable components, and
          > help us converge on technology best practices and solving problems the same
          > way across all projects. But, especially at first, these projects are
          > pretty independent of one another.
          >
          > How have other organizations dealt with this issue? What techniques do you
          > use to ensure that each project gets its fair share of attention based on
          > how much they're contributing to the funding (or whatever other metric is
          > important in your org for this)? How did the developers react to this - did
          > they start working on each other's projects, or did they all just continue
          > to focus on the stories from the project(s) they used to be servicing when
          > there were separate small project teams? How did the customers react - was
          > it a hard sell to them (and to management)?
          >
          > Thoughts and opinions from others who have already tackled this would be
          > greatly appreciated. I haven't come across a lot of material that addresses
          > agile in multi-project environments - much more material assumes that there
          > is a single large project and/or customer being serviced by the team.
          >
          > Thanks,
          > Fran
          >
          >
          >


          [Non-text portions of this message have been removed]
        • Fran Fabrizio
          ... Yes, came across it in some of my research on this topic earlier this week. Definitely resonated with me. ... [snip] ... These two comments made me realize
          Message 4 of 9 , Aug 21, 2013
          • 0 Attachment
            > Right now we're

            > > split into teams of 1-3 members, each servicing one or more projects
            > > (typically, one main project and some smaller ancillary projects for each
            > > team).
            > >
            > >
            > Are you familiar with Conway's Law?
            >

            Yes, came across it in some of my research on this topic earlier this week.
            Definitely resonated with me.


            > I'm not sure that this is actually a problem.
            >

            [snip]


            > Individual/small-team biases are worth listening to, because those are the
            > folks who have to live in the context in which the problem came up.
            >

            These two comments made me realize that I didn't actually explain clearly
            why the current setup is problematic.

            If I may ask for your patience, I'll give the necessary background context.
            We're a large research center and we work on a set of federally-funded
            research projects. As an IT group, our customers are the groups of
            researchers who work on these grants. Our current setup is that we have a
            team for each of our large grants. (This is a slight simplification but
            it's close enough to be valid for the purposes here). Twenty or so years
            ago, we got our first grant, and a dev group was formed to work on the
            project, which was/is essentially to build a large data processing and
            dissemination system. That project continues today. About fifteen years
            ago, a second grant came along, and a second team was born to do much the
            same thing but for a different set of data and customers. About a decade
            ago we got a third major grant. A third team was born. The project was
            dealing with a different kind of data and a different customer yet again,
            but from the dev side, we were building another large data processing and
            dissemination system. That project also is still active. Last year, a
            fourth grant was won. A fourth team was born, to build (you guessed it) a
            large data processing and dissemination system (again, different data
            domain and different customers).

            We now have four teams building or maintaining large data processing and
            dissemination systems in various stages of their lifecycles. Because they
            were all born many years apart, they are pretty divergent in terms of their
            design approach to solving the core components of such systems. The second
            team figured out how to do some things better than the first (after making
            many of the same mistakes the first team did), and the third team did
            better than the second (after again making the same mistakes), and the
            fourth is now in the process of doing that as well. But as the newer teams
            come up with better approaches, these are not "backported" into the
            existing projects. So, we're not scaling well, and we're maintaining a lot
            of code for which we've since learned better ways to achieve the same
            thing.

            Pros
            - From each individual project's point of view, things look great - they
            have a dedicated dev team taking care of them.
            - Those devs have deep expertise on their projects.

            Cons
            - From the organization's point of view, this is very inefficient. We
            aren't scaling any better than linearly even though we're building systems
            which share a lot of characteristics with each other.
            - We're not using best practices everywhere, and we're not using the
            insights gained over time to shed some of these legacy approaches and
            technical debt so we can work smarter and be better positioned to take on
            new things.
            - From the dev team's point of view, it's unsatisfying to continue to
            support stuff for which better solutions are right in front of us.
            Particularly for those working on older projects, it has an effect on
            morale and retention.
            - When a project has only 1-2 people on it, it's hard to step back from
            daily feature development and refactor or port in some of these better
            solutions. When a team is 1 web guy and 1 db guy, there's no breathing
            space to do so.
            - Project funding and demand ebbs and flows, and it's often hard to keep
            one team busy while other teams are starved for resources. Right now, our
            devs are not very portable between projects because of how different the
            codebases are.
            - With 1-2 person teams, a person's departure has a big negative impact.
            There are many single points of failure with small teams, and a lot of
            knowledge walks out the door with that person.
            - With 1-2 person teams, a product ends up being a very strong reflection
            of the individual personalities - we see it in choice of technologies, code
            approaches, which parts of the project get the most attention based on
            personal interest, etc...
            - From my point of view (IT management), it's a whole lot of cats to
            herd, and it's very hard to form a team-wide practice.

            We're starting to feel that the current way of doing things is not
            sustainable. The data that comes into these projects (census, demographic,
            environmental, geospatial) is exploding in quantity and complexity. The
            core problem we need to solve is to figure out how to scale to continue to
            be able to process and disseminate that data. We need to find ways to
            simplify and converge the portfolio we're currently managing so that we
            have capacity and ability to meet the challenges the next 3-5 years will
            bring to us. There are currently four different data ingest processes,
            four different data storage engines, four different metadata management
            setups, etc...

            So, that's a summary of the challenge. :-) It's an organization-wide
            challenge, but my role as IT management is to ask whether we are the most
            efficient IT group we can be, and whether or not we can align the dev group
            in a way that helps us meet that challenge. While asking that, I started
            to ponder the idea of larger, Agile teams as a response.

            Sounds like it's worth a shot. My own personal experience suggests that
            > smaller teams focused on individual products sometimes get better results
            > sooner. However, it sounds like that isn't working for you.
            >

            Actually this is true for us. The first 1-2 years of a new project, the
            small focused team always seems like a great idea, people are motivated,
            they're pumping out new features left and right, it feels like we're making
            a lot of progress.... and then the project matures, and now it looks
            different than anything else we have, it starts to be bigger in scope than
            what 1-2 people can handle or some of those initial team members leave, and
            that's where the problems tend to become more pronounced.

            Again, I am not convinced, based on my own experience, that convergence of
            > codebases or practices are beneficial. I certainly don't believe that they
            > are worthy business goals in and of themselves. If each team is serving its
            > customers effectively those customers aren't likely to care if they are
            > doing a variety of things to accomplish it. It is certainly possible that
            > teams can learn valuable lessons from one another and the broader community
            > but from a business perspective it is at most a secondary goal.
            >

            This is the main reason why we haven't changed yet. We're known for
            providing very good service to our customers and the individual customers
            are largely happy with status quo. However, senior leadership senses the
            fundamental change in the nature/quantity/complexity/velocity of the data
            coming at us, combined with how research funding is changing, and knows
            we're not going to be able to scale to the challenge easily in our current
            form. We're not going to be able to simply throw a linear amount of new
            labor at the problem as we have been able to do to date. So we're trying
            to get out ahead of it. Transforming while keeping the level of customer
            service at the same high level will be a huge challenge, regardless of what
            we do.

            Hopefully that provides more context. Perhaps it's a bit far afield from
            the main purpose of this list, but it does seem like Agile development
            practices could provide a number of defenses and opportunities to us moving
            forward. It's just the part about one team, multiple projects that I'm
            trying to get my head around, how that would actually work in practice.

            Thanks,
            Fran


            [Non-text portions of this message have been removed]
          • Fran Fabrizio
            ... Certainly (as my last reply hopefully shows) the problem is bigger than a dev problem, but I do think that the current and historical approach to the dev
            Message 5 of 9 , Aug 21, 2013
            • 0 Attachment
              On Wed, Aug 21, 2013 at 4:47 PM, Steven Gordon <sgordonphd@...> wrote:

              > Fran,
              >
              > This smells more like a business problem that a methodology or software
              > development problem.
              >

              Certainly (as my last reply hopefully shows) the problem is bigger than a
              dev problem, but I do think that the current and historical approach to the
              dev has contributed to the problem, and hopefully can have a role in
              addressing it moving forward.


              > Is it really best to fragment the company's capabilities on so many
              > parallel projects? Wouldn't concentrating the capabilities on getting
              > fewer projects completely done and earning revenue, and then coming back to
              > the other projects be more profitable?
              >

              Yes. This model would have a hard time flying in the corporate world, that
              much is certain.

              Higher ed research centers are a different universe. :-) Win a grant ->
              build a team and consider each grant in isolation from one another is
              generally actually a decent approach (with the somewhat unfortunate side
              effect that many grants don't get renewed, so people are out of a job when
              it ends). In some sense, our problem is that we're actually unusually
              successful in what we do - we have a phenomenal grant renewal rate relative
              to peers and federal research as a whole - so the projects never die. :-)
              We just keep piling on new ones.


              > Even if these projects are for clients, you will not get them done faster
              > by spreading your capabilities so thinly. Prioritizing gets some projects
              > done sooner and none any later than doing them all at the same time. Draw
              > them out on graph paper, and you will see this is the case. When you
              > factor in context-switching time, even the last projects done in priority
              > order get done earlier than when they are all worked on in parallel.
              >

              Interesting point. We work in terms of research grants (which are generally
              no more granular than "we'll do X in year one, Y in year two, etc..." for a
              typical 5 year grant.) The challenge is that even though logically some
              are higher priority than others, there are entire teams of researchers that
              would be stuck waiting if we decided we weren't going to work on their
              project for say, 6 months, and we're obligated to our funding sources to be
              making more or less consistent progress over the life of the grant. Once
              the clock starts ticking on one of them, you can't really say "well, we'll
              get to it next year" - you can hedge a little bit (try to front-load some
              of the non-IT portions of the work, etc...) but eventually you need to get
              working on each. And I simplified it in my earlier post - we actually have
              20+ grants all active right now which all include dev deliverables.


              > I would ask how many of these projects are new development vs. mostly
              > maintenance? This is a factor in how you might go about organizing.
              >

              All feature new development regularly (that's how you keep the grant
              machine rolling - "renew our grant and we'll add features X,Y and Z over
              the next 5 years"). The older projects also require a lot of maintenance
              of code implemented long ago. The main difference is that in a young
              project, you're likely building everything up at the same time (web UI,
              data processing pipelines, metadata tools) whereas on a more mature project
              you may be able to be more focused on just one or two areas of the product
              where you choose to make refinements.

              -Fran


              [Non-text portions of this message have been removed]
            • Steven Gordon
              Definitely, a different context than I expected. ... [Non-text portions of this message have been removed]
              Message 6 of 9 , Aug 21, 2013
              • 0 Attachment
                Definitely, a different context than I expected.


                On Wed, Aug 21, 2013 at 3:09 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

                > **
                >
                >
                > On Wed, Aug 21, 2013 at 4:47 PM, Steven Gordon <sgordonphd@...>
                > wrote:
                >
                > > Fran,
                > >
                > > This smells more like a business problem that a methodology or software
                > > development problem.
                > >
                >
                > Certainly (as my last reply hopefully shows) the problem is bigger than a
                > dev problem, but I do think that the current and historical approach to the
                > dev has contributed to the problem, and hopefully can have a role in
                > addressing it moving forward.
                >
                >
                > > Is it really best to fragment the company's capabilities on so many
                > > parallel projects? Wouldn't concentrating the capabilities on getting
                > > fewer projects completely done and earning revenue, and then coming back
                > to
                > > the other projects be more profitable?
                > >
                >
                > Yes. This model would have a hard time flying in the corporate world, that
                > much is certain.
                >
                > Higher ed research centers are a different universe. :-) Win a grant ->
                > build a team and consider each grant in isolation from one another is
                > generally actually a decent approach (with the somewhat unfortunate side
                > effect that many grants don't get renewed, so people are out of a job when
                > it ends). In some sense, our problem is that we're actually unusually
                > successful in what we do - we have a phenomenal grant renewal rate relative
                > to peers and federal research as a whole - so the projects never die. :-)
                > We just keep piling on new ones.
                >
                >
                > > Even if these projects are for clients, you will not get them done faster
                > > by spreading your capabilities so thinly. Prioritizing gets some projects
                > > done sooner and none any later than doing them all at the same time. Draw
                > > them out on graph paper, and you will see this is the case. When you
                > > factor in context-switching time, even the last projects done in priority
                > > order get done earlier than when they are all worked on in parallel.
                > >
                >
                > Interesting point. We work in terms of research grants (which are generally
                > no more granular than "we'll do X in year one, Y in year two, etc..." for a
                > typical 5 year grant.) The challenge is that even though logically some
                > are higher priority than others, there are entire teams of researchers that
                > would be stuck waiting if we decided we weren't going to work on their
                > project for say, 6 months, and we're obligated to our funding sources to be
                > making more or less consistent progress over the life of the grant. Once
                > the clock starts ticking on one of them, you can't really say "well, we'll
                > get to it next year" - you can hedge a little bit (try to front-load some
                > of the non-IT portions of the work, etc...) but eventually you need to get
                > working on each. And I simplified it in my earlier post - we actually have
                > 20+ grants all active right now which all include dev deliverables.
                >
                >
                > > I would ask how many of these projects are new development vs. mostly
                > > maintenance? This is a factor in how you might go about organizing.
                > >
                >
                > All feature new development regularly (that's how you keep the grant
                > machine rolling - "renew our grant and we'll add features X,Y and Z over
                > the next 5 years"). The older projects also require a lot of maintenance
                > of code implemented long ago. The main difference is that in a young
                > project, you're likely building everything up at the same time (web UI,
                > data processing pipelines, metadata tools) whereas on a more mature project
                > you may be able to be more focused on just one or two areas of the product
                > where you choose to make refinements.
                >
                > -Fran
                >
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >


                [Non-text portions of this message have been removed]
              • Chris Hulan
                Any chances of getting a computer science researcher to get a grant to research a software framework for science research? ... [Non-text portions of this
                Message 7 of 9 , Aug 22, 2013
                • 0 Attachment
                  Any chances of getting a computer science researcher to get a grant to
                  research a software framework for science research?




                  On Wed, Aug 21, 2013 at 5:57 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

                  > **
                  >
                  >
                  > > Right now we're
                  >
                  > > > split into teams of 1-3 members, each servicing one or more projects
                  > > > (typically, one main project and some smaller ancillary projects for
                  > each
                  > > > team).
                  > > >
                  > > >
                  > > Are you familiar with Conway's Law?
                  > >
                  >
                  > Yes, came across it in some of my research on this topic earlier this week.
                  > Definitely resonated with me.
                  >
                  >
                  > > I'm not sure that this is actually a problem.
                  > >
                  >
                  > [snip]
                  >
                  >
                  > > Individual/small-team biases are worth listening to, because those are
                  > the
                  > > folks who have to live in the context in which the problem came up.
                  > >
                  >
                  > These two comments made me realize that I didn't actually explain clearly
                  > why the current setup is problematic.
                  >
                  > If I may ask for your patience, I'll give the necessary background context.
                  > We're a large research center and we work on a set of federally-funded
                  > research projects. As an IT group, our customers are the groups of
                  > researchers who work on these grants. Our current setup is that we have a
                  > team for each of our large grants. (This is a slight simplification but
                  > it's close enough to be valid for the purposes here). Twenty or so years
                  > ago, we got our first grant, and a dev group was formed to work on the
                  > project, which was/is essentially to build a large data processing and
                  > dissemination system. That project continues today. About fifteen years
                  > ago, a second grant came along, and a second team was born to do much the
                  > same thing but for a different set of data and customers. About a decade
                  > ago we got a third major grant. A third team was born. The project was
                  > dealing with a different kind of data and a different customer yet again,
                  > but from the dev side, we were building another large data processing and
                  > dissemination system. That project also is still active. Last year, a
                  > fourth grant was won. A fourth team was born, to build (you guessed it) a
                  > large data processing and dissemination system (again, different data
                  > domain and different customers).
                  >
                  > We now have four teams building or maintaining large data processing and
                  > dissemination systems in various stages of their lifecycles. Because they
                  > were all born many years apart, they are pretty divergent in terms of their
                  > design approach to solving the core components of such systems. The second
                  > team figured out how to do some things better than the first (after making
                  > many of the same mistakes the first team did), and the third team did
                  > better than the second (after again making the same mistakes), and the
                  > fourth is now in the process of doing that as well. But as the newer teams
                  > come up with better approaches, these are not "backported" into the
                  > existing projects. So, we're not scaling well, and we're maintaining a lot
                  > of code for which we've since learned better ways to achieve the same
                  > thing.
                  >
                  > Pros
                  > - From each individual project's point of view, things look great - they
                  > have a dedicated dev team taking care of them.
                  > - Those devs have deep expertise on their projects.
                  >
                  > Cons
                  > - From the organization's point of view, this is very inefficient. We
                  > aren't scaling any better than linearly even though we're building systems
                  > which share a lot of characteristics with each other.
                  > - We're not using best practices everywhere, and we're not using the
                  > insights gained over time to shed some of these legacy approaches and
                  > technical debt so we can work smarter and be better positioned to take on
                  > new things.
                  > - From the dev team's point of view, it's unsatisfying to continue to
                  > support stuff for which better solutions are right in front of us.
                  > Particularly for those working on older projects, it has an effect on
                  > morale and retention.
                  > - When a project has only 1-2 people on it, it's hard to step back from
                  > daily feature development and refactor or port in some of these better
                  > solutions. When a team is 1 web guy and 1 db guy, there's no breathing
                  > space to do so.
                  > - Project funding and demand ebbs and flows, and it's often hard to keep
                  > one team busy while other teams are starved for resources. Right now, our
                  > devs are not very portable between projects because of how different the
                  > codebases are.
                  > - With 1-2 person teams, a person's departure has a big negative impact.
                  > There are many single points of failure with small teams, and a lot of
                  > knowledge walks out the door with that person.
                  > - With 1-2 person teams, a product ends up being a very strong reflection
                  > of the individual personalities - we see it in choice of technologies, code
                  > approaches, which parts of the project get the most attention based on
                  > personal interest, etc...
                  > - From my point of view (IT management), it's a whole lot of cats to
                  > herd, and it's very hard to form a team-wide practice.
                  >
                  > We're starting to feel that the current way of doing things is not
                  > sustainable. The data that comes into these projects (census, demographic,
                  > environmental, geospatial) is exploding in quantity and complexity. The
                  > core problem we need to solve is to figure out how to scale to continue to
                  > be able to process and disseminate that data. We need to find ways to
                  > simplify and converge the portfolio we're currently managing so that we
                  > have capacity and ability to meet the challenges the next 3-5 years will
                  > bring to us. There are currently four different data ingest processes,
                  > four different data storage engines, four different metadata management
                  > setups, etc...
                  >
                  > So, that's a summary of the challenge. :-) It's an organization-wide
                  > challenge, but my role as IT management is to ask whether we are the most
                  > efficient IT group we can be, and whether or not we can align the dev group
                  > in a way that helps us meet that challenge. While asking that, I started
                  > to ponder the idea of larger, Agile teams as a response.
                  >
                  >
                  > Sounds like it's worth a shot. My own personal experience suggests that
                  > > smaller teams focused on individual products sometimes get better results
                  > > sooner. However, it sounds like that isn't working for you.
                  > >
                  >
                  > Actually this is true for us. The first 1-2 years of a new project, the
                  > small focused team always seems like a great idea, people are motivated,
                  > they're pumping out new features left and right, it feels like we're making
                  > a lot of progress.... and then the project matures, and now it looks
                  > different than anything else we have, it starts to be bigger in scope than
                  > what 1-2 people can handle or some of those initial team members leave, and
                  > that's where the problems tend to become more pronounced.
                  >
                  >
                  > Again, I am not convinced, based on my own experience, that convergence of
                  > > codebases or practices are beneficial. I certainly don't believe that
                  > they
                  > > are worthy business goals in and of themselves. If each team is serving
                  > its
                  > > customers effectively those customers aren't likely to care if they are
                  > > doing a variety of things to accomplish it. It is certainly possible that
                  > > teams can learn valuable lessons from one another and the broader
                  > community
                  > > but from a business perspective it is at most a secondary goal.
                  > >
                  >
                  > This is the main reason why we haven't changed yet. We're known for
                  > providing very good service to our customers and the individual customers
                  > are largely happy with status quo. However, senior leadership senses the
                  > fundamental change in the nature/quantity/complexity/velocity of the data
                  > coming at us, combined with how research funding is changing, and knows
                  > we're not going to be able to scale to the challenge easily in our current
                  > form. We're not going to be able to simply throw a linear amount of new
                  > labor at the problem as we have been able to do to date. So we're trying
                  > to get out ahead of it. Transforming while keeping the level of customer
                  > service at the same high level will be a huge challenge, regardless of what
                  > we do.
                  >
                  > Hopefully that provides more context. Perhaps it's a bit far afield from
                  > the main purpose of this list, but it does seem like Agile development
                  > practices could provide a number of defenses and opportunities to us moving
                  > forward. It's just the part about one team, multiple projects that I'm
                  > trying to get my head around, how that would actually work in practice.
                  >
                  > Thanks,
                  > Fran
                  >
                  >
                  > [Non-text portions of this message have been removed]
                  >
                  >
                  >


                  [Non-text portions of this message have been removed]
                • Wouter Lagerweij
                  Hi Fran, We like to say that a framework is something extracted after you ve build the same type of system a few times. This seems to have some relevance to
                  Message 8 of 9 , Aug 22, 2013
                  • 0 Attachment
                    Hi Fran,

                    We like to say that a framework is something extracted after you've build
                    the same type of system a few times. This seems to have some relevance to
                    your situation.
                    You also might consider that you've been piecemeal transforming from a
                    project company to a product company. You just haven't made it explicit
                    yet. Your remarks on expectations of changing requirements indicate that
                    you see a market, and a need to develop expertise or a product to be able
                    to serve those new customer needs.

                    Steven mentioned Conway's Law. I think that also works in reverse. By
                    increasing the communication intensity between your developers, you can
                    expect changes in your architecture. Take care though that you don't start
                    re-writing existing systems that satisfy customer needs. That can be a
                    killer move... In the bad way.

                    Wouter




                    On Wed, Aug 21, 2013 at 11:57 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

                    > **
                    >
                    >
                    > > Right now we're
                    >
                    > > > split into teams of 1-3 members, each servicing one or more projects
                    > > > (typically, one main project and some smaller ancillary projects for
                    > each
                    > > > team).
                    > > >
                    > > >
                    > > Are you familiar with Conway's Law?
                    > >
                    >
                    > Yes, came across it in some of my research on this topic earlier this week.
                    > Definitely resonated with me.
                    >
                    >
                    > > I'm not sure that this is actually a problem.
                    > >
                    >
                    > [snip]
                    >
                    >
                    > > Individual/small-team biases are worth listening to, because those are
                    > the
                    > > folks who have to live in the context in which the problem came up.
                    > >
                    >
                    > These two comments made me realize that I didn't actually explain clearly
                    > why the current setup is problematic.
                    >
                    > If I may ask for your patience, I'll give the necessary background context.
                    > We're a large research center and we work on a set of federally-funded
                    > research projects. As an IT group, our customers are the groups of
                    > researchers who work on these grants. Our current setup is that we have a
                    > team for each of our large grants. (This is a slight simplification but
                    > it's close enough to be valid for the purposes here). Twenty or so years
                    > ago, we got our first grant, and a dev group was formed to work on the
                    > project, which was/is essentially to build a large data processing and
                    > dissemination system. That project continues today. About fifteen years
                    > ago, a second grant came along, and a second team was born to do much the
                    > same thing but for a different set of data and customers. About a decade
                    > ago we got a third major grant. A third team was born. The project was
                    > dealing with a different kind of data and a different customer yet again,
                    > but from the dev side, we were building another large data processing and
                    > dissemination system. That project also is still active. Last year, a
                    > fourth grant was won. A fourth team was born, to build (you guessed it) a
                    > large data processing and dissemination system (again, different data
                    > domain and different customers).
                    >
                    > We now have four teams building or maintaining large data processing and
                    > dissemination systems in various stages of their lifecycles. Because they
                    > were all born many years apart, they are pretty divergent in terms of their
                    > design approach to solving the core components of such systems. The second
                    > team figured out how to do some things better than the first (after making
                    > many of the same mistakes the first team did), and the third team did
                    > better than the second (after again making the same mistakes), and the
                    > fourth is now in the process of doing that as well. But as the newer teams
                    > come up with better approaches, these are not "backported" into the
                    > existing projects. So, we're not scaling well, and we're maintaining a lot
                    > of code for which we've since learned better ways to achieve the same
                    > thing.
                    >
                    > Pros
                    > - From each individual project's point of view, things look great - they
                    > have a dedicated dev team taking care of them.
                    > - Those devs have deep expertise on their projects.
                    >
                    > Cons
                    > - From the organization's point of view, this is very inefficient. We
                    > aren't scaling any better than linearly even though we're building systems
                    > which share a lot of characteristics with each other.
                    > - We're not using best practices everywhere, and we're not using the
                    > insights gained over time to shed some of these legacy approaches and
                    > technical debt so we can work smarter and be better positioned to take on
                    > new things.
                    > - From the dev team's point of view, it's unsatisfying to continue to
                    > support stuff for which better solutions are right in front of us.
                    > Particularly for those working on older projects, it has an effect on
                    > morale and retention.
                    > - When a project has only 1-2 people on it, it's hard to step back from
                    > daily feature development and refactor or port in some of these better
                    > solutions. When a team is 1 web guy and 1 db guy, there's no breathing
                    > space to do so.
                    > - Project funding and demand ebbs and flows, and it's often hard to keep
                    > one team busy while other teams are starved for resources. Right now, our
                    > devs are not very portable between projects because of how different the
                    > codebases are.
                    > - With 1-2 person teams, a person's departure has a big negative impact.
                    > There are many single points of failure with small teams, and a lot of
                    > knowledge walks out the door with that person.
                    > - With 1-2 person teams, a product ends up being a very strong reflection
                    > of the individual personalities - we see it in choice of technologies, code
                    > approaches, which parts of the project get the most attention based on
                    > personal interest, etc...
                    > - From my point of view (IT management), it's a whole lot of cats to
                    > herd, and it's very hard to form a team-wide practice.
                    >
                    > We're starting to feel that the current way of doing things is not
                    > sustainable. The data that comes into these projects (census, demographic,
                    > environmental, geospatial) is exploding in quantity and complexity. The
                    > core problem we need to solve is to figure out how to scale to continue to
                    > be able to process and disseminate that data. We need to find ways to
                    > simplify and converge the portfolio we're currently managing so that we
                    > have capacity and ability to meet the challenges the next 3-5 years will
                    > bring to us. There are currently four different data ingest processes,
                    > four different data storage engines, four different metadata management
                    > setups, etc...
                    >
                    > So, that's a summary of the challenge. :-) It's an organization-wide
                    > challenge, but my role as IT management is to ask whether we are the most
                    > efficient IT group we can be, and whether or not we can align the dev group
                    > in a way that helps us meet that challenge. While asking that, I started
                    > to ponder the idea of larger, Agile teams as a response.
                    >
                    >
                    > Sounds like it's worth a shot. My own personal experience suggests that
                    > > smaller teams focused on individual products sometimes get better results
                    > > sooner. However, it sounds like that isn't working for you.
                    > >
                    >
                    > Actually this is true for us. The first 1-2 years of a new project, the
                    > small focused team always seems like a great idea, people are motivated,
                    > they're pumping out new features left and right, it feels like we're making
                    > a lot of progress.... and then the project matures, and now it looks
                    > different than anything else we have, it starts to be bigger in scope than
                    > what 1-2 people can handle or some of those initial team members leave, and
                    > that's where the problems tend to become more pronounced.
                    >
                    >
                    > Again, I am not convinced, based on my own experience, that convergence of
                    > > codebases or practices are beneficial. I certainly don't believe that
                    > they
                    > > are worthy business goals in and of themselves. If each team is serving
                    > its
                    > > customers effectively those customers aren't likely to care if they are
                    > > doing a variety of things to accomplish it. It is certainly possible that
                    > > teams can learn valuable lessons from one another and the broader
                    > community
                    > > but from a business perspective it is at most a secondary goal.
                    > >
                    >
                    > This is the main reason why we haven't changed yet. We're known for
                    > providing very good service to our customers and the individual customers
                    > are largely happy with status quo. However, senior leadership senses the
                    > fundamental change in the nature/quantity/complexity/velocity of the data
                    > coming at us, combined with how research funding is changing, and knows
                    > we're not going to be able to scale to the challenge easily in our current
                    > form. We're not going to be able to simply throw a linear amount of new
                    > labor at the problem as we have been able to do to date. So we're trying
                    > to get out ahead of it. Transforming while keeping the level of customer
                    > service at the same high level will be a huge challenge, regardless of what
                    > we do.
                    >
                    > Hopefully that provides more context. Perhaps it's a bit far afield from
                    > the main purpose of this list, but it does seem like Agile development
                    > practices could provide a number of defenses and opportunities to us moving
                    > forward. It's just the part about one team, multiple projects that I'm
                    > trying to get my head around, how that would actually work in practice.
                    >
                    > Thanks,
                    > Fran
                    >
                    >
                    > [Non-text portions of this message have been removed]
                    >
                    >
                    >



                    --
                    Wouter Lagerweij | wouter@...
                    http://www.lagerweij.com | @wouterla <http://twitter.com/#!/wouterla>
                    linkedin.com/in/lagerweij | +31(0)6 28553506


                    [Non-text portions of this message have been removed]
                  • David Allsopp
                    All, This is very interesting and resonates with me, as much of my work has been from research funding. In this context, some of the industrial assumptions,
                    Message 9 of 9 , Aug 22, 2013
                    • 0 Attachment
                      All,

                      This is very interesting and resonates with me, as much of my work has been from research funding.

                      In this context, some of the industrial assumptions, like "Finishing early is good" are invalid, as projects are typically of fixed duration.
                      Research projects rarely 'finish' - they merely run out of time and money.
                      Your 'customer' is not the one paying the bills. You may not have any customer at all, in the sense of someone who can provide requirements. Value is not generally measured in terms of revenue, RoI, etc.

                      This is why I would like to see people being more explicit about their context, e.g.
                      Are you developing a product? For a mass market or a niche market?
                      Or are you developing in-house software? Is it user-centric UI stuff, or backend systems?
                      Are you doing contract development? Is it commercial or research? Is it fixed-goal or fixed-duration? Are you embedded with the customer, or at 'home base'?

                      Without this context, it is extremely difficult to understand where people are coming from, and why they hold certain views or use certain practices.

                      Regards,

                      David.



                      ________________________________
                      From: Fran Fabrizio <fran.fabrizio@...>
                      To: extremeprogramming <extremeprogramming@yahoogroups.com>
                      Sent: Wednesday, 21 August 2013, 23:09
                      Subject: Re: [XP] Agile development with many projects and one team


                      On Wed, Aug 21, 2013 at 4:47 PM, Steven Gordon <sgordonphd@...> wrote:

                      > Fran,
                      >
                      > This smells more like a business problem that a methodology or software
                      > development problem.
                      >

                      Certainly (as my last reply hopefully shows) the problem is bigger than a
                      dev problem, but I do think that the current and historical approach to the
                      dev has contributed to the problem, and hopefully can have a role in
                      addressing it moving forward.


                      > Is it really best to fragment the company's capabilities on so many
                      > parallel projects?  Wouldn't concentrating the capabilities on getting
                      > fewer projects completely done and earning revenue, and then coming back to
                      > the other projects be more profitable?
                      >

                      Yes.  This model would have a hard time flying in the corporate world, that
                      much is certain.

                      Higher ed research centers are a different universe.  :-)  Win a grant ->
                      build a team and consider each grant in isolation from one another is
                      generally actually a decent approach (with the somewhat unfortunate side
                      effect that many grants don't get renewed, so people are out of a job when
                      it ends).  In some sense, our problem is that we're actually unusually
                      successful in what we do - we have a phenomenal grant renewal rate relative
                      to peers and federal research as a whole - so the projects never die.  :-)
                      We just keep piling on new ones.


                      > Even if these projects are for clients, you will not get them done faster
                      > by spreading your capabilities so thinly.  Prioritizing gets some projects
                      > done sooner and none any later than doing them all at the same time.  Draw
                      > them out on graph paper, and you will see this is the case.  When you
                      > factor in context-switching time, even the last projects done in priority
                      > order get done earlier than when they are all worked on in parallel.
                      >

                      Interesting point. We work in terms of research grants (which are generally
                      no more granular than "we'll do X in year one, Y in year two, etc..." for a
                      typical 5 year grant.)  The challenge is that even though logically some
                      are higher priority than others, there are entire teams of researchers that
                      would be stuck waiting if we decided we weren't going to work on their
                      project for say, 6 months, and we're obligated to our funding sources to be
                      making more or less consistent progress over the life of the grant.  Once
                      the clock starts ticking on one of them, you can't really say "well, we'll
                      get to it next year" - you can hedge a little bit (try to front-load some
                      of the non-IT portions of the work, etc...) but eventually you need to get
                      working on each.  And I simplified it in my earlier post - we actually have
                      20+ grants all active right now which all include dev deliverables.


                      > I would ask how many of these projects are new development vs. mostly
                      > maintenance? This is a factor in how you might go about organizing.
                      >

                      All feature new development regularly (that's how you keep the grant
                      machine rolling - "renew our grant and we'll add features X,Y and Z over
                      the next 5 years").  The older projects also require a lot of maintenance
                      of code implemented long ago.  The main difference is that in a young
                      project, you're likely building everything up at the same time (web UI,
                      data processing pipelines, metadata tools) whereas on a more mature project
                      you may be able to be more focused on just one or two areas of the product
                      where you choose to make refinements.

                      -Fran


                      [Non-text portions of this message have been removed]



                      ------------------------------------

                      To Post a message, send it to:  extremeprogramming@...

                      To Unsubscribe, send a blank message to: extremeprogramming-unsubscribe@...

                      ad-free courtesy of objectmentor.comYahoo! Groups Links



                      [Non-text portions of this message have been removed]
                    Your message has been successfully submitted and would be delivered to recipients shortly.