Loading ...
Sorry, an error occurred while loading the content.

Re: [XP] Agile development with many projects and one team

Expand Messages
  • Chris Hulan
    Any chances of getting a computer science researcher to get a grant to research a software framework for science research? ... [Non-text portions of this
    Message 1 of 9 , Aug 22 7:10 AM
    • 0 Attachment
      Any chances of getting a computer science researcher to get a grant to
      research a software framework for science research?




      On Wed, Aug 21, 2013 at 5:57 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

      > **
      >
      >
      > > Right now we're
      >
      > > > split into teams of 1-3 members, each servicing one or more projects
      > > > (typically, one main project and some smaller ancillary projects for
      > each
      > > > team).
      > > >
      > > >
      > > Are you familiar with Conway's Law?
      > >
      >
      > Yes, came across it in some of my research on this topic earlier this week.
      > Definitely resonated with me.
      >
      >
      > > I'm not sure that this is actually a problem.
      > >
      >
      > [snip]
      >
      >
      > > Individual/small-team biases are worth listening to, because those are
      > the
      > > folks who have to live in the context in which the problem came up.
      > >
      >
      > These two comments made me realize that I didn't actually explain clearly
      > why the current setup is problematic.
      >
      > If I may ask for your patience, I'll give the necessary background context.
      > We're a large research center and we work on a set of federally-funded
      > research projects. As an IT group, our customers are the groups of
      > researchers who work on these grants. Our current setup is that we have a
      > team for each of our large grants. (This is a slight simplification but
      > it's close enough to be valid for the purposes here). Twenty or so years
      > ago, we got our first grant, and a dev group was formed to work on the
      > project, which was/is essentially to build a large data processing and
      > dissemination system. That project continues today. About fifteen years
      > ago, a second grant came along, and a second team was born to do much the
      > same thing but for a different set of data and customers. About a decade
      > ago we got a third major grant. A third team was born. The project was
      > dealing with a different kind of data and a different customer yet again,
      > but from the dev side, we were building another large data processing and
      > dissemination system. That project also is still active. Last year, a
      > fourth grant was won. A fourth team was born, to build (you guessed it) a
      > large data processing and dissemination system (again, different data
      > domain and different customers).
      >
      > We now have four teams building or maintaining large data processing and
      > dissemination systems in various stages of their lifecycles. Because they
      > were all born many years apart, they are pretty divergent in terms of their
      > design approach to solving the core components of such systems. The second
      > team figured out how to do some things better than the first (after making
      > many of the same mistakes the first team did), and the third team did
      > better than the second (after again making the same mistakes), and the
      > fourth is now in the process of doing that as well. But as the newer teams
      > come up with better approaches, these are not "backported" into the
      > existing projects. So, we're not scaling well, and we're maintaining a lot
      > of code for which we've since learned better ways to achieve the same
      > thing.
      >
      > Pros
      > - From each individual project's point of view, things look great - they
      > have a dedicated dev team taking care of them.
      > - Those devs have deep expertise on their projects.
      >
      > Cons
      > - From the organization's point of view, this is very inefficient. We
      > aren't scaling any better than linearly even though we're building systems
      > which share a lot of characteristics with each other.
      > - We're not using best practices everywhere, and we're not using the
      > insights gained over time to shed some of these legacy approaches and
      > technical debt so we can work smarter and be better positioned to take on
      > new things.
      > - From the dev team's point of view, it's unsatisfying to continue to
      > support stuff for which better solutions are right in front of us.
      > Particularly for those working on older projects, it has an effect on
      > morale and retention.
      > - When a project has only 1-2 people on it, it's hard to step back from
      > daily feature development and refactor or port in some of these better
      > solutions. When a team is 1 web guy and 1 db guy, there's no breathing
      > space to do so.
      > - Project funding and demand ebbs and flows, and it's often hard to keep
      > one team busy while other teams are starved for resources. Right now, our
      > devs are not very portable between projects because of how different the
      > codebases are.
      > - With 1-2 person teams, a person's departure has a big negative impact.
      > There are many single points of failure with small teams, and a lot of
      > knowledge walks out the door with that person.
      > - With 1-2 person teams, a product ends up being a very strong reflection
      > of the individual personalities - we see it in choice of technologies, code
      > approaches, which parts of the project get the most attention based on
      > personal interest, etc...
      > - From my point of view (IT management), it's a whole lot of cats to
      > herd, and it's very hard to form a team-wide practice.
      >
      > We're starting to feel that the current way of doing things is not
      > sustainable. The data that comes into these projects (census, demographic,
      > environmental, geospatial) is exploding in quantity and complexity. The
      > core problem we need to solve is to figure out how to scale to continue to
      > be able to process and disseminate that data. We need to find ways to
      > simplify and converge the portfolio we're currently managing so that we
      > have capacity and ability to meet the challenges the next 3-5 years will
      > bring to us. There are currently four different data ingest processes,
      > four different data storage engines, four different metadata management
      > setups, etc...
      >
      > So, that's a summary of the challenge. :-) It's an organization-wide
      > challenge, but my role as IT management is to ask whether we are the most
      > efficient IT group we can be, and whether or not we can align the dev group
      > in a way that helps us meet that challenge. While asking that, I started
      > to ponder the idea of larger, Agile teams as a response.
      >
      >
      > Sounds like it's worth a shot. My own personal experience suggests that
      > > smaller teams focused on individual products sometimes get better results
      > > sooner. However, it sounds like that isn't working for you.
      > >
      >
      > Actually this is true for us. The first 1-2 years of a new project, the
      > small focused team always seems like a great idea, people are motivated,
      > they're pumping out new features left and right, it feels like we're making
      > a lot of progress.... and then the project matures, and now it looks
      > different than anything else we have, it starts to be bigger in scope than
      > what 1-2 people can handle or some of those initial team members leave, and
      > that's where the problems tend to become more pronounced.
      >
      >
      > Again, I am not convinced, based on my own experience, that convergence of
      > > codebases or practices are beneficial. I certainly don't believe that
      > they
      > > are worthy business goals in and of themselves. If each team is serving
      > its
      > > customers effectively those customers aren't likely to care if they are
      > > doing a variety of things to accomplish it. It is certainly possible that
      > > teams can learn valuable lessons from one another and the broader
      > community
      > > but from a business perspective it is at most a secondary goal.
      > >
      >
      > This is the main reason why we haven't changed yet. We're known for
      > providing very good service to our customers and the individual customers
      > are largely happy with status quo. However, senior leadership senses the
      > fundamental change in the nature/quantity/complexity/velocity of the data
      > coming at us, combined with how research funding is changing, and knows
      > we're not going to be able to scale to the challenge easily in our current
      > form. We're not going to be able to simply throw a linear amount of new
      > labor at the problem as we have been able to do to date. So we're trying
      > to get out ahead of it. Transforming while keeping the level of customer
      > service at the same high level will be a huge challenge, regardless of what
      > we do.
      >
      > Hopefully that provides more context. Perhaps it's a bit far afield from
      > the main purpose of this list, but it does seem like Agile development
      > practices could provide a number of defenses and opportunities to us moving
      > forward. It's just the part about one team, multiple projects that I'm
      > trying to get my head around, how that would actually work in practice.
      >
      > Thanks,
      > Fran
      >
      >
      > [Non-text portions of this message have been removed]
      >
      >
      >


      [Non-text portions of this message have been removed]
    • Wouter Lagerweij
      Hi Fran, We like to say that a framework is something extracted after you ve build the same type of system a few times. This seems to have some relevance to
      Message 2 of 9 , Aug 22 1:40 PM
      • 0 Attachment
        Hi Fran,

        We like to say that a framework is something extracted after you've build
        the same type of system a few times. This seems to have some relevance to
        your situation.
        You also might consider that you've been piecemeal transforming from a
        project company to a product company. You just haven't made it explicit
        yet. Your remarks on expectations of changing requirements indicate that
        you see a market, and a need to develop expertise or a product to be able
        to serve those new customer needs.

        Steven mentioned Conway's Law. I think that also works in reverse. By
        increasing the communication intensity between your developers, you can
        expect changes in your architecture. Take care though that you don't start
        re-writing existing systems that satisfy customer needs. That can be a
        killer move... In the bad way.

        Wouter




        On Wed, Aug 21, 2013 at 11:57 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

        > **
        >
        >
        > > Right now we're
        >
        > > > split into teams of 1-3 members, each servicing one or more projects
        > > > (typically, one main project and some smaller ancillary projects for
        > each
        > > > team).
        > > >
        > > >
        > > Are you familiar with Conway's Law?
        > >
        >
        > Yes, came across it in some of my research on this topic earlier this week.
        > Definitely resonated with me.
        >
        >
        > > I'm not sure that this is actually a problem.
        > >
        >
        > [snip]
        >
        >
        > > Individual/small-team biases are worth listening to, because those are
        > the
        > > folks who have to live in the context in which the problem came up.
        > >
        >
        > These two comments made me realize that I didn't actually explain clearly
        > why the current setup is problematic.
        >
        > If I may ask for your patience, I'll give the necessary background context.
        > We're a large research center and we work on a set of federally-funded
        > research projects. As an IT group, our customers are the groups of
        > researchers who work on these grants. Our current setup is that we have a
        > team for each of our large grants. (This is a slight simplification but
        > it's close enough to be valid for the purposes here). Twenty or so years
        > ago, we got our first grant, and a dev group was formed to work on the
        > project, which was/is essentially to build a large data processing and
        > dissemination system. That project continues today. About fifteen years
        > ago, a second grant came along, and a second team was born to do much the
        > same thing but for a different set of data and customers. About a decade
        > ago we got a third major grant. A third team was born. The project was
        > dealing with a different kind of data and a different customer yet again,
        > but from the dev side, we were building another large data processing and
        > dissemination system. That project also is still active. Last year, a
        > fourth grant was won. A fourth team was born, to build (you guessed it) a
        > large data processing and dissemination system (again, different data
        > domain and different customers).
        >
        > We now have four teams building or maintaining large data processing and
        > dissemination systems in various stages of their lifecycles. Because they
        > were all born many years apart, they are pretty divergent in terms of their
        > design approach to solving the core components of such systems. The second
        > team figured out how to do some things better than the first (after making
        > many of the same mistakes the first team did), and the third team did
        > better than the second (after again making the same mistakes), and the
        > fourth is now in the process of doing that as well. But as the newer teams
        > come up with better approaches, these are not "backported" into the
        > existing projects. So, we're not scaling well, and we're maintaining a lot
        > of code for which we've since learned better ways to achieve the same
        > thing.
        >
        > Pros
        > - From each individual project's point of view, things look great - they
        > have a dedicated dev team taking care of them.
        > - Those devs have deep expertise on their projects.
        >
        > Cons
        > - From the organization's point of view, this is very inefficient. We
        > aren't scaling any better than linearly even though we're building systems
        > which share a lot of characteristics with each other.
        > - We're not using best practices everywhere, and we're not using the
        > insights gained over time to shed some of these legacy approaches and
        > technical debt so we can work smarter and be better positioned to take on
        > new things.
        > - From the dev team's point of view, it's unsatisfying to continue to
        > support stuff for which better solutions are right in front of us.
        > Particularly for those working on older projects, it has an effect on
        > morale and retention.
        > - When a project has only 1-2 people on it, it's hard to step back from
        > daily feature development and refactor or port in some of these better
        > solutions. When a team is 1 web guy and 1 db guy, there's no breathing
        > space to do so.
        > - Project funding and demand ebbs and flows, and it's often hard to keep
        > one team busy while other teams are starved for resources. Right now, our
        > devs are not very portable between projects because of how different the
        > codebases are.
        > - With 1-2 person teams, a person's departure has a big negative impact.
        > There are many single points of failure with small teams, and a lot of
        > knowledge walks out the door with that person.
        > - With 1-2 person teams, a product ends up being a very strong reflection
        > of the individual personalities - we see it in choice of technologies, code
        > approaches, which parts of the project get the most attention based on
        > personal interest, etc...
        > - From my point of view (IT management), it's a whole lot of cats to
        > herd, and it's very hard to form a team-wide practice.
        >
        > We're starting to feel that the current way of doing things is not
        > sustainable. The data that comes into these projects (census, demographic,
        > environmental, geospatial) is exploding in quantity and complexity. The
        > core problem we need to solve is to figure out how to scale to continue to
        > be able to process and disseminate that data. We need to find ways to
        > simplify and converge the portfolio we're currently managing so that we
        > have capacity and ability to meet the challenges the next 3-5 years will
        > bring to us. There are currently four different data ingest processes,
        > four different data storage engines, four different metadata management
        > setups, etc...
        >
        > So, that's a summary of the challenge. :-) It's an organization-wide
        > challenge, but my role as IT management is to ask whether we are the most
        > efficient IT group we can be, and whether or not we can align the dev group
        > in a way that helps us meet that challenge. While asking that, I started
        > to ponder the idea of larger, Agile teams as a response.
        >
        >
        > Sounds like it's worth a shot. My own personal experience suggests that
        > > smaller teams focused on individual products sometimes get better results
        > > sooner. However, it sounds like that isn't working for you.
        > >
        >
        > Actually this is true for us. The first 1-2 years of a new project, the
        > small focused team always seems like a great idea, people are motivated,
        > they're pumping out new features left and right, it feels like we're making
        > a lot of progress.... and then the project matures, and now it looks
        > different than anything else we have, it starts to be bigger in scope than
        > what 1-2 people can handle or some of those initial team members leave, and
        > that's where the problems tend to become more pronounced.
        >
        >
        > Again, I am not convinced, based on my own experience, that convergence of
        > > codebases or practices are beneficial. I certainly don't believe that
        > they
        > > are worthy business goals in and of themselves. If each team is serving
        > its
        > > customers effectively those customers aren't likely to care if they are
        > > doing a variety of things to accomplish it. It is certainly possible that
        > > teams can learn valuable lessons from one another and the broader
        > community
        > > but from a business perspective it is at most a secondary goal.
        > >
        >
        > This is the main reason why we haven't changed yet. We're known for
        > providing very good service to our customers and the individual customers
        > are largely happy with status quo. However, senior leadership senses the
        > fundamental change in the nature/quantity/complexity/velocity of the data
        > coming at us, combined with how research funding is changing, and knows
        > we're not going to be able to scale to the challenge easily in our current
        > form. We're not going to be able to simply throw a linear amount of new
        > labor at the problem as we have been able to do to date. So we're trying
        > to get out ahead of it. Transforming while keeping the level of customer
        > service at the same high level will be a huge challenge, regardless of what
        > we do.
        >
        > Hopefully that provides more context. Perhaps it's a bit far afield from
        > the main purpose of this list, but it does seem like Agile development
        > practices could provide a number of defenses and opportunities to us moving
        > forward. It's just the part about one team, multiple projects that I'm
        > trying to get my head around, how that would actually work in practice.
        >
        > Thanks,
        > Fran
        >
        >
        > [Non-text portions of this message have been removed]
        >
        >
        >



        --
        Wouter Lagerweij | wouter@...
        http://www.lagerweij.com | @wouterla <http://twitter.com/#!/wouterla>
        linkedin.com/in/lagerweij | +31(0)6 28553506


        [Non-text portions of this message have been removed]
      • David Allsopp
        All, This is very interesting and resonates with me, as much of my work has been from research funding. In this context, some of the industrial assumptions,
        Message 3 of 9 , Aug 22 3:13 PM
        • 0 Attachment
          All,

          This is very interesting and resonates with me, as much of my work has been from research funding.

          In this context, some of the industrial assumptions, like "Finishing early is good" are invalid, as projects are typically of fixed duration.
          Research projects rarely 'finish' - they merely run out of time and money.
          Your 'customer' is not the one paying the bills. You may not have any customer at all, in the sense of someone who can provide requirements. Value is not generally measured in terms of revenue, RoI, etc.

          This is why I would like to see people being more explicit about their context, e.g.
          Are you developing a product? For a mass market or a niche market?
          Or are you developing in-house software? Is it user-centric UI stuff, or backend systems?
          Are you doing contract development? Is it commercial or research? Is it fixed-goal or fixed-duration? Are you embedded with the customer, or at 'home base'?

          Without this context, it is extremely difficult to understand where people are coming from, and why they hold certain views or use certain practices.

          Regards,

          David.



          ________________________________
          From: Fran Fabrizio <fran.fabrizio@...>
          To: extremeprogramming <extremeprogramming@yahoogroups.com>
          Sent: Wednesday, 21 August 2013, 23:09
          Subject: Re: [XP] Agile development with many projects and one team


          On Wed, Aug 21, 2013 at 4:47 PM, Steven Gordon <sgordonphd@...> wrote:

          > Fran,
          >
          > This smells more like a business problem that a methodology or software
          > development problem.
          >

          Certainly (as my last reply hopefully shows) the problem is bigger than a
          dev problem, but I do think that the current and historical approach to the
          dev has contributed to the problem, and hopefully can have a role in
          addressing it moving forward.


          > Is it really best to fragment the company's capabilities on so many
          > parallel projects?  Wouldn't concentrating the capabilities on getting
          > fewer projects completely done and earning revenue, and then coming back to
          > the other projects be more profitable?
          >

          Yes.  This model would have a hard time flying in the corporate world, that
          much is certain.

          Higher ed research centers are a different universe.  :-)  Win a grant ->
          build a team and consider each grant in isolation from one another is
          generally actually a decent approach (with the somewhat unfortunate side
          effect that many grants don't get renewed, so people are out of a job when
          it ends).  In some sense, our problem is that we're actually unusually
          successful in what we do - we have a phenomenal grant renewal rate relative
          to peers and federal research as a whole - so the projects never die.  :-)
          We just keep piling on new ones.


          > Even if these projects are for clients, you will not get them done faster
          > by spreading your capabilities so thinly.  Prioritizing gets some projects
          > done sooner and none any later than doing them all at the same time.  Draw
          > them out on graph paper, and you will see this is the case.  When you
          > factor in context-switching time, even the last projects done in priority
          > order get done earlier than when they are all worked on in parallel.
          >

          Interesting point. We work in terms of research grants (which are generally
          no more granular than "we'll do X in year one, Y in year two, etc..." for a
          typical 5 year grant.)  The challenge is that even though logically some
          are higher priority than others, there are entire teams of researchers that
          would be stuck waiting if we decided we weren't going to work on their
          project for say, 6 months, and we're obligated to our funding sources to be
          making more or less consistent progress over the life of the grant.  Once
          the clock starts ticking on one of them, you can't really say "well, we'll
          get to it next year" - you can hedge a little bit (try to front-load some
          of the non-IT portions of the work, etc...) but eventually you need to get
          working on each.  And I simplified it in my earlier post - we actually have
          20+ grants all active right now which all include dev deliverables.


          > I would ask how many of these projects are new development vs. mostly
          > maintenance? This is a factor in how you might go about organizing.
          >

          All feature new development regularly (that's how you keep the grant
          machine rolling - "renew our grant and we'll add features X,Y and Z over
          the next 5 years").  The older projects also require a lot of maintenance
          of code implemented long ago.  The main difference is that in a young
          project, you're likely building everything up at the same time (web UI,
          data processing pipelines, metadata tools) whereas on a more mature project
          you may be able to be more focused on just one or two areas of the product
          where you choose to make refinements.

          -Fran


          [Non-text portions of this message have been removed]



          ------------------------------------

          To Post a message, send it to:  extremeprogramming@...

          To Unsubscribe, send a blank message to: extremeprogramming-unsubscribe@...

          ad-free courtesy of objectmentor.comYahoo! Groups Links



          [Non-text portions of this message have been removed]
        Your message has been successfully submitted and would be delivered to recipients shortly.