Loading ...
Sorry, an error occurred while loading the content.

158760Re: [XP] Agile development with many projects and one team

Expand Messages
  • Chris Hulan
    Aug 22, 2013
      Any chances of getting a computer science researcher to get a grant to
      research a software framework for science research?




      On Wed, Aug 21, 2013 at 5:57 PM, Fran Fabrizio <fran.fabrizio@...>wrote:

      > **
      >
      >
      > > Right now we're
      >
      > > > split into teams of 1-3 members, each servicing one or more projects
      > > > (typically, one main project and some smaller ancillary projects for
      > each
      > > > team).
      > > >
      > > >
      > > Are you familiar with Conway's Law?
      > >
      >
      > Yes, came across it in some of my research on this topic earlier this week.
      > Definitely resonated with me.
      >
      >
      > > I'm not sure that this is actually a problem.
      > >
      >
      > [snip]
      >
      >
      > > Individual/small-team biases are worth listening to, because those are
      > the
      > > folks who have to live in the context in which the problem came up.
      > >
      >
      > These two comments made me realize that I didn't actually explain clearly
      > why the current setup is problematic.
      >
      > If I may ask for your patience, I'll give the necessary background context.
      > We're a large research center and we work on a set of federally-funded
      > research projects. As an IT group, our customers are the groups of
      > researchers who work on these grants. Our current setup is that we have a
      > team for each of our large grants. (This is a slight simplification but
      > it's close enough to be valid for the purposes here). Twenty or so years
      > ago, we got our first grant, and a dev group was formed to work on the
      > project, which was/is essentially to build a large data processing and
      > dissemination system. That project continues today. About fifteen years
      > ago, a second grant came along, and a second team was born to do much the
      > same thing but for a different set of data and customers. About a decade
      > ago we got a third major grant. A third team was born. The project was
      > dealing with a different kind of data and a different customer yet again,
      > but from the dev side, we were building another large data processing and
      > dissemination system. That project also is still active. Last year, a
      > fourth grant was won. A fourth team was born, to build (you guessed it) a
      > large data processing and dissemination system (again, different data
      > domain and different customers).
      >
      > We now have four teams building or maintaining large data processing and
      > dissemination systems in various stages of their lifecycles. Because they
      > were all born many years apart, they are pretty divergent in terms of their
      > design approach to solving the core components of such systems. The second
      > team figured out how to do some things better than the first (after making
      > many of the same mistakes the first team did), and the third team did
      > better than the second (after again making the same mistakes), and the
      > fourth is now in the process of doing that as well. But as the newer teams
      > come up with better approaches, these are not "backported" into the
      > existing projects. So, we're not scaling well, and we're maintaining a lot
      > of code for which we've since learned better ways to achieve the same
      > thing.
      >
      > Pros
      > - From each individual project's point of view, things look great - they
      > have a dedicated dev team taking care of them.
      > - Those devs have deep expertise on their projects.
      >
      > Cons
      > - From the organization's point of view, this is very inefficient. We
      > aren't scaling any better than linearly even though we're building systems
      > which share a lot of characteristics with each other.
      > - We're not using best practices everywhere, and we're not using the
      > insights gained over time to shed some of these legacy approaches and
      > technical debt so we can work smarter and be better positioned to take on
      > new things.
      > - From the dev team's point of view, it's unsatisfying to continue to
      > support stuff for which better solutions are right in front of us.
      > Particularly for those working on older projects, it has an effect on
      > morale and retention.
      > - When a project has only 1-2 people on it, it's hard to step back from
      > daily feature development and refactor or port in some of these better
      > solutions. When a team is 1 web guy and 1 db guy, there's no breathing
      > space to do so.
      > - Project funding and demand ebbs and flows, and it's often hard to keep
      > one team busy while other teams are starved for resources. Right now, our
      > devs are not very portable between projects because of how different the
      > codebases are.
      > - With 1-2 person teams, a person's departure has a big negative impact.
      > There are many single points of failure with small teams, and a lot of
      > knowledge walks out the door with that person.
      > - With 1-2 person teams, a product ends up being a very strong reflection
      > of the individual personalities - we see it in choice of technologies, code
      > approaches, which parts of the project get the most attention based on
      > personal interest, etc...
      > - From my point of view (IT management), it's a whole lot of cats to
      > herd, and it's very hard to form a team-wide practice.
      >
      > We're starting to feel that the current way of doing things is not
      > sustainable. The data that comes into these projects (census, demographic,
      > environmental, geospatial) is exploding in quantity and complexity. The
      > core problem we need to solve is to figure out how to scale to continue to
      > be able to process and disseminate that data. We need to find ways to
      > simplify and converge the portfolio we're currently managing so that we
      > have capacity and ability to meet the challenges the next 3-5 years will
      > bring to us. There are currently four different data ingest processes,
      > four different data storage engines, four different metadata management
      > setups, etc...
      >
      > So, that's a summary of the challenge. :-) It's an organization-wide
      > challenge, but my role as IT management is to ask whether we are the most
      > efficient IT group we can be, and whether or not we can align the dev group
      > in a way that helps us meet that challenge. While asking that, I started
      > to ponder the idea of larger, Agile teams as a response.
      >
      >
      > Sounds like it's worth a shot. My own personal experience suggests that
      > > smaller teams focused on individual products sometimes get better results
      > > sooner. However, it sounds like that isn't working for you.
      > >
      >
      > Actually this is true for us. The first 1-2 years of a new project, the
      > small focused team always seems like a great idea, people are motivated,
      > they're pumping out new features left and right, it feels like we're making
      > a lot of progress.... and then the project matures, and now it looks
      > different than anything else we have, it starts to be bigger in scope than
      > what 1-2 people can handle or some of those initial team members leave, and
      > that's where the problems tend to become more pronounced.
      >
      >
      > Again, I am not convinced, based on my own experience, that convergence of
      > > codebases or practices are beneficial. I certainly don't believe that
      > they
      > > are worthy business goals in and of themselves. If each team is serving
      > its
      > > customers effectively those customers aren't likely to care if they are
      > > doing a variety of things to accomplish it. It is certainly possible that
      > > teams can learn valuable lessons from one another and the broader
      > community
      > > but from a business perspective it is at most a secondary goal.
      > >
      >
      > This is the main reason why we haven't changed yet. We're known for
      > providing very good service to our customers and the individual customers
      > are largely happy with status quo. However, senior leadership senses the
      > fundamental change in the nature/quantity/complexity/velocity of the data
      > coming at us, combined with how research funding is changing, and knows
      > we're not going to be able to scale to the challenge easily in our current
      > form. We're not going to be able to simply throw a linear amount of new
      > labor at the problem as we have been able to do to date. So we're trying
      > to get out ahead of it. Transforming while keeping the level of customer
      > service at the same high level will be a huge challenge, regardless of what
      > we do.
      >
      > Hopefully that provides more context. Perhaps it's a bit far afield from
      > the main purpose of this list, but it does seem like Agile development
      > practices could provide a number of defenses and opportunities to us moving
      > forward. It's just the part about one team, multiple projects that I'm
      > trying to get my head around, how that would actually work in practice.
      >
      > Thanks,
      > Fran
      >
      >
      > [Non-text portions of this message have been removed]
      >
      >
      >


      [Non-text portions of this message have been removed]
    • Show all 9 messages in this topic