Loading ...
Sorry, an error occurred while loading the content.

Re: Architecture for XP and scalable web sites

Expand Messages
  • Chad Fowler
    ... XP ... automate ... XP ... be ... I think the point of a lot of these messages is that EJB is not, at least at the beginning of a project, (in most cases)
    Message 1 of 22 , Jan 1, 2001
    • 0 Attachment
      --- In extremeprogramming@egroups.com, "Keith Richardson"
      <keith@s...> wrote:
      > Hello,
      > This forum has included several messages describing problems using
      XP
      > with EJB, describing EJB as non-OO and other difficulties with the
      > EJB architecture. A highly scalable web site must be able to
      automate
      > persistance of session and environment data, allow deployment to be
      > easily adjusted and many other features that EJBs (and more
      > completely J2EE) provides. Are there other environments that have
      > proven to be better for developing highly scalable solutions with
      XP
      > or should I interpret these messages as saying that XP is not
      > applicable for these needs? Any XP success stories in this area to
      be
      > shared?
      > Keith Richardson

      I think the point of a lot of these messages is that EJB is not, at
      least at the beginning of a project, (in most cases) the simplest
      thing that could possibly work. A well factored servlet application
      provides session persistence (not sure what you mean by "environment"
      in this context) and (being well factored) would also give
      flexibility for deployment options (things are logically decoupled,
      so they can be physically separated down the road).

      In the past, our developers were assuming that scalable-web-
      application == J2EE-all-the-way and just going with EJB from the
      start. This instroduced complexity in the code, the
      development/build process, and the application server deployment.
      This complexity invariably lead to various problems (more points of
      potential failure).

      The approach we've taken semi-recently with our developers is to
      say, "You are free to use EJB (and the rest of the J2EE baggage) if
      you can explain why you need it." What we've found so far is that no
      one has ever needed it, and we haven't had any related problems.
      (And, our applications are *so* much easier to deploy and maintain
      now!) In the mean time, we've been trying to keep things well
      factored, so they could be easily moved to the more complex solution
      should we ever hit the scalability barriers of this approach. My
      guess is that we won't ever get there.

      Of course, we're no amazon.com, but we get some pretty heavy traffic.

      Just as an addendum, here's my XP zealot answer:
      "must be able to automate persistance of session and environment
      data, allow deployment to be easily adjusted and many other features
      that EJBs (and more completely J2EE) provides" wasn't in my user
      stories. :)

      Chad
    • Paul Hodgetts
      ... Would you define what you mean by highly scalable? How often do you need to adjust the scale that the site must support? What are the increments of
      Message 2 of 22 , Jan 1, 2001
      • 0 Attachment
        Keith Richardson wrote:

        > A highly scalable web site

        Would you define what you mean by highly scalable? How often do
        you need to adjust the "scale" that the site must support? What
        are the increments of scale (how many scale targets)?

        My experience is that a site typically only goes through a few
        scale adjustments in its life cycle. It may start out with an
        initial deployment that supports a few thousand users, then as
        success warrants it gets bumped up to support some large target
        that marketing sets and probably never gets reached. If you're
        really successful, you may bump it once more to support a huge
        amount of users. The need to increase the scale is usually
        known before the system is maxed out, if we're paying attention
        to the usage stats and trends and doing some load testing.

        My point is that the ability to scale is important, but that
        the scale adjustments don't happen suddenly or very often. So
        if we're carrying along a lot of baggage to allow quick and
        frequent scale adjustments, it may not be worth the continuous
        cost of that baggage for the few times we bump the scale.

        > must be able to automate persistance of session and environment
        > data,

        How much automation do you want?

        There's that cost thing again. There's often more complexity
        with the "automated" persistent mechanisms, and I think many of
        them have additional build and deploy cycles that can limit our
        ability to rapidly and continuously integrate.

        > allow deployment to be easily adjusted and many other features
        > that EJBs (and more completely J2EE) provides.

        Similar to scaling, deployment adjustments usually won't happen
        all that often. In most sites, the deployment needs aren't all
        that complex and most everything gets deployed as just one or
        maybe a few sets of components. It's only when we get to much
        larger sites with a wider range of deployed components that the
        deployments get complicated. When the site's in development,
        there's probably more thrashing with moving things around. This
        stabilizes quite a bit as time goes on. Again, is it worth a lot
        of day-to-day baggage for the few times that components get moved
        around in a deployment?

        > Are there other environments that have proven to be better for
        > developing highly scalable solutions with XP or should I
        > interpret these messages as saying that XP is not applicable for
        > these needs? Any XP success stories in this area to be shared?

        I don't think any of the distributed environments (J2EE, MTS,
        CORBA) are incompatible with XP per se. XP would ask us to
        implement the simplest solution to meet the current requirements,
        and to evolve that solution as the requirements change. I'm not
        sure incrementally working from the initial requirements to an
        eventual solution would necessarily lead us to choose these
        technologies, or if we'd end up with something else entirely.

        So far our team is learning that it's best to use good design
        principles that reduce the coupling and allow the specific
        implementations of the various technologies to be changed without
        thrashing the whole system. Then you've got flexibility and you
        don't need to commit the project to a specific technology early.

        Most of these technologies have a lot of overhead in terms of
        building, deploying, and starting/stopping the environments. This
        cuts into the speed at which we can integrate new code. IMHO,
        it's extremely important to be able to test the core business
        classes outside the environment, and only pull the environment in
        when needed for more comprehensive testing. Good decoupled
        designs that are kept that way through continuous refactoring help.

        We made the mistake of coupling our design too closely to the
        technologies and now we have to fix that. We also relied a lot
        on EJB entity beans, and they have not lived up to our needs
        (performance problems, added complexity, inflexibility of the
        persistent object model). Our primary symptoms are that we
        can't implement adequate unit testing, and our code/build/test
        cycles are too long for continuous integration.

        I'm starting to be convinced that using XP principles from the
        start on a web site development project would probably lead to
        a surprisingly simple solution. Maybe just some front end
        servlets or JSPs, perhaps using XML and XSLT, accessing the
        persistent data via a simple mapping layer. If I had a more
        open mind, maybe I'd end up with a non-Java based solution. ;-)

        I see no reason a simple solution like this can't be scalable,
        deployable, and all the other -ilities you want from a site.
        It doesn't sound as cool as a grand J2EE implementation, but it
        might get a whole lot of business value implemented a whole lot
        faster.

        -Paul Hodgetts
      • John Brewer
        ... Don t believe all of Sun s hype. EJBs aren t done yet. In particular, the current state of entity beans leaves lots to be desired. I ve heard of at least
        Message 3 of 22 , Jan 1, 2001
        • 0 Attachment
          --- In extremeprogramming@egroups.com, "Keith Richardson" <keith@s...>
          wrote:
          > A highly scalable web site must be able to automate
          > persistance of session and environment data, allow deployment to be
          > easily adjusted and many other features that EJBs (and more
          > completely J2EE) provides.

          Don't believe all of Sun's hype. EJBs aren't done yet. In
          particular, the current state of entity beans leaves lots to be
          desired.

          I've heard of at least one XP project that ended up removing the
          entity beans from their system because of performance problems.
          Fortunately, they had the well-factored code and corresponding tests
          to make that possible. I believe they eventually eliminated all the
          fancy app-server stuff from their code, allowing them to deploy using
          an inexpensive servlet container.

          John Brewer
          Jera Design
        • Keith Richardson
          ... ... be ... tests ... the ... using ... I have also heard of some projects having these problems. I would love to hear about architectures that
          Message 4 of 22 , Jan 1, 2001
          • 0 Attachment
            --- In extremeprogramming@egroups.com, "John Brewer" <jbrewer@j...>
            wrote:
            > --- In extremeprogramming@egroups.com, "Keith Richardson"
            <keith@s...>
            > wrote:
            > > A highly scalable web site must be able to automate
            > > persistance of session and environment data, allow deployment to
            be
            > > easily adjusted and many other features that EJBs (and more
            > > completely J2EE) provides.
            >
            > Don't believe all of Sun's hype. EJBs aren't done yet. In
            > particular, the current state of entity beans leaves lots to be
            > desired.
            >
            > I've heard of at least one XP project that ended up removing the
            > entity beans from their system because of performance problems.
            > Fortunately, they had the well-factored code and corresponding
            tests
            > to make that possible. I believe they eventually eliminated all
            the
            > fancy app-server stuff from their code, allowing them to deploy
            using
            > an inexpensive servlet container.
            >

            I have also heard of some projects having these problems. I would
            love to hear about architectures that have worked. I will be
            developing Web site frameworks that will be deployed by our clients
            so the solution must scale to fit a wide range of clients. I have
            heard many success stories of J2EE/WebLogic scaling exceptionally
            well and any alternative architecture must be just as good.
            Suggestions?
            Keith Richardson
          • Keith Richardson
            We are developing a Web application framework for a specialized vertical market that will be deployed in our client s environments. The solution must scale to
            Message 5 of 22 , Jan 1, 2001
            • 0 Attachment
              We are developing a Web application framework for a specialized
              vertical market that will be deployed in our client's environments.
              The solution must scale to fit a wide range of clients. We can not
              accurately predict the way future clients will use the solution
              (think "shrink wrapped software"). The most important customers are
              the biggest ones and we can not afford to find that the solution does
              not scale to meet the requirements when we get interest from a new
              major client. i.e. We will get a sudden major jump in performance
              requirements is a new customer comes along who pushes the limits of
              what we are doing.

              For these reasons I consider scalability to be a fundamental
              requirement. We are not in a position to develop to meet just the
              performance needs in front of us right now so I beleive we must
              select technologies to give us very flexible deployment. Before
              considering XP I had selected J2EE/Weblogic as a good solution. Other
              related complanies in our field are using the same environment which
              is a double plus - the technology is proven and we have the
              opportunity to share code in the future. (i.e. they could become
              customers by deploying components of our software.)

              Automated persistence is a plus because we have no way of predicting
              what state the users will be in when the system gets an unexpected
              burst of heavy use. Without automated persistence, we would have to
              do all the coding to make decisions on which session components get
              swapped out when things get busy. It is easy building a Web site but
              not easy building one to handle bursts of use that are orders of
              magnitude higher than average. Everything else being equal, I would
              prefer the deployment architecture to take care of this. This again
              leads to the J2EE/Weblogic option.

              Now I am trying to decide how XP fits in this environment of whether
              an alternative should be considered to better fit XP. I also came to
              your conclusion that XP should lead to simple solutions. There seems
              no reason why J2EE can't be kept simple. Or is there?

              Keith Richardson

              --- In extremeprogramming@egroups.com, Paul Hodgetts <prh@z...> wrote:
              > Keith Richardson wrote:
              >
              > > A highly scalable web site
              >
              > Would you define what you mean by highly scalable? How often do
              > you need to adjust the "scale" that the site must support? What
              > are the increments of scale (how many scale targets)?
              >
              > My experience is that a site typically only goes through a few
              > scale adjustments in its life cycle. It may start out with an
              > initial deployment that supports a few thousand users, then as
              > success warrants it gets bumped up to support some large target
              > that marketing sets and probably never gets reached. If you're
              > really successful, you may bump it once more to support a huge
              > amount of users. The need to increase the scale is usually
              > known before the system is maxed out, if we're paying attention
              > to the usage stats and trends and doing some load testing.
              >
              > My point is that the ability to scale is important, but that
              > the scale adjustments don't happen suddenly or very often. So
              > if we're carrying along a lot of baggage to allow quick and
              > frequent scale adjustments, it may not be worth the continuous
              > cost of that baggage for the few times we bump the scale.
              >
              > > must be able to automate persistance of session and environment
              > > data,
              >
              > How much automation do you want?
              >
              > There's that cost thing again. There's often more complexity
              > with the "automated" persistent mechanisms, and I think many of
              > them have additional build and deploy cycles that can limit our
              > ability to rapidly and continuously integrate.
              >
              > > allow deployment to be easily adjusted and many other features
              > > that EJBs (and more completely J2EE) provides.
              >
              > Similar to scaling, deployment adjustments usually won't happen
              > all that often. In most sites, the deployment needs aren't all
              > that complex and most everything gets deployed as just one or
              > maybe a few sets of components. It's only when we get to much
              > larger sites with a wider range of deployed components that the
              > deployments get complicated. When the site's in development,
              > there's probably more thrashing with moving things around. This
              > stabilizes quite a bit as time goes on. Again, is it worth a lot
              > of day-to-day baggage for the few times that components get moved
              > around in a deployment?
              >
              > > Are there other environments that have proven to be better for
              > > developing highly scalable solutions with XP or should I
              > > interpret these messages as saying that XP is not applicable for
              > > these needs? Any XP success stories in this area to be shared?
              >
              > I don't think any of the distributed environments (J2EE, MTS,
              > CORBA) are incompatible with XP per se. XP would ask us to
              > implement the simplest solution to meet the current requirements,
              > and to evolve that solution as the requirements change. I'm not
              > sure incrementally working from the initial requirements to an
              > eventual solution would necessarily lead us to choose these
              > technologies, or if we'd end up with something else entirely.
              >
              > So far our team is learning that it's best to use good design
              > principles that reduce the coupling and allow the specific
              > implementations of the various technologies to be changed without
              > thrashing the whole system. Then you've got flexibility and you
              > don't need to commit the project to a specific technology early.
              >
              > Most of these technologies have a lot of overhead in terms of
              > building, deploying, and starting/stopping the environments. This
              > cuts into the speed at which we can integrate new code. IMHO,
              > it's extremely important to be able to test the core business
              > classes outside the environment, and only pull the environment in
              > when needed for more comprehensive testing. Good decoupled
              > designs that are kept that way through continuous refactoring help.
              >
              > We made the mistake of coupling our design too closely to the
              > technologies and now we have to fix that. We also relied a lot
              > on EJB entity beans, and they have not lived up to our needs
              > (performance problems, added complexity, inflexibility of the
              > persistent object model). Our primary symptoms are that we
              > can't implement adequate unit testing, and our code/build/test
              > cycles are too long for continuous integration.
              >
              > I'm starting to be convinced that using XP principles from the
              > start on a web site development project would probably lead to
              > a surprisingly simple solution. Maybe just some front end
              > servlets or JSPs, perhaps using XML and XSLT, accessing the
              > persistent data via a simple mapping layer. If I had a more
              > open mind, maybe I'd end up with a non-Java based solution. ;-)
              >
              > I see no reason a simple solution like this can't be scalable,
              > deployable, and all the other -ilities you want from a site.
              > It doesn't sound as cool as a grand J2EE implementation, but it
              > might get a whole lot of business value implemented a whole lot
              > faster.
              >
              > -Paul Hodgetts
            • Keith Richardson
              ... application ... by environment ... no ... solution ... traffic. ... features ... How would you have handled the user story: Must be able to scale
              Message 6 of 22 , Jan 1, 2001
              • 0 Attachment
                --- In extremeprogramming@egroups.com, "Chad Fowler"
                <chadfowler@y...> wrote:
                > I think the point of a lot of these messages is that EJB is not, at
                > least at the beginning of a project, (in most cases) the simplest
                > thing that could possibly work. A well factored servlet
                application
                > provides session persistence (not sure what you mean
                by "environment"
                > in this context) and (being well factored) would also give
                > flexibility for deployment options (things are logically decoupled,
                > so they can be physically separated down the road).
                >
                > In the past, our developers were assuming that scalable-web-
                > application == J2EE-all-the-way and just going with EJB from the
                > start. This instroduced complexity in the code, the
                > development/build process, and the application server deployment.
                > This complexity invariably lead to various problems (more points of
                > potential failure).
                >
                > The approach we've taken semi-recently with our developers is to
                > say, "You are free to use EJB (and the rest of the J2EE baggage) if
                > you can explain why you need it." What we've found so far is that
                no
                > one has ever needed it, and we haven't had any related problems.
                > (And, our applications are *so* much easier to deploy and maintain
                > now!) In the mean time, we've been trying to keep things well
                > factored, so they could be easily moved to the more complex
                solution
                > should we ever hit the scalability barriers of this approach. My
                > guess is that we won't ever get there.
                >
                > Of course, we're no amazon.com, but we get some pretty heavy
                traffic.
                >
                > Just as an addendum, here's my XP zealot answer:
                > "must be able to automate persistance of session and environment
                > data, allow deployment to be easily adjusted and many other
                features
                > that EJBs (and more completely J2EE) provides" wasn't in my user
                > stories. :)
                >
                > Chad

                How would you have handled the user story: "Must be able to scale
                quickly to unpredictable levels when a new client wants to deploy in
                a high load environment"? Maybe there is a good reason by this is a
                bad user story but I am having a hard time not having this need lead
                to a technology selection.
                Keith Richardson
              • Ron Jeffries
                ... Well, this is a copout, but that story can t be estimated (at least not by me). I d have to break it down into stories like must be able to run 100 hits
                Message 7 of 22 , Jan 1, 2001
                • 0 Attachment
                  At 11:27 PM 1/1/2001 +0000, it seemed like Keith Richardson wrote:
                  >How would you have handled the user story: "Must be able to scale
                  >quickly to unpredictable levels when a new client wants to deploy in
                  >a high load environment"? Maybe there is a good reason by this is a
                  >bad user story but I am having a hard time not having this need lead
                  >to a technology selection.

                  Well, this is a copout, but that story can't be estimated (at least not by
                  me). I'd have to break it down into stories like "must be able to run 100
                  hits per second on a Pentium IV" or something.

                  Then I'd have to do some experiments. This process might look almost
                  exactly like technology selection.

                  Ronald E Jeffries
                  http://www.XProgramming.com
                  http://www.objectmentor.com
                • Chad Fowler
                  ... at ... in ... lead ... Perhaps, your project doesn t fall into the in most cases category. If it were me, I would choose the simplest thing that I
                  Message 8 of 22 , Jan 1, 2001
                  • 0 Attachment
                    --- In extremeprogramming@egroups.com, "Keith Richardson"
                    <keith@s...> wrote:
                    > --- In extremeprogramming@egroups.com, "Chad Fowler"
                    > <chadfowler@y...> wrote:
                    > > I think the point of a lot of these messages is that EJB is not,
                    at
                    > > least at the beginning of a project, (in most cases) the simplest
                    > > thing that could possibly work.

                    > How would you have handled the user story: "Must be able to scale
                    > quickly to unpredictable levels when a new client wants to deploy
                    in
                    > a high load environment"? Maybe there is a good reason by this is a
                    > bad user story but I am having a hard time not having this need
                    lead
                    > to a technology selection.

                    Perhaps, your project doesn't fall into the "in most cases"
                    category. If it were me, I would choose the simplest thing that I
                    *believe* could possibly work and follow up with a performance spike
                    using load testing software. Of course, as Ron mentioned in a
                    previous response, words like "unpredictable" don't lend themselves
                    to estimation. I would first try to get a solid idea of
                    what "unpredictable" might mean.
                  • Keith Richardson
                    ... in ... lead ... not by ... run 100 ... almost ... I think I am faced with having to take a copout of some form. We will be creating solutions that are
                    Message 9 of 22 , Jan 1, 2001
                    • 0 Attachment
                      --- In extremeprogramming@egroups.com, Ron Jeffries
                      <ronjeffries@a...> wrote:
                      > At 11:27 PM 1/1/2001 +0000, it seemed like Keith Richardson wrote:
                      > >How would you have handled the user story: "Must be able to scale
                      > >quickly to unpredictable levels when a new client wants to deploy
                      in
                      > >a high load environment"? Maybe there is a good reason by this is a
                      > >bad user story but I am having a hard time not having this need
                      lead
                      > >to a technology selection.
                      >
                      > Well, this is a copout, but that story can't be estimated (at least
                      not by
                      > me). I'd have to break it down into stories like "must be able to
                      run 100
                      > hits per second on a Pentium IV" or something.
                      >
                      > Then I'd have to do some experiments. This process might look
                      almost
                      > exactly like technology selection.
                      >
                      > Ronald E Jeffries
                      > http://www.XProgramming.com
                      > http://www.objectmentor.com

                      I think I am faced with having to take a copout of some form. We will
                      be creating solutions that are installed by our clients and used by
                      their customers. Some clients are large with jobs running through
                      hundreds of plants that are accessed though a single Web site. I have
                      absolutely no idea how often their customers will be accessing the
                      site - the clients don't know either. Trying to put a number of this
                      would be a wild guess at best. The worst that can happen if the most
                      scalable technology is selected is the project takes longer to
                      implement and the deployment costs are slightly higher. The worst
                      that can happen if we go for lightweight solutions is that we find
                      the system is unusable when it eventually gets deployed. That would
                      be bad!

                      There have been several messages in this forum describing situations
                      where EJB was found to be overkill and unnecessary. I am worried that
                      selecting another technology could leave me doing major rework
                      against a deployment deadline when I run in to the eventual
                      performance bottleneck. What success have other Web developers had in
                      passing performance limits?
                      Keith Richardson
                    • Dossy
                      ... This does sound like a big cop-out. Can we not apply Yesterday s Weather to this problem? Do you have ANY customers? Or are you still in the haven t
                      Message 10 of 22 , Jan 1, 2001
                      • 0 Attachment
                        On 2001.01.02, Keith Richardson <keith@...> wrote:
                        > I think I am faced with having to take a copout of some form. We will
                        > be creating solutions that are installed by our clients and used by
                        > their customers. Some clients are large with jobs running through
                        > hundreds of plants that are accessed though a single Web site. I have
                        > absolutely no idea how often their customers will be accessing the
                        > site - the clients don't know either. Trying to put a number of this
                        > would be a wild guess at best.

                        This does sound like a big cop-out. Can we not apply Yesterday's
                        Weather to this problem?

                        Do you have ANY customers? Or are you still in the "haven't sold
                        product" stage and we're just doing a lot of what-if? (If your
                        answers are "no" and "yes", then I'm tempted to say YAGNI. ;-) )

                        If you already have a customer, can you not use them as a "model"
                        for other customers in their size/scale/class, at least to create
                        estimates that are even slightly better than WAGs?

                        Can you not use measurements from smaller customers to approximate
                        what measurements might look like for larger customers? Can you
                        not create a spike to at least verify that these approximations
                        are "close enough"? (Complexologists would suggest building
                        simulations - I tend to agree.)

                        It might be very costly to build a worthwhile simulation of a
                        theoretical "huge" customer, but with the kind of dollars on
                        the line for that kind of customer, wouldn't it be worth it?
                        What would be more costly, the simulation or the loss of the
                        customer because you didn't fully understand your own product
                        that you're selling?

                        > The worst that can happen if the most
                        > scalable technology is selected is the project takes longer to
                        > implement and the deployment costs are slightly higher. The worst
                        > that can happen if we go for lightweight solutions is that we find
                        > the system is unusable when it eventually gets deployed. That would
                        > be bad!

                        If you're really doing XP, the cost of change curve should be
                        "flat enough" such that moving from a lightweight solution to a
                        "scalable technology" shouldn't have an oppressive cost.

                        If you implement the lightweight solution poorly, sure, it'll
                        be that much tougher to enhance the system to meet scalability
                        needs later on. But that's not a technology selection problem...

                        Make the system usable when it's deployed, and make it deployable
                        ASAP for the customers you DO know about. When a big fat customer
                        comes along, the money they should pay should justify any serious
                        enhancements to the system, and as long as you've kept the system
                        as simple as possible, enhancing where it's necessary (and only
                        where it's necessary) shouldn't be too hard.


                        - Dossy

                        --
                        Dossy Shiobara mail: dossy@...
                        Panoptic Computer Network web: http://www.panoptic.com/
                      • Ron Jeffries
                        ... With all due respect, I believe the above to be almost exactly incorrect. The worst that can happen if a big technology is chosen is that you get to market
                        Message 11 of 22 , Jan 1, 2001
                        • 0 Attachment
                          At 02:55 AM 1/2/2001 +0000, it seemed like Keith Richardson wrote:
                          >The worst that can happen if the most
                          >scalable technology is selected is the project takes longer to
                          >implement and the deployment costs are slightly higher. The worst
                          >that can happen if we go for lightweight solutions is that we find
                          >the system is unusable when it eventually gets deployed. That would
                          >be bad!

                          With all due respect, I believe the above to be almost exactly incorrect.

                          The worst that can happen if a big technology is chosen is that you get to
                          market late, lose market share, and don't learn what people really want
                          until it is too late. The worst that can happen if you go for lightweight
                          solutions is that you have to beef it up upon deployment. But you find that
                          out about a week after you start.

                          Flexibility is better than planning, every time.

                          Ronald E Jeffries
                          http://www.XProgramming.com
                          http://www.objectmentor.com
                        • Glen B. Alleman
                          Flexibility versus planning? ... critical (realtime shop floor control) or is it informational to a non-critical set of users? That is a big range of
                          Message 12 of 22 , Jan 1, 2001
                          • 0 Attachment
                            Flexibility versus planning?

                            Here's some questions that may help clarify the situation:

                            >>What is the context of the problem here?
                            >>Where are the boundaries of your recommendation here?
                            >>Where in the overall scheme of things does this system fit? Is it mission
                            critical (realtime shop floor control) or is it informational to a
                            non-critical set of users? That is a big range of "domain."

                            The problem described by the original poster was a bit vague I agree. The
                            manufacturing domain problems of "scalability estimates" are common,
                            serious, and at the same time unknown without a lot of leg work. In a large
                            manufacturing systems, "massive customization" is supported by a
                            "configurator" based BOM system. See http://www.configsc.com/logia2.htm as
                            an example of such a system. This is not the best, there are others (I2
                            being a better one). This process can whipsaw the transaction rates by 2 to
                            3 orders of magnitude when a simple "build to order" configuration is
                            changed by marketing in an attempt to reposition a product line. The sizing
                            impacts on this type of plant or shop floor system are a challenge for the
                            experts, let along someone just getting started in the development of a
                            plant data management system.

                            This fellow needs to perform some analysis to determine the boundaries of
                            the problem before embarking on ANY development method. He needs to get a
                            strategy of how to scale the system "if and when" scaling is needed. If it
                            is not needed, fine, but if it is then the system must be capable of
                            performing this scaling without disruption to the ongoing business.

                            If it is an on going business, then there are some CRUD stats somewhere,
                            even if they have to be gathered by hand. Those then are used to "size" the
                            solutions and determine the underlying technology needs. This is really
                            simple IT Strategy stuff. There is definitely missing information in this
                            case, which will most likely cause problems down stream. He's in no position
                            to pick any technology without first understanding the processes and the
                            data that they touch.

                            Glen B Alleman
                            Niwot Ridge Consulting

                            "Every even number greater than 2 is the sum of two primes"


                            >-----Original Message-----
                            >From: Ron Jeffries [mailto:ronjeffries@...]
                            >Sent: Monday, January 01, 2001 8:28 PM
                            >To: extremeprogramming@egroups.com
                            >Subject: Re: [XP] Re: Architecture for XP and scalable web sites
                            >
                            >
                            >At 02:55 AM 1/2/2001 +0000, it seemed like Keith Richardson wrote:
                            >>The worst that can happen if the most
                            >>scalable technology is selected is the project takes longer to
                            >>implement and the deployment costs are slightly higher. The worst
                            >>that can happen if we go for lightweight solutions is that we find
                            >>the system is unusable when it eventually gets deployed. That would
                            >>be bad!
                            >
                            >With all due respect, I believe the above to be almost exactly incorrect.
                            >
                            >The worst that can happen if a big technology is chosen is that you get to
                            >market late, lose market share, and don't learn what people really want
                            >until it is too late. The worst that can happen if you go for lightweight
                            >solutions is that you have to beef it up upon deployment. But you
                            >find that
                            >out about a week after you start.
                            >
                            >Flexibility is better than planning, every time.
                            >
                            >Ronald E Jeffries
                            >
                          • Patrick Logan
                            ... I would notice that this story is ambiguous.
                            Message 13 of 22 , Jan 1, 2001
                            • 0 Attachment
                              --- "Keith Richardson" <keith@s...> wrote:

                              > How would you have handled the user story: "Must be able to
                              > scale quickly to unpredictable levels when a new client wants
                              > to deploy in a high load environment"?

                              I would notice that this story is ambiguous.
                            • Laurent Bossavit
                              ... Wouldn t it be worse if you selected the most scalable technology, took longer to implement the project, *and* found the system didn t respond well to high
                              Message 14 of 22 , Jan 2, 2001
                              • 0 Attachment
                                > The worst that can happen if the most scalable technology is
                                > selected is the project takes longer to implement and the
                                > deployment costs are slightly higher. The worst that can happen if
                                > we go for lightweight solutions is that we find the system is
                                > unusable when it eventually gets deployed. That would be bad!

                                Wouldn't it be worse if you selected the most scalable technology,
                                took longer to implement the project, *and* found the system didn't
                                respond well to high loads when deployed ?

                                Do you have any guarantee that won't happen ? What steps can be
                                taken to prevent this happening ?


                                ========================================
                                We aim to make simple things simple and
                                complex things possible.
                                ========================================
                                Laurent Bossavit - Software Architect
                                >>> laurent.bossavit@... <<<
                                >>> 06 68 15 11 44 <<<
                                >> ICQ#39281367 <<
                                Agence Bless http://www.agencebless.com/
                                ========================================
                              • Keith Richardson
                                ... What a great forum - I posted a message at 10pm and got many great responses by 7am!! Either you guys never sleep or most of you are not is the EST time
                                Message 15 of 22 , Jan 2, 2001
                                • 0 Attachment
                                  --- In extremeprogramming@egroups.com, "Laurent Bossavit"
                                  <laurent.bossavit@a...> wrote:
                                  > > The worst that can happen if the most scalable technology is
                                  > > selected is the project takes longer to implement and the
                                  > > deployment costs are slightly higher. The worst that can happen if
                                  > > we go for lightweight solutions is that we find the system is
                                  > > unusable when it eventually gets deployed. That would be bad!
                                  >
                                  > Wouldn't it be worse if you selected the most scalable technology,
                                  > took longer to implement the project, *and* found the system didn't
                                  > respond well to high loads when deployed ?
                                  >
                                  > Do you have any guarantee that won't happen ? What steps can be
                                  > taken to prevent this happening ?

                                  What a great forum - I posted a message at 10pm and got many great
                                  responses by 7am!! Either you guys never sleep or most of you are not
                                  is the EST time zone!

                                  The clear consensus is to keep the technology simple. Now to pick a
                                  robust and simple technology that works well with XP. Java is a much
                                  cleaner environment than ActiveX which would suggest not using MS-
                                  based development environment for IIS. A new question: what
                                  combination of development tools and deployment environment is best
                                  for a Web application built using XP? It should have a clean object
                                  model and a build environment that allows for rapid development cycle
                                  times. It must also be robust and a good performer because I don't
                                  want to be kludging around limitations or speed bottlenecks. What is
                                  being used for XP and how is it working?
                                  Keith Richardson
                                • Chad Fowler
                                  ... cycle ... I would suggest that keeping your technology choice as open as possible is a good plan. This enables you to start with simplicity and grow when
                                  Message 16 of 22 , Jan 2, 2001
                                  • 0 Attachment
                                    >
                                    > The clear consensus is to keep the technology simple. Now to pick a
                                    > robust and simple technology that works well with XP. Java is a much
                                    > cleaner environment than ActiveX which would suggest not using MS-
                                    > based development environment for IIS. A new question: what
                                    > combination of development tools and deployment environment is best
                                    > for a Web application built using XP? It should have a clean object
                                    > model and a build environment that allows for rapid development
                                    cycle
                                    > times. It must also be robust and a good performer because I don't
                                    > want to be kludging around limitations or speed bottlenecks. What is
                                    > being used for XP and how is it working?
                                    > Keith Richardson

                                    I would suggest that keeping your technology choice as open as
                                    possible is a good plan. This enables you to start with simplicity
                                    and grow when the need arises with less investment than a proprietary
                                    alternative would afford. We are successfully using and deploying
                                    Enhydra (http://www.enhydra.org) in a change-heavy iterative
                                    development environment. We have been very happy with its performance
                                    and flexibility (as well as its price--Open Source).
                                  • Stefan Schmiedl
                                    They work 40 hour weeks and hence have lots of free time on their hands.... :-) Stefan +---------+------------------------- ...
                                    Message 17 of 22 , Jan 2, 2001
                                    • 0 Attachment
                                      They work 40 hour weeks and hence have lots of free time on their
                                      hands.... :-)

                                      Stefan

                                      +---------+-------------------------
                                      | from | Keith Richardson <keith@...>
                                      | to | extremeprogramming@egroups.com <extremeprogramming@egroups.com>
                                      | date | 02.01.2001 16:13
                                      | subject | [XP] Re: Architecture for XP and scalable web sites
                                      +---------+-------------------------

                                      K> What a great forum - I posted a message at 10pm and got many great
                                      K> responses by 7am!! Either you guys never sleep or most of you are not
                                      K> is the EST time zone!
                                    • astl@home.com
                                      ... using ... the ... be ... to ... application ... by environment ... no ... solution ... traffic. ... features ... Do the user stories say don t lose my
                                      Message 18 of 22 , Jan 12, 2001
                                      • 0 Attachment
                                        --- In extremeprogramming@egroups.com, "Chad Fowler"
                                        <chadfowler@y...> wrote:
                                        > --- In extremeprogramming@egroups.com, "Keith Richardson"
                                        > <keith@s...> wrote:
                                        > > Hello,
                                        > > This forum has included several messages describing problems
                                        using
                                        > XP
                                        > > with EJB, describing EJB as non-OO and other difficulties with
                                        the
                                        > > EJB architecture. A highly scalable web site must be able to
                                        > automate
                                        > > persistance of session and environment data, allow deployment to
                                        be
                                        > > easily adjusted and many other features that EJBs (and more
                                        > > completely J2EE) provides. Are there other environments that have
                                        > > proven to be better for developing highly scalable solutions with
                                        > XP
                                        > > or should I interpret these messages as saying that XP is not
                                        > > applicable for these needs? Any XP success stories in this area
                                        to
                                        > be
                                        > > shared?
                                        > > Keith Richardson
                                        >
                                        > I think the point of a lot of these messages is that EJB is not, at
                                        > least at the beginning of a project, (in most cases) the simplest
                                        > thing that could possibly work. A well factored servlet
                                        application
                                        > provides session persistence (not sure what you mean
                                        by "environment"
                                        > in this context) and (being well factored) would also give
                                        > flexibility for deployment options (things are logically decoupled,
                                        > so they can be physically separated down the road).
                                        >
                                        > In the past, our developers were assuming that scalable-web-
                                        > application == J2EE-all-the-way and just going with EJB from the
                                        > start. This instroduced complexity in the code, the
                                        > development/build process, and the application server deployment.
                                        > This complexity invariably lead to various problems (more points of
                                        > potential failure).
                                        >
                                        > The approach we've taken semi-recently with our developers is to
                                        > say, "You are free to use EJB (and the rest of the J2EE baggage) if
                                        > you can explain why you need it." What we've found so far is that
                                        no
                                        > one has ever needed it, and we haven't had any related problems.
                                        > (And, our applications are *so* much easier to deploy and maintain
                                        > now!) In the mean time, we've been trying to keep things well
                                        > factored, so they could be easily moved to the more complex
                                        solution
                                        > should we ever hit the scalability barriers of this approach. My
                                        > guess is that we won't ever get there.
                                        >
                                        > Of course, we're no amazon.com, but we get some pretty heavy
                                        traffic.
                                        >
                                        > Just as an addendum, here's my XP zealot answer:
                                        > "must be able to automate persistance of session and environment
                                        > data, allow deployment to be easily adjusted and many other
                                        features
                                        > that EJBs (and more completely J2EE) provides" wasn't in my user
                                        > stories. :)
                                        >
                                        > Chad

                                        Do the user stories say "don't lose my data that I am entering just
                                        because you want to update the servers?"

                                        Ken.
                                      • Alan Francis
                                        ... ... Can we PLEASE try to keep the copied text to a minimum ? Even if the two new lines are the most
                                        Message 19 of 22 , Jan 12, 2001
                                        • 0 Attachment
                                          --- In extremeprogramming@egroups.com, astl@h... wrote:
                                          <a zillion lines of copied message>
                                          <followed by....>
                                          > Do the user stories say "don't lose my data that I am entering just
                                          > because you want to update the servers?"

                                          Can we PLEASE try to keep the copied text to a minimum ? Even if the
                                          two new lines are the most insightful piece of observation yet, it's
                                          still anooying to get massive emails full of copied stuff.

                                          Maybe OMI could host an extremeprogramming wiki so conversations
                                          could be conversations ? :-)

                                          Alan
                                        • Chad Fowler
                                          ... If they do, then it s time to find the simplest solution that can satisfy the story--again, not limited to EJB (nor very likely to be).
                                          Message 20 of 22 , Jan 13, 2001
                                          • 0 Attachment
                                            > > Just as an addendum, here's my XP zealot answer:
                                            > > "must be able to automate persistance of session and environment
                                            > > data, allow deployment to be easily adjusted and many other
                                            > features
                                            > > that EJBs (and more completely J2EE) provides" wasn't in my user
                                            > > stories. :)
                                            > >
                                            > Do the user stories say "don't lose my data that I am entering just
                                            > because you want to update the servers?"
                                            >
                                            > Ken.

                                            If they do, then it's time to find the simplest solution that can
                                            satisfy the story--again, not limited to EJB (nor very likely to be).
                                          • Erik Meade
                                            I mentioned this to Ward a few weeks ago, he said that Extreme Programming is on topic for his (The) Wiki http://www.c2.com/cgi/wiki?FrontPage -- Erik Meade
                                            Message 21 of 22 , Jan 17, 2001
                                            • 0 Attachment
                                              I mentioned this to Ward a few weeks ago, he said that Extreme
                                              Programming is on topic for his (The) Wiki
                                              http://www.c2.com/cgi/wiki?FrontPage

                                              --
                                              Erik Meade emeade@...
                                              Senior Consultant Object Mentor, Inc.
                                              http://www.junit.org


                                              -----Original Message-----
                                              From: "Alan Francis" <alan@...>
                                              Subject: Re: Architecture for XP and scalable web sites

                                              Maybe OMI could host an extremeprogramming wiki so conversations
                                              could be conversations ? :-)

                                              Alan
                                            Your message has been successfully submitted and would be delivered to recipients shortly.