Loading ...
Sorry, an error occurred while loading the content.

Re: Architecture for XP and scalable web sites

Expand Messages
  • Keith Richardson
    ... application ... by environment ... no ... solution ... traffic. ... features ... How would you have handled the user story: Must be able to scale
    Message 1 of 22 , Jan 1, 2001
    • 0 Attachment
      --- In extremeprogramming@egroups.com, "Chad Fowler"
      <chadfowler@y...> wrote:
      > I think the point of a lot of these messages is that EJB is not, at
      > least at the beginning of a project, (in most cases) the simplest
      > thing that could possibly work. A well factored servlet
      application
      > provides session persistence (not sure what you mean
      by "environment"
      > in this context) and (being well factored) would also give
      > flexibility for deployment options (things are logically decoupled,
      > so they can be physically separated down the road).
      >
      > In the past, our developers were assuming that scalable-web-
      > application == J2EE-all-the-way and just going with EJB from the
      > start. This instroduced complexity in the code, the
      > development/build process, and the application server deployment.
      > This complexity invariably lead to various problems (more points of
      > potential failure).
      >
      > The approach we've taken semi-recently with our developers is to
      > say, "You are free to use EJB (and the rest of the J2EE baggage) if
      > you can explain why you need it." What we've found so far is that
      no
      > one has ever needed it, and we haven't had any related problems.
      > (And, our applications are *so* much easier to deploy and maintain
      > now!) In the mean time, we've been trying to keep things well
      > factored, so they could be easily moved to the more complex
      solution
      > should we ever hit the scalability barriers of this approach. My
      > guess is that we won't ever get there.
      >
      > Of course, we're no amazon.com, but we get some pretty heavy
      traffic.
      >
      > Just as an addendum, here's my XP zealot answer:
      > "must be able to automate persistance of session and environment
      > data, allow deployment to be easily adjusted and many other
      features
      > that EJBs (and more completely J2EE) provides" wasn't in my user
      > stories. :)
      >
      > Chad

      How would you have handled the user story: "Must be able to scale
      quickly to unpredictable levels when a new client wants to deploy in
      a high load environment"? Maybe there is a good reason by this is a
      bad user story but I am having a hard time not having this need lead
      to a technology selection.
      Keith Richardson
    • Ron Jeffries
      ... Well, this is a copout, but that story can t be estimated (at least not by me). I d have to break it down into stories like must be able to run 100 hits
      Message 2 of 22 , Jan 1, 2001
      • 0 Attachment
        At 11:27 PM 1/1/2001 +0000, it seemed like Keith Richardson wrote:
        >How would you have handled the user story: "Must be able to scale
        >quickly to unpredictable levels when a new client wants to deploy in
        >a high load environment"? Maybe there is a good reason by this is a
        >bad user story but I am having a hard time not having this need lead
        >to a technology selection.

        Well, this is a copout, but that story can't be estimated (at least not by
        me). I'd have to break it down into stories like "must be able to run 100
        hits per second on a Pentium IV" or something.

        Then I'd have to do some experiments. This process might look almost
        exactly like technology selection.

        Ronald E Jeffries
        http://www.XProgramming.com
        http://www.objectmentor.com
      • Chad Fowler
        ... at ... in ... lead ... Perhaps, your project doesn t fall into the in most cases category. If it were me, I would choose the simplest thing that I
        Message 3 of 22 , Jan 1, 2001
        • 0 Attachment
          --- In extremeprogramming@egroups.com, "Keith Richardson"
          <keith@s...> wrote:
          > --- In extremeprogramming@egroups.com, "Chad Fowler"
          > <chadfowler@y...> wrote:
          > > I think the point of a lot of these messages is that EJB is not,
          at
          > > least at the beginning of a project, (in most cases) the simplest
          > > thing that could possibly work.

          > How would you have handled the user story: "Must be able to scale
          > quickly to unpredictable levels when a new client wants to deploy
          in
          > a high load environment"? Maybe there is a good reason by this is a
          > bad user story but I am having a hard time not having this need
          lead
          > to a technology selection.

          Perhaps, your project doesn't fall into the "in most cases"
          category. If it were me, I would choose the simplest thing that I
          *believe* could possibly work and follow up with a performance spike
          using load testing software. Of course, as Ron mentioned in a
          previous response, words like "unpredictable" don't lend themselves
          to estimation. I would first try to get a solid idea of
          what "unpredictable" might mean.
        • Keith Richardson
          ... in ... lead ... not by ... run 100 ... almost ... I think I am faced with having to take a copout of some form. We will be creating solutions that are
          Message 4 of 22 , Jan 1, 2001
          • 0 Attachment
            --- In extremeprogramming@egroups.com, Ron Jeffries
            <ronjeffries@a...> wrote:
            > At 11:27 PM 1/1/2001 +0000, it seemed like Keith Richardson wrote:
            > >How would you have handled the user story: "Must be able to scale
            > >quickly to unpredictable levels when a new client wants to deploy
            in
            > >a high load environment"? Maybe there is a good reason by this is a
            > >bad user story but I am having a hard time not having this need
            lead
            > >to a technology selection.
            >
            > Well, this is a copout, but that story can't be estimated (at least
            not by
            > me). I'd have to break it down into stories like "must be able to
            run 100
            > hits per second on a Pentium IV" or something.
            >
            > Then I'd have to do some experiments. This process might look
            almost
            > exactly like technology selection.
            >
            > Ronald E Jeffries
            > http://www.XProgramming.com
            > http://www.objectmentor.com

            I think I am faced with having to take a copout of some form. We will
            be creating solutions that are installed by our clients and used by
            their customers. Some clients are large with jobs running through
            hundreds of plants that are accessed though a single Web site. I have
            absolutely no idea how often their customers will be accessing the
            site - the clients don't know either. Trying to put a number of this
            would be a wild guess at best. The worst that can happen if the most
            scalable technology is selected is the project takes longer to
            implement and the deployment costs are slightly higher. The worst
            that can happen if we go for lightweight solutions is that we find
            the system is unusable when it eventually gets deployed. That would
            be bad!

            There have been several messages in this forum describing situations
            where EJB was found to be overkill and unnecessary. I am worried that
            selecting another technology could leave me doing major rework
            against a deployment deadline when I run in to the eventual
            performance bottleneck. What success have other Web developers had in
            passing performance limits?
            Keith Richardson
          • Dossy
            ... This does sound like a big cop-out. Can we not apply Yesterday s Weather to this problem? Do you have ANY customers? Or are you still in the haven t
            Message 5 of 22 , Jan 1, 2001
            • 0 Attachment
              On 2001.01.02, Keith Richardson <keith@...> wrote:
              > I think I am faced with having to take a copout of some form. We will
              > be creating solutions that are installed by our clients and used by
              > their customers. Some clients are large with jobs running through
              > hundreds of plants that are accessed though a single Web site. I have
              > absolutely no idea how often their customers will be accessing the
              > site - the clients don't know either. Trying to put a number of this
              > would be a wild guess at best.

              This does sound like a big cop-out. Can we not apply Yesterday's
              Weather to this problem?

              Do you have ANY customers? Or are you still in the "haven't sold
              product" stage and we're just doing a lot of what-if? (If your
              answers are "no" and "yes", then I'm tempted to say YAGNI. ;-) )

              If you already have a customer, can you not use them as a "model"
              for other customers in their size/scale/class, at least to create
              estimates that are even slightly better than WAGs?

              Can you not use measurements from smaller customers to approximate
              what measurements might look like for larger customers? Can you
              not create a spike to at least verify that these approximations
              are "close enough"? (Complexologists would suggest building
              simulations - I tend to agree.)

              It might be very costly to build a worthwhile simulation of a
              theoretical "huge" customer, but with the kind of dollars on
              the line for that kind of customer, wouldn't it be worth it?
              What would be more costly, the simulation or the loss of the
              customer because you didn't fully understand your own product
              that you're selling?

              > The worst that can happen if the most
              > scalable technology is selected is the project takes longer to
              > implement and the deployment costs are slightly higher. The worst
              > that can happen if we go for lightweight solutions is that we find
              > the system is unusable when it eventually gets deployed. That would
              > be bad!

              If you're really doing XP, the cost of change curve should be
              "flat enough" such that moving from a lightweight solution to a
              "scalable technology" shouldn't have an oppressive cost.

              If you implement the lightweight solution poorly, sure, it'll
              be that much tougher to enhance the system to meet scalability
              needs later on. But that's not a technology selection problem...

              Make the system usable when it's deployed, and make it deployable
              ASAP for the customers you DO know about. When a big fat customer
              comes along, the money they should pay should justify any serious
              enhancements to the system, and as long as you've kept the system
              as simple as possible, enhancing where it's necessary (and only
              where it's necessary) shouldn't be too hard.


              - Dossy

              --
              Dossy Shiobara mail: dossy@...
              Panoptic Computer Network web: http://www.panoptic.com/
            • Ron Jeffries
              ... With all due respect, I believe the above to be almost exactly incorrect. The worst that can happen if a big technology is chosen is that you get to market
              Message 6 of 22 , Jan 1, 2001
              • 0 Attachment
                At 02:55 AM 1/2/2001 +0000, it seemed like Keith Richardson wrote:
                >The worst that can happen if the most
                >scalable technology is selected is the project takes longer to
                >implement and the deployment costs are slightly higher. The worst
                >that can happen if we go for lightweight solutions is that we find
                >the system is unusable when it eventually gets deployed. That would
                >be bad!

                With all due respect, I believe the above to be almost exactly incorrect.

                The worst that can happen if a big technology is chosen is that you get to
                market late, lose market share, and don't learn what people really want
                until it is too late. The worst that can happen if you go for lightweight
                solutions is that you have to beef it up upon deployment. But you find that
                out about a week after you start.

                Flexibility is better than planning, every time.

                Ronald E Jeffries
                http://www.XProgramming.com
                http://www.objectmentor.com
              • Glen B. Alleman
                Flexibility versus planning? ... critical (realtime shop floor control) or is it informational to a non-critical set of users? That is a big range of
                Message 7 of 22 , Jan 1, 2001
                • 0 Attachment
                  Flexibility versus planning?

                  Here's some questions that may help clarify the situation:

                  >>What is the context of the problem here?
                  >>Where are the boundaries of your recommendation here?
                  >>Where in the overall scheme of things does this system fit? Is it mission
                  critical (realtime shop floor control) or is it informational to a
                  non-critical set of users? That is a big range of "domain."

                  The problem described by the original poster was a bit vague I agree. The
                  manufacturing domain problems of "scalability estimates" are common,
                  serious, and at the same time unknown without a lot of leg work. In a large
                  manufacturing systems, "massive customization" is supported by a
                  "configurator" based BOM system. See http://www.configsc.com/logia2.htm as
                  an example of such a system. This is not the best, there are others (I2
                  being a better one). This process can whipsaw the transaction rates by 2 to
                  3 orders of magnitude when a simple "build to order" configuration is
                  changed by marketing in an attempt to reposition a product line. The sizing
                  impacts on this type of plant or shop floor system are a challenge for the
                  experts, let along someone just getting started in the development of a
                  plant data management system.

                  This fellow needs to perform some analysis to determine the boundaries of
                  the problem before embarking on ANY development method. He needs to get a
                  strategy of how to scale the system "if and when" scaling is needed. If it
                  is not needed, fine, but if it is then the system must be capable of
                  performing this scaling without disruption to the ongoing business.

                  If it is an on going business, then there are some CRUD stats somewhere,
                  even if they have to be gathered by hand. Those then are used to "size" the
                  solutions and determine the underlying technology needs. This is really
                  simple IT Strategy stuff. There is definitely missing information in this
                  case, which will most likely cause problems down stream. He's in no position
                  to pick any technology without first understanding the processes and the
                  data that they touch.

                  Glen B Alleman
                  Niwot Ridge Consulting

                  "Every even number greater than 2 is the sum of two primes"


                  >-----Original Message-----
                  >From: Ron Jeffries [mailto:ronjeffries@...]
                  >Sent: Monday, January 01, 2001 8:28 PM
                  >To: extremeprogramming@egroups.com
                  >Subject: Re: [XP] Re: Architecture for XP and scalable web sites
                  >
                  >
                  >At 02:55 AM 1/2/2001 +0000, it seemed like Keith Richardson wrote:
                  >>The worst that can happen if the most
                  >>scalable technology is selected is the project takes longer to
                  >>implement and the deployment costs are slightly higher. The worst
                  >>that can happen if we go for lightweight solutions is that we find
                  >>the system is unusable when it eventually gets deployed. That would
                  >>be bad!
                  >
                  >With all due respect, I believe the above to be almost exactly incorrect.
                  >
                  >The worst that can happen if a big technology is chosen is that you get to
                  >market late, lose market share, and don't learn what people really want
                  >until it is too late. The worst that can happen if you go for lightweight
                  >solutions is that you have to beef it up upon deployment. But you
                  >find that
                  >out about a week after you start.
                  >
                  >Flexibility is better than planning, every time.
                  >
                  >Ronald E Jeffries
                  >
                • Patrick Logan
                  ... I would notice that this story is ambiguous.
                  Message 8 of 22 , Jan 1, 2001
                  • 0 Attachment
                    --- "Keith Richardson" <keith@s...> wrote:

                    > How would you have handled the user story: "Must be able to
                    > scale quickly to unpredictable levels when a new client wants
                    > to deploy in a high load environment"?

                    I would notice that this story is ambiguous.
                  • Laurent Bossavit
                    ... Wouldn t it be worse if you selected the most scalable technology, took longer to implement the project, *and* found the system didn t respond well to high
                    Message 9 of 22 , Jan 2, 2001
                    • 0 Attachment
                      > The worst that can happen if the most scalable technology is
                      > selected is the project takes longer to implement and the
                      > deployment costs are slightly higher. The worst that can happen if
                      > we go for lightweight solutions is that we find the system is
                      > unusable when it eventually gets deployed. That would be bad!

                      Wouldn't it be worse if you selected the most scalable technology,
                      took longer to implement the project, *and* found the system didn't
                      respond well to high loads when deployed ?

                      Do you have any guarantee that won't happen ? What steps can be
                      taken to prevent this happening ?


                      ========================================
                      We aim to make simple things simple and
                      complex things possible.
                      ========================================
                      Laurent Bossavit - Software Architect
                      >>> laurent.bossavit@... <<<
                      >>> 06 68 15 11 44 <<<
                      >> ICQ#39281367 <<
                      Agence Bless http://www.agencebless.com/
                      ========================================
                    • Keith Richardson
                      ... What a great forum - I posted a message at 10pm and got many great responses by 7am!! Either you guys never sleep or most of you are not is the EST time
                      Message 10 of 22 , Jan 2, 2001
                      • 0 Attachment
                        --- In extremeprogramming@egroups.com, "Laurent Bossavit"
                        <laurent.bossavit@a...> wrote:
                        > > The worst that can happen if the most scalable technology is
                        > > selected is the project takes longer to implement and the
                        > > deployment costs are slightly higher. The worst that can happen if
                        > > we go for lightweight solutions is that we find the system is
                        > > unusable when it eventually gets deployed. That would be bad!
                        >
                        > Wouldn't it be worse if you selected the most scalable technology,
                        > took longer to implement the project, *and* found the system didn't
                        > respond well to high loads when deployed ?
                        >
                        > Do you have any guarantee that won't happen ? What steps can be
                        > taken to prevent this happening ?

                        What a great forum - I posted a message at 10pm and got many great
                        responses by 7am!! Either you guys never sleep or most of you are not
                        is the EST time zone!

                        The clear consensus is to keep the technology simple. Now to pick a
                        robust and simple technology that works well with XP. Java is a much
                        cleaner environment than ActiveX which would suggest not using MS-
                        based development environment for IIS. A new question: what
                        combination of development tools and deployment environment is best
                        for a Web application built using XP? It should have a clean object
                        model and a build environment that allows for rapid development cycle
                        times. It must also be robust and a good performer because I don't
                        want to be kludging around limitations or speed bottlenecks. What is
                        being used for XP and how is it working?
                        Keith Richardson
                      • Chad Fowler
                        ... cycle ... I would suggest that keeping your technology choice as open as possible is a good plan. This enables you to start with simplicity and grow when
                        Message 11 of 22 , Jan 2, 2001
                        • 0 Attachment
                          >
                          > The clear consensus is to keep the technology simple. Now to pick a
                          > robust and simple technology that works well with XP. Java is a much
                          > cleaner environment than ActiveX which would suggest not using MS-
                          > based development environment for IIS. A new question: what
                          > combination of development tools and deployment environment is best
                          > for a Web application built using XP? It should have a clean object
                          > model and a build environment that allows for rapid development
                          cycle
                          > times. It must also be robust and a good performer because I don't
                          > want to be kludging around limitations or speed bottlenecks. What is
                          > being used for XP and how is it working?
                          > Keith Richardson

                          I would suggest that keeping your technology choice as open as
                          possible is a good plan. This enables you to start with simplicity
                          and grow when the need arises with less investment than a proprietary
                          alternative would afford. We are successfully using and deploying
                          Enhydra (http://www.enhydra.org) in a change-heavy iterative
                          development environment. We have been very happy with its performance
                          and flexibility (as well as its price--Open Source).
                        • Stefan Schmiedl
                          They work 40 hour weeks and hence have lots of free time on their hands.... :-) Stefan +---------+------------------------- ...
                          Message 12 of 22 , Jan 2, 2001
                          • 0 Attachment
                            They work 40 hour weeks and hence have lots of free time on their
                            hands.... :-)

                            Stefan

                            +---------+-------------------------
                            | from | Keith Richardson <keith@...>
                            | to | extremeprogramming@egroups.com <extremeprogramming@egroups.com>
                            | date | 02.01.2001 16:13
                            | subject | [XP] Re: Architecture for XP and scalable web sites
                            +---------+-------------------------

                            K> What a great forum - I posted a message at 10pm and got many great
                            K> responses by 7am!! Either you guys never sleep or most of you are not
                            K> is the EST time zone!
                          • astl@home.com
                            ... using ... the ... be ... to ... application ... by environment ... no ... solution ... traffic. ... features ... Do the user stories say don t lose my
                            Message 13 of 22 , Jan 12, 2001
                            • 0 Attachment
                              --- In extremeprogramming@egroups.com, "Chad Fowler"
                              <chadfowler@y...> wrote:
                              > --- In extremeprogramming@egroups.com, "Keith Richardson"
                              > <keith@s...> wrote:
                              > > Hello,
                              > > This forum has included several messages describing problems
                              using
                              > XP
                              > > with EJB, describing EJB as non-OO and other difficulties with
                              the
                              > > EJB architecture. A highly scalable web site must be able to
                              > automate
                              > > persistance of session and environment data, allow deployment to
                              be
                              > > easily adjusted and many other features that EJBs (and more
                              > > completely J2EE) provides. Are there other environments that have
                              > > proven to be better for developing highly scalable solutions with
                              > XP
                              > > or should I interpret these messages as saying that XP is not
                              > > applicable for these needs? Any XP success stories in this area
                              to
                              > be
                              > > shared?
                              > > Keith Richardson
                              >
                              > I think the point of a lot of these messages is that EJB is not, at
                              > least at the beginning of a project, (in most cases) the simplest
                              > thing that could possibly work. A well factored servlet
                              application
                              > provides session persistence (not sure what you mean
                              by "environment"
                              > in this context) and (being well factored) would also give
                              > flexibility for deployment options (things are logically decoupled,
                              > so they can be physically separated down the road).
                              >
                              > In the past, our developers were assuming that scalable-web-
                              > application == J2EE-all-the-way and just going with EJB from the
                              > start. This instroduced complexity in the code, the
                              > development/build process, and the application server deployment.
                              > This complexity invariably lead to various problems (more points of
                              > potential failure).
                              >
                              > The approach we've taken semi-recently with our developers is to
                              > say, "You are free to use EJB (and the rest of the J2EE baggage) if
                              > you can explain why you need it." What we've found so far is that
                              no
                              > one has ever needed it, and we haven't had any related problems.
                              > (And, our applications are *so* much easier to deploy and maintain
                              > now!) In the mean time, we've been trying to keep things well
                              > factored, so they could be easily moved to the more complex
                              solution
                              > should we ever hit the scalability barriers of this approach. My
                              > guess is that we won't ever get there.
                              >
                              > Of course, we're no amazon.com, but we get some pretty heavy
                              traffic.
                              >
                              > Just as an addendum, here's my XP zealot answer:
                              > "must be able to automate persistance of session and environment
                              > data, allow deployment to be easily adjusted and many other
                              features
                              > that EJBs (and more completely J2EE) provides" wasn't in my user
                              > stories. :)
                              >
                              > Chad

                              Do the user stories say "don't lose my data that I am entering just
                              because you want to update the servers?"

                              Ken.
                            • Alan Francis
                              ... ... Can we PLEASE try to keep the copied text to a minimum ? Even if the two new lines are the most
                              Message 14 of 22 , Jan 12, 2001
                              • 0 Attachment
                                --- In extremeprogramming@egroups.com, astl@h... wrote:
                                <a zillion lines of copied message>
                                <followed by....>
                                > Do the user stories say "don't lose my data that I am entering just
                                > because you want to update the servers?"

                                Can we PLEASE try to keep the copied text to a minimum ? Even if the
                                two new lines are the most insightful piece of observation yet, it's
                                still anooying to get massive emails full of copied stuff.

                                Maybe OMI could host an extremeprogramming wiki so conversations
                                could be conversations ? :-)

                                Alan
                              • Chad Fowler
                                ... If they do, then it s time to find the simplest solution that can satisfy the story--again, not limited to EJB (nor very likely to be).
                                Message 15 of 22 , Jan 13, 2001
                                • 0 Attachment
                                  > > Just as an addendum, here's my XP zealot answer:
                                  > > "must be able to automate persistance of session and environment
                                  > > data, allow deployment to be easily adjusted and many other
                                  > features
                                  > > that EJBs (and more completely J2EE) provides" wasn't in my user
                                  > > stories. :)
                                  > >
                                  > Do the user stories say "don't lose my data that I am entering just
                                  > because you want to update the servers?"
                                  >
                                  > Ken.

                                  If they do, then it's time to find the simplest solution that can
                                  satisfy the story--again, not limited to EJB (nor very likely to be).
                                • Erik Meade
                                  I mentioned this to Ward a few weeks ago, he said that Extreme Programming is on topic for his (The) Wiki http://www.c2.com/cgi/wiki?FrontPage -- Erik Meade
                                  Message 16 of 22 , Jan 17, 2001
                                  • 0 Attachment
                                    I mentioned this to Ward a few weeks ago, he said that Extreme
                                    Programming is on topic for his (The) Wiki
                                    http://www.c2.com/cgi/wiki?FrontPage

                                    --
                                    Erik Meade emeade@...
                                    Senior Consultant Object Mentor, Inc.
                                    http://www.junit.org


                                    -----Original Message-----
                                    From: "Alan Francis" <alan@...>
                                    Subject: Re: Architecture for XP and scalable web sites

                                    Maybe OMI could host an extremeprogramming wiki so conversations
                                    could be conversations ? :-)

                                    Alan
                                  Your message has been successfully submitted and would be delivered to recipients shortly.