Loading ...
Sorry, an error occurred while loading the content.
 

Re: [XP] "Databases" (was: summary of the discusion

Expand Messages
  • Sean Corfield
    Thank you William! Two good, solid, real-world examples showing it is possible. I m more convinced now :)
    Message 1 of 4 , Nov 7, 2009
      Thank you William! Two good, solid, real-world examples showing it is
      possible. I'm more convinced now :)

      On Sat, Nov 7, 2009 at 3:14 PM, William Pietri <william@...> wrote:
      > I don't think many people actually do it, but I think a lot more should.
      > I can give two public examples, and I've seen a lot more private ones.
      >
      > One is Long Bets: http://www.longbets.org/
      >
      > I build that in 2002. I started out, DTSTTCPW style, with just
      > serializing Java objects out to the filesystem. That was a pain to
      > develop with, so I quickly switched to XML persistence, with one
      > directory per type and one file per object or tightly bound object graph.
      >
      > That was only a provisional step, until persistence needs became
      > clearer. But it turned out that worked just fine, and has been working
      > fine since then. Having no database has simplified a lot of things,
      > including development, maintenance, and site operations.
      >
      > Another is Sidereel: http://www.sidereel.com/
      >
      > Again, we started out by keeping all data hot in RAM and writing out
      > files with XML object dumps. The theory was we'd wait until the needs
      > became clearer before we picked a database-ish thing to use. Nearly
      > three years later, they run roughly on the original model, with a number
      > of improvements.
      >
      > That's not out of laziness either; every year or so the question of
      > using an official database comes up. Developers will pick some plausible
      > candidate and do a spike to see how it works. So far, each time they've
      > discovered that the grass is not actually any greener on the other side
      > of the fence. Using a database would save them from certain sorts of
      > hassle, but it introduces other kinds of hassle. Worse, from their
      > perspective, is that databases have big performance costs.
      >
      > So from my perspective, you can often delay the "what database do we
      > use" question a long time, and perhaps forever.
      >
      > William
    • daswartz@prodigy
      Hello Sean, ... In 1999, we were beginning to build a relatively small system; our first XP project. We had Ron Jeffries in to mentor/coach us. We were all
      Message 2 of 4 , Nov 8, 2009
        Hello Sean,

        > I'm curious about real-world practices here. How many people really do
        > start out with a flat file and switch to a database only when they
        > need it?

        > I can understand wanting to defer the decision about the *type* of
        > database (relational, object, schema-less/no-SQL, etc) but how far
        > into a project can you really defer the decision about whether you
        > *need* a database? How many projects really go live without a
        > database?

        > As I say, just curious about people's experiences...

        In 1999, we were beginning to build a relatively small system; our
        first XP project. We had Ron Jeffries in to mentor/coach us. We were
        all experienced folks with a minimum of 15 years of experience. the
        system was sort of a big multi-dimensional spreadsheet with a bunch of
        prebuilt formulas, and data coming from file feeds from various
        systems and web entry.

        Shortly after we built a one day spike of the system that we threw
        away, we sat down to talk about the first set of cards we were
        going to work on. One of our discussion items was which DBMS we
        were going to use for persistence. Ron spoke up and said "When did the
        customer say you had to store the data in a database? Couldn't you
        just keep it in memory and stream it to disk occasionally to save it."
        While being a little uncomfortable with the idea, we decided to give
        it a try.

        We ended up building a minimal XML object broker framework to save and
        retrieve objects from flat files in directories named after object
        classes.

        Today, that original system is a product family with 10, or so,
        instantiations, all continuing to share the vast majority of their
        code. All but one of them still use the (quite a bit expanded) XML in
        flat files scheme. The one exception stores one of its classes in a
        relational table. One of the current systems calculates and manages
        about 40-50 million "spreadsheet cells" each day.

        Occasionally we talk about replacing the flat files with a relational
        store, but, except for the one instance, where we plugged in a
        relational store after the fact, it hasn't been worth the effort.


        __
        Doug swartz
      Your message has been successfully submitted and would be delivered to recipients shortly.