Loading ...
Sorry, an error occurred while loading the content.
 

"Databases" (was: summary of the discusion

Expand Messages
  • Sean Corfield
    ... I m curious about real-world practices here. How many people really do start out with a flat file and switch to a database only when they need it? I can
    Message 1 of 4 , Nov 7, 2009
      On Fri, Nov 6, 2009 at 3:31 AM, Phlip <phlip2005@...> wrote:
      > The books about "Lean Software Development" explain that. When you analyze,
      > without any code yet, you are tempted to make decisions about how to implement
      > things. For example, "we will obviously need to install a big, heavy, expensive
      > database engine, first, to support all these records".
      >
      > You are making those decisions before you have hard data to support them. If you
      > instead start writing your most important features, when the time comes to store
      > their data, you can first try to put them into a flat file. If you then discover
      > you need a database, you can replace the flat file and then pass all your tests.
      > "DeprecationRefactor". The tests pin down your important features, making the
      > change easy and safe.
      >
      > Because you already have live features, you can easily prove what parameters of
      > database you need (not just "the biggest one!"), and you can measure the results.
      >
      > However, if you prove you don't need a big heavy database engine there, then you
      > won. The flat file was the simplest path to a useful system. And if any future
      > requirements force you to switch to a database engine, then you also won. You
      > successfully delivered the high value features, and got customer feedback on
      > them, without making them wait for you to install a database.

      I'm curious about real-world practices here. How many people really do
      start out with a flat file and switch to a database only when they
      need it?

      I can understand wanting to defer the decision about the *type* of
      database (relational, object, schema-less/no-SQL, etc) but how far
      into a project can you really defer the decision about whether you
      *need* a database? How many projects really go live without a
      database?

      As I say, just curious about people's experiences...
      --
      Sean A Corfield -- (904) 302-SEAN
      Railo Technologies US -- http://getrailo.com/
      An Architect's View -- http://corfield.org/

      "If you're not annoying somebody, you're not really alive."
      -- Margaret Atwood
    • William Pietri
      ... I don t think many people actually do it, but I think a lot more should. I can give two public examples, and I ve seen a lot more private ones. One is Long
      Message 2 of 4 , Nov 7, 2009
        Sean Corfield wrote:
        > I'm curious about real-world practices here. How many people really do
        > start out with a flat file and switch to a database only when they
        > need it?
        >
        > I can understand wanting to defer the decision about the *type* of
        > database (relational, object, schema-less/no-SQL, etc) but how far
        > into a project can you really defer the decision about whether you
        > *need* a database? How many projects really go live without a
        > database?
        >
        > As I say, just curious about people's experiences...
        >

        I don't think many people actually do it, but I think a lot more should.
        I can give two public examples, and I've seen a lot more private ones.

        One is Long Bets: http://www.longbets.org/

        I build that in 2002. I started out, DTSTTCPW style, with just
        serializing Java objects out to the filesystem. That was a pain to
        develop with, so I quickly switched to XML persistence, with one
        directory per type and one file per object or tightly bound object graph.

        That was only a provisional step, until persistence needs became
        clearer. But it turned out that worked just fine, and has been working
        fine since then. Having no database has simplified a lot of things,
        including development, maintenance, and site operations.


        Another is Sidereel: http://www.sidereel.com/

        Again, we started out by keeping all data hot in RAM and writing out
        files with XML object dumps. The theory was we'd wait until the needs
        became clearer before we picked a database-ish thing to use. Nearly
        three years later, they run roughly on the original model, with a number
        of improvements.

        That's not out of laziness either; every year or so the question of
        using an official database comes up. Developers will pick some plausible
        candidate and do a spike to see how it works. So far, each time they've
        discovered that the grass is not actually any greener on the other side
        of the fence. Using a database would save them from certain sorts of
        hassle, but it introduces other kinds of hassle. Worse, from their
        perspective, is that databases have big performance costs.


        So from my perspective, you can often delay the "what database do we
        use" question a long time, and perhaps forever.

        William
      • Sean Corfield
        Thank you William! Two good, solid, real-world examples showing it is possible. I m more convinced now :)
        Message 3 of 4 , Nov 7, 2009
          Thank you William! Two good, solid, real-world examples showing it is
          possible. I'm more convinced now :)

          On Sat, Nov 7, 2009 at 3:14 PM, William Pietri <william@...> wrote:
          > I don't think many people actually do it, but I think a lot more should.
          > I can give two public examples, and I've seen a lot more private ones.
          >
          > One is Long Bets: http://www.longbets.org/
          >
          > I build that in 2002. I started out, DTSTTCPW style, with just
          > serializing Java objects out to the filesystem. That was a pain to
          > develop with, so I quickly switched to XML persistence, with one
          > directory per type and one file per object or tightly bound object graph.
          >
          > That was only a provisional step, until persistence needs became
          > clearer. But it turned out that worked just fine, and has been working
          > fine since then. Having no database has simplified a lot of things,
          > including development, maintenance, and site operations.
          >
          > Another is Sidereel: http://www.sidereel.com/
          >
          > Again, we started out by keeping all data hot in RAM and writing out
          > files with XML object dumps. The theory was we'd wait until the needs
          > became clearer before we picked a database-ish thing to use. Nearly
          > three years later, they run roughly on the original model, with a number
          > of improvements.
          >
          > That's not out of laziness either; every year or so the question of
          > using an official database comes up. Developers will pick some plausible
          > candidate and do a spike to see how it works. So far, each time they've
          > discovered that the grass is not actually any greener on the other side
          > of the fence. Using a database would save them from certain sorts of
          > hassle, but it introduces other kinds of hassle. Worse, from their
          > perspective, is that databases have big performance costs.
          >
          > So from my perspective, you can often delay the "what database do we
          > use" question a long time, and perhaps forever.
          >
          > William
        • daswartz@prodigy
          Hello Sean, ... In 1999, we were beginning to build a relatively small system; our first XP project. We had Ron Jeffries in to mentor/coach us. We were all
          Message 4 of 4 , Nov 8, 2009
            Hello Sean,

            > I'm curious about real-world practices here. How many people really do
            > start out with a flat file and switch to a database only when they
            > need it?

            > I can understand wanting to defer the decision about the *type* of
            > database (relational, object, schema-less/no-SQL, etc) but how far
            > into a project can you really defer the decision about whether you
            > *need* a database? How many projects really go live without a
            > database?

            > As I say, just curious about people's experiences...

            In 1999, we were beginning to build a relatively small system; our
            first XP project. We had Ron Jeffries in to mentor/coach us. We were
            all experienced folks with a minimum of 15 years of experience. the
            system was sort of a big multi-dimensional spreadsheet with a bunch of
            prebuilt formulas, and data coming from file feeds from various
            systems and web entry.

            Shortly after we built a one day spike of the system that we threw
            away, we sat down to talk about the first set of cards we were
            going to work on. One of our discussion items was which DBMS we
            were going to use for persistence. Ron spoke up and said "When did the
            customer say you had to store the data in a database? Couldn't you
            just keep it in memory and stream it to disk occasionally to save it."
            While being a little uncomfortable with the idea, we decided to give
            it a try.

            We ended up building a minimal XML object broker framework to save and
            retrieve objects from flat files in directories named after object
            classes.

            Today, that original system is a product family with 10, or so,
            instantiations, all continuing to share the vast majority of their
            code. All but one of them still use the (quite a bit expanded) XML in
            flat files scheme. The one exception stores one of its classes in a
            relational table. One of the current systems calculates and manages
            about 40-50 million "spreadsheet cells" each day.

            Occasionally we talk about replacing the flat files with a relational
            store, but, except for the one instance, where we plugged in a
            relational store after the fact, it hasn't been worth the effort.


            __
            Doug swartz
          Your message has been successfully submitted and would be delivered to recipients shortly.