Loading ...
Sorry, an error occurred while loading the content.

Re: sqlite database

Expand Messages
  • pcsol1996
    ... Thanks for the offer. I will see if he can cook something up.
    Message 1 of 12 , Nov 1, 2004
    • 0 Attachment
      --- In solarisx86@yahoogroups.com, John D Groenveld <jdg117@e...> wrote:
      > In message <cm5rcg+e8mu@e...>, pcsol1996 writes:
      > >Does anyone have any experience with Solaris 9x86/sqlite
      > >and SVM RAID 5 (or RAID 1+0)? My experience with SVM is
      >
      > I don't.
      >
      > >with RAID 0 on data ingest machines. Would SVM extract too much
      > >of a penalty? Expensive external RAID (Stor Edge 3310, 3510..etc)
      >
      > Probably, but I'm just guessing.
      >
      > If you develop a generic benchmark for sqlite which mirrors the size
      > and type of transactions that you're planning, I might be willing to
      > test on some systems here.
      >
      > John
      > groenveld@a...

      Thanks for the offer. I will see if he can cook something up.
    • Al Hopper
      ... Why not use the same LsiLogic card that Sun is using? The driver and raidctl have been backported to Sol9, but I think, per John G, it s still an
      Message 2 of 12 , Nov 1, 2004
      • 0 Attachment
        On Mon, 1 Nov 2004, pcsol1996 wrote:

        >
        >
        > We have a developer that will be developing an sqlite database
        > that will store customer info. The database will be about 20%
        > writes and 80% reads. There would be about 10 users/apps
        > hitting the database at once time.
        >
        > I am planning on using Solaris of course, but
        > I am a little leary of Solaris 10 with the new RAID drivers.
        > (Dell 2650 with PERC4, DC)

        Why not use the same LsiLogic card that Sun is using? The driver and
        raidctl have been backported to Sol9, but I think, per John G, it's still
        an unsupported solution.

        But you can always use the "Al Test" on it: Beat the living daylights out
        of it for 3 days, round the clock. If it survives, and you keep the drives
        cool enough, it's a done deal!

        I posted recently that the Tekram DC-390U4B_U4W *might* work in place of
        the LsiLogic card. It's worth a ~ $220 gamble. If it does not work you
        can always find another use for it.

        > Does anyone have any experience with Solaris 9x86/sqlite
        > and SVM RAID 5 (or RAID 1+0)? My experience with SVM is
        > with RAID 0 on data ingest machines. Would SVM extract too much
        > of a penalty? Expensive external RAID (Stor Edge 3310, 3510..etc)
        > is NOT an option.

        Hard to say - but as the CPU "horsepower" continues to climb, you don't
        mind giving up some CPU cycles to get the I/O bandwidth you need. Since
        you're sticking with SCSI, you'll exhaust the I/O Ops/Sec of the SCSI
        busses, long before you'll see a big impact on your CPU - given a P4 2.8
        or faster.

        Like John G says: try it and see. I may be able to assist in setting up
        some benchmarks also. Am planning to try the Tekram - when I get some free
        time.... Yeah Right! Have some nice "cost effective" SCSI drives on hard
        to try: Fujitsu MAP series MAP3367NP (10K RPM 36Gb ~ $120). PS: one of
        these replaced that lack-luster IBM ATA/IDE drive that is standard in a
        w1100z.

        PS: It's been ~ 3 years since I messed with SCSI. Have been using IDE and
        FC. FC solves the IOPS problem that SCSI suffers from. But at a cost....

        Al Hopper Logical Approach Inc, Plano, TX. al@...
        Voice: 972.379.2133 Fax: 972.379.2134
        Solaris Express 10/04: Is it there yet? Now is it there? It *is*! :)
      • John D Groenveld
        In message
        Message 3 of 12 , Nov 1, 2004
        • 0 Attachment
          In message <Pine.GSO.4.33.0411011613530.27248-100000@...-approach.c
          om>, Al Hopper writes:
          >But you can always use the "Al Test" on it: Beat the living daylights out
          >of it for 3 days, round the clock. If it survives, and you keep the drives
          >cool enough, it's a done deal!

          More justification to develop a reproducible test / benchmark application
          that will work with easy to produce test data: it also serves to certify
          the system before going into production.

          John
          groenveld@...
        • Al Hopper
          ... Agreed entirely. I m reading JUnit Recipes (ISBN 1932394230) that stresses that very point and is an excellent practical book IMHO. Highly recommended.
          Message 4 of 12 , Nov 1, 2004
          • 0 Attachment
            On Mon, 1 Nov 2004, John D Groenveld wrote:

            > In message <Pine.GSO.4.33.0411011613530.27248-100000@...-approach.c
            > om>, Al Hopper writes:
            > >But you can always use the "Al Test" on it: Beat the living daylights out
            > >of it for 3 days, round the clock. If it survives, and you keep the drives
            > >cool enough, it's a done deal!
            >
            > More justification to develop a reproducible test / benchmark application
            > that will work with easy to produce test data: it also serves to certify
            > the system before going into production.

            Agreed entirely. I'm reading JUnit Recipes (ISBN 1932394230) that stresses
            that very point and is an excellent practical book IMHO. Highly
            recommended. Couple JUnit with Ant and you test/resolv issues before they
            become bugs. The author discusses this approach ... do you want to resolve
            a problem with a functional test that fails, where the test code (self)
            documents the condition being tested and the expected result and
            immediately puts you at a known point in your code; or do you want to have
            a bug reported and now you have to try to figure out exactly what the bug
            is, and where it *might* be in your code. And then you still need to
            retest to ensure that the bugfix did'nt break something else ... but with
            what? (assuming that you have not been using JUnit all along).

            Don't know if Sqlite supports JDBC - but that would bring it far enough
            into the Java domain to where you could apply JUnit and Ant.

            PS: I have mixed feelings about Ant. The functionality and breath of the
            existing Ant tasks is amazing[1] ... and expanding all the time. But the
            learning curve is like learning a new language. You "program" with one new
            "word"[2] at a time. It's kind of like learning a language one word at a
            time. You learn the word, apply it, debug it - because you don't really
            understand it - then learn another word etc. It's a pleasant surprise when
            you can re-use words; but if it's been months since you last used that word
            - you're back to the dictionary *again*. And the dictionary keeps
            growing....

            It's definately a much better tool than Make ever was or will be. But when
            I get jammed up with Ant I get tempted to simply write a Python script and
            move on. So far I'm still with Ant. With Tomcat you can't beat the
            convenience of "ant reload" - build the app, and ask the Tomcat container
            to reload it. Or "ant war" (calls "ant clean", "ant build" with
            optimization) when you're ready to deliver a WAR file to the production
            environment. That takes a whole 3 Seconds on an aging dual AMD MP box
            these days for an app with 20,000+ lines of Java code!

            [1] C++ Ant task
            [2] this is the same argument/objection to using XML as a programming
            language.

            Al Hopper Logical Approach Inc, Plano, TX. al@...
            Voice: 972.379.2133 Fax: 972.379.2134
            Solaris Express 10/04: Is it there yet? Now is it there? It *is*! :)
          • pcsol1996
            ... still ... That is not a RAID card per se. It only supports RAID 1 (mirroring) on the two disks in the V20z. RAID 1+0 or RAID 5 isn t possible. ...
            Message 5 of 12 , Nov 1, 2004
            • 0 Attachment
              --- In solarisx86@yahoogroups.com, Al Hopper <al@l...> wrote:
              > On Mon, 1 Nov 2004, pcsol1996 wrote:
              >
              > >
              > >
              > > We have a developer that will be developing an sqlite database
              > > that will store customer info. The database will be about 20%
              > > writes and 80% reads. There would be about 10 users/apps
              > > hitting the database at once time.
              > >
              > > I am planning on using Solaris of course, but
              > > I am a little leary of Solaris 10 with the new RAID drivers.
              > > (Dell 2650 with PERC4, DC)
              >
              > Why not use the same LsiLogic card that Sun is using? The driver and
              > raidctl have been backported to Sol9, but I think, per John G, it's
              still
              > an unsupported solution.

              That is not a RAID card per se. It only supports RAID 1
              (mirroring) on the two disks in the V20z. RAID 1+0 or RAID
              5 isn't possible.

              >
              > But you can always use the "Al Test" on it: Beat the living
              daylights out
              > of it for 3 days, round the clock. If it survives, and you keep the
              drives
              > cool enough, it's a done deal!
              >
              > I posted recently that the Tekram DC-390U4B_U4W *might* work in place of
              > the LsiLogic card. It's worth a ~ $220 gamble. If it does not work you
              > can always find another use for it.

              Once again, it's not a RAID card. The LSI U320 SCSI driver in the
              Dell 2600 works very well. I have 3 running.
              >
              > > Does anyone have any experience with Solaris 9x86/sqlite
              > > and SVM RAID 5 (or RAID 1+0)? My experience with SVM is
              > > with RAID 0 on data ingest machines. Would SVM extract too much
              > > of a penalty? Expensive external RAID (Stor Edge 3310, 3510..etc)
              > > is NOT an option.
              >
              > Hard to say - but as the CPU "horsepower" continues to climb, you don't
              > mind giving up some CPU cycles to get the I/O bandwidth you need. Since
              > you're sticking with SCSI, you'll exhaust the I/O Ops/Sec of the SCSI
              > busses, long before you'll see a big impact on your CPU - given a P4 2.8
              > or faster.
              >
              > Like John G says: try it and see. I may be able to assist in setting up
              > some benchmarks also. Am planning to try the Tekram - when I get
              some free
              > time.... Yeah Right! Have some nice "cost effective" SCSI drives
              on hard
              > to try: Fujitsu MAP series MAP3367NP (10K RPM 36Gb ~ $120). PS: one of
              > these replaced that lack-luster IBM ATA/IDE drive that is standard in a
              > w1100z.

              I can't try it until I buy the box. I have nothing to spare right
              now. If I go the SW RAID route I would buy something different
              than the HW RAID route,
              >
              > PS: It's been ~ 3 years since I messed with SCSI. Have been using
              IDE and
              > FC. FC solves the IOPS problem that SCSI suffers from. But at a
              cost....

              I have been severely burned by the poor (relative to cost) of the
              Sun T3. No way am I going to use FC unless forced.
              >
              >
              Anyway thanks for the help. I will see if the developer can put
              together a test case (his spare time is very limited). We may just
              have to go with my best judgement on the box and test it on that
              box, then buy the backup box.
            • Al Hopper
              ... It supports RAID 0, 1, 5, RAID 10 (RAID 1 + RAID 0) and RAID50 (RAID 5 + RAID 0). The Solaris raidctl interface (program) only supports RAID 1
              Message 6 of 12 , Nov 1, 2004
              • 0 Attachment
                On Tue, 2 Nov 2004, pcsol1996 wrote:

                >
                >
                > --- In solarisx86@yahoogroups.com, Al Hopper <al@l...> wrote:
                > > On Mon, 1 Nov 2004, pcsol1996 wrote:
                > >
                > > >
                > > >
                > > > We have a developer that will be developing an sqlite database
                > > > that will store customer info. The database will be about 20%
                > > > writes and 80% reads. There would be about 10 users/apps
                > > > hitting the database at once time.
                > > >
                > > > I am planning on using Solaris of course, but
                > > > I am a little leary of Solaris 10 with the new RAID drivers.
                > > > (Dell 2650 with PERC4, DC)
                > >
                > > Why not use the same LsiLogic card that Sun is using? The driver and
                > > raidctl have been backported to Sol9, but I think, per John G, it's
                > still
                > > an unsupported solution.
                >
                > That is not a RAID card per se. It only supports RAID 1
                > (mirroring) on the two disks in the V20z. RAID 1+0 or RAID
                > 5 isn't possible.

                It supports RAID 0, 1, 5, RAID 10 (RAID 1 + RAID 0) and RAID50 (RAID 5 +
                RAID 0). The Solaris raidctl interface (program) only supports RAID 1
                (mirroring). At least that is the story *today*. To use the other RAID
                modes, you must configure the card via the BIOS before the OS loads.

                There are docs on sun.com that show configuring multiple cards in the same
                box with arrays that span different controllers. All done via the BIOS.
                IIRC the limit is 4 cards per system. The main point of those docs is to
                figure out, from a Solaris perspective, what drive is mapped to what.

                ... snip ...
                > Once again, it's not a RAID card. The LSI U320 SCSI driver in the

                As above. Same chip.

                > Dell 2600 works very well. I have 3 running.

                Theres good sense in going with what you know. I'm only trying to save you
                money. And I think that the raidctl program will have expanded
                functionality one of these months.

                ... snip ....
                > I have been severely burned by the poor (relative to cost) of the
                > Sun T3. No way am I going to use FC unless forced.

                Yep. It's well known fact that Suns storage products are lame.
                Especially the low end products with the high end price tags. When Sun
                builds a decent FC RAID box that is cost/performance competitive - I'll buy
                one! The 3510FC is the closest they have come yet ... but it's overpriced
                IMO.

                Al Hopper Logical Approach Inc, Plano, TX. al@...
                Voice: 972.379.2133 Fax: 972.379.2134
                Solaris Express 10/04: Is it there yet? Now is it there? It *is*! :)
              • Ian Collins
                ... My usual advice is to avoid SVM RAID5 at all costs. RAID0 performance is as one would expect, slight hit on writes, slight gain on reads. These are the
                Message 7 of 12 , Nov 1, 2004
                • 0 Attachment
                  pcsol1996 wrote:

                  >We have a developer that will be developing an sqlite database
                  >that will store customer info. The database will be about 20%
                  >writes and 80% reads. There would be about 10 users/apps
                  >hitting the database at once time.
                  >
                  > I am planning on using Solaris of course, but
                  >I am a little leary of Solaris 10 with the new RAID drivers.
                  >(Dell 2650 with PERC4, DC)
                  >Does anyone have any experience with Solaris 9x86/sqlite
                  >and SVM RAID 5 (or RAID 1+0)? My experience with SVM is
                  >with RAID 0 on data ingest machines. Would SVM extract too much
                  >of a penalty? Expensive external RAID (Stor Edge 3310, 3510..etc)
                  >is NOT an option.
                  >
                  >
                  >
                  My usual advice is to avoid SVM RAID5 at all costs. RAID0 performance
                  is as one would expect, slight hit on writes, slight gain on reads.

                  These are the bonnie results form a Quantum 72G U160 drive on a 700MHz
                  PIII system:

                  Raw:

                  -------Sequential Output-------- ---Sequential Input--
                  --Random--
                  -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                  --Seeks---
                  Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                  /sec %CPU
                  1000 20685 95.0 31794 84.8 10655 34.8 25115 96.3 30316 25.1
                  191.6 5.1

                  SVM RAID5:

                  -------Sequential Output-------- ---Sequential Input--
                  --Random--
                  -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                  --Seeks---
                  Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                  /sec %CPU
                  1000 4621 30.8 4657 21.3 2255 11.0 22184 90.7 27688 32.4
                  278.5 8.4

                  SVM Mirror:
                  -------Sequential Output-------- ---Sequential Input--
                  --Random--
                  -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                  --Seeks---
                  Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                  /sec %CPU
                  1000 17691 90.3 18087 55.1 6719 25.0 17651 68.4 31831 33.6
                  280.9 7.0

                  Ian
                • Ian Collins
                  ... For this, the Ian test is at least 6 bonnie sessions with file sizes between 1.5 and 2GB, plus a format surface analysis. Locks my RAID card river in 5-10
                  Message 8 of 12 , Nov 1, 2004
                  • 0 Attachment
                    Al Hopper wrote:

                    >On Mon, 1 Nov 2004, pcsol1996 wrote:
                    >
                    >
                    >
                    >>We have a developer that will be developing an sqlite database
                    >>that will store customer info. The database will be about 20%
                    >>writes and 80% reads. There would be about 10 users/apps
                    >>hitting the database at once time.
                    >>
                    >> I am planning on using Solaris of course, but
                    >>I am a little leary of Solaris 10 with the new RAID drivers.
                    >>(Dell 2650 with PERC4, DC)
                    >>
                    >>
                    >
                    >Why not use the same LsiLogic card that Sun is using? The driver and
                    >raidctl have been backported to Sol9, but I think, per John G, it's still
                    >an unsupported solution.
                    >
                    >But you can always use the "Al Test" on it: Beat the living daylights out
                    >of it for 3 days, round the clock. If it survives, and you keep the drives
                    >cool enough, it's a done deal!
                    >
                    >

                    For this, the Ian test is at least 6 bonnie sessions with file sizes
                    between 1.5 and 2GB, plus a format surface analysis. Locks my RAID card
                    river in 5-10 minutes!

                    Ian
                  • palowoda
                    ... still ... daylights out ... the drives ... card ... Worse use the bonnie++ tests as it heats up the servo on the random file creation time tests. Either
                    Message 9 of 12 , Nov 2, 2004
                    • 0 Attachment
                      --- In solarisx86@yahoogroups.com, Ian Collins <ian@i...> wrote:
                      > Al Hopper wrote:
                      >
                      > >On Mon, 1 Nov 2004, pcsol1996 wrote:
                      > >
                      > >
                      > >
                      > >>We have a developer that will be developing an sqlite database
                      > >>that will store customer info. The database will be about 20%
                      > >>writes and 80% reads. There would be about 10 users/apps
                      > >>hitting the database at once time.
                      > >>
                      > >> I am planning on using Solaris of course, but
                      > >>I am a little leary of Solaris 10 with the new RAID drivers.
                      > >>(Dell 2650 with PERC4, DC)
                      > >>
                      > >>
                      > >
                      > >Why not use the same LsiLogic card that Sun is using? The driver and
                      > >raidctl have been backported to Sol9, but I think, per John G, it's
                      still
                      > >an unsupported solution.
                      > >
                      > >But you can always use the "Al Test" on it: Beat the living
                      daylights out
                      > >of it for 3 days, round the clock. If it survives, and you keep
                      the drives
                      > >cool enough, it's a done deal!
                      > >
                      > >
                      >
                      > For this, the Ian test is at least 6 bonnie sessions with file sizes
                      > between 1.5 and 2GB, plus a format surface analysis. Locks my RAID
                      card
                      > river in 5-10 minutes!
                      >

                      Worse use the 'bonnie++' tests as it heats up the servo
                      on the random file creation time tests. Either one
                      you choose it did wonders to identify the the old weak
                      DeathStar disks. And bonnie++ is not limited to the 2g
                      file size.

                      ---Bob
                    • pcsol1996
                      - ... Wow, now that is a hit. I think that pretty much answers my question. That s pretty bad. Thanks for sharing that.
                      Message 10 of 12 , Nov 2, 2004
                      • 0 Attachment
                        -
                        > >
                        > My usual advice is to avoid SVM RAID5 at all costs. RAID0 performance
                        > is as one would expect, slight hit on writes, slight gain on reads.
                        >
                        > These are the bonnie results form a Quantum 72G U160 drive on a 700MHz
                        > PIII system:
                        >
                        > Raw:
                        >
                        > -------Sequential Output-------- ---Sequential Input--
                        > --Random--
                        > -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                        > --Seeks---
                        > Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                        > /sec %CPU
                        > 1000 20685 95.0 31794 84.8 10655 34.8 25115 96.3 30316 25.1
                        > 191.6 5.1
                        >
                        > SVM RAID5:
                        >
                        > -------Sequential Output-------- ---Sequential Input--
                        > --Random--
                        > -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                        > --Seeks---
                        > Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                        > /sec %CPU
                        > 1000 4621 30.8 4657 21.3 2255 11.0 22184 90.7 27688 32.4
                        > 278.5 8.4
                        >
                        > SVM Mirror:
                        > -------Sequential Output-------- ---Sequential Input--
                        > --Random--
                        > -Per Char- --Block--- -Rewrite-- -Per Char- --Block---
                        > --Seeks---
                        > Machine MB K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU K/sec %CPU
                        > /sec %CPU
                        > 1000 17691 90.3 18087 55.1 6719 25.0 17651 68.4 31831 33.6
                        > 280.9 7.0
                        >
                        > Ian

                        Wow, now that is a hit. I think that pretty much answers my
                        question. That's pretty bad.

                        Thanks for sharing that.
                      Your message has been successfully submitted and would be delivered to recipients shortly.