Loading ...
Sorry, an error occurred while loading the content.

Re: [PanoToolsNG] PTgui -- why doesnt it just use RAM?

Expand Messages
  • mrjimbo
    Hi Matt, I m a little rusty at this but here goes.. First ram is not intended to replicate a hard disk .. It is used as a place to hold portions of data to be
    Message 1 of 19 , Dec 1, 2007
    View Source
    • 0 Attachment
      Hi Matt,
      I'm a little rusty at this but here goes.. First ram is not intended to replicate a hard disk .. It is used as a place to hold portions of data to be processed.. Another item that hasn't really been spoken about so far is CPU cache.. That also helps a bunch on a CPU that is being used soley for crunching and actually much more then most realize.. But it too has a high cost penality.. Ok your machine can only go as fast as it's smallest bottleneck.. So after readin your posts and a few others I think you may have a few things slowing you down. I don't knwo what you use your machine fo rbut if it's loaded with programs and thing sthat ar erunning at start up then you may hav eissues.. Actually theri are a few viruses that really slow a machine down.. CHeck to see how many services are runing.. If you hav e50 or above you may want to look at that.. A well oiled workstation has usually less then 30.. Rememebr that all the services that are runnng require allocation for attention.
      Just for the record a really great hard drive set up screams.. remember that things are not only being written to it but also read and that's a lot of work. So if your only using 5-10% of your CPU then yup something is slowing you down and it sounds like you need to find out what. When your crunching 16 bit files you are using a ton of overhead.. Ru you trying to make what I call a home computer function like a CAD workstation? I don't know your answer, sorry, bu tno way does it take 5 hours to crunch what you've described.. Your system has a bottle neck and I think you may need to find it. Checking your services is a start..

      Jim

      ----- Original Message -----
      From: matt_nolan_uaf
      To: PanoToolsNG@yahoogroups.com
      Sent: Saturday, December 01, 2007 3:28 PM
      Subject: [PanoToolsNG] PTgui -- why doesnt it just use RAM?


      To create a hi-res spherical, PTgui says 7 tiffs at 70MB each requires
      about 1GB of scratch disk -- if I have 4 GB of RAM, why does it need a
      disk at all except to save the final image? Wouldn't doing everything
      in RAM dramatically speed things up, especially since disk I/O seems to
      be the bottleneck?

      I'm no computer guru, so maybe this is a naive question and I'm missing
      something simple.

      What I'm fairly certain of is that in the 5 hours it takes my dual core
      2GHz machine (2:30 minutes on Milko's speedtest) to stitch a 16bit
      11,000x5500 image, my CPU reads 5-10% most of the time and my disks are
      not being accessed much of this time (and could write 1GB a few hundred
      times in that time anyway, if that were truly the bottleneck).

      -Matt





      [Non-text portions of this message have been removed]
    • Hans Nyberg
      ... In PTGui Preferences/Plugins write: -m 3000 in the command line parameters If your mac and Enblend 3.0 behaves the same as mine enblend will crash after a
      Message 2 of 19 , Dec 1, 2007
      View Source
      • 0 Attachment
        --- In PanoToolsNG@yahoogroups.com, "AYRTON - avi" <avi@...> wrote:
        >
        > On 12/1/07, Hans Nyberg <hans@...> wrote:
        >
        > > The 11400x5700 16bit takes 31 minutes using PTguiWarp + Ptgui Blend.
        > >
        > > Using PTGuiWarp + Enblend with 3 GB Ram applied I get down to 19.20 minutes.
        >
        > Sorry I could NOT figure out HOW to aplly more RAM when using Enblend ???
        > Pls some directions will be appreciatted :-)
        > Thanks

        In PTGui Preferences/Plugins write: -m 3000 in the command line parameters

        If your mac and Enblend 3.0 behaves the same as mine enblend will crash after a while if
        you try to apply 3600 or more.
        How much Ram did you get?
        I just found out that I can update my G5 with 4 gb and get 8 in all for just $130.
        I guess I paid 4 btimes as much at least for the 4 GB I got when I bought the G5.

        Hans
      • AYRTON - avi
        ... Thanks and sorry but can someone point me to where could I learn that ??? I mean, how I would know that by myself ??? ... 4GB ... here in Brazil RAM is
        Message 3 of 19 , Dec 1, 2007
        View Source
        • 0 Attachment
          On 12/1/07, Hans Nyberg <hans@...> wrote:

          >
          > In PTGui Preferences/Plugins write: -m 3000 in the command line parameters

          Thanks
          and sorry but can someone point me to where could I learn that ???
          I mean, how I would know that by myself ???

          >
          > If your mac and Enblend 3.0 behaves the same as mine enblend will crash after a while if
          > you try to apply 3600 or more.
          > How much Ram did you get?

          4GB

          > I just found out that I can update my G5 with 4 gb and get 8 in all for just $130.
          > I guess I paid 4 btimes as much at least for the 4 GB I got when I bought the G5.

          here in Brazil RAM is getting very cheap everyday too
          I remember the first MAC I use in the 90's and the price was 1 dolar
          for each 1mb :-(

          AYRTON

          >
          > Hans
          >
          >
          >
          > --
          >
          > Yahoo! Groups Links
          >
          >
          >
          >


          --
          AYRTON 21-9982.6313 www.ayrton.com
          Ladeira de Nossa Senhora, 214 / sl. 101 www.vrfolio.com
          Outeiro da Glória - RJ - 22211-100 - Brasil www.vr-images.com
          Panoramas do Rio de Janeiro www.rio360.com.br
        • Hans Nyberg
          ... http://enblend.sourceforge.net/ Has all the commandline parameters But I think PTGui should have a little more about it on the tutorial page.
          Message 4 of 19 , Dec 1, 2007
          View Source
          • 0 Attachment
            --- In PanoToolsNG@yahoogroups.com, "AYRTON - avi" <avi@...> wrote:
            >
            > On 12/1/07, Hans Nyberg <hans@...> wrote:
            >
            > >
            > > In PTGui Preferences/Plugins write: -m 3000 in the command line parameters
            >
            > Thanks
            > and sorry but can someone point me to where could I learn that ???
            > I mean, how I would know that by myself ???
            >

            http://enblend.sourceforge.net/
            Has all the commandline parameters

            But I think PTGui should have a little more about it on the tutorial page.
            http://www.ptgui.com/plugins.html

            As far as I know there are some commands prebuilt into PTGui at least the parameter for
            blending around the 360 degree wrap.

            Hans
          • AYRTON - avi
            THANKS !!!! ayrton ... -- AYRTON 21-9982.6313 www.ayrton.com Ladeira de Nossa Senhora, 214 / sl. 101 www.vrfolio.com
            Message 5 of 19 , Dec 1, 2007
            View Source
            • 0 Attachment
              THANKS !!!!

              ayrton


              On 12/1/07, Hans Nyberg <hans@...> wrote:
              > --- In PanoToolsNG@yahoogroups.com, "AYRTON - avi" <avi@...> wrote:
              > >
              > > On 12/1/07, Hans Nyberg <hans@...> wrote:
              > >
              > > >
              > > > In PTGui Preferences/Plugins write: -m 3000 in the command line parameters
              > >
              > > Thanks
              > > and sorry but can someone point me to where could I learn that ???
              > > I mean, how I would know that by myself ???
              > >
              >
              > http://enblend.sourceforge.net/
              > Has all the commandline parameters
              >
              > But I think PTGui should have a little more about it on the tutorial page.
              > http://www.ptgui.com/plugins.html
              >
              > As far as I know there are some commands prebuilt into PTGui at least the parameter for
              > blending around the 360 degree wrap.
              >
              >
              > Hans
              >
              >
              >
              >
              >
              >
              > --
              >
              > Yahoo! Groups Links
              >
              >
              >
              >


              --
              AYRTON 21-9982.6313 www.ayrton.com
              Ladeira de Nossa Senhora, 214 / sl. 101 www.vrfolio.com
              Outeiro da Glória - RJ - 22211-100 - Brasil www.vr-images.com
              Panoramas do Rio de Janeiro www.rio360.com.br
            • Charlie Hubbard
              ... Yes it still amazes me how cheap computer equipment has become. There seems to be no end. I m a bit older than you, I think. I got my first computer
              Message 6 of 19 , Dec 1, 2007
              View Source
              • 0 Attachment
                > here in Brazil RAM is getting very cheap everyday too
                > I remember the first MAC I use in the 90's and the price was 1 dolar
                > for each 1mb :-(

                Yes it still amazes me how cheap computer equipment has become. There
                seems to be no end. I'm a bit older than you, I think. I got my first
                computer around 1979 or 1980. RAM cost 200 dollars (US) for 16 kilobytes
                (but dropped to $100 for 16kB before the thing was completely obsolete).
                And prices have been plummeting ever since. I remember when the first
                1GB hard disks came out. They were big, noisy, power hungry, and very
                expensive. I remember thinking at the time "How could I ever possibly
                use up an entire gigabyte?!" I knew a guy who had one. He had it
                partitioned into 16 or so logical drives to keep the file system's block
                size reasonably small. Today it's pretty routine to have single files
                larger than 1GB. I remember getting a 3GB external SCSI drive out at
                work in the early 90's. We had it sitting on a typing table next to the
                computer. It was a monster. The table actually shook when the drive
                was spinning up. Today I carry an 8GB thumb drive in my pocket. Who
                can forget Bill Gates' 1983 quote "Nobody will ever need more than 640
                kB of RAM." Today even modest computers have 1- or 2,000 times that
                much RAM. I remember when 640x480 reigned supreme and JPGs were
                extremely annoying because it took SO DAMN LONG for them to decode.
                Times have changed. It will be interesting to see what the future holds.
              • panotools@360image.de
                Hi Matt, ... 5 hours seems a bit excessive. Although, you work on a laptop which have low speed hdd usually. On your speedtest results you mention an external
                Message 7 of 19 , Dec 1, 2007
                View Source
                • 0 Attachment
                  Hi Matt,
                  >
                  >
                  > What I'm fairly certain of is that in the 5 hours it takes my dual core
                  > 2GHz machine (2:30 minutes on Milko's speedtest) to stitch a 16bit
                  > 11,000x5500 image, my CPU reads 5-10% most of the time and my disks are
                  > not being accessed much of this time (and could write 1GB a few hundred
                  > times in that time anyway, if that were truly the bottleneck).
                  >
                  5 hours seems a bit excessive. Although, you work on a laptop which have
                  low speed hdd usually.
                  On your speedtest results you mention an external 500gig drive connected
                  with USB2.
                  Are you hosting the panorama project files and sources on that drive?
                  And where are you sending your temp files to?

                  Cheers, Milko
                • matt_nolan_uaf
                  Milko, I guess I need to do some further controlled tests, but as you know my machine performed your speed test at 2.5 minutes, which is comparable to others.
                  Message 8 of 19 , Dec 1, 2007
                  View Source
                  • 0 Attachment
                    Milko,

                    I guess I need to do some further controlled tests, but as you know
                    my machine performed your speed test at 2.5 minutes, which is
                    comparable to others. Using the same configuration, a 11,000 x 5,500
                    pix pano (7 images at 70 MB each) at 8 bit takes about 30-45 minutes
                    to process (need to check). Going to 16 bit more than doubles it to
                    5 hours or more (dont know, I always kill it first). But there could
                    easily be something buggering up my system that has nothing to do
                    with PTgui itself. Glad to hear that others are doing this much
                    faster. Maybe we need another speed test -- one with a large file
                    size, as there could be different issues associated with this?

                    To all, yes, my laptop has tons of stuff loaded on it, and I have
                    seen the light of buying a dedicated stitching box. But in
                    researching the fastest hard drives, I read about hard disks built
                    from RAM that plug into a standard SATA slot. These are not flash
                    disks, but literally using RAM and I began thinking, why is there any
                    disk access at all for a project that can fit into real RAM anyway?
                    I'm still not sure why.

                    In case others are interested, visit this page to learn about the
                    apparent benefits of setting up two such RAM disks in RAID 0.
                    http://www.tomshardware.com/2005/12/05/hyperos_dram_hard_drive_on_the_
                    block/page6.html The big winner seems to be iRAM, which you can buy
                    for about $140 for the card and up to 4 GB of cheap slow RAM (buying
                    fast RAM doesnt help, because the cheap stuff is already maxing out
                    the 1.5 GB/s SATA channel, and there is no 3.0GB/s disk yet
                    apparently).

                    For spherical fisheye stitching, where project needs are never higher
                    than 3 or 4 GB max of scratch space, this could be the silver bullet
                    that could get Milko's test down to near zero. These disks in RAID 0
                    are apparently hundreds to thousands of times faster than Raptors or
                    SAS!

                    Another trick I've read about is called quarter stroking your disk.
                    Apparently you can create a partition on a 7500 RPM disk such that
                    the partition is only using the inner portion of the disk, so that
                    the arm doesnt move so far. Doing this is apparently faster than
                    using a 15k SAS disk, and much cheaper, and plus you still get the
                    slow part of the disk to use for storage.

                    So I'm beginning to think of a system like this:
                    - XP 64 bit loaded onto a 16 GB flash disk
                    - RAID 0 iRAM 2 GB disks for stitching fisheye sphericals
                    - RAID 0 750 GB disks SATA II quarter stroked to provide about 375 GB
                    of fast scratch space for gigapixel panos, with the rest (about 1TB)
                    for storage.
                    - eSATA external disks for backup

                    My other constraint is that I work from home and live off the
                    electrical grid. So these flash and RAM disks make great sense for
                    me, as thinking of a computer with a 500W power supply give me the
                    shudders...


                    -Matt




                    --- In PanoToolsNG@yahoogroups.com, "panotools@..." <panotools@...>
                    wrote:
                    >
                    > Hi Matt,
                    > >
                    > >
                    > > What I'm fairly certain of is that in the 5 hours it takes my
                    dual core
                    > > 2GHz machine (2:30 minutes on Milko's speedtest) to stitch a 16bit
                    > > 11,000x5500 image, my CPU reads 5-10% most of the time and my
                    disks are
                    > > not being accessed much of this time (and could write 1GB a few
                    hundred
                    > > times in that time anyway, if that were truly the bottleneck).
                    > >
                    > 5 hours seems a bit excessive. Although, you work on a laptop which
                    have
                    > low speed hdd usually.
                    > On your speedtest results you mention an external 500gig drive
                    connected
                    > with USB2.
                    > Are you hosting the panorama project files and sources on that
                    drive?
                    > And where are you sending your temp files to?
                    >
                    > Cheers, Milko
                    >
                  • matt_nolan_uaf
                    I created a RAM disk using http://www.ramdisk.tk/, changed my scratch disk and write disk to the ram disk, and ran Milko s speed test and got exactly the same
                    Message 9 of 19 , Dec 2, 2007
                    View Source
                    • 0 Attachment
                      I created a RAM disk using http://www.ramdisk.tk/, changed my scratch
                      disk and write disk to the ram disk, and ran Milko's speed test and got
                      exactly the same results -- 2 minutes 30 seconds. Wierd. How could it
                      not be incredibly faster? My dual CPUs ran at 50%, with one near 100%
                      and the other near idle; I checked the affinity for PTgui was set to
                      both.

                      -Matt


                      --- In PanoToolsNG@yahoogroups.com, Bruno Postle <bruno@...> wrote:
                      >
                      > On Sat 01-Dec-2007 at 22:28 -0000, matt_nolan_uaf wrote:
                      > > To create a hi-res spherical, PTgui says 7 tiffs at 70MB each
                      > > requires about 1GB of scratch disk -- if I have 4 GB of RAM, why
                      > > does it need a disk at all except to save the final image?
                      > > Wouldn't doing everything in RAM dramatically speed things up,
                      > > especially since disk I/O seems to be the bottleneck?
                      >
                      > Good point, try setting up a 2GiB RAM disk for the scratch data and
                      > see if that makes an improvement.
                      >
                      > --
                      > Bruno
                      >
                    • Joost Nieuwenhuijse
                      This is because PTGui was already using RAM.. PTGui stores temporary data in temp files, but due to the OS s file system caching, the files never really get
                      Message 10 of 19 , Dec 2, 2007
                      View Source
                      • 0 Attachment
                        This is because PTGui was already using RAM..

                        PTGui stores temporary data in temp files, but due to the OS's file
                        system caching, the files never really get written to disk but instead
                        they are cached in RAM. Only for large panoramas the OS will not have
                        enough cache memory and that's when the temporary files actually start
                        being written to disk.

                        This also means that PTGui should use all available RAM as long as the
                        OS is not doing anything else.

                        These 'stitching speed' questions are always difficult to answer since
                        there are so many factors that influence the speed.

                        Joost


                        matt_nolan_uaf wrote:
                        > I created a RAM disk using http://www.ramdisk.tk/, changed my scratch
                        > disk and write disk to the ram disk, and ran Milko's speed test and got
                        > exactly the same results -- 2 minutes 30 seconds. Wierd. How could it
                        > not be incredibly faster? My dual CPUs ran at 50%, with one near 100%
                        > and the other near idle; I checked the affinity for PTgui was set to
                        > both.
                        >
                        > -Matt
                        >
                        >
                        > --- In PanoToolsNG@yahoogroups.com, Bruno Postle <bruno@...> wrote:
                        >> On Sat 01-Dec-2007 at 22:28 -0000, matt_nolan_uaf wrote:
                        >>> To create a hi-res spherical, PTgui says 7 tiffs at 70MB each
                        >>> requires about 1GB of scratch disk -- if I have 4 GB of RAM, why
                        >>> does it need a disk at all except to save the final image?
                        >>> Wouldn't doing everything in RAM dramatically speed things up,
                        >>> especially since disk I/O seems to be the bottleneck?
                        >> Good point, try setting up a 2GiB RAM disk for the scratch data and
                        >> see if that makes an improvement.
                        >>
                        >> --
                        >> Bruno
                        >>
                        >
                        >
                        >
                        >
                      • Pat Swovelin
                        ... Is that also a good number for WinXP with 4GB RAM and the 3GB switch flipped? ... Pat Swovelin Cool Guy @ Large
                        Message 11 of 19 , Dec 2, 2007
                        View Source
                        • 0 Attachment
                          On 12/1/2007 3:44 PM, Hans Nyberg rambled on about ...:
                          > --- In PanoToolsNG@yahoogroups.com, "AYRTON - avi" <avi@...> wrote:
                          >> On 12/1/07, Hans Nyberg <hans@...> wrote:
                          >>
                          >>> The 11400x5700 16bit takes 31 minutes using PTguiWarp + Ptgui Blend.
                          >>>
                          >>> Using PTGuiWarp + Enblend with 3 GB Ram applied I get down to 19.20
                          >>> minutes.
                          >> Sorry I could NOT figure out HOW to aplly more RAM when using Enblend ???
                          >> Pls some directions will be appreciatted :-)
                          >> Thanks
                          >
                          > In PTGui Preferences/Plugins write: -m 3000 in the command line parameters

                          Is that also a good number for WinXP with 4GB RAM and the 3GB switch
                          flipped?

                          > If your mac and Enblend 3.0 behaves the same as mine enblend will
                          > crash after a while if
                          > you try to apply 3600 or more.
                          > How much Ram did you get?
                          > I just found out that I can update my G5 with 4 gb and get 8 in all
                          > for just $130.
                          > I guess I paid 4 btimes as much at least for the 4 GB I got when I
                          > bought the G5.
                          >
                          > Hans




                          Pat Swovelin
                          Cool Guy @ Large
                        • matt_nolan_uaf
                          This is because PTGui was already using RAM.. ... instead ... have ... start ... I guess I m missing something easy and fundamental. For fisheye sperhical
                          Message 12 of 19 , Dec 4, 2007
                          View Source
                          • 0 Attachment
                            This is because PTGui was already using RAM..
                            >
                            > PTGui stores temporary data in temp files, but due to the OS's file
                            > system caching, the files never really get written to disk but
                            instead
                            > they are cached in RAM. Only for large panoramas the OS will not
                            have
                            > enough cache memory and that's when the temporary files actually
                            start
                            > being written to disk.
                            >

                            I guess I'm missing something easy and fundamental. For fisheye
                            sperhical projects using only a GB or so, does this mean that using a
                            virtual RAM disk will have no benefit on any computer, not just my
                            slow, bloated one? Also, if everything is happening in RAM already
                            with a small stitch, why is there a hard disk bottleneck at all?



                            > This also means that PTGui should use all available RAM as long as
                            the
                            > OS is not doing anything else.
                            >
                            > These 'stitching speed' questions are always difficult to answer
                            since
                            > there are so many factors that influence the speed.

                            I bet. But I guess my interest in understanding the existing messy
                            hardware issues is so that I can purchase something that is optimized
                            for stitching, with a minimum of unknowns involved.

                            What I'm trying to decide at the moment is whether it's worth
                            investing in something like the iRAM disks, or whether virtual RAM
                            disks would do the same thing but better. These iRAM disks seem to
                            be an order of magnitude faster or more than the fastest spinning
                            disks. Any insights would be appreciated.

                            Thanks,
                            Matt
                          • Joost Nieuwenhuijse
                            ... If you have enough RAM, yes. Windows will effectively emulate a RAM disk by means of file caching. ... For small panoramas there should not be a hard disk
                            Message 13 of 19 , Dec 4, 2007
                            View Source
                            • 0 Attachment
                              matt_nolan_uaf wrote:
                              > This is because PTGui was already using RAM..
                              >> PTGui stores temporary data in temp files, but due to the OS's file
                              >> system caching, the files never really get written to disk but
                              > instead
                              >> they are cached in RAM. Only for large panoramas the OS will not
                              > have
                              >> enough cache memory and that's when the temporary files actually
                              > start
                              >> being written to disk.
                              >>
                              >
                              > I guess I'm missing something easy and fundamental. For fisheye
                              > sperhical projects using only a GB or so, does this mean that using a
                              > virtual RAM disk will have no benefit on any computer, not just my
                              > slow, bloated one?

                              If you have enough RAM, yes. Windows will effectively emulate a RAM disk
                              by means of file caching.

                              > Also, if everything is happening in RAM already
                              > with a small stitch, why is there a hard disk bottleneck at all?

                              For small panoramas there should not be a hard disk bottleneck, assuming
                              that disk caching is enabled for your temp drive (thanks to Bernhard for
                              pointing that out..)

                              > What I'm trying to decide at the moment is whether it's worth
                              > investing in something like the iRAM disks, or whether virtual RAM
                              > disks would do the same thing but better. These iRAM disks seem to
                              > be an order of magnitude faster or more than the fastest spinning
                              > disks. Any insights would be appreciated.

                              I think it will not make much difference, but the only way to find out
                              is by trying. But that could be an expensive experiment, I agree..

                              Joost
                            • Bernhard Vogl
                              ... Well it s not as simple ;-) - quarter stroking *can* speed up head positioning as long as you don t use the other portions of the disk. But similar
                              Message 14 of 19 , Dec 5, 2007
                              View Source
                              • 0 Attachment
                                > Another trick I've read about is called quarter stroking your disk.
                                > Apparently you can create a partition on a 7500 RPM disk such that
                                > the partition is only using the inner portion of the disk, so that
                                > the arm doesnt move so far. Doing this is apparently faster than
                                > using a 15k SAS disk, and much cheaper, and plus you still get the
                                > slow part of the disk to use for storage.

                                Well it's not as simple ;-) - "quarter stroking" *can* speed up head positioning as long as you don't use the other portions of the disk. But similar concepts are implemented in nearly every file system driver. E.g. the middle of the disk is used to store directory information and the data is arranged around it. Chunks of a file are stored near to each other as long as there's enough space left for an optimal placement. Only if the filesystem is nearly full, the chunks are placed non-optimal.
                                This is the reason why you never should fill a files system above approx 70%.
                                Another thing to point out: It's not the inner side of the disk which is the fastest, it's the outer side. The disk surface below the head has an higher speed here, so you can store more data in a given time than on the inner side.

                                Just in case i didn't already mention: I strongly suggest everyone who is not perfectly sure what he's doing to use a standard setup instead of tweaking the system. There are chances that the system become worse than before...

                                Best regards
                                Bernhard
                              Your message has been successfully submitted and would be delivered to recipients shortly.