Loading ...
Sorry, an error occurred while loading the content.

Life after NSLU2?

Expand Messages
  • Carl Zetie
    The NSLU2 that I use for its originally intended purpose, i.e. as a networked file share, is on its last legs. What would people suggest as a replacement for
    Message 1 of 18 , Mar 27 11:28 AM
    • 0 Attachment
      The NSLU2 that I use for its originally intended purpose, i.e. as a
      networked file share, is on its last legs. What would people suggest as a
      replacement for basic network-attached file-sharing and backup?

      I'd like to be able to reuse my existing disks if possible, consisting of a
      500GB LaCie BigDisk (the old version of
      this<http://www.lacie.com/us/products/product.htm?pid=10922>)
      and two 250GB Fantom drives, all in USB2.0 (only) enclosures. I don't mind
      cracking open the enclosures and installing the drives in a NAS enclosure if
      people think that's a good route to go, but I don't know if the LaCie will
      fit...

      I don't care about wireless as my house has CAT5E everywhere!

      TIA,
      Carl


      --
      View my personal blog at TheCzarDictates.blogspot.com

      The contents of this message represent the
      personal opinion of Carl Zetie and do not
      reflect the opinions of his employer


      [Non-text portions of this message have been removed]
    • docbillnet
      ... as a ... Why not buy another NSLU2? Bill
      Message 2 of 18 , Mar 27 1:14 PM
      • 0 Attachment
        --- In nslu2-general@yahoogroups.com, "Carl Zetie" <carl.zetie@...> wrote:
        >
        > The NSLU2 that I use for its originally intended purpose, i.e. as a
        > networked file share, is on its last legs. What would people suggest
        as a
        > replacement for basic network-attached file-sharing and backup?

        Why not buy another NSLU2?

        Bill
      • Klaus Ruebsam
        How about the Longshine LCS-8311, Gigabit SATA NAS, AV-Server, Itunes-Server, Bittorrent-Client Network Storage Server - 10/100/1000 MBit/s - 1 x RJ-45 Port -
        Message 3 of 18 , Mar 27 1:27 PM
        • 0 Attachment
          How about the Longshine

          LCS-8311, Gigabit SATA NAS, AV-Server, Itunes-Server, Bittorrent-Client

          Network Storage Server
          - 10/100/1000 MBit/s
          - 1 x RJ-45 Port
          - 2 x USB 2.0 Ports
          - 1 x eSATA Port
          - SATA HD with up to 750GB
          - Webserver; FTP Server; Printserver
          - AV Server; ITunes Server; P2P Client

          Sells here in Europe for abt 100 EUR and has double the size of RAM (Flash :
          4M, SDRAM : 64M).

          More details:
          http://www.longshine.de/longshine/products/storage/8311/8311_eng.pdf

          BTW: I´m still happy with what I have (NSLU2)

          Best regards,

          Klaus

          -----Ursprüngliche Nachricht-----
          Von: nslu2-general@yahoogroups.com [mailto:nslu2-general@yahoogroups.com] Im
          Auftrag von docbillnet
          Gesendet: Donnerstag, 27. März 2008 21:14
          An: nslu2-general@yahoogroups.com
          Betreff: [nslu2-general] Re: Life after NSLU2?

          --- In nslu2-general@yahoogroups.com, "Carl Zetie" <carl.zetie@...> wrote:
          >
          > The NSLU2 that I use for its originally intended purpose, i.e. as a
          > networked file share, is on its last legs. What would people suggest
          as a
          > replacement for basic network-attached file-sharing and backup?

          Why not buy another NSLU2?

          Bill



          ------------------------------------

          Yahoo! Groups Links
        • Carl Zetie
          ... Because the file transfer speed is just too slow. It s OK for overnight backups where throughput really doesn t matter, and as a learning platform for
          Message 4 of 18 , Mar 28 6:40 PM
          • 0 Attachment
            docbillnet:
            >Why not buy another NSLU2?
            Because the file transfer speed is just too slow. It's OK for overnight
            backups where throughput really doesn't matter, and as a learning platform
            for Linux, but as a file server it just doesn't cut it. When the USB
            interface runs at (theoretically) 480Mbps and the network interface is
            capable of 100Mbps (and I'm thinking of upgrading to gigabit Ethernet), file
            transfer speeds that top out at just a handful of Mbps is pretty
            disappointing.

            By the way, if anybody does want another one thingfling.com has them in its
            rotation again at $60 (new, not refurb.)

            regards,
            Carl
            --
            View my personal blog at TheCzarDictates.blogspot.com

            The contents of this message represent the
            personal opinion of Carl Zetie and do not
            reflect the opinions of his employer


            [Non-text portions of this message have been removed]
          • Carl Lowenstein
            ... By the way, one should be careful about megabits and megabytes. The USB interface is 480 megabits per second, which is 60 megabytes per second. The
            Message 5 of 18 , Mar 28 9:32 PM
            • 0 Attachment
              On 3/28/08, Carl Zetie <carl.zetie@...> wrote:
              >
              > docbillnet:
              > >Why not buy another NSLU2?
              > Because the file transfer speed is just too slow. It's OK for overnight
              > backups where throughput really doesn't matter, and as a learning platform
              > for Linux, but as a file server it just doesn't cut it. When the USB
              > interface runs at (theoretically) 480Mbps and the network interface is
              > capable of 100Mbps (and I'm thinking of upgrading to gigabit Ethernet), file
              > transfer speeds that top out at just a handful of Mbps is pretty
              > disappointing.

              By the way, one should be careful about megabits and megabytes. The
              USB interface is 480 megabits per second, which is 60 megabytes per
              second. The ethernet at 100 megabits per second actually tops out at
              about 10 megabytes per second.

              Achieved data transfer rates using dd(1) between two USB disks seem to
              top out at 10 megabytes per second. Every byte is handled twice by
              the CPU, once for reading and once for writing.. If one uses rsync,
              the extra checksum calculations saturate the slow CPU in the NSLU2 and
              the disk to disk rate drops to 3 megabytes per second.

              So yes, it isn't the worlds fastest CPU and asking it to do all this
              extra stuff makes the over-all transfer rate pretty slow.

              But remeber that a handful of megabytes is eight hands full of megabits.

              carl
              --
              carl lowenstein
              marine physical lab, u.c. san diego
              clowenstein@...
            • hanifadhel
              ... overnight ... learning platform ... the USB ... interface is ... Ethernet), file ... The ... at ... seem to ... by ... rsync, ... and ... this ...
              Message 6 of 18 , Mar 28 10:08 PM
              • 0 Attachment
                --- In nslu2-general@yahoogroups.com, "Carl Lowenstein"
                <clowenstein@...> wrote:
                >
                > On 3/28/08, Carl Zetie <carl.zetie@...> wrote:
                > >
                > > docbillnet:
                > > >Why not buy another NSLU2?
                > > Because the file transfer speed is just too slow. It's OK for
                overnight
                > > backups where throughput really doesn't matter, and as a
                learning platform
                > > for Linux, but as a file server it just doesn't cut it. When
                the USB
                > > interface runs at (theoretically) 480Mbps and the network
                interface is
                > > capable of 100Mbps (and I'm thinking of upgrading to gigabit
                Ethernet), file
                > > transfer speeds that top out at just a handful of Mbps is pretty
                > > disappointing.
                >
                > By the way, one should be careful about megabits and megabytes.
                The
                > USB interface is 480 megabits per second, which is 60 megabytes per
                > second. The ethernet at 100 megabits per second actually tops out
                at
                > about 10 megabytes per second.
                >
                > Achieved data transfer rates using dd(1) between two USB disks
                seem to
                > top out at 10 megabytes per second. Every byte is handled twice
                by
                > the CPU, once for reading and once for writing.. If one uses
                rsync,
                > the extra checksum calculations saturate the slow CPU in the NSLU2
                and
                > the disk to disk rate drops to 3 megabytes per second.
                >
                > So yes, it isn't the worlds fastest CPU and asking it to do all
                this
                > extra stuff makes the over-all transfer rate pretty slow.
                >
                > But remeber that a handful of megabytes is eight hands full of
                megabits.
                >
                > carl
                > --
                > carl lowenstein
                > marine physical lab, u.c. san diego
                > clowenstein@...
                >
              • Paul Bonsor
                I can thoroughly recommend the Synology DS107, and I d all there other products are the same. Paul B On Sat, Mar 29, 2008 at 4:32 AM, Carl Lowenstein
                Message 7 of 18 , Mar 29 4:25 AM
                • 0 Attachment
                  I can thoroughly recommend the Synology DS107, and I'd all there other
                  products are the same.

                  Paul B

                  On Sat, Mar 29, 2008 at 4:32 AM, Carl Lowenstein <clowenstein@...>
                  wrote:

                  > On 3/28/08, Carl Zetie <carl.zetie@... <carl.zetie%40gmail.com>>
                  > wrote:
                  > >
                  > > docbillnet:
                  > > >Why not buy another NSLU2?
                  > > Because the file transfer speed is just too slow. It's OK for overnight
                  > > backups where throughput really doesn't matter, and as a learning
                  > platform
                  > > for Linux, but as a file server it just doesn't cut it. When the USB
                  > > interface runs at (theoretically) 480Mbps and the network interface is
                  > > capable of 100Mbps (and I'm thinking of upgrading to gigabit Ethernet),
                  > file
                  > > transfer speeds that top out at just a handful of Mbps is pretty
                  > > disappointing.
                  >
                  > By the way, one should be careful about megabits and megabytes. The
                  > USB interface is 480 megabits per second, which is 60 megabytes per
                  > second. The ethernet at 100 megabits per second actually tops out at
                  > about 10 megabytes per second.
                  >
                  > Achieved data transfer rates using dd(1) between two USB disks seem to
                  > top out at 10 megabytes per second. Every byte is handled twice by
                  > the CPU, once for reading and once for writing.. If one uses rsync,
                  > the extra checksum calculations saturate the slow CPU in the NSLU2 and
                  > the disk to disk rate drops to 3 megabytes per second.
                  >
                  > So yes, it isn't the worlds fastest CPU and asking it to do all this
                  > extra stuff makes the over-all transfer rate pretty slow.
                  >
                  > But remeber that a handful of megabytes is eight hands full of megabits.
                  >
                  > carl
                  > --
                  > carl lowenstein
                  > marine physical lab, u.c. san diego
                  > clowenstein@... <clowenstein%40ucsd.edu>
                  >
                  >


                  [Non-text portions of this message have been removed]
                • docbillnet
                  OK. Then it is worth doing some measurements to see what is acceptable to you. First off, note that you can speed up rsync by running it over nfs, rather
                  Message 8 of 18 , Mar 29 8:35 AM
                  • 0 Attachment
                    OK. Then it is worth doing some measurements to see what is
                    acceptable to you. First off, note that you can speed up rsync by
                    running it over nfs, rather than directly writing to the NSLU2. This
                    approximately doubles your speed. I just ran a quick test on my
                    Fedora8 desktop:

                    [docbill@hartnell ~]$ sudo rsync --progress /tmp/1956\ 1\ Ten\
                    Commandments\ \(Part\ 1\).avi /net/192.168.2.11/share/.
                    1956 1 Ten Commandments (Part 1).avi
                    1688447022 100% 4.40MB/s 0:06:05 (xfer#1, to-check=0/1)

                    sent 1688653247 bytes received 42 bytes 4280489.96 bytes/sec

                    Then it is worth noting how much better a faster CPU will help. I can
                    test this by connecting to a second desktop machine in my home office.
                    (I have one for work, and one for personal use.)

                    [docbill@hartnell ~]$ sudo rsync --progress /tmp/1956\ 1\ Ten\
                    Commandments\ \(Part\ 1\).avi /net/192.168.2.202/share/.
                    1956 1 Ten Commandments (Part 1).avi
                    1688447022 100% 9.58MB/s 0:02:48 (xfer#1, to-check=0/1)

                    sent 1688653247 bytes received 42 bytes 9355419.88 bytes/sec

                    It looks like you can double your speed with a device that has a
                    faster CPU. To gain further performance, you will need to use 1000
                    Mbps networking. That means your computer, your NAS device, and
                    router will all need to support this speed. I would expect that to
                    approximately double your backup speed, because at that point you will
                    be bottlenecked by your USB enclosure. I have yet to find a USB
                    enclosure that can transfer the data faster than about 20 megabytes
                    per second. You might be able to get another factor of two
                    improvement if your NAS supports Sata or eSata. However, none of he
                    NAS devices I have tested get those kind of speed...

                    How much of a speed improvement are you looking for?

                    2x - mount the drives with NFS.
                    4x - NAS with a faster CPU and NFS support.
                    8x - NAS with a faster CPU, NFS support, and 1000 Mbps.
                    more than 8x - NAS with a faster CPU, NFS support, 1000 Mbps, and
                    Sata/eSata

                    I hope this helps you make a decision on what to buy.

                    Bill

                    p.s. Don't be fooled by 802.11n wireless devices. You won't get
                    speeds any faster than 100Mbps.
                  • bloedmann999
                    ... overnight ... interface is ... Ethernet), ... megabits. ... Hi, ... I posted my test results about this yesterday. But, I am running rsync between the two
                    Message 9 of 18 , Mar 29 10:08 AM
                    • 0 Attachment
                      --- In nslu2-general@yahoogroups.com, "Paul Bonsor" <pb89552@...> wrote:
                      >
                      > I can thoroughly recommend the Synology DS107, and I'd all there other
                      > products are the same.
                      >
                      > Paul B
                      >
                      > On Sat, Mar 29, 2008 at 4:32 AM, Carl Lowenstein <clowenstein@...>
                      > wrote:
                      >
                      > > On 3/28/08, Carl Zetie <carl.zetie@... <carl.zetie%40gmail.com>>
                      > > wrote:
                      > > >
                      > > > docbillnet:
                      > > > >Why not buy another NSLU2?
                      > > > Because the file transfer speed is just too slow. It's OK for
                      overnight
                      > > > backups where throughput really doesn't matter, and as a learning
                      > > platform
                      > > > for Linux, but as a file server it just doesn't cut it. When the USB
                      > > > interface runs at (theoretically) 480Mbps and the network
                      interface is
                      > > > capable of 100Mbps (and I'm thinking of upgrading to gigabit
                      Ethernet),
                      > > file
                      > > > transfer speeds that top out at just a handful of Mbps is pretty
                      > > > disappointing.
                      > >
                      > > By the way, one should be careful about megabits and megabytes. The
                      > > USB interface is 480 megabits per second, which is 60 megabytes per
                      > > second. The ethernet at 100 megabits per second actually tops out at
                      > > about 10 megabytes per second.
                      > >
                      > > Achieved data transfer rates using dd(1) between two USB disks seem to
                      > > top out at 10 megabytes per second. Every byte is handled twice by
                      > > the CPU, once for reading and once for writing.. If one uses rsync,
                      > > the extra checksum calculations saturate the slow CPU in the NSLU2 and
                      > > the disk to disk rate drops to 3 megabytes per second.
                      > >
                      > > So yes, it isn't the worlds fastest CPU and asking it to do all this
                      > > extra stuff makes the over-all transfer rate pretty slow.
                      > >
                      > > But remeber that a handful of megabytes is eight hands full of
                      megabits.
                      > >
                      > > carl
                      > > --
                      > > carl lowenstein
                      > > marine physical lab, u.c. san diego
                      > > clowenstein@... <clowenstein%40ucsd.edu>
                      > >
                      > >
                      >
                      >
                      > [Non-text portions of this message have been removed]
                      >
                      Hi,


                      > Achieved data transfer rates using dd(1) between two USB disks seem to
                      > top out at 10 megabytes per second. Every byte is handled twice by
                      > the CPU, once for reading and once for writing.. If one uses rsync,
                      > the extra checksum calculations saturate the slow CPU in the NSLU2 and
                      > the disk to disk rate drops to 3 megabytes per second.
                      >

                      I posted my test results about this yesterday. But, I am running rsync
                      between the two USB drives with --size-only specified, therefore there
                      should be very little extra work for rsync to do, apart from copying
                      the files. Still, my CPU is pegged at 100% and the xfer rate goes down
                      to the approx. 3MByte/Sec seen.

                      I wonder why?

                      Cheers Brian
                    • Rob Lockhart
                      On Sat, Mar 29, 2008 at 12:32 AM, Carl Lowenstein wrote: {SNIP} Achieved data transfer rates using dd(1) between two USB disks seem to
                      Message 10 of 18 , Mar 29 11:34 AM
                      • 0 Attachment
                        On Sat, Mar 29, 2008 at 12:32 AM, Carl Lowenstein <clowenstein@...>
                        wrote:

                        {SNIP}

                        Achieved data transfer rates using dd(1) between two USB disks seem to
                        > top out at 10 megabytes per second. Every byte is handled twice by
                        > the CPU, once for reading and once for writing.. If one uses rsync,
                        > the extra checksum calculations saturate the slow CPU in the NSLU2 and
                        > the disk to disk rate drops to 3 megabytes per second.
                        >

                        Note that I use rsync to mirror between two EXT3 USB drives on NSLU2, and
                        don't use the "-c" option (uses CRC checksum). If you use "rsync -va $SRC
                        $DST" then it would only look at the dates, mod times, sizes, etc. in
                        determining if files would be transferred or not. The "-c" option would
                        first calculate the CRC of both files (or just the source, I believe, if
                        dest doesn't exist), and then transfer the file from SRC to DST if CRCs were
                        different. So, for one transfer of slightly different files (assume file
                        size, date, etc. were all the same), the SRC file is read, CRC check done,
                        the DST file is read, CRC done, then the file is transferred from SRC to
                        DST. Indeed, the NSLU2 is transferring one file 4 times, versus one file 2
                        times for the non-CRC checked version. I doubt you'll be able to get
                        throughput of 3MB/sec in CRC check mode (more likely <1MB/sec or worse due
                        to overhead). I often see ~2-3MB/sec in non-CRC check mode though
                        (drive-to-drive rsync). The fastest transfers are from my linux server to
                        the NSLU2 via NFS at about 4MB/sec.


                        [Non-text portions of this message have been removed]
                      • Carl Zetie
                        I dunno, maybe I m just cranky because I ve always loved the NSLU2 in concept, and I m disappointed that Linksys didn t follow through. When I think how much
                        Message 11 of 18 , Mar 29 3:22 PM
                        • 0 Attachment
                          I dunno, maybe I'm just cranky because I've always loved the NSLU2 in
                          concept, and I'm disappointed that Linksys didn't follow through. When I
                          think how much the cost of memory has come down and the performance of
                          processors has gone up since the NSLU2 was first designed, I'd kinda hoped
                          we'd be looking at a $100 NSLU3 or NSLU4 by now, with performance limited
                          more by the Ethernet i/f than the CPU, maybe with Firewire800 or even eSata
                          sockets.

                          While I'm dreaming, I'd also like a pony.


                          carl lowenstein pointed out (among other things)...
                          >But reme[m]ber that a handful of megabytes is eight hands full of megabits.
                          Fair enough, but... I would like to get a bit closer to the theoretical
                          maximum. I cracked open the two USB2 enclosures I'm thinking of repurposing
                          and found vintage DiamondMax Plux 9 SATA 1 drives inside. So the claimed
                          performance for the drive itself is 150Mbps, (a long way short of the SATA 1
                          limit) although this
                          test<http://modtown.co.uk/mt/review2.php?id=maxtor200sata&p=3>says
                          that 110Mbps is more realistic. Either way, that's a pretty good match
                          to the theoretical network capacity of 100Mbps, or 12.5MBps (and yeah,
                          nobody is really going to get that in the real world for lots of reasons).
                          It seems like the NSLU2 gets about one quarter to one third of that, i.e. 3
                          to 4 MBps (and the NAS200, which I guess is Linksys' intended
                          successor, is barely
                          better <http://www.smallnetbuilder.com/content/view/30127/75/1/2/>). It
                          looks like d-link, synology, and some of the other vendors other posters
                          mentioned can achieve more like 6 to 8 MBps.

                          So in answer to the poster who wisely asked, "what do you really need from
                          the device?", my list would look something like this:

                          - Performance much closer to the interface's limit, but at least 6MBps
                          - Performs "adequately" from native Windows apps like Winamp, not just rsync
                          - Doesn't crap out during intensive disk-to-disk copies, e.g. syncing the
                          MP3 library with synctoys

                          I can't really justify the cost of gigabit ethernet and, more importantly,
                          disks fast enough to task it (having just got done upgrading my wife's tower
                          to Sata II and 1TB of disk...). At least, not this year...



                          --
                          View my personal blog at TheCzarDictates.blogspot.com

                          The contents of this message represent the
                          personal opinion of Carl Zetie and do not
                          reflect the opinions of his employer


                          [Non-text portions of this message have been removed]
                        • dongilles
                          I ve been running my NSLU2 non-stop 24-7 since they first came out several years ago. I ve still to see anything as simple & reliable as it is. Go buy
                          Message 12 of 18 , Mar 29 7:52 PM
                          • 0 Attachment
                            I've been running my NSLU2 non-stop 24-7 since they first came out
                            several years ago. I've still to see anything as simple & reliable as
                            it is. Go buy another! My only thoughts are to stick with the ext
                            file system and load the 2.3R29 firmware - rock solid. R63 still
                            seems to have problems with large files. I flashed back within hours.
                          • docbillnet
                            ... No matter what options you specify, rsync always computes checksums for files that it transfers. So if you want to transfer files between two USB drives
                            Message 13 of 18 , Mar 31 7:09 AM
                            • 0 Attachment
                              --- In nslu2-general@yahoogroups.com, "bloedmann999"
                              <Brian_Dorling@...> wrote:
                              >
                              > --- In nslu2-general@yahoogroups.com, "Paul Bonsor" <pb89552@> wrote:
                              > I posted my test results about this yesterday. But, I am running rsync
                              > between the two USB drives with --size-only specified, therefore there
                              > should be very little extra work for rsync to do, apart from copying
                              > the files. Still, my CPU is pegged at 100% and the xfer rate goes down
                              > to the approx. 3MByte/Sec seen.
                              >
                              > I wonder why?

                              No matter what options you specify, rsync always computes checksums
                              for files that it transfers. So if you want to transfer files between
                              two USB drives with rysnc, you are best off to at least do the initial
                              sync with the drives plugged into a desktop machine, or have lots of
                              patients.

                              From the rsync manual page:

                              Note that rsync always verifies that each transferred file was
                              correctly reconstructed on the receiving side by checking its
                              whole-file checksum, but that automatic after-the-transfer
                              verification has nothing to do with this option's before-the-transfer
                              "Does this file need to be updated?" check.


                              Bill
                            • Rob Lockhart
                              ... Interesting... I use rsync between two USB HDDs on NSLU2, and use it with cygwin at work and at home. When specifying rsync -vca $SRC $DST it takes a
                              Message 14 of 18 , Mar 31 10:03 PM
                              • 0 Attachment
                                On Mon, Mar 31, 2008 at 10:09 AM, docbillnet <yahoo@...> wrote:

                                > --- In nslu2-general@yahoogroups.com, "bloedmann999"
                                > <Brian_Dorling@...> wrote:
                                > >
                                > > --- In nslu2-general@yahoogroups.com, "Paul Bonsor" <pb89552@> wrote:
                                > > I posted my test results about this yesterday. But, I am running rsync
                                > > between the two USB drives with --size-only specified, therefore there
                                > > should be very little extra work for rsync to do, apart from copying
                                > > the files. Still, my CPU is pegged at 100% and the xfer rate goes down
                                > > to the approx. 3MByte/Sec seen.
                                > >
                                > > I wonder why?
                                >
                                > No matter what options you specify, rsync always computes checksums
                                > for files that it transfers. So if you want to transfer files between
                                > two USB drives with rysnc, you are best off to at least do the initial
                                > sync with the drives plugged into a desktop machine, or have lots of
                                > patients.
                                >
                                > From the rsync manual page:
                                >
                                > Note that rsync always verifies that each transferred file was
                                > correctly reconstructed on the receiving side by checking its
                                > whole-file checksum, but that automatic after-the-transfer
                                > verification has nothing to do with this option's before-the-transfer
                                > "Does this file need to be updated?" check.
                                >

                                Interesting... I use rsync between two USB HDDs on NSLU2, and use it with
                                cygwin at work and at home. When specifying "rsync -vca $SRC $DST" it takes
                                a whole lot longer to do the synchronization than if I specified "rsync -va
                                $SRC $DST". Because of the transfer speeds, there's almost no way that
                                without the "-c" switch, a CRC checksum is being generated.

                                From what I gathered here:

                                http://samba.org/rsync/how-rsync-works.html

                                "The file list includes not only the pathnames but also ownership, mode,
                                permissions, size and modtime. If the --checksum option has been specified
                                it also includes the file checksums."

                                Further on down, it states:
                                "The file's checksum is generated as the temp-file is built. At the end of
                                the file, this checksum is compared with the file checksum from the sender.
                                If the file checksums do not match the temp-file is deleted."

                                I'm wondering if that is correct, as surely CRC checksums can't be generated
                                "quickly" (as compared to multi-GHz processors) with NSLU2. The exact
                                command I use for mirroring from an NFS server is:

                                nice -n 10 /usr/bin/rsync -va --delete --force --stats \
                                --exclude=System\ Volume\ Information --exclude=lost+found \
                                ${SRC}/ ${DST}/ >>/var/log/rsyncbackup.log

                                with $SRC and $DST being of appropriately-named source and destination
                                locations. If you specify "--progress" it'll also show you the transfer
                                stats as they are happening. As was mentioned, however, if "-c" or
                                "--checksum" parameter is given, files existing on both $SRC and $DST are
                                MD4 checksum'ed versus diffs of size and mod times only (below from man
                                page):

                                "-c, --checksum skip based on checksum, not mod-time & size"

                                In cygwin, I use "rsa" and "rsac" depending if I want to transfer files
                                based on mod times or CRC check. This is especially useful as sometimes
                                (for whatever reason) people open and close MS-Excel or MS-Word files (with
                                no apparent modifications made), and the file date isn't modified but the
                                CRC checks are different.

                                $ alias rsa
                                alias rsa='rsync -vrlpt --delete --force --progress --stats
                                --exclude=System\ Volume\ Information --exclude=lost+found -e "ssh -p 23" '

                                $ alias rsac
                                alias rsac='rsync -vrlptc --delete --force --progress --stats
                                --block-size=8192 --exclude=System\ Volume\ Information --exclude=lost+found
                                -e "ssh -p 23" '

                                As an experiment, I tried something below. I generated a 100MB file via
                                /dev/urandom on each of a linux box and NSLU2. Then I forced the time to be
                                the same on both via "touch". Then, I used parameters "-vca" and "-va" as
                                NSLU2 as SRC and LINUXbox as DST. Note that I used the "-n" parameter,
                                which means "dry run" (no file transfer, just go through the motions as if
                                it did).

                                =======================================
                                [rob@Linux ~]$ dd if=/dev/urandom of=100MiB_file bs=100k count=1024
                                1024+0 records in
                                1024+0 records out
                                104857600 bytes (105 MB) copied, 44.3161 seconds, 2.4 MB/s
                                [rob@Linux ~]$ touch -t 200804010026.00 100MiB_file
                                [rob@Linux ~]$ ls -la 100MiB_file
                                -rw-r--r-- 1 rob rob 104857600 Apr 1 00:26 100MiB_file


                                rob@NSLU2:~$ dd if=/dev/urandom of=100MiB_file bs=100k count=1024
                                1024+0 records in
                                1024+0 records out
                                104857600 bytes (105 MB) copied, 264.031 seconds, 397 kB/s
                                rob@NSLU2:~$ touch -t 200804010026.00 100MiB_file
                                rob@NSLU2:~$ ls -la 100MiB_file
                                -rw-r--r-- 1 rob rob 104857600 Apr 1 00:26 100MiB_file

                                [rob@Linux ~]$ date; rsync -van rob@NSLU2:100MiB_file 100MiB_file ; date
                                Tue Apr 1 00:33:05 EDT 2008
                                receiving file list ... done
                                sent 20 bytes received 82 bytes 40.80 bytes/sec
                                total size is 104857600 speedup is 1028015.69
                                Tue Apr 1 00:33:07 EDT 2008

                                [rob@Linux ~]$ date; rsync -vacn rob@NSLU2:100MiB_file 100MiB_file ; date
                                Tue Apr 1 00:33:49 EDT 2008
                                receiving file list ... done
                                100MiB_file
                                sent 26 bytes received 104 bytes 7.43 bytes/sec
                                total size is 104857600 speedup is 806596.92
                                Tue Apr 1 00:34:06 EDT 2008

                                [rob@Linux ~]$ date; rsync -vacn rob@NSLU2:100MiB_file 100MiB_file ; date
                                Tue Apr 1 00:34:34 EDT 2008
                                receiving file list ... done
                                100MiB_file
                                sent 26 bytes received 104 bytes 7.43 bytes/sec
                                total size is 104857600 speedup is 806596.92
                                Tue Apr 1 00:34:51 EDT 2008

                                [rob@Linux ~]$ date; rsync -van rob@NSLU2:100MiB_file 100MiB_file ; date
                                Tue Apr 1 00:35:38 EDT 2008
                                receiving file list ... done
                                sent 20 bytes received 82 bytes 29.14 bytes/sec
                                total size is 104857600 speedup is 1028015.69
                                Tue Apr 1 00:35:41 EDT 2008

                                rob@NSLU2:~$ date; md5sum 100MiB_file ; date
                                Tue Apr 1 00:46:33 EDT 2008
                                62b2c27a678fd3bf07418c9f4f161c93 100MiB_file
                                Tue Apr 1 00:46:42 EDT 2008

                                [rob@Linux ~]$ date; md5sum 100MiB_file ; date
                                Tue Apr 1 00:47:25 EDT 2008
                                2ed67109979cec75d1b155581cdb7cf4 100MiB_file
                                Tue Apr 1 00:47:26 EDT 2008
                                =======================================

                                Observations:
                                1. Note that the bytes received and sent is different with the CRC option:
                                CRC: sent = 6, recd = 22 bytes extra
                                2. Note that with different file contents but identical
                                UID/GID/moddate/size, the file doesn't transfer. The start/stop time is
                                2-3s. This proves a checksum is not being performed, as the file contents
                                are different.
                                3. Note that with the "-c" checksum option, the file is transferred (as
                                UID/GID/moddate/size are ignored per man page), but after a significantly
                                longer time of 17s . This must predominantly be the time it takes to
                                generate the checksum on both ends.
                                4. Note from the MD5 checksums (rsync uses MD4), it is hard to believe that
                                the NSLU2 can read a 100MiB file and generate an MD5 sum in less than 10
                                seconds (>10MiBps); some caching must be occurring? I had heard throughput
                                of ~3-5MiBps for the NSLU2.
                                5. If I understand the process correctly, it appears that the MAN page is
                                wrong in regards to checksum always being calculated for rsync. It won't be
                                the first time a man page is wrong or clear as mud. :-)


                                [Non-text portions of this message have been removed]
                              • docbillnet
                                ... with ... it takes ... rsync -va ... No. That is correct. With the -c option, transfers will be much slower still. The reason being with the -c option
                                Message 15 of 18 , Apr 1, 2008
                                • 0 Attachment
                                  --- In nslu2-general@yahoogroups.com, "Rob Lockhart" <rlockhar@...> wrote:
                                  >
                                  > On Mon, Mar 31, 2008 at 10:09 AM, docbillnet <yahoo@...> wrote:
                                  > > Note that rsync always verifies that each transferred file was
                                  > > correctly reconstructed on the receiving side by checking its
                                  > > whole-file checksum, but that automatic after-the-transfer
                                  > > verification has nothing to do with this option's before-the-transfer
                                  > > "Does this file need to be updated?" check.
                                  >
                                  > Interesting... I use rsync between two USB HDDs on NSLU2, and use it
                                  with
                                  > cygwin at work and at home. When specifying "rsync -vca $SRC $DST"
                                  it takes
                                  > a whole lot longer to do the synchronization than if I specified
                                  "rsync -va
                                  > $SRC $DST". Because of the transfer speeds, there's almost no way that
                                  > without the "-c" switch, a CRC checksum is being generated.

                                  No. That is correct. With the -c option, transfers will be much
                                  slower still. The reason being with the -c option the checksum is
                                  generated by reading each file prior to transfer. This effectively
                                  doubles your IO time, because each file will be read in twice.
                                  Without the -c, rsync computes the checksum while transferring. So
                                  the amount of CPU time is larger than say "cp", but there is no extra
                                  IO time. However, since the CPU on the NSLU2 is bottlenecked by CPU
                                  speed, any additional computations slow down the copy operation.
                                • docbillnet
                                  ... page is ... won t be ... The man page says the checksums are always computed to compare the transfered files. Since you used -n, there was no transfer, so
                                  Message 16 of 18 , Apr 1, 2008
                                  • 0 Attachment
                                    --- In nslu2-general@yahoogroups.com, "Rob Lockhart" <rlockhar@...> wrote:
                                    >
                                    > On Mon, Mar 31, 2008 at 10:09 AM, docbillnet <yahoo@...> wrote:
                                    >
                                    > > --- In nslu2-general@yahoogroups.com, "bloedmann999"
                                    > > <Brian_Dorling@> wrote:
                                    > 5. If I understand the process correctly, it appears that the MAN
                                    page is
                                    > wrong in regards to checksum always being calculated for rsync. It
                                    won't be
                                    > the first time a man page is wrong or clear as mud. :-)

                                    The man page says the checksums are always computed to compare the
                                    transfered files. Since you used -n, there was no transfer, so there
                                    was no checksum. If it doesn't write the temp file, it has nothing to
                                    compute the checksum on.

                                    Bill
                                  • Rob Lockhart
                                    ... I just tried it again, and with touch -t 200804010026.00 100MiB_file on both sides, with *different CRCs*, there was no transfer, even without the -n
                                    Message 17 of 18 , Apr 1, 2008
                                    • 0 Attachment
                                      On Tue, Apr 1, 2008 at 1:42 PM, docbillnet <yahoo@...> wrote:

                                      > --- In nslu2-general@yahoogroups.com, "Rob Lockhart" <rlockhar@...> wrote:
                                      > >
                                      > > On Mon, Mar 31, 2008 at 10:09 AM, docbillnet <yahoo@...> wrote:
                                      > >
                                      > > > --- In nslu2-general@yahoogroups.com, "bloedmann999"
                                      > > > <Brian_Dorling@> wrote:
                                      > > 5. If I understand the process correctly, it appears that the MAN
                                      > page is
                                      > > wrong in regards to checksum always being calculated for rsync.
                                      >
                                      > The man page says the checksums are always computed to compare the
                                      > transfered files. Since you used -n, there was no transfer, so there
                                      > was no checksum. If it doesn't write the temp file, it has nothing to
                                      > compute the checksum on.
                                      >

                                      I just tried it again, and with "touch -t 200804010026.00 100MiB_file" on
                                      both sides, with *different CRCs*, there was no transfer, even without the
                                      "-n" switch. No temp file created, no checksum generated, as there was no
                                      transfer.

                                      rob@NSLU2:~$ md5sum -b 100MiB_file
                                      62b2c27a678fd3bf07418c9f4f161c93 *100MiB_file
                                      rob@NSLU2:~$ touch -t 200804010026.00 100MiB_file

                                      [rob@Linux ~]$ md5sum -b 100MiB_file
                                      2ed67109979cec75d1b155581cdb7cf4 *100MiB_file
                                      [rob@Linux ~]$ touch -t 200804010026.00 100MiB_file

                                      [rob@Linux ~]$ date; rsync -va --delete --progress --stats
                                      rob@NSLU2:100MiB_file
                                      100MiB_file ; date
                                      Tue Apr 1 22:05:01 EDT 2008
                                      receiving file list ...
                                      1 file to consider

                                      Number of files: 1
                                      Number of files transferred: 0
                                      Total file size: 104857600 bytes
                                      Total transferred file size: 0 bytes
                                      Literal data: 0 bytes
                                      Matched data: 0 bytes
                                      File list size: 62
                                      File list generation time: 0.012 seconds
                                      File list transfer time: 0.000 seconds
                                      Total bytes sent: 20
                                      Total bytes received: 82

                                      sent 20 bytes received 82 bytes 40.80 bytes/sec
                                      total size is 104857600 speedup is 1028015.69
                                      Tue Apr 1 22:05:03 EDT 2008

                                      [rob@Linux ~]$ md5sum -b 100MiB_file
                                      2ed67109979cec75d1b155581cdb7cf4 *100MiB_file

                                      rob@NSLU2:~$ md5sum -b 100MiB_file
                                      62b2c27a678fd3bf07418c9f4f161c93 *100MiB_file

                                      Assuming the checksum is generated while the file is being transferred, that
                                      doesn't seem to be the determining factor for whether to transfer files of
                                      identical UID/date/time/size (but different content). It appears that
                                      without the "-c" switch, the CRC is not calculated on files for which there
                                      is no pending transfer. I'm not sure what would be gained in calculating
                                      the CRC on a file during transfer, perhaps only to ensure that it matches
                                      the destination file CRC when being created? Indeed, the man page is
                                      correct regarding CRC of transferred files (based on criterion given to
                                      rsync with which to compare the files).

                                      -Rob

                                      P.S. Apologies for the diversion (thread hijacking), folks. Should've
                                      changed the subject to rsync or similar.


                                      [Non-text portions of this message have been removed]
                                    • docbillnet
                                      ... 100MiB_file on ... without the ... was no ... As predicted. Exactly what I said, the checksum is always computed for transfered files. I am glad you
                                      Message 18 of 18 , Apr 2, 2008
                                      • 0 Attachment
                                        --- In nslu2-general@yahoogroups.com, "Rob Lockhart" <rlockhar@...> wrote:
                                        >
                                        > On Tue, Apr 1, 2008 at 1:42 PM, docbillnet <yahoo@...> wrote:
                                        >
                                        > > --- In nslu2-general@yahoogroups.com, "Rob Lockhart" <rlockhar@>
                                        wrote:
                                        > > The man page says the checksums are always computed to compare the
                                        > > transfered files. Since you used -n, there was no transfer, so there
                                        > > was no checksum. If it doesn't write the temp file, it has nothing to
                                        > > compute the checksum on.
                                        > >
                                        >
                                        > I just tried it again, and with "touch -t 200804010026.00
                                        100MiB_file" on
                                        > both sides, with *different CRCs*, there was no transfer, even
                                        without the
                                        > "-n" switch. No temp file created, no checksum generated, as there
                                        was no
                                        > transfer.

                                        As predicted. Exactly what I said, the checksum is always computed
                                        for transfered files. I am glad you could validate that the manual
                                        page is correct.

                                        > is no pending transfer. I'm not sure what would be gained in
                                        calculating
                                        > the CRC on a file during transfer, perhaps only to ensure that it
                                        matches
                                        > the destination file CRC when being created? Indeed, the man page is

                                        I am assuming that the same code is used to compute checksums while
                                        reading regardless if the file transfer is local or remote. The same
                                        code is also probably used to compute checksums when saving,
                                        regardless of transfer mode. I can see absolutely no advantage in
                                        computing a checksum as your read a file and a checksum as you write a
                                        file locally. If the checksum is changing in your memory blocks, then
                                        no amount of sanity checking will help... I can see the advantage
                                        across a socket or a pipe. TCP-IP packets only use a very small
                                        checksum (I think it is one byte). Which means if you have enough
                                        errors occur you are almost guaranteed that one of them will be
                                        received as valid data. When you consider people will use rsync to
                                        copy many terabytes of data, that means they would regularly corrupt
                                        files with even the rarest of network errors if not for the additional
                                        checksums.

                                        Bill
                                      Your message has been successfully submitted and would be delivered to recipients shortly.