Loading ...
Sorry, an error occurred while loading the content.

Re: [nslu2-general] Backup thoughts

Expand Messages
  • Peter Chant
    ... Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for anything but rsync, gallery was too slow (not overclocked this one yet). No contraints
    Message 1 of 9 , Oct 31, 2007
    • 0 Attachment
      On Tuesday 30 October 2007, Mike (mwester) wrote:

      >Could you share with us what firmware you are currently running?  Do you
      >have constraints on which firmware would be acceptable?  Does the device
      >need to do anything other than the backup you speak of?  How many disks do
      >you currently have, and how are they partitioned?  Do you have swap space
      >enabled?

      Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for anything
      but rsync, gallery was too slow (not overclocked this one yet). No
      contraints on acceptable firmware - been meaning to try debian slug (or
      another variant but not got around to it. Really, backup is the primary use
      for it - I don't think the distro would affect that too much.

      1 disk, from fdisk -l /dev/sda


      Disk /dev/sda: 255 heads, 63 sectors, 36483 cylinders
      Units = cylinders of 16065 * 512 bytes

      Device Boot Start End Blocks Id System
      /dev/sda1 1 36461 292872951 83 Linux
      /dev/sda2 36462 36476 120487+ 83 Linux
      /dev/sda3 36477 36483 56227+ 82 Linux swap

      Swap is listed in fstab, but is not being used (swapon -s) nor can I find
      anywhere in the init scripts that enables the swap. Hmm need to sort that.

      >There's no reason that the slug cannot be a fine backup device.  My main
      >NSLU2 file server recently served as the temporary storage for my main Linux
      >desktop system's data while that system was being replaced -- back-up and
      >restore of that involved approx 100 GBytes of data, most of which was
      >transferred via rsync running as a daemon on the NSLU2.  The NSLU2 performed
      >that task with no problems at all, including doing its own internal
      >drive-to-drive backup of the data to its second drive.  (The uptime for that
      >NSLU2 is now slightly longer than one year, so it has proven to be
      >remarkably stable in its role as a file server here.)

      Good to know. Did you just do something along the lines of:

      rsync -avz --exclude-from=<exclude file> / slug.<localdomain>::backup/ ?

      That is certainly what I want to be able to do. In fact the slug did save me
      a month or two back when my data drive went down - just that rsync is playing
      up again on my home directory.

      Pete



      --
      Peter Chant
      http://www.petezilla.co.uk
    • Marcel Nijenhof
      ... What version of rsync? ... Please verify it with free . A other test would be to run top during the rsync. -- marceln
      Message 2 of 9 , Nov 2, 2007
      • 0 Attachment
        On Wed, 2007-10-31 at 09:35 +0000, Peter Chant wrote:

        > Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for
        > anything but rsync, gallery was too slow (not overclocked this one
        > yet).

        What version of rsync?

        > Swap is listed in fstab, but is not being used (swapon -s) nor can I
        > find anywhere in the init scripts that enables the swap. Hmm need to
        > sort that.

        Please verify it with "free".

        A other test would be to run "top" during the rsync.

        --
        marceln
      • Peter Chant
        ... On the slug: bash-2.05b# rsync --version rsync version 2.6.3 protocol version 28 On slackware 12.0: root@phoenix:~# rsync --version rsync version 2.6.9
        Message 3 of 9 , Nov 6, 2007
        • 0 Attachment
          On Friday 02 November 2007, Marcel Nijenhof wrote:
          > On Wed, 2007-10-31 at 09:35 +0000, Peter Chant wrote:
          > > Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for
          > > anything but rsync, gallery was too slow (not overclocked this one
          > > yet).
          >
          > What version of rsync?

          On the slug:

          bash-2.05b# rsync --version
          rsync version 2.6.3 protocol version 28

          On slackware 12.0:

          root@phoenix:~# rsync --version
          rsync version 2.6.9 protocol version 29

          >
          > > Swap is listed in fstab, but is not being used (swapon -s) nor can I
          > > find anywhere in the init scripts that enables the swap. Hmm need to
          > > sort that.
          >
          > Please verify it with "free".

          BusyBox v0.60.4 (2004.07.01-03:05+0000) Built-in shell (ash)
          Enter 'help' for a list of built-in commands.

          # bash
          bash-2.05b# free
          total used free shared buffers
          Mem: 30520 29796 724 0 11556
          Swap: 56220 20596 35624
          Total: 86740 50392 36348
          bash-2.05b#

          with rsync running, backing up /usr on the big machine - not the particular
          invocation that caused the problem, but I would expect a fairly heavy load.

          Hmm, going down:

          bash-2.05b# free
          total used free shared buffers
          Mem: 30520 30000 520 0 9396
          Swap: 56220 36916 19304
          Total: 86740 66916 19824
          bash-2.05b#

          >
          > A other test would be to run "top" during the rsync.

          On the slug - not particularly high.

          Tasks: 50 total, 2 running, 47 sleeping, 0 stopped, 1 zombie
          Cpu(s): 3.4% user, 2.4% system, 0.0% nice, 94.2% idle
          Mem: 30520k total, 29960k used, 560k free, 13332k buffers
          Swap: 56220k total, 46344k used, 9876k free, 1900k cached

          PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
          4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
          4676 root 14 0 876 876 688 R 13.6 2.9 0:00.22 top
          10 root 9 0 0 0 0 S 1.7 0.0 10:25.04 usb-storage-1
          1 root 8 0 68 8 8 S 0.0 0.0 0:09.64 init
          2 root 9 0 0 0 0 S 0.0 0.0 0:00.55 keventd
          3 root 19 19 0 0 0 R 0.0 0.0 3:34.82 ksoftirqd_CPU0
          4 root 9 0 0 0 0 S 0.0 0.0 16:46.14 kswapd
          5 root 9 0 0 0 0 S 0.0 0.0 0:05.24 bdflush
          6 root 9 0 0 0 0 S 0.0 0.0 0:01.28 kupdated
          7 root 9 0 0 0 0 S 0.0 0.0 0:00.00 mtdblockd
          8 root 9 0 0 0 0 S 0.0 0.0 0:00.07 khubd
          9 root 14 10 0 0 0 S 0.0 0.0 0:00.00 jffs2_gcd_mtd4
          11 root 9 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
          29 root 9 0 0 0 0 S 0.0 0.0 0:55.04 kjournald
          53 root 9 0 0 0 0 D 0.0 0.0 0:00.31 ixp425_csr
          54 root 9 0 0 0 0 S 0.0 0.0 0:00.30 ixp425 ixp0
          57 root 9 0 84 0 0 S 0.0 0.0 0:00.02 sh
          bash-2.05b#

          OK, got 30% memory used by rsync a little later on.

          Hmm, seems to get < 30%

          --
          Peter Chant
          http://www.petezilla.co.uk
        • Marcel Nijenhof
          ... The version i used in my test (and the current ipkg version as well): $ rsync --version rsync version 2.6.9 protocol version 29 ... So swap usage
          Message 4 of 9 , Nov 6, 2007
          • 0 Attachment
            On Tue, 2007-11-06 at 20:45 +0000, Peter Chant wrote:

            > On the slug:
            >
            > bash-2.05b# rsync --version
            > rsync version 2.6.3 protocol version 28
            >

            The version i used in my test (and the current ipkg version as well):
            $ rsync --version
            rsync version 2.6.9 protocol version 29


            > # bash
            > bash-2.05b# free
            > total used free shared buffers
            > Mem: 30520 29796 724 0 11556
            > Swap: 56220 20596 35624
            > Total: 86740 50392 36348
            > bash-2.05b#
            >
            > with rsync running, backing up /usr on the big machine - not the
            > particular invocation that caused the problem, but I would expect a
            > fairly heavy load.
            >
            > Hmm, going down:
            >
            > bash-2.05b# free
            > total used free shared buffers
            > Mem: 30520 30000 520 0 9396
            > Swap: 56220 36916 19304
            > Total: 86740 66916 19824

            So swap usage increase 16 mb during this run!
            >
            >
            > >
            > > A other test would be to run "top" during the rsync.
            >
            > On the slug - not particularly high.
            >
            > Tasks: 50 total, 2 running, 47 sleeping, 0 stopped, 1 zombie
            > Cpu(s): 3.4% user, 2.4% system, 0.0% nice, 94.2% idle
            > Mem: 30520k total, 29960k used, 560k free, 13332k buffers
            > Swap: 56220k total, 46344k used, 9876k free, 1900k cached
            >
            > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
            > 4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
            ^^^^^

            So indeed rsync is using 26 mb of virtual memory.

            This is just the proof we need that rsync is causing the problems!

            Could you try to upgrade to version 2.6.9 which should be installed
            after:
            ipkg update
            ipkg upgrade rsync

            and see if the problem still exists.

            Because i can't reproduce your problem with that version and there is
            a comment in the changelog which says:
            - Reduced the amount of stack memory needed for each level of
            directory recursion by nearly MAXPATHLEN bytes.

            See:
            http://rsync.samba.org/ftp/rsync/old-versions/rsync-2.6.7-NEWS

            --
            marceln
          • Peter Chant
            ... Done. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync OK, while it was
            Message 5 of 9 , Nov 6, 2007
            • 0 Attachment
              On Tuesday 06 November 2007, Marcel Nijenhof wrote:

              > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
              > > 4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
              >
              > ^^^^^
              >
              > So indeed rsync is using 26 mb of virtual memory.
              >
              > This is just the proof we need that rsync is causing the problems!
              >
              > Could you try to upgrade to version 2.6.9 which should be installed
              > after:

              Done.

              PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
              4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync

              OK, while it was scanning the slug's hard disk when backing up /usr. 11MB is
              a bit difference on a machine with not much ram. Will report back when the
              backup job completes.

              Pete

              --
              Peter Chant
              http://www.petezilla.co.uk
            • Peter Chant
              ... Looks like it did the trick, just re-running the backup script piping its output to a log file to confirm. Thanks, Pete -- Peter Chant
              Message 6 of 9 , Nov 7, 2007
              • 0 Attachment
                > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                > 4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync
                >
                > OK, while it was scanning the slug's hard disk when backing up /usr. 11MB
                > is a bit difference on a machine with not much ram. Will report back when
                > the backup job completes.

                Looks like it did the trick, just re-running the backup script piping its
                output to a log file to confirm.

                Thanks,

                Pete




                --
                Peter Chant
                http://www.petezilla.co.uk
              • Peter Chant
                ... Yep, did the trick. Pete -- Peter Chant http://www.petezilla.co.uk
                Message 7 of 9 , Nov 7, 2007
                • 0 Attachment
                  On Wednesday 07 November 2007, Peter Chant wrote:

                  > Looks like it did the trick, just re-running the backup script piping its
                  > output to a log file to confirm.

                  Yep, did the trick.

                  Pete


                  --
                  Peter Chant
                  http://www.petezilla.co.uk
                Your message has been successfully submitted and would be delivered to recipients shortly.