Loading ...
Sorry, an error occurred while loading the content.
 

Backup thoughts

Expand Messages
  • Peter Chant
    I have been backing up my slackware box to my slug using rsync. rsync is running in daemon mode on the slug. I ve had to split the backup into a number of
    Message 1 of 9 , Oct 30, 2007
      I have been backing up my slackware box to my slug using rsync. rsync is
      running in daemon mode on the slug. I've had to split the backup into a
      number of chunks as otherwise rsync crashes - I think the slug runs out of
      memory. Lately this has become a problem again as part of the filesystem on
      my Slackware box has grown. This piecemeal approach is getting v. ackward to
      manage.

      Any thoughts? One plan would seem to be to ignore rsync on the slug and use
      nfs to mount a backup directory on the slug straight to the slack box and
      just use rsync on the slackbox alone to perform the backup. Hopefully this
      will not cause the slug memory problems.

      Alternately I could just plug the usb drive direct to the slack box cutting
      the slug out the loop!

      I am wondering whether I ought to partition my data a bit more, do something
      along the lines of the following:

      /home/<user> Normal run of the mill files,
      backed up daily

      /data/<user + other data> Data that changes less frequently,
      scanned photos etc. Backed up weekly

      /transient/<user + other data>
      ISO's, downloaded packages, file system images
      Large things that are no major issue if lost yet
      take inordinate time and space to backup.

      I do note this is non-standard and a little fiddly. Fitting my data storage
      around my backup method - not the other way.

      I am wonding this as backing up a large collection of my scanned photos
      takes quite a while yet is changes little. I also wonder, even rsync
      seems to give the disk a good workout, more than regular use, does my
      backup routine do as much harm as good.

      I'm looking for a backup solution that enables me to simply mount and copy
      back data in the event of a drive failure.

      Thoughts?

      Thanks,

      Pete

      --
      Peter Chant
      http://www.petezilla.co.uk
    • Mike (mwester)
      ... From: Peter Chant To: Sent: Tuesday, October 30, 2007 5:55 PM Subject: [nslu2-general] Backup
      Message 2 of 9 , Oct 30, 2007
        ----- Original Message -----
        From: "Peter Chant" <pete@...>
        To: <nslu2-general@yahoogroups.com>
        Sent: Tuesday, October 30, 2007 5:55 PM
        Subject: [nslu2-general] Backup thoughts


        > I have been backing up my slackware box to my slug using rsync. rsync is
        > running in daemon mode on the slug. I've had to split the backup into a
        > number of chunks as otherwise rsync crashes - I think the slug runs out of
        > memory. Lately this has become a problem again as part of the filesystem
        on
        > my Slackware box has grown. This piecemeal approach is getting v. ackward
        to
        > manage.
        >
        > Any thoughts? One plan would seem to be to ignore rsync on the slug and
        use
        > nfs to mount a backup directory on the slug straight to the slack box and
        > just use rsync on the slackbox alone to perform the backup. Hopefully
        this
        > will not cause the slug memory problems.
        >
        > Alternately I could just plug the usb drive direct to the slack box
        cutting
        > the slug out the loop!
        >
        > I am wondering whether I ought to partition my data a bit more, do
        something
        > along the lines of the following:
        >
        > /home/<user> Normal run of the mill files,
        > backed up daily
        >
        > /data/<user + other data> Data that changes less frequently,
        > scanned photos etc. Backed up weekly
        >
        > /transient/<user + other data>
        > ISO's, downloaded packages, file system images
        > Large things that are no major issue if lost yet
        > take inordinate time and space to backup.
        >
        > I do note this is non-standard and a little fiddly. Fitting my data
        storage
        > around my backup method - not the other way.
        >
        > I am wonding this as backing up a large collection of my scanned photos
        > takes quite a while yet is changes little. I also wonder, even rsync
        > seems to give the disk a good workout, more than regular use, does my
        > backup routine do as much harm as good.
        >
        > I'm looking for a backup solution that enables me to simply mount and copy
        > back data in the event of a drive failure.
        >
        > Thoughts?

        Could you share with us what firmware you are currently running? Do you
        have constraints on which firmware would be acceptable? Does the device
        need to do anything other than the backup you speak of? How many disks do
        you currently have, and how are they partitioned? Do you have swap space
        enabled?

        There's no reason that the slug cannot be a fine backup device. My main
        NSLU2 file server recently served as the temporary storage for my main Linux
        desktop system's data while that system was being replaced -- back-up and
        restore of that involved approx 100 GBytes of data, most of which was
        transferred via rsync running as a daemon on the NSLU2. The NSLU2 performed
        that task with no problems at all, including doing its own internal
        drive-to-drive backup of the data to its second drive. (The uptime for that
        NSLU2 is now slightly longer than one year, so it has proven to be
        remarkably stable in its role as a file server here.)

        > Thanks,
        >
        > Pete
        >
        > --
        > Peter Chant
        > http://www.petezilla.co.uk
        >

        Mike (mwester)
      • Peter Chant
        ... Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for anything but rsync, gallery was too slow (not overclocked this one yet). No contraints
        Message 3 of 9 , Oct 31, 2007
          On Tuesday 30 October 2007, Mike (mwester) wrote:

          >Could you share with us what firmware you are currently running?  Do you
          >have constraints on which firmware would be acceptable?  Does the device
          >need to do anything other than the backup you speak of?  How many disks do
          >you currently have, and how are they partitioned?  Do you have swap space
          >enabled?

          Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for anything
          but rsync, gallery was too slow (not overclocked this one yet). No
          contraints on acceptable firmware - been meaning to try debian slug (or
          another variant but not got around to it. Really, backup is the primary use
          for it - I don't think the distro would affect that too much.

          1 disk, from fdisk -l /dev/sda


          Disk /dev/sda: 255 heads, 63 sectors, 36483 cylinders
          Units = cylinders of 16065 * 512 bytes

          Device Boot Start End Blocks Id System
          /dev/sda1 1 36461 292872951 83 Linux
          /dev/sda2 36462 36476 120487+ 83 Linux
          /dev/sda3 36477 36483 56227+ 82 Linux swap

          Swap is listed in fstab, but is not being used (swapon -s) nor can I find
          anywhere in the init scripts that enables the swap. Hmm need to sort that.

          >There's no reason that the slug cannot be a fine backup device.  My main
          >NSLU2 file server recently served as the temporary storage for my main Linux
          >desktop system's data while that system was being replaced -- back-up and
          >restore of that involved approx 100 GBytes of data, most of which was
          >transferred via rsync running as a daemon on the NSLU2.  The NSLU2 performed
          >that task with no problems at all, including doing its own internal
          >drive-to-drive backup of the data to its second drive.  (The uptime for that
          >NSLU2 is now slightly longer than one year, so it has proven to be
          >remarkably stable in its role as a file server here.)

          Good to know. Did you just do something along the lines of:

          rsync -avz --exclude-from=<exclude file> / slug.<localdomain>::backup/ ?

          That is certainly what I want to be able to do. In fact the slug did save me
          a month or two back when my data drive went down - just that rsync is playing
          up again on my home directory.

          Pete



          --
          Peter Chant
          http://www.petezilla.co.uk
        • Marcel Nijenhof
          ... What version of rsync? ... Please verify it with free . A other test would be to run top during the rsync. -- marceln
          Message 4 of 9 , Nov 2, 2007
            On Wed, 2007-10-31 at 09:35 +0000, Peter Chant wrote:

            > Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for
            > anything but rsync, gallery was too slow (not overclocked this one
            > yet).

            What version of rsync?

            > Swap is listed in fstab, but is not being used (swapon -s) nor can I
            > find anywhere in the init scripts that enables the swap. Hmm need to
            > sort that.

            Please verify it with "free".

            A other test would be to run "top" during the rsync.

            --
            marceln
          • Peter Chant
            ... On the slug: bash-2.05b# rsync --version rsync version 2.6.3 protocol version 28 On slackware 12.0: root@phoenix:~# rsync --version rsync version 2.6.9
            Message 5 of 9 , Nov 6, 2007
              On Friday 02 November 2007, Marcel Nijenhof wrote:
              > On Wed, 2007-10-31 at 09:35 +0000, Peter Chant wrote:
              > > Quite old, V2.3R29-uNSLUng-5.5-beta, I have not been using it for
              > > anything but rsync, gallery was too slow (not overclocked this one
              > > yet).
              >
              > What version of rsync?

              On the slug:

              bash-2.05b# rsync --version
              rsync version 2.6.3 protocol version 28

              On slackware 12.0:

              root@phoenix:~# rsync --version
              rsync version 2.6.9 protocol version 29

              >
              > > Swap is listed in fstab, but is not being used (swapon -s) nor can I
              > > find anywhere in the init scripts that enables the swap. Hmm need to
              > > sort that.
              >
              > Please verify it with "free".

              BusyBox v0.60.4 (2004.07.01-03:05+0000) Built-in shell (ash)
              Enter 'help' for a list of built-in commands.

              # bash
              bash-2.05b# free
              total used free shared buffers
              Mem: 30520 29796 724 0 11556
              Swap: 56220 20596 35624
              Total: 86740 50392 36348
              bash-2.05b#

              with rsync running, backing up /usr on the big machine - not the particular
              invocation that caused the problem, but I would expect a fairly heavy load.

              Hmm, going down:

              bash-2.05b# free
              total used free shared buffers
              Mem: 30520 30000 520 0 9396
              Swap: 56220 36916 19304
              Total: 86740 66916 19824
              bash-2.05b#

              >
              > A other test would be to run "top" during the rsync.

              On the slug - not particularly high.

              Tasks: 50 total, 2 running, 47 sleeping, 0 stopped, 1 zombie
              Cpu(s): 3.4% user, 2.4% system, 0.0% nice, 94.2% idle
              Mem: 30520k total, 29960k used, 560k free, 13332k buffers
              Swap: 56220k total, 46344k used, 9876k free, 1900k cached

              PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
              4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
              4676 root 14 0 876 876 688 R 13.6 2.9 0:00.22 top
              10 root 9 0 0 0 0 S 1.7 0.0 10:25.04 usb-storage-1
              1 root 8 0 68 8 8 S 0.0 0.0 0:09.64 init
              2 root 9 0 0 0 0 S 0.0 0.0 0:00.55 keventd
              3 root 19 19 0 0 0 R 0.0 0.0 3:34.82 ksoftirqd_CPU0
              4 root 9 0 0 0 0 S 0.0 0.0 16:46.14 kswapd
              5 root 9 0 0 0 0 S 0.0 0.0 0:05.24 bdflush
              6 root 9 0 0 0 0 S 0.0 0.0 0:01.28 kupdated
              7 root 9 0 0 0 0 S 0.0 0.0 0:00.00 mtdblockd
              8 root 9 0 0 0 0 S 0.0 0.0 0:00.07 khubd
              9 root 14 10 0 0 0 S 0.0 0.0 0:00.00 jffs2_gcd_mtd4
              11 root 9 0 0 0 0 S 0.0 0.0 0:00.00 scsi_eh_1
              29 root 9 0 0 0 0 S 0.0 0.0 0:55.04 kjournald
              53 root 9 0 0 0 0 D 0.0 0.0 0:00.31 ixp425_csr
              54 root 9 0 0 0 0 S 0.0 0.0 0:00.30 ixp425 ixp0
              57 root 9 0 84 0 0 S 0.0 0.0 0:00.02 sh
              bash-2.05b#

              OK, got 30% memory used by rsync a little later on.

              Hmm, seems to get < 30%

              --
              Peter Chant
              http://www.petezilla.co.uk
            • Marcel Nijenhof
              ... The version i used in my test (and the current ipkg version as well): $ rsync --version rsync version 2.6.9 protocol version 29 ... So swap usage
              Message 6 of 9 , Nov 6, 2007
                On Tue, 2007-11-06 at 20:45 +0000, Peter Chant wrote:

                > On the slug:
                >
                > bash-2.05b# rsync --version
                > rsync version 2.6.3 protocol version 28
                >

                The version i used in my test (and the current ipkg version as well):
                $ rsync --version
                rsync version 2.6.9 protocol version 29


                > # bash
                > bash-2.05b# free
                > total used free shared buffers
                > Mem: 30520 29796 724 0 11556
                > Swap: 56220 20596 35624
                > Total: 86740 50392 36348
                > bash-2.05b#
                >
                > with rsync running, backing up /usr on the big machine - not the
                > particular invocation that caused the problem, but I would expect a
                > fairly heavy load.
                >
                > Hmm, going down:
                >
                > bash-2.05b# free
                > total used free shared buffers
                > Mem: 30520 30000 520 0 9396
                > Swap: 56220 36916 19304
                > Total: 86740 66916 19824

                So swap usage increase 16 mb during this run!
                >
                >
                > >
                > > A other test would be to run "top" during the rsync.
                >
                > On the slug - not particularly high.
                >
                > Tasks: 50 total, 2 running, 47 sleeping, 0 stopped, 1 zombie
                > Cpu(s): 3.4% user, 2.4% system, 0.0% nice, 94.2% idle
                > Mem: 30520k total, 29960k used, 560k free, 13332k buffers
                > Swap: 56220k total, 46344k used, 9876k free, 1900k cached
                >
                > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                > 4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
                ^^^^^

                So indeed rsync is using 26 mb of virtual memory.

                This is just the proof we need that rsync is causing the problems!

                Could you try to upgrade to version 2.6.9 which should be installed
                after:
                ipkg update
                ipkg upgrade rsync

                and see if the problem still exists.

                Because i can't reproduce your problem with that version and there is
                a comment in the changelog which says:
                - Reduced the amount of stack memory needed for each level of
                directory recursion by nearly MAXPATHLEN bytes.

                See:
                http://rsync.samba.org/ftp/rsync/old-versions/rsync-2.6.7-NEWS

                --
                marceln
              • Peter Chant
                ... Done. PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND 4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync OK, while it was
                Message 7 of 9 , Nov 6, 2007
                  On Tuesday 06 November 2007, Marcel Nijenhof wrote:

                  > > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                  > > 4651 root 15 0 26812 1804 192 D 17.0 5.9 0:49.78 rsync
                  >
                  > ^^^^^
                  >
                  > So indeed rsync is using 26 mb of virtual memory.
                  >
                  > This is just the proof we need that rsync is causing the problems!
                  >
                  > Could you try to upgrade to version 2.6.9 which should be installed
                  > after:

                  Done.

                  PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                  4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync

                  OK, while it was scanning the slug's hard disk when backing up /usr. 11MB is
                  a bit difference on a machine with not much ram. Will report back when the
                  backup job completes.

                  Pete

                  --
                  Peter Chant
                  http://www.petezilla.co.uk
                • Peter Chant
                  ... Looks like it did the trick, just re-running the backup script piping its output to a log file to confirm. Thanks, Pete -- Peter Chant
                  Message 8 of 9 , Nov 7, 2007
                    > PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
                    > 4743 root 18 0 15548 3680 3580 D 19.5 12.1 0:38.72 rsync
                    >
                    > OK, while it was scanning the slug's hard disk when backing up /usr. 11MB
                    > is a bit difference on a machine with not much ram. Will report back when
                    > the backup job completes.

                    Looks like it did the trick, just re-running the backup script piping its
                    output to a log file to confirm.

                    Thanks,

                    Pete




                    --
                    Peter Chant
                    http://www.petezilla.co.uk
                  • Peter Chant
                    ... Yep, did the trick. Pete -- Peter Chant http://www.petezilla.co.uk
                    Message 9 of 9 , Nov 7, 2007
                      On Wednesday 07 November 2007, Peter Chant wrote:

                      > Looks like it did the trick, just re-running the backup script piping its
                      > output to a log file to confirm.

                      Yep, did the trick.

                      Pete


                      --
                      Peter Chant
                      http://www.petezilla.co.uk
                    Your message has been successfully submitted and would be delivered to recipients shortly.