Loading ...
Sorry, an error occurred while loading the content.

Kernel nasties during backup

Expand Messages
  • Drew Kirkpatrick
    This morning I noticed my slug wasn t responding (lights were still on), so I gave it the old reboot and started checking the logs. Looks like I got some some
    Message 1 of 11 , Aug 14, 2008
    • 0 Attachment
      This morning I noticed my slug wasn't responding (lights were still
      on), so I gave it the old reboot and started checking the logs. Looks
      like I got some some bad mojo with my kernel last night while it was
      running it's cron'd backups. These backups have been working
      flawlessly for a few weeks now, so I'm at a loss as to what really
      happened here. Here's the log:

      Aug 14 00:52:35 (none) syslog.info -- MARK --
      Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
      (/root/runDailyBackupScript.sh > /dev/null 2>&1)
      Aug 14 01:12:35 (none) syslog.info -- MARK --
      Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
      (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
      Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
      pointer dereference at virtual address 00000200
      Aug 14 01:16:00 (none) user.alert kernel: pgd = c17fc000
      Aug 14 01:16:00 (none) user.alert kernel: [00000200] *pgd=00902031,
      *pte=00000000, *ppte=00000000
      Aug 14 01:16:00 (none) user.warn kernel: Internal error: Oops: 17 [#1]
      Aug 14 01:16:00 (none) user.warn kernel: Modules linked in: ixp4xx_mac
      ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
      Aug 14 01:16:00 (none) user.warn kernel: CPU: 0
      Aug 14 01:16:00 (none) user.warn kernel: pc : [<c00921a4>] lr :
      [<c0092274>] Not tainted
      Aug 14 01:16:00 (none) user.warn kernel: sp : c0ac3d14 ip : c0ac3d30
      fp : c0ac3d2c
      Aug 14 01:16:00 (none) user.warn kernel: r10: c026a8c4 r9 : c0269320
      r8 : 042445d8
      Aug 14 01:16:00 (none) user.warn kernel: r7 : c19a3400 r6 : c19a3400
      r5 : c026a8c4 r4 : 042445d8
      Aug 14 01:16:00 (none) user.warn kernel: r3 : 0454c52e r2 : 00000200
      r1 : c0683b50 r0 : 00000200
      Aug 14 01:16:00 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
      Mode SVC_32 Segment user
      Aug 14 01:16:00 (none) user.warn kernel: Control: 39FF
      Aug 14 01:16:00 (none) user.warn kernel: Table: 017FC000 DAC: 00000015
      Aug 14 01:16:00 (none) user.warn kernel: Process chown (pid: 10579,
      stack limit = 0xc0ac2250)
      Aug 14 01:16:00 (none) user.warn kernel: Stack: (0xc0ac3d14 to 0xc0ac4000)
      Aug 14 01:16:00 (none) user.warn kernel: 3d00:
      c0ac2000 c19a3400 c144e534
      Aug 14 01:16:00 (none) user.warn kernel: 3d20: c0ac3d44 c0ac3d30
      c0092274 c0092164 042445d8 c19a3400 c0ac3d74 c0ac3d48
      Aug 14 01:16:00 (none) user.warn kernel: 3d40: c0092604 c009225c
      000015a4 042445d8 c19a3400 c144e534 c1e176a8 c0ac3ecc
      Aug 14 01:16:00 (none) user.warn kernel: 3d60: c0ac3dd0 c0ac3ddc
      c0ac3d94 c0ac3d78 bf02abc0 c00925cc c13e6acc c144e534
      Aug 14 01:16:00 (none) user.warn kernel: 3d80: c109724c c1e17640
      c0ac3dc4 c0ac3d98 c008522c bf02ab1c c0277d20 000041ed
      Aug 14 01:16:00 (none) user.warn kernel: 3da0: c0ac3ecc c1e17640
      c0ac2000 00000000 c1dfd03f 00029c38 c0ac3e0c c0ac3dc8
      Aug 14 01:16:00 (none) user.warn kernel: 3dc0: c00871f8 c008517c
      00000000 bf033058 3cc2f422 0000000c c1dfd033 c0277d20
      Aug 14 01:16:00 (none) user.warn kernel: 3de0: c109724c 00000001
      c0ac3ecc c0277f20 c1e22190 c1dfd000 c0ac2000 00029c38
      Aug 14 01:16:00 (none) user.warn kernel: 3e00: c0ac3e80 c0ac3e10
      c008784c c008673c c1e22190 c0277f20 000000d4 c001de04
      Aug 14 01:16:00 (none) user.warn kernel: 3e20: 00029c38 00000000
      00000001 00000000 c01bd00c 00000046 c0ac3f00 00000000
      Aug 14 01:16:00 (none) user.warn kernel: 3e40: c0ac3f24 c0ac3f24
      c0ac3f10 c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38
      Aug 14 01:16:00 (none) user.warn kernel: 3e60: 00000001 c029e2c0
      c0ac3ecc c0ac3ecc c1dfd000 c0ac3ea8 c0ac3e84 c0087dc8
      Aug 14 01:16:00 (none) user.warn kernel: 3e80: c00877cc c0ac3e90
      c1dfd000 00000000 c0ac3ecc c1dfd000 c001de04 c0ac3ec8
      Aug 14 01:16:00 (none) user.warn kernel: 3ea0: c0ac3eac c00886e0
      c0087ad4 c0ac3f44 c0ac3ecc 0008f608 000000c4 c0ac3f30
      Aug 14 01:16:00 (none) user.warn kernel: 3ec0: c0ac3ecc c0080b70
      c00886a8 c109724c c0277d20 000000d4 c001de04 00029c38
      Aug 14 01:16:00 (none) user.warn kernel: 3ee0: 00000000 00000001
      00000000 c01bd00c 00000046 c0ac3f00 00000000 c0ac3f24
      Aug 14 01:16:00 (none) user.warn kernel: 3f00: c0ac3f24 c0ac3f10
      c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38 c0ac3f44
      Aug 14 01:16:00 (none) user.warn kernel: 3f20: bea97b90 c0ac3f40
      c0ac3f34 c0080c08 c0080b58 c0ac3fa4 c0ac3f44 c0080c2c
      Aug 14 01:16:00 (none) user.warn kernel: 3f40: c0080bf8 00000000
      042445d7 00800011 00000001 00000001 00000000 00000000
      Aug 14 01:16:00 (none) user.warn kernel: 3f60: 00000000 00000000
      000bbf4d 488bcae3 00000000 48875ddd 00000000 48a38742
      Aug 14 01:16:00 (none) user.warn kernel: 3f80: 00000000 00001000
      00000000 0008f608 00000001 0008f608 00000000 c0ac3fa8
      Aug 14 01:16:00 (none) user.warn kernel: 3fa0: c001dc60 c0080c18
      00000001 0008f608 0008f608 bea97b90 bea97b90 00000000
      Aug 14 01:16:00 (none) user.warn kernel: 3fc0: 00000001 0008f608
      0008f608 00000001 00000000 00000000 00029c38 00029c38
      Aug 14 01:16:00 (none) user.warn kernel: 3fe0: 0007e610 bea97b78
      bea97b90 40175ec0 60000010 0008f608 00000000 00000000
      Aug 14 01:16:00 (none) user.warn kernel: Backtrace:
      Aug 14 01:16:00 (none) user.warn kernel: Function entered at
      [<c0092158>] from [<c0092274>]
      Aug 14 01:16:00 (none) user.warn kernel: r6 = C144E534 r5 = C19A3400
      r4 = C0AC2000
      Aug 14 01:16:00 (none) user.warn kernel: Function entered at
      [<c0092250>] from [<c0092604>]
      Aug 14 01:16:00 (none) user.warn kernel: r5 = C19A3400 r4 = 042445D8
      Aug 14 01:16:00 (none) user.warn kernel: Function entered at
      [<c00925c0>] from [<bf02abc0>]
      Aug 14 01:16:00 (none) user.warn kernel: Function entered at
      [<bf02ab10>] from [<c008522c>]
      Aug 14 01:16:00 (none) user.warn kernel: r6 = C1E17640 r5 = C109724C
      r4 = C144E534
      ...
      ...
      ...


      That backups script simply calls two other scripts, run data backup,
      and run system backup. The system backup didn't happen. Looks like the
      data backup happened, but the permissions on them are wrong, so I
      think that's my clue as to where in the backup script the crash
      happened. Here's a copy of my databackup script:


      #!/bin/bash
      # ----------------------------------------------------------------------
      # mikes handy rotating-filesystem-snapshot utility
      # Edited by Drew for my nslu2 system
      # ----------------------------------------------------------------------
      # this needs to be a lot more general, but the basic idea is it makes
      # rotating backup-snapshots whenever called
      # ----------------------------------------------------------------------

      unset PATH # suggestion from H. Milz: avoid accidental use of $PATH

      # ------------- system commands used by this script --------------------
      ID=/usr/bin/id;
      ECHO=/bin/echo;

      MOUNT=/bin/mount;
      RM=/bin/rm;
      MV=/bin/mv;
      CP=/bin/cp;
      CPIO=/bin/cpio;
      FIND=/usr/bin/find;
      TOUCH=/bin/touch;
      CHOWN=/bin/chown;
      CHMOD=/bin/chmod;

      RSYNC=/usr/bin/rsync;


      # ------------- file locations -----------------------------------------

      SNAPSHOT_RW=/mnt/backups/snapshots/data;
      STUFF_TO_BACKUP=/mnt/storage/
      EXCLUDES=/root/dataBackupExcludes.cfg;


      # ------------- the script itself --------------------------------------

      # make sure we're running as root
      if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.
      Exiting..."; exit; } fi

      # rotating snapshots of / (fixme: this should be more general)

      # step 1: delete the oldest snapshot, if it exists:
      if [ -d $SNAPSHOT_RW/daily.3 ] ; then \
      $RM -rf $SNAPSHOT_RW/daily.3 ; \
      fi ;

      # step 2: shift the middle snapshots(s) back by one, if they exist
      if [ -d $SNAPSHOT_RW/daily.2 ] ; then \
      $MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ; \
      fi;

      if [ -d $SNAPSHOT_RW/daily.1 ] ; then \
      $MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ; \
      fi;

      # step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
      # if that exists
      # Note: Since the slug does not have GNU cp, it doesn't support the
      # -l option, so this has been replaced with a combination of
      # find and cpio as described at www.mikerubel.org/computers/rsync_snapshots

      # Old GNU cp version
      #if [ -d $SNAPSHOT_RW/daily.0 ] ; then \
      #$CP -al $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ; \
      #fi;

      # Version that should work on slug
      if [ -d $SNAPSHOT_RW/daily.0 ] ; then
      cd $SNAPSHOT_RW/daily.0; $FIND . -print | $CPIO -dpl $SNAPSHOT_RW/daily.1; \
      fi;



      # step 4: rsync from the system into the latest snapshot (notice that
      # rsync behaves like cp --remove-destination by default, so the destination
      # is unlinked first. If it were not so, this would copy over the other
      # snapshot(s) too!
      $RSYNC \
      -va --delete --delete-excluded \
      --exclude-from="$EXCLUDES" \
      $STUFF_TO_BACKUP $SNAPSHOT_RW/daily.0 ;


      # Change data file permissions so that no one but root can modify the files.
      # Only do this to files shared through samba, i.e. /mnt/storage crap
      # File permissions will be preserved on the rest of the files
      # (i.e. full system backup)
      if [ -d $SNAPSHOT_RW/daily.0 ] ; then
      $CHOWN -R root:root $SNAPSHOT_RW/daily.0; \
      $CHMOD -R 644 $SNAPSHOT_RW/daily.0; \
      fi;

      if [ -d $SNAPSHOT_RW/daily.1 ] ; then
      $CHOWN -R root:root $SNAPSHOT_RW/daily.1; \
      $CHMOD -R 644 $SNAPSHOT_RW/daily.1; \
      fi;

      if [ -d $SNAPSHOT_RW/daily.2 ] ; then
      $CHOWN -R root:root $SNAPSHOT_RW/daily.2; \
      $CHMOD -R 644 $SNAPSHOT_RW/daily.2; \
      fi;

      if [ -d $SNAPSHOT_RW/daily.3 ] ; then
      $CHOWN -R root:root $SNAPSHOT_RW/daily.3; \
      $CHMOD -R 644 $SNAPSHOT_RW/daily.3; \
      fi;

      $FIND $SNAPSHOT_RW -type d -exec $CHMOD 755 {} \;


      # step 5: update the mtime of daily.0 to reflect the snapshot time
      $TOUCH $SNAPSHOT_RW/daily.0 ;








      Any gurus out there see anything wrong with this? Like I said, this
      has been running perfectly, and I haven't changed my scripts in some
      time. Thanks!


      -Drew
    • Mike (mwester)
      Good info, all, but sadly incomplete. I can deduce from the log and the script that it would appear you are not running Unslung. But apart from that, it
      Message 2 of 11 , Aug 14, 2008
      • 0 Attachment
        Good info, all, but sadly incomplete. I can deduce from the log and the
        script that it would appear you are not running Unslung. But apart from
        that, it could be anything else. If you can share what firmware you are
        running, what version of that firmware, and in order to be able to make
        some sense of the addresses in the kernel crash dump, the specific
        kernel version (uname -a), that would be helpful.

        Mike (mwester)

        Drew Kirkpatrick wrote:
        > This morning I noticed my slug wasn't responding (lights were still
        > on), so I gave it the old reboot and started checking the logs. Looks
        > like I got some some bad mojo with my kernel last night while it was
        > running it's cron'd backups. These backups have been working
        > flawlessly for a few weeks now, so I'm at a loss as to what really
        > happened here. Here's the log:
        >
        > Aug 14 00:52:35 (none) syslog.info -- MARK --
        > Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
        > (/root/runDailyBackupScript.sh > /dev/null 2>&1)
        > Aug 14 01:12:35 (none) syslog.info -- MARK --
        > Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
        > (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
        > Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
        > pointer dereference at virtual address 00000200
        > Aug 14 01:16:00 (none) user.alert kernel: pgd = c17fc000
        > Aug 14 01:16:00 (none) user.alert kernel: [00000200] *pgd=00902031,
        > *pte=00000000, *ppte=00000000
        > Aug 14 01:16:00 (none) user.warn kernel: Internal error: Oops: 17 [#1]
        > Aug 14 01:16:00 (none) user.warn kernel: Modules linked in: ixp4xx_mac
        > ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
        > Aug 14 01:16:00 (none) user.warn kernel: CPU: 0
        > Aug 14 01:16:00 (none) user.warn kernel: pc : [<c00921a4>] lr :
        > [<c0092274>] Not tainted
        > Aug 14 01:16:00 (none) user.warn kernel: sp : c0ac3d14 ip : c0ac3d30
        > fp : c0ac3d2c
        > Aug 14 01:16:00 (none) user.warn kernel: r10: c026a8c4 r9 : c0269320
        > r8 : 042445d8
        > Aug 14 01:16:00 (none) user.warn kernel: r7 : c19a3400 r6 : c19a3400
        > r5 : c026a8c4 r4 : 042445d8
        > Aug 14 01:16:00 (none) user.warn kernel: r3 : 0454c52e r2 : 00000200
        > r1 : c0683b50 r0 : 00000200
        > Aug 14 01:16:00 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
        > Mode SVC_32 Segment user
        > Aug 14 01:16:00 (none) user.warn kernel: Control: 39FF
        > Aug 14 01:16:00 (none) user.warn kernel: Table: 017FC000 DAC: 00000015
        > Aug 14 01:16:00 (none) user.warn kernel: Process chown (pid: 10579,
        > stack limit = 0xc0ac2250)
        > Aug 14 01:16:00 (none) user.warn kernel: Stack: (0xc0ac3d14 to 0xc0ac4000)
        > Aug 14 01:16:00 (none) user.warn kernel: 3d00:
        > c0ac2000 c19a3400 c144e534
        > Aug 14 01:16:00 (none) user.warn kernel: 3d20: c0ac3d44 c0ac3d30
        > c0092274 c0092164 042445d8 c19a3400 c0ac3d74 c0ac3d48
        > Aug 14 01:16:00 (none) user.warn kernel: 3d40: c0092604 c009225c
        > 000015a4 042445d8 c19a3400 c144e534 c1e176a8 c0ac3ecc
        > Aug 14 01:16:00 (none) user.warn kernel: 3d60: c0ac3dd0 c0ac3ddc
        > c0ac3d94 c0ac3d78 bf02abc0 c00925cc c13e6acc c144e534
        > Aug 14 01:16:00 (none) user.warn kernel: 3d80: c109724c c1e17640
        > c0ac3dc4 c0ac3d98 c008522c bf02ab1c c0277d20 000041ed
        > Aug 14 01:16:00 (none) user.warn kernel: 3da0: c0ac3ecc c1e17640
        > c0ac2000 00000000 c1dfd03f 00029c38 c0ac3e0c c0ac3dc8
        > Aug 14 01:16:00 (none) user.warn kernel: 3dc0: c00871f8 c008517c
        > 00000000 bf033058 3cc2f422 0000000c c1dfd033 c0277d20
        > Aug 14 01:16:00 (none) user.warn kernel: 3de0: c109724c 00000001
        > c0ac3ecc c0277f20 c1e22190 c1dfd000 c0ac2000 00029c38
        > Aug 14 01:16:00 (none) user.warn kernel: 3e00: c0ac3e80 c0ac3e10
        > c008784c c008673c c1e22190 c0277f20 000000d4 c001de04
        > Aug 14 01:16:00 (none) user.warn kernel: 3e20: 00029c38 00000000
        > 00000001 00000000 c01bd00c 00000046 c0ac3f00 00000000
        > Aug 14 01:16:00 (none) user.warn kernel: 3e40: c0ac3f24 c0ac3f24
        > c0ac3f10 c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38
        > Aug 14 01:16:00 (none) user.warn kernel: 3e60: 00000001 c029e2c0
        > c0ac3ecc c0ac3ecc c1dfd000 c0ac3ea8 c0ac3e84 c0087dc8
        > Aug 14 01:16:00 (none) user.warn kernel: 3e80: c00877cc c0ac3e90
        > c1dfd000 00000000 c0ac3ecc c1dfd000 c001de04 c0ac3ec8
        > Aug 14 01:16:00 (none) user.warn kernel: 3ea0: c0ac3eac c00886e0
        > c0087ad4 c0ac3f44 c0ac3ecc 0008f608 000000c4 c0ac3f30
        > Aug 14 01:16:00 (none) user.warn kernel: 3ec0: c0ac3ecc c0080b70
        > c00886a8 c109724c c0277d20 000000d4 c001de04 00029c38
        > Aug 14 01:16:00 (none) user.warn kernel: 3ee0: 00000000 00000001
        > 00000000 c01bd00c 00000046 c0ac3f00 00000000 c0ac3f24
        > Aug 14 01:16:00 (none) user.warn kernel: 3f00: c0ac3f24 c0ac3f10
        > c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38 c0ac3f44
        > Aug 14 01:16:00 (none) user.warn kernel: 3f20: bea97b90 c0ac3f40
        > c0ac3f34 c0080c08 c0080b58 c0ac3fa4 c0ac3f44 c0080c2c
        > Aug 14 01:16:00 (none) user.warn kernel: 3f40: c0080bf8 00000000
        > 042445d7 00800011 00000001 00000001 00000000 00000000
        > Aug 14 01:16:00 (none) user.warn kernel: 3f60: 00000000 00000000
        > 000bbf4d 488bcae3 00000000 48875ddd 00000000 48a38742
        > Aug 14 01:16:00 (none) user.warn kernel: 3f80: 00000000 00001000
        > 00000000 0008f608 00000001 0008f608 00000000 c0ac3fa8
        > Aug 14 01:16:00 (none) user.warn kernel: 3fa0: c001dc60 c0080c18
        > 00000001 0008f608 0008f608 bea97b90 bea97b90 00000000
        > Aug 14 01:16:00 (none) user.warn kernel: 3fc0: 00000001 0008f608
        > 0008f608 00000001 00000000 00000000 00029c38 00029c38
        > Aug 14 01:16:00 (none) user.warn kernel: 3fe0: 0007e610 bea97b78
        > bea97b90 40175ec0 60000010 0008f608 00000000 00000000
        > Aug 14 01:16:00 (none) user.warn kernel: Backtrace:
        > Aug 14 01:16:00 (none) user.warn kernel: Function entered at
        > [<c0092158>] from [<c0092274>]
        > Aug 14 01:16:00 (none) user.warn kernel: r6 = C144E534 r5 = C19A3400
        > r4 = C0AC2000
        > Aug 14 01:16:00 (none) user.warn kernel: Function entered at
        > [<c0092250>] from [<c0092604>]
        > Aug 14 01:16:00 (none) user.warn kernel: r5 = C19A3400 r4 = 042445D8
        > Aug 14 01:16:00 (none) user.warn kernel: Function entered at
        > [<c00925c0>] from [<bf02abc0>]
        > Aug 14 01:16:00 (none) user.warn kernel: Function entered at
        > [<bf02ab10>] from [<c008522c>]
        > Aug 14 01:16:00 (none) user.warn kernel: r6 = C1E17640 r5 = C109724C
        > r4 = C144E534
        > ...
        > ...
        > ...
        >
        >
        > That backups script simply calls two other scripts, run data backup,
        > and run system backup. The system backup didn't happen. Looks like the
        > data backup happened, but the permissions on them are wrong, so I
        > think that's my clue as to where in the backup script the crash
        > happened. Here's a copy of my databackup script:
        >
        >
        > #!/bin/bash
        > # ----------------------------------------------------------------------
        > # mikes handy rotating-filesystem-snapshot utility
        > # Edited by Drew for my nslu2 system
        > # ----------------------------------------------------------------------
        > # this needs to be a lot more general, but the basic idea is it makes
        > # rotating backup-snapshots whenever called
        > # ----------------------------------------------------------------------
        >
        > unset PATH # suggestion from H. Milz: avoid accidental use of $PATH
        >
        > # ------------- system commands used by this script --------------------
        > ID=/usr/bin/id;
        > ECHO=/bin/echo;
        >
        > MOUNT=/bin/mount;
        > RM=/bin/rm;
        > MV=/bin/mv;
        > CP=/bin/cp;
        > CPIO=/bin/cpio;
        > FIND=/usr/bin/find;
        > TOUCH=/bin/touch;
        > CHOWN=/bin/chown;
        > CHMOD=/bin/chmod;
        >
        > RSYNC=/usr/bin/rsync;
        >
        >
        > # ------------- file locations -----------------------------------------
        >
        > SNAPSHOT_RW=/mnt/backups/snapshots/data;
        > STUFF_TO_BACKUP=/mnt/storage/
        > EXCLUDES=/root/dataBackupExcludes.cfg;
        >
        >
        > # ------------- the script itself --------------------------------------
        >
        > # make sure we're running as root
        > if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.
        > Exiting..."; exit; } fi
        >
        > # rotating snapshots of / (fixme: this should be more general)
        >
        > # step 1: delete the oldest snapshot, if it exists:
        > if [ -d $SNAPSHOT_RW/daily.3 ] ; then \
        > $RM -rf $SNAPSHOT_RW/daily.3 ; \
        > fi ;
        >
        > # step 2: shift the middle snapshots(s) back by one, if they exist
        > if [ -d $SNAPSHOT_RW/daily.2 ] ; then \
        > $MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ; \
        > fi;
        >
        > if [ -d $SNAPSHOT_RW/daily.1 ] ; then \
        > $MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ; \
        > fi;
        >
        > # step 3: make a hard-link-only (except for dirs) copy of the latest snapshot,
        > # if that exists
        > # Note: Since the slug does not have GNU cp, it doesn't support the
        > # -l option, so this has been replaced with a combination of
        > # find and cpio as described at www.mikerubel.org/computers/rsync_snapshots
        >
        > # Old GNU cp version
        > #if [ -d $SNAPSHOT_RW/daily.0 ] ; then \
        > #$CP -al $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ; \
        > #fi;
        >
        > # Version that should work on slug
        > if [ -d $SNAPSHOT_RW/daily.0 ] ; then
        > cd $SNAPSHOT_RW/daily.0; $FIND . -print | $CPIO -dpl $SNAPSHOT_RW/daily.1; \
        > fi;
        >
        >
        >
        > # step 4: rsync from the system into the latest snapshot (notice that
        > # rsync behaves like cp --remove-destination by default, so the destination
        > # is unlinked first. If it were not so, this would copy over the other
        > # snapshot(s) too!
        > $RSYNC \
        > -va --delete --delete-excluded \
        > --exclude-from="$EXCLUDES" \
        > $STUFF_TO_BACKUP $SNAPSHOT_RW/daily.0 ;
        >
        >
        > # Change data file permissions so that no one but root can modify the files.
        > # Only do this to files shared through samba, i.e. /mnt/storage crap
        > # File permissions will be preserved on the rest of the files
        > # (i.e. full system backup)
        > if [ -d $SNAPSHOT_RW/daily.0 ] ; then
        > $CHOWN -R root:root $SNAPSHOT_RW/daily.0; \
        > $CHMOD -R 644 $SNAPSHOT_RW/daily.0; \
        > fi;
        >
        > if [ -d $SNAPSHOT_RW/daily.1 ] ; then
        > $CHOWN -R root:root $SNAPSHOT_RW/daily.1; \
        > $CHMOD -R 644 $SNAPSHOT_RW/daily.1; \
        > fi;
        >
        > if [ -d $SNAPSHOT_RW/daily.2 ] ; then
        > $CHOWN -R root:root $SNAPSHOT_RW/daily.2; \
        > $CHMOD -R 644 $SNAPSHOT_RW/daily.2; \
        > fi;
        >
        > if [ -d $SNAPSHOT_RW/daily.3 ] ; then
        > $CHOWN -R root:root $SNAPSHOT_RW/daily.3; \
        > $CHMOD -R 644 $SNAPSHOT_RW/daily.3; \
        > fi;
        >
        > $FIND $SNAPSHOT_RW -type d -exec $CHMOD 755 {} \;
        >
        >
        > # step 5: update the mtime of daily.0 to reflect the snapshot time
        > $TOUCH $SNAPSHOT_RW/daily.0 ;
        >
        >
        >
        >
        >
        >
        >
        >
        > Any gurus out there see anything wrong with this? Like I said, this
        > has been running perfectly, and I haven't changed my scripts in some
        > time. Thanks!
        >
        >
        > -Drew
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
      • Drew Kirkpatrick
        Doh. I shouldn t write technical emails in the morning :) I m running slugos 4.8 beta BE. It s running the stock kernel, 2.6.21.7 preemptive. Thanks :)
        Message 3 of 11 , Aug 14, 2008
        • 0 Attachment
          Doh. I shouldn't write technical emails in the morning :)

          I'm running slugos 4.8 beta BE. It's running the stock kernel,
          2.6.21.7 preemptive.

          Thanks :)

          On Thu, Aug 14, 2008 at 8:03 AM, Mike (mwester) <mwester@...> wrote:
          > Good info, all, but sadly incomplete. I can deduce from the log and the
          > script that it would appear you are not running Unslung. But apart from
          > that, it could be anything else. If you can share what firmware you are
          > running, what version of that firmware, and in order to be able to make
          > some sense of the addresses in the kernel crash dump, the specific
          > kernel version (uname -a), that would be helpful.
          >
          > Mike (mwester)
          >
          > Drew Kirkpatrick wrote:
          >> This morning I noticed my slug wasn't responding (lights were still
          >> on), so I gave it the old reboot and started checking the logs. Looks
          >> like I got some some bad mojo with my kernel last night while it was
          >> running it's cron'd backups. These backups have been working
          >> flawlessly for a few weeks now, so I'm at a loss as to what really
          >> happened here. Here's the log:
          >>
          >> Aug 14 00:52:35 (none) syslog.info -- MARK --
          >> Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
          >> (/root/runDailyBackupScript.sh > /dev/null 2>&1)
          >> Aug 14 01:12:35 (none) syslog.info -- MARK --
          >> Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
          >> (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
          >> Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
          >> pointer dereference at virtual address 00000200
          >> Aug 14 01:16:00 (none) user.alert kernel: pgd = c17fc000
          >> Aug 14 01:16:00 (none) user.alert kernel: [00000200] *pgd=00902031,
          >> *pte=00000000, *ppte=00000000
          >> Aug 14 01:16:00 (none) user.warn kernel: Internal error: Oops: 17 [#1]
          >> Aug 14 01:16:00 (none) user.warn kernel: Modules linked in: ixp4xx_mac
          >> ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
          >> Aug 14 01:16:00 (none) user.warn kernel: CPU: 0
          >> Aug 14 01:16:00 (none) user.warn kernel: pc : [<c00921a4>] lr :
          >> [<c0092274>] Not tainted
          >> Aug 14 01:16:00 (none) user.warn kernel: sp : c0ac3d14 ip : c0ac3d30
          >> fp : c0ac3d2c
          >> Aug 14 01:16:00 (none) user.warn kernel: r10: c026a8c4 r9 : c0269320
          >> r8 : 042445d8
          >> Aug 14 01:16:00 (none) user.warn kernel: r7 : c19a3400 r6 : c19a3400
          >> r5 : c026a8c4 r4 : 042445d8
          >> Aug 14 01:16:00 (none) user.warn kernel: r3 : 0454c52e r2 : 00000200
          >> r1 : c0683b50 r0 : 00000200
          >> Aug 14 01:16:00 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
          >> Mode SVC_32 Segment user
          >> Aug 14 01:16:00 (none) user.warn kernel: Control: 39FF
          >> Aug 14 01:16:00 (none) user.warn kernel: Table: 017FC000 DAC: 00000015
          >> Aug 14 01:16:00 (none) user.warn kernel: Process chown (pid: 10579,
          >> stack limit = 0xc0ac2250)
          >> Aug 14 01:16:00 (none) user.warn kernel: Stack: (0xc0ac3d14 to 0xc0ac4000)
          >> Aug 14 01:16:00 (none) user.warn kernel: 3d00:
          >> c0ac2000 c19a3400 c144e534
          >> Aug 14 01:16:00 (none) user.warn kernel: 3d20: c0ac3d44 c0ac3d30
          >> c0092274 c0092164 042445d8 c19a3400 c0ac3d74 c0ac3d48
          >> Aug 14 01:16:00 (none) user.warn kernel: 3d40: c0092604 c009225c
          >> 000015a4 042445d8 c19a3400 c144e534 c1e176a8 c0ac3ecc
          >> Aug 14 01:16:00 (none) user.warn kernel: 3d60: c0ac3dd0 c0ac3ddc
          >> c0ac3d94 c0ac3d78 bf02abc0 c00925cc c13e6acc c144e534
          >> Aug 14 01:16:00 (none) user.warn kernel: 3d80: c109724c c1e17640
          >> c0ac3dc4 c0ac3d98 c008522c bf02ab1c c0277d20 000041ed
          >> Aug 14 01:16:00 (none) user.warn kernel: 3da0: c0ac3ecc c1e17640
          >> c0ac2000 00000000 c1dfd03f 00029c38 c0ac3e0c c0ac3dc8
          >> Aug 14 01:16:00 (none) user.warn kernel: 3dc0: c00871f8 c008517c
          >> 00000000 bf033058 3cc2f422 0000000c c1dfd033 c0277d20
          >> Aug 14 01:16:00 (none) user.warn kernel: 3de0: c109724c 00000001
          >> c0ac3ecc c0277f20 c1e22190 c1dfd000 c0ac2000 00029c38
          >> Aug 14 01:16:00 (none) user.warn kernel: 3e00: c0ac3e80 c0ac3e10
          >> c008784c c008673c c1e22190 c0277f20 000000d4 c001de04
          >> Aug 14 01:16:00 (none) user.warn kernel: 3e20: 00029c38 00000000
          >> 00000001 00000000 c01bd00c 00000046 c0ac3f00 00000000
          >> Aug 14 01:16:00 (none) user.warn kernel: 3e40: c0ac3f24 c0ac3f24
          >> c0ac3f10 c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38
          >> Aug 14 01:16:00 (none) user.warn kernel: 3e60: 00000001 c029e2c0
          >> c0ac3ecc c0ac3ecc c1dfd000 c0ac3ea8 c0ac3e84 c0087dc8
          >> Aug 14 01:16:00 (none) user.warn kernel: 3e80: c00877cc c0ac3e90
          >> c1dfd000 00000000 c0ac3ecc c1dfd000 c001de04 c0ac3ec8
          >> Aug 14 01:16:00 (none) user.warn kernel: 3ea0: c0ac3eac c00886e0
          >> c0087ad4 c0ac3f44 c0ac3ecc 0008f608 000000c4 c0ac3f30
          >> Aug 14 01:16:00 (none) user.warn kernel: 3ec0: c0ac3ecc c0080b70
          >> c00886a8 c109724c c0277d20 000000d4 c001de04 00029c38
          >> Aug 14 01:16:00 (none) user.warn kernel: 3ee0: 00000000 00000001
          >> 00000000 c01bd00c 00000046 c0ac3f00 00000000 c0ac3f24
          >> Aug 14 01:16:00 (none) user.warn kernel: 3f00: c0ac3f24 c0ac3f10
          >> c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38 c0ac3f44
          >> Aug 14 01:16:00 (none) user.warn kernel: 3f20: bea97b90 c0ac3f40
          >> c0ac3f34 c0080c08 c0080b58 c0ac3fa4 c0ac3f44 c0080c2c
          >> Aug 14 01:16:00 (none) user.warn kernel: 3f40: c0080bf8 00000000
          >> 042445d7 00800011 00000001 00000001 00000000 00000000
          >> Aug 14 01:16:00 (none) user.warn kernel: 3f60: 00000000 00000000
          >> 000bbf4d 488bcae3 00000000 48875ddd 00000000 48a38742
          >> Aug 14 01:16:00 (none) user.warn kernel: 3f80: 00000000 00001000
          >> 00000000 0008f608 00000001 0008f608 00000000 c0ac3fa8
          >> Aug 14 01:16:00 (none) user.warn kernel: 3fa0: c001dc60 c0080c18
          >> 00000001 0008f608 0008f608 bea97b90 bea97b90 00000000
          >> Aug 14 01:16:00 (none) user.warn kernel: 3fc0: 00000001 0008f608
          >> 0008f608 00000001 00000000 00000000 00029c38 00029c38
          >> Aug 14 01:16:00 (none) user.warn kernel: 3fe0: 0007e610 bea97b78
          >> bea97b90 40175ec0 60000010 0008f608 00000000 00000000
          >> Aug 14 01:16:00 (none) user.warn kernel: Backtrace:
          >> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
          >> [<c0092158>] from [<c0092274>]
          >> Aug 14 01:16:00 (none) user.warn kernel: r6 = C144E534 r5 = C19A3400
          >> r4 = C0AC2000
          >> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
          >> [<c0092250>] from [<c0092604>]
          >> Aug 14 01:16:00 (none) user.warn kernel: r5 = C19A3400 r4 = 042445D8
          >> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
          >> [<c00925c0>] from [<bf02abc0>]
          >> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
          >> [<bf02ab10>] from [<c008522c>]
          >> Aug 14 01:16:00 (none) user.warn kernel: r6 = C1E17640 r5 = C109724C
          >> r4 = C144E534
          >> ...
          >> ...
          >> ...
          >>
          >>
          >> That backups script simply calls two other scripts, run data backup,
          >> and run system backup. The system backup didn't happen. Looks like the
          >> data backup happened, but the permissions on them are wrong, so I
          >> think that's my clue as to where in the backup script the crash
          >> happened. Here's a copy of my databackup script:
          >>
          >>
          >> #!/bin/bash
          >> # ----------------------------------------------------------
          >> # mikes handy rotating-filesystem-snapshot utility
          >> # Edited by Drew for my nslu2 system
          >> # ----------------------------------------------------------
          >> # this needs to be a lot more general, but the basic idea is it makes
          >> # rotating backup-snapshots whenever called
          >> # ----------------------------------------------------------
          >>
          >> unset PATH # suggestion from H. Milz: avoid accidental use of $PATH
          >>
          >> # ------------- system commands used by this script --------------------
          >> ID=/usr/bin/id;
          >> ECHO=/bin/echo;
          >>
          >> MOUNT=/bin/mount;
          >> RM=/bin/rm;
          >> MV=/bin/mv;
          >> CP=/bin/cp;
          >> CPIO=/bin/cpio;
          >> FIND=/usr/bin/find;
          >> TOUCH=/bin/touch;
          >> CHOWN=/bin/chown;
          >> CHMOD=/bin/chmod;
          >>
          >> RSYNC=/usr/bin/rsync;
          >>
          >>
          >> # ------------- file locations -----------------------------------------
          >>
          >> SNAPSHOT_RW=/mnt/backups/snapshots/data;
          >> STUFF_TO_BACKUP=/mnt/storage/
          >> EXCLUDES=/root/dataBackupExcludes.cfg;
          >>
          >>
          >> # ------------- the script itself --------------------------------------
          >>
          >> # make sure we're running as root
          >> if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.
          >> Exiting..."; exit; } fi
          >>
          >> # rotating snapshots of / (fixme: this should be more general)
          >>
          >> # step 1: delete the oldest snapshot, if it exists:
          >> if [ -d $SNAPSHOT_RW/daily.3 ] ; then \
          >> $RM -rf $SNAPSHOT_RW/daily.3 ; \
          >> fi ;
          >>
          >> # step 2: shift the middle snapshots(s) back by one, if they exist
          >> if [ -d $SNAPSHOT_RW/daily.2 ] ; then \
          >> $MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ; \
          >> fi;
          >>
          >> if [ -d $SNAPSHOT_RW/daily.1 ] ; then \
          >> $MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ; \
          >> fi;
          >>
          >> # step 3: make a hard-link-only (except for dirs) copy of the latest
          >> snapshot,
          >> # if that exists
          >> # Note: Since the slug does not have GNU cp, it doesn't support the
          >> # -l option, so this has been replaced with a combination of
          >> # find and cpio as described at
          >> www.mikerubel.org/computers/rsync_snapshots
          >>
          >> # Old GNU cp version
          >> #if [ -d $SNAPSHOT_RW/daily.0 ] ; then \
          >> #$CP -al $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ; \
          >> #fi;
          >>
          >> # Version that should work on slug
          >> if [ -d $SNAPSHOT_RW/daily.0 ] ; then
          >> cd $SNAPSHOT_RW/daily.0; $FIND . -print | $CPIO -dpl $SNAPSHOT_RW/daily.1;
          >> \
          >> fi;
          >>
          >>
          >>
          >> # step 4: rsync from the system into the latest snapshot (notice that
          >> # rsync behaves like cp --remove-destination by default, so the
          >> destination
          >> # is unlinked first. If it were not so, this would copy over the other
          >> # snapshot(s) too!
          >> $RSYNC \
          >> -va --delete --delete-excluded \
          >> --exclude-from="$EXCLUDES" \
          >> $STUFF_TO_BACKUP $SNAPSHOT_RW/daily.0 ;
          >>
          >>
          >> # Change data file permissions so that no one but root can modify the
          >> files.
          >> # Only do this to files shared through samba, i.e. /mnt/storage crap
          >> # File permissions will be preserved on the rest of the files
          >> # (i.e. full system backup)
          >> if [ -d $SNAPSHOT_RW/daily.0 ] ; then
          >> $CHOWN -R root:root $SNAPSHOT_RW/daily.0; \
          >> $CHMOD -R 644 $SNAPSHOT_RW/daily.0; \
          >> fi;
          >>
          >> if [ -d $SNAPSHOT_RW/daily.1 ] ; then
          >> $CHOWN -R root:root $SNAPSHOT_RW/daily.1; \
          >> $CHMOD -R 644 $SNAPSHOT_RW/daily.1; \
          >> fi;
          >>
          >> if [ -d $SNAPSHOT_RW/daily.2 ] ; then
          >> $CHOWN -R root:root $SNAPSHOT_RW/daily.2; \
          >> $CHMOD -R 644 $SNAPSHOT_RW/daily.2; \
          >> fi;
          >>
          >> if [ -d $SNAPSHOT_RW/daily.3 ] ; then
          >> $CHOWN -R root:root $SNAPSHOT_RW/daily.3; \
          >> $CHMOD -R 644 $SNAPSHOT_RW/daily.3; \
          >> fi;
          >>
          >> $FIND $SNAPSHOT_RW -type d -exec $CHMOD 755 {} \;
          >>
          >>
          >> # step 5: update the mtime of daily.0 to reflect the snapshot time
          >> $TOUCH $SNAPSHOT_RW/daily.0 ;
          >>
          >>
          >>
          >>
          >>
          >>
          >>
          >>
          >> Any gurus out there see anything wrong with this? Like I said, this
          >> has been running perfectly, and I haven't changed my scripts in some
          >> time. Thanks!
          >>
          >>
          >> -Drew
          >>
          >> ------------------------------------
          >>
          >> Yahoo! Groups Links
          >>
          >>
          >>
          >>
          >
        • Drew Kirkpatrick
          Well, looks like it has nothing to do with my backup scripts. It just happened again. I did have it under a fairly heavy load, copying lots of big files to the
          Message 4 of 11 , Aug 14, 2008
          • 0 Attachment
            Well, looks like it has nothing to do with my backup scripts. It just
            happened again. I did have it under a fairly heavy load, copying lots
            of big files to the system through samba. Hopefully I don't have a bad
            slug, this is my second one.


            -Drew

            On Thu, Aug 14, 2008 at 9:06 AM, Drew Kirkpatrick
            <drew.kirkpatrick@...> wrote:
            > Doh. I shouldn't write technical emails in the morning :)
            >
            > I'm running slugos 4.8 beta BE. It's running the stock kernel,
            > 2.6.21.7 preemptive.
            >
            > Thanks :)
            >
            > On Thu, Aug 14, 2008 at 8:03 AM, Mike (mwester) <mwester@...> wrote:
            >> Good info, all, but sadly incomplete. I can deduce from the log and the
            >> script that it would appear you are not running Unslung. But apart from
            >> that, it could be anything else. If you can share what firmware you are
            >> running, what version of that firmware, and in order to be able to make
            >> some sense of the addresses in the kernel crash dump, the specific
            >> kernel version (uname -a), that would be helpful.
            >>
            >> Mike (mwester)
            >>
            >> Drew Kirkpatrick wrote:
            >>> This morning I noticed my slug wasn't responding (lights were still
            >>> on), so I gave it the old reboot and started checking the logs. Looks
            >>> like I got some some bad mojo with my kernel last night while it was
            >>> running it's cron'd backups. These backups have been working
            >>> flawlessly for a few weeks now, so I'm at a loss as to what really
            >>> happened here. Here's the log:
            >>>
            >>> Aug 14 00:52:35 (none) syslog.info -- MARK --
            >>> Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
            >>> (/root/runDailyBackupScript.sh > /dev/null 2>&1)
            >>> Aug 14 01:12:35 (none) syslog.info -- MARK --
            >>> Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
            >>> (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
            >>> Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
            >>> pointer dereference at virtual address 00000200
            >>> Aug 14 01:16:00 (none) user.alert kernel: pgd = c17fc000
            >>> Aug 14 01:16:00 (none) user.alert kernel: [00000200] *pgd=00902031,
            >>> *pte=00000000, *ppte=00000000
            >>> Aug 14 01:16:00 (none) user.warn kernel: Internal error: Oops: 17 [#1]
            >>> Aug 14 01:16:00 (none) user.warn kernel: Modules linked in: ixp4xx_mac
            >>> ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
            >>> Aug 14 01:16:00 (none) user.warn kernel: CPU: 0
            >>> Aug 14 01:16:00 (none) user.warn kernel: pc : [<c00921a4>] lr :
            >>> [<c0092274>] Not tainted
            >>> Aug 14 01:16:00 (none) user.warn kernel: sp : c0ac3d14 ip : c0ac3d30
            >>> fp : c0ac3d2c
            >>> Aug 14 01:16:00 (none) user.warn kernel: r10: c026a8c4 r9 : c0269320
            >>> r8 : 042445d8
            >>> Aug 14 01:16:00 (none) user.warn kernel: r7 : c19a3400 r6 : c19a3400
            >>> r5 : c026a8c4 r4 : 042445d8
            >>> Aug 14 01:16:00 (none) user.warn kernel: r3 : 0454c52e r2 : 00000200
            >>> r1 : c0683b50 r0 : 00000200
            >>> Aug 14 01:16:00 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
            >>> Mode SVC_32 Segment user
            >>> Aug 14 01:16:00 (none) user.warn kernel: Control: 39FF
            >>> Aug 14 01:16:00 (none) user.warn kernel: Table: 017FC000 DAC: 00000015
            >>> Aug 14 01:16:00 (none) user.warn kernel: Process chown (pid: 10579,
            >>> stack limit = 0xc0ac2250)
            >>> Aug 14 01:16:00 (none) user.warn kernel: Stack: (0xc0ac3d14 to 0xc0ac4000)
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3d00:
            >>> c0ac2000 c19a3400 c144e534
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3d20: c0ac3d44 c0ac3d30
            >>> c0092274 c0092164 042445d8 c19a3400 c0ac3d74 c0ac3d48
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3d40: c0092604 c009225c
            >>> 000015a4 042445d8 c19a3400 c144e534 c1e176a8 c0ac3ecc
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3d60: c0ac3dd0 c0ac3ddc
            >>> c0ac3d94 c0ac3d78 bf02abc0 c00925cc c13e6acc c144e534
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3d80: c109724c c1e17640
            >>> c0ac3dc4 c0ac3d98 c008522c bf02ab1c c0277d20 000041ed
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3da0: c0ac3ecc c1e17640
            >>> c0ac2000 00000000 c1dfd03f 00029c38 c0ac3e0c c0ac3dc8
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3dc0: c00871f8 c008517c
            >>> 00000000 bf033058 3cc2f422 0000000c c1dfd033 c0277d20
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3de0: c109724c 00000001
            >>> c0ac3ecc c0277f20 c1e22190 c1dfd000 c0ac2000 00029c38
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3e00: c0ac3e80 c0ac3e10
            >>> c008784c c008673c c1e22190 c0277f20 000000d4 c001de04
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3e20: 00029c38 00000000
            >>> 00000001 00000000 c01bd00c 00000046 c0ac3f00 00000000
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3e40: c0ac3f24 c0ac3f24
            >>> c0ac3f10 c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3e60: 00000001 c029e2c0
            >>> c0ac3ecc c0ac3ecc c1dfd000 c0ac3ea8 c0ac3e84 c0087dc8
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3e80: c00877cc c0ac3e90
            >>> c1dfd000 00000000 c0ac3ecc c1dfd000 c001de04 c0ac3ec8
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3ea0: c0ac3eac c00886e0
            >>> c0087ad4 c0ac3f44 c0ac3ecc 0008f608 000000c4 c0ac3f30
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3ec0: c0ac3ecc c0080b70
            >>> c00886a8 c109724c c0277d20 000000d4 c001de04 00029c38
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3ee0: 00000000 00000001
            >>> 00000000 c01bd00c 00000046 c0ac3f00 00000000 c0ac3f24
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3f00: c0ac3f24 c0ac3f10
            >>> c0095494 c00e2214 c0ac3f3c 00000000 c0ac3f38 c0ac3f44
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3f20: bea97b90 c0ac3f40
            >>> c0ac3f34 c0080c08 c0080b58 c0ac3fa4 c0ac3f44 c0080c2c
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3f40: c0080bf8 00000000
            >>> 042445d7 00800011 00000001 00000001 00000000 00000000
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3f60: 00000000 00000000
            >>> 000bbf4d 488bcae3 00000000 48875ddd 00000000 48a38742
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3f80: 00000000 00001000
            >>> 00000000 0008f608 00000001 0008f608 00000000 c0ac3fa8
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3fa0: c001dc60 c0080c18
            >>> 00000001 0008f608 0008f608 bea97b90 bea97b90 00000000
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3fc0: 00000001 0008f608
            >>> 0008f608 00000001 00000000 00000000 00029c38 00029c38
            >>> Aug 14 01:16:00 (none) user.warn kernel: 3fe0: 0007e610 bea97b78
            >>> bea97b90 40175ec0 60000010 0008f608 00000000 00000000
            >>> Aug 14 01:16:00 (none) user.warn kernel: Backtrace:
            >>> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
            >>> [<c0092158>] from [<c0092274>]
            >>> Aug 14 01:16:00 (none) user.warn kernel: r6 = C144E534 r5 = C19A3400
            >>> r4 = C0AC2000
            >>> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
            >>> [<c0092250>] from [<c0092604>]
            >>> Aug 14 01:16:00 (none) user.warn kernel: r5 = C19A3400 r4 = 042445D8
            >>> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
            >>> [<c00925c0>] from [<bf02abc0>]
            >>> Aug 14 01:16:00 (none) user.warn kernel: Function entered at
            >>> [<bf02ab10>] from [<c008522c>]
            >>> Aug 14 01:16:00 (none) user.warn kernel: r6 = C1E17640 r5 = C109724C
            >>> r4 = C144E534
            >>> ...
            >>> ...
            >>> ...
            >>>
            >>>
            >>> That backups script simply calls two other scripts, run data backup,
            >>> and run system backup. The system backup didn't happen. Looks like the
            >>> data backup happened, but the permissions on them are wrong, so I
            >>> think that's my clue as to where in the backup script the crash
            >>> happened. Here's a copy of my databackup script:
            >>>
            >>>
            >>> #!/bin/bash
            >>> # ----------------------------------------------------------
            >>> # mikes handy rotating-filesystem-snapshot utility
            >>> # Edited by Drew for my nslu2 system
            >>> # ----------------------------------------------------------
            >>> # this needs to be a lot more general, but the basic idea is it makes
            >>> # rotating backup-snapshots whenever called
            >>> # ----------------------------------------------------------
            >>>
            >>> unset PATH # suggestion from H. Milz: avoid accidental use of $PATH
            >>>
            >>> # ------------- system commands used by this script --------------------
            >>> ID=/usr/bin/id;
            >>> ECHO=/bin/echo;
            >>>
            >>> MOUNT=/bin/mount;
            >>> RM=/bin/rm;
            >>> MV=/bin/mv;
            >>> CP=/bin/cp;
            >>> CPIO=/bin/cpio;
            >>> FIND=/usr/bin/find;
            >>> TOUCH=/bin/touch;
            >>> CHOWN=/bin/chown;
            >>> CHMOD=/bin/chmod;
            >>>
            >>> RSYNC=/usr/bin/rsync;
            >>>
            >>>
            >>> # ------------- file locations -----------------------------------------
            >>>
            >>> SNAPSHOT_RW=/mnt/backups/snapshots/data;
            >>> STUFF_TO_BACKUP=/mnt/storage/
            >>> EXCLUDES=/root/dataBackupExcludes.cfg;
            >>>
            >>>
            >>> # ------------- the script itself --------------------------------------
            >>>
            >>> # make sure we're running as root
            >>> if (( `$ID -u` != 0 )); then { $ECHO "Sorry, must be root.
            >>> Exiting..."; exit; } fi
            >>>
            >>> # rotating snapshots of / (fixme: this should be more general)
            >>>
            >>> # step 1: delete the oldest snapshot, if it exists:
            >>> if [ -d $SNAPSHOT_RW/daily.3 ] ; then \
            >>> $RM -rf $SNAPSHOT_RW/daily.3 ; \
            >>> fi ;
            >>>
            >>> # step 2: shift the middle snapshots(s) back by one, if they exist
            >>> if [ -d $SNAPSHOT_RW/daily.2 ] ; then \
            >>> $MV $SNAPSHOT_RW/daily.2 $SNAPSHOT_RW/daily.3 ; \
            >>> fi;
            >>>
            >>> if [ -d $SNAPSHOT_RW/daily.1 ] ; then \
            >>> $MV $SNAPSHOT_RW/daily.1 $SNAPSHOT_RW/daily.2 ; \
            >>> fi;
            >>>
            >>> # step 3: make a hard-link-only (except for dirs) copy of the latest
            >>> snapshot,
            >>> # if that exists
            >>> # Note: Since the slug does not have GNU cp, it doesn't support the
            >>> # -l option, so this has been replaced with a combination of
            >>> # find and cpio as described at
            >>> www.mikerubel.org/computers/rsync_snapshots
            >>>
            >>> # Old GNU cp version
            >>> #if [ -d $SNAPSHOT_RW/daily.0 ] ; then \
            >>> #$CP -al $SNAPSHOT_RW/daily.0 $SNAPSHOT_RW/daily.1 ; \
            >>> #fi;
            >>>
            >>> # Version that should work on slug
            >>> if [ -d $SNAPSHOT_RW/daily.0 ] ; then
            >>> cd $SNAPSHOT_RW/daily.0; $FIND . -print | $CPIO -dpl $SNAPSHOT_RW/daily.1;
            >>> \
            >>> fi;
            >>>
            >>>
            >>>
            >>> # step 4: rsync from the system into the latest snapshot (notice that
            >>> # rsync behaves like cp --remove-destination by default, so the
            >>> destination
            >>> # is unlinked first. If it were not so, this would copy over the other
            >>> # snapshot(s) too!
            >>> $RSYNC \
            >>> -va --delete --delete-excluded \
            >>> --exclude-from="$EXCLUDES" \
            >>> $STUFF_TO_BACKUP $SNAPSHOT_RW/daily.0 ;
            >>>
            >>>
            >>> # Change data file permissions so that no one but root can modify the
            >>> files.
            >>> # Only do this to files shared through samba, i.e. /mnt/storage crap
            >>> # File permissions will be preserved on the rest of the files
            >>> # (i.e. full system backup)
            >>> if [ -d $SNAPSHOT_RW/daily.0 ] ; then
            >>> $CHOWN -R root:root $SNAPSHOT_RW/daily.0; \
            >>> $CHMOD -R 644 $SNAPSHOT_RW/daily.0; \
            >>> fi;
            >>>
            >>> if [ -d $SNAPSHOT_RW/daily.1 ] ; then
            >>> $CHOWN -R root:root $SNAPSHOT_RW/daily.1; \
            >>> $CHMOD -R 644 $SNAPSHOT_RW/daily.1; \
            >>> fi;
            >>>
            >>> if [ -d $SNAPSHOT_RW/daily.2 ] ; then
            >>> $CHOWN -R root:root $SNAPSHOT_RW/daily.2; \
            >>> $CHMOD -R 644 $SNAPSHOT_RW/daily.2; \
            >>> fi;
            >>>
            >>> if [ -d $SNAPSHOT_RW/daily.3 ] ; then
            >>> $CHOWN -R root:root $SNAPSHOT_RW/daily.3; \
            >>> $CHMOD -R 644 $SNAPSHOT_RW/daily.3; \
            >>> fi;
            >>>
            >>> $FIND $SNAPSHOT_RW -type d -exec $CHMOD 755 {} \;
            >>>
            >>>
            >>> # step 5: update the mtime of daily.0 to reflect the snapshot time
            >>> $TOUCH $SNAPSHOT_RW/daily.0 ;
            >>>
            >>>
            >>>
            >>>
            >>>
            >>>
            >>>
            >>>
            >>> Any gurus out there see anything wrong with this? Like I said, this
            >>> has been running perfectly, and I haven't changed my scripts in some
            >>> time. Thanks!
            >>>
            >>>
            >>> -Drew
            >>>
            >>> ------------------------------------
            >>>
            >>> Yahoo! Groups Links
            >>>
            >>>
            >>>
            >>>
            >>
            >
          • chris_burghart
            Most interesting to me is the log entry at 01:14:13, where there s a kernel Alignment trap caused by rtorrent a couple minutes before the backup problems
            Message 5 of 11 , Aug 14, 2008
            • 0 Attachment
              Most interesting to me is the log entry at 01:14:13,
              where there's a kernel "Alignment trap" caused by rtorrent a
              couple minutes before the backup problems began. I think I
              would look for rtorrent problems before anything else.

              Chris

              --- In nslu2-linux@yahoogroups.com, "Drew Kirkpatrick"
              <drew.kirkpatrick@...> wrote:
              >
              > This morning I noticed my slug wasn't responding (lights were still
              > on), so I gave it the old reboot and started checking the logs. Looks
              > like I got some some bad mojo with my kernel last night while it was
              > running it's cron'd backups. These backups have been working
              > flawlessly for a few weeks now, so I'm at a loss as to what really
              > happened here. Here's the log:
              >
              > Aug 14 00:52:35 (none) syslog.info -- MARK --
              > Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
              > (/root/runDailyBackupScript.sh > /dev/null 2>&1)
              > Aug 14 01:12:35 (none) syslog.info -- MARK --
              > Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
              > (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
              > Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
            • Drew Kirkpatrick
              Rtorrent does that all the time. Everytime it has crashed, Rtorrent was running, so I ve stopped. I ll see if it happens again...
              Message 6 of 11 , Aug 14, 2008
              • 0 Attachment
                Rtorrent does that all the time. Everytime it has crashed, Rtorrent
                was running, so I've stopped. I'll see if it happens again...

                On Thu, Aug 14, 2008 at 4:19 PM, chris_burghart <burghart@...> wrote:
                > Most interesting to me is the log entry at 01:14:13,
                > where there's a kernel "Alignment trap" caused by rtorrent a
                > couple minutes before the backup problems began. I think I
                > would look for rtorrent problems before anything else.
                >
                > Chris
                >
                > --- In nslu2-linux@yahoogroups.com, "Drew Kirkpatrick"
                >
                > <drew.kirkpatrick@...> wrote:
                >>
                >> This morning I noticed my slug wasn't responding (lights were still
                >> on), so I gave it the old reboot and started checking the logs. Looks
                >> like I got some some bad mojo with my kernel last night while it was
                >> running it's cron'd backups. These backups have been working
                >> flawlessly for a few weeks now, so I'm at a loss as to what really
                >> happened here. Here's the log:
                >>
                >> Aug 14 00:52:35 (none) syslog.info -- MARK --
                >> Aug 14 01:00:01 (none) cron.info /opt/sbin/cron[10564]: (root) CMD
                >> (/root/runDailyBackupScript.sh > /dev/null 2>&1)
                >> Aug 14 01:12:35 (none) syslog.info -- MARK --
                >> Aug 14 01:14:13 (none) user.warn kernel: Alignment trap: rtorrent
                >> (1184) PC=0x401a34e8 Instr=0xe8be000f Address=0x002548b3 FSR 0x013
                >> Aug 14 01:16:00 (none) user.alert kernel: Unable to handle kernel NULL
                >
                >
              • David Given
                ... There are many known issues with the SLUG going wrong when it runs out of memory; but those usually show up in the logs with an OOM Killer message, so it
                Message 7 of 11 , Aug 14, 2008
                • 0 Attachment
                  Drew Kirkpatrick wrote:
                  > Rtorrent does that all the time. Everytime it has crashed, Rtorrent
                  > was running, so I've stopped. I'll see if it happens again...

                  There are many known issues with the SLUG going wrong when it runs out
                  of memory; but those usually show up in the logs with an 'OOM Killer'
                  message, so it could be unrelated. You may like to try turning off
                  overcommit in the VM:

                  echo -1 > /proc/sys/vm/overcommit_memory

                  (I think.)

                  This may help. Or it may not. Or it may cause your system to crash
                  differently...

                  --
                  ┌─── dg@cowlark.com ───── http://www.cowlark.com ─────

                  │ "I love the way Microsoft follows standards. In much the same manner
                  │ that fish follow migrating caribou." --- Paul Tomblin
                • Drew Kirkpatrick
                  Thanks for the tip. At the least you have me something else to google around on. I have allocated a bunch of swap space, but if the memory management is out of
                  Message 8 of 11 , Aug 14, 2008
                  • 0 Attachment
                    Thanks for the tip. At the least you have me something else to google around on. I have allocated a bunch of swap space, but if the memory management is out of whack a bit it might not help. Worst comes to worst, I can throw gentoo back on there and roll my own kernel, but I have enough gentoo boxes to contend with at the moment. I'm getting lazy in my old age. Thanks to all for the help...


                    -----Original Message-----
                    From: David Given <dg@...>

                    Date: Fri, 15 Aug 2008 00:56:15
                    To: <nslu2-linux@yahoogroups.com>
                    Subject: Re: [nslu2-linux] Re: Kernel nasties during backup


                    Drew Kirkpatrick wrote:
                    > Rtorrent does that all the time. Everytime it has crashed, Rtorrent
                    > was running, so I've stopped. I'll see if it happens again...

                    There are many known issues with the SLUG going wrong when it runs out
                    of memory; but those usually show up in the logs with an 'OOM Killer'
                    message, so it could be unrelated. You may like to try turning off
                    overcommit in the VM:

                    echo -1 > /proc/sys/vm/overcommit_memory

                    (I think.)

                    This may help. Or it may not. Or it may cause your system to crash
                    differently...

                    --
                    ┌─── dg@cowlark.com ───── http://www.cowlark.com ─────

                    │ "I love the way Microsoft follows standards. In much the same manner
                    │ that fish follow migrating caribou." --- Paul Tomblin
                  • Drew Kirkpatrick
                    Well, my problems with my slugos 4.8 beta BE kernel has continued even without running rtorrents. Now all the slug does is samba, ssh, and nightly rsync
                    Message 9 of 11 , Aug 26, 2008
                    • 0 Attachment
                      Well, my problems with my slugos 4.8 beta BE kernel has continued even
                      without running rtorrents. Now all the slug does is samba, ssh, and
                      nightly rsync backups to a second drive. I still get kernel oop's
                      regularly. Sometimes it's the chown process. Often it's the kswapd
                      process. They always start like this:

                      Aug 26 07:31:55 (none) user.alert kernel: Unable to handle kernel NULL
                      pointer dereference at virtual address 00000204
                      Aug 26 07:31:55 (none) user.alert kernel: pgd = c0004000
                      Aug 26 07:31:55 (none) user.alert kernel: [00000204] *pgd=00000000
                      Aug 26 07:31:55 (none) user.warn kernel: Internal error: Oops: 817 [#2]
                      Aug 26 07:31:55 (none) user.warn kernel: Modules linked in: ixp4xx_mac
                      ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
                      Aug 26 07:31:55 (none) user.warn kernel: CPU: 0
                      Aug 26 07:31:55 (none) user.warn kernel: pc : [<c0092da4>] lr :
                      [<bf02c5e0>] Not tainted
                      Aug 26 07:31:55 (none) user.warn kernel: sp : c02d1eb8 ip : c02d1e88
                      fp : c02d1ed8
                      Aug 26 07:31:55 (none) user.warn kernel: r10: 00000080 r9 : 00000080
                      r8 : c02d1ee0
                      Aug 26 07:31:55 (none) user.warn kernel: r7 : 0000003c r6 : c02d0000
                      r5 : c0683b50 r4 : c0683b58
                      Aug 26 07:31:55 (none) user.warn kernel: r3 : 00000200 r2 : c070bd00
                      r1 : c0683b60 r0 : 00000000
                      Aug 26 07:31:55 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
                      Mode SVC_32 Segment kernel
                      Aug 26 07:31:55 (none) user.warn kernel: Control: 39FF
                      Aug 26 07:31:55 (none) user.warn kernel: Table: 00B24000 DAC: 00000017
                      Aug 26 07:31:55 (none) user.warn kernel: Process kswapd0 (pid: 58,
                      stack limit = 0xc02d0250)
                      Aug 26 07:31:55 (none) user.warn kernel: Stack: (0xc02d1eb8 to 0xc02d2000)
                      Aug 26 07:31:55 (none) user.warn kernel: 1ea0:
                      c19db980 c19db988
                      Aug 26 07:31:55 (none) user.warn kernel: 1ec0: c02d0000 c0206df0
                      00000000 c02d1f10 c02d1edc c0093054 c0092d30 00000080
                      ...

                      As you can see, this particular kernel oops was from kswapd. This is
                      with kernel 2.6.21.7, which I'm guessing is the only option in slugos
                      4.8. There was nothing high memory use going on at this time. The
                      crashes on process chown look exactly the same (I have some chown's in
                      my backup scripts). This is a new slug. I bricked my old one, and just
                      put this one in about 2 weeks ago. The slug is on a ups, so it should
                      be getting clean power.

                      I've updated and upgraded both ipkg, and ipkg-opt feeds hoping that
                      newer versions of packages would help. It hasn't.

                      I'm about at the point that I think I might need to try another
                      firmware. Is slugos commonly this unstable, or do I have something
                      screwy going on (most likely)? Should I be trying debian instead? All
                      I need is openssh (not dropbear, I do a lot of ssh tunneling and
                      such), samba, rsync, and a system that is stable. Crashing every other
                      day just doesn't cut it for what I need this little server to do.

                      I was going to try setting the overcommit_memory to -1 as David Given
                      suggested, but after googling around I don't think -1 turns off the
                      overcommit. I think 0 (the default) is the off setting for overcommit.
                      I see a number of references to setting this to 1 or 2, usually in
                      regards to problems to the OOM killer. But I have yet to have an out
                      of memory/process killed event. I just get these damn crashes.

                      Any tips out there?




                      On Thu, Aug 14, 2008 at 7:04 PM, Drew Kirkpatrick
                      <drew.kirkpatrick@...> wrote:
                      > Thanks for the tip. At the least you have me something else to google around on. I have allocated a bunch of swap space, but if the memory management is out of whack a bit it might not help. Worst comes to worst, I can throw gentoo back on there and roll my own kernel, but I have enough gentoo boxes to contend with at the moment. I'm getting lazy in my old age. Thanks to all for the help...
                      >
                      >
                      > -----Original Message-----
                      > From: David Given <dg@...>
                      >
                      > Date: Fri, 15 Aug 2008 00:56:15
                      > To: <nslu2-linux@yahoogroups.com>
                      > Subject: Re: [nslu2-linux] Re: Kernel nasties during backup
                      >
                      >
                      > Drew Kirkpatrick wrote:
                      >> Rtorrent does that all the time. Everytime it has crashed, Rtorrent
                      >> was running, so I've stopped. I'll see if it happens again...
                      >
                      > There are many known issues with the SLUG going wrong when it runs out
                      > of memory; but those usually show up in the logs with an 'OOM Killer'
                      > message, so it could be unrelated. You may like to try turning off
                      > overcommit in the VM:
                      >
                      > echo -1 > /proc/sys/vm/overcommit_memory
                      >
                      > (I think.)
                      >
                      > This may help. Or it may not. Or it may cause your system to crash
                      > differently...
                      >
                      > --
                      > ┌─── dg@cowlark.com ───── http://www.cowlark.com ─────
                      > │
                      > │ "I love the way Microsoft follows standards. In much the same manner
                      > │ that fish follow migrating caribou." --- Paul Tomblin
                      >
                      >
                      >
                    • Mike (mwester)
                      ... That s the first page of the kernel. Odd. ... c0092d88 t shrink_icache_memory This is code in the middle of a bunch of inode-management code. ... Yes.
                      Message 10 of 11 , Aug 26, 2008
                      • 0 Attachment
                        Drew Kirkpatrick wrote:
                        > Well, my problems with my slugos 4.8 beta BE kernel has continued even
                        > without running rtorrents. Now all the slug does is samba, ssh, and
                        > nightly rsync backups to a second drive. I still get kernel oop's
                        > regularly. Sometimes it's the chown process. Often it's the kswapd
                        > process. They always start like this:
                        >
                        > Aug 26 07:31:55 (none) user.alert kernel: Unable to handle kernel NULL
                        > pointer dereference at virtual address 00000204
                        > Aug 26 07:31:55 (none) user.alert kernel: pgd = c0004000

                        That's the first page of the kernel. Odd.

                        > Aug 26 07:31:55 (none) user.alert kernel: [00000204] *pgd=00000000
                        > Aug 26 07:31:55 (none) user.warn kernel: Internal error: Oops: 817 [#2]
                        > Aug 26 07:31:55 (none) user.warn kernel: Modules linked in: ixp4xx_mac
                        > ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
                        > Aug 26 07:31:55 (none) user.warn kernel: CPU: 0
                        > Aug 26 07:31:55 (none) user.warn kernel: pc : [<c0092da4>] lr :
                        > [<bf02c5e0>] Not tainted

                        c0092d88 t shrink_icache_memory

                        This is code in the middle of a bunch of inode-management code.


                        > Aug 26 07:31:55 (none) user.warn kernel: sp : c02d1eb8 ip : c02d1e88
                        > fp : c02d1ed8
                        > Aug 26 07:31:55 (none) user.warn kernel: r10: 00000080 r9 : 00000080
                        > r8 : c02d1ee0
                        > Aug 26 07:31:55 (none) user.warn kernel: r7 : 0000003c r6 : c02d0000
                        > r5 : c0683b50 r4 : c0683b58
                        > Aug 26 07:31:55 (none) user.warn kernel: r3 : 00000200 r2 : c070bd00
                        > r1 : c0683b60 r0 : 00000000
                        > Aug 26 07:31:55 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
                        > Mode SVC_32 Segment kernel
                        > Aug 26 07:31:55 (none) user.warn kernel: Control: 39FF
                        > Aug 26 07:31:55 (none) user.warn kernel: Table: 00B24000 DAC: 00000017
                        > Aug 26 07:31:55 (none) user.warn kernel: Process kswapd0 (pid: 58,
                        > stack limit = 0xc02d0250)
                        > Aug 26 07:31:55 (none) user.warn kernel: Stack: (0xc02d1eb8 to 0xc02d2000)
                        > Aug 26 07:31:55 (none) user.warn kernel: 1ea0:
                        > c19db980 c19db988
                        > Aug 26 07:31:55 (none) user.warn kernel: 1ec0: c02d0000 c0206df0
                        > 00000000 c02d1f10 c02d1edc c0093054 c0092d30 00000080
                        > ...
                        >
                        > As you can see, this particular kernel oops was from kswapd. This is
                        > with kernel 2.6.21.7, which I'm guessing is the only option in slugos
                        > 4.8.

                        Yes. (But of course you build a more recent SlugOS with newer kernel,
                        if you wished).

                        There was nothing high memory use going on at this time. The
                        > crashes on process chown look exactly the same (I have some chown's in
                        > my backup scripts). This is a new slug. I bricked my old one, and just
                        > put this one in about 2 weeks ago. The slug is on a ups, so it should
                        > be getting clean power.
                        >

                        What about the USB devices? Have you swapped any of those? Assuming
                        that there was indeed nothing going on (which isn't really true, of
                        coure, otherwise the slug wouldn't be doing I/O, but I know what you
                        mean...), anyway, assuming there was nothing other than normal
                        background and housekeeping stuff happening, then I'm beginning to
                        wonder if perhaps you have a USB device that's going to sleep, and
                        something goes terribly wrong when SlugOS attempts I/O, which presumably
                        usually wakes up the device.

                        The stack dump is quite clear that this is I/O-related. Corruption such
                        as this may also be an event that happened some time earlier; the stack
                        dump then would reflect nothing unusual. So another question would be
                        if there *was* some I/O activity, possibly intensive, that may have
                        caused swapping or some other memory shortfall, perhaps some time in the
                        past -- and the system just had the misfortune to finally get around to
                        touching the damaged data structure.

                        I'd run an fsck on all partitions, testing for bad-blocks as well. I'd
                        also format and test the swap partitions thoroughly before returning
                        them to being swap space. If you have done a turnup to disk, you'll
                        have a syslog that survives reboots -- it won't have data from close to
                        that crash obviously, but it might have other events from earlier -
                        paging errors, disk errors, etc.

                        > I've updated and upgraded both ipkg, and ipkg-opt feeds hoping that
                        > newer versions of packages would help. It hasn't.
                        >

                        That's good; that's expected -- userspace shouldn't be able to cause a
                        kernel oops ever.

                        > I'm about at the point that I think I might need to try another
                        > firmware. Is slugos commonly this unstable, or do I have something
                        > screwy going on (most likely)? Should I be trying debian instead? All
                        > I need is openssh (not dropbear, I do a lot of ssh tunneling and
                        > such), samba, rsync, and a system that is stable. Crashing every other
                        > day just doesn't cut it for what I need this little server to do.
                        >

                        SlugOS is very stable. There are known issues with the OOM stuff in
                        ARM, and have been for a very loooong time -- other than that there are
                        no known problems with the stock-memory NSLU2s.

                        Looking at your software list, though, you are certainly stressing the
                        device, so I think looking more closely at the swap and OOM behavior is
                        the most likely direction to find a solution. OpenSSH takes a lot of
                        CPU, Samba is just I/O intensive -- the combination can stress a system.
                        But I suspect rsync may be the issue. It builds its entire comparison
                        list in memory, and if you are rsyncing anything like a typical rootfs
                        for a backup, or such, you'll certainly be at or beyond the amount of
                        RAM the little box has before you even transfer a single byte of file data.

                        It would be interesting to note if Debian behaves differently, but I
                        certainly have no expectation of that based on the stack trace.

                        > I was going to try setting the overcommit_memory to -1 as David Given
                        > suggested, but after googling around I don't think -1 turns off the
                        > overcommit. I think 0 (the default) is the off setting for overcommit.
                        > I see a number of references to setting this to 1 or 2, usually in
                        > regards to problems to the OOM killer. But I have yet to have an out
                        > of memory/process killed event. I just get these damn crashes.
                        >
                        > Any tips out there?

                        I've attached the System.map file for that kernel for your reference.
                        I'd keep a close watch on my I/O for errors, fsck and test *all* disk
                        and flash memory space, do whatever is necessary to make sure that
                        *nothing* is going to sleep or spinning down on me, and I'd persue the
                        OOM settings -- and I'd start to monitor the rsync stuff closely to see
                        if there is a correlation that can be drawn between the rsync activity
                        and future crashes.

                        Personally, just to perturb things, I'd turn off my swap space and see
                        what happened. That'll tell you in a hurry if you're swapping. If I
                        was, I'd try a swap file -- and move the file about on the various
                        partitions just for fun.

                        A serial console is a must, but netconsole might be acceptable.

                        And I'd try a different set of USB devices if available.

                        Mike (mwester)
                      • Drew Kirkpatrick
                        I haven t swapped my USB devices, I assumed this was memory related. Much thanks for helping me interpret that data. I never would have thought to look at the
                        Message 11 of 11 , Aug 26, 2008
                        • 0 Attachment
                          I haven't swapped my USB devices, I assumed this was memory related.
                          Much thanks for helping me interpret that data. I never would have
                          thought to look at the I/O stuff.

                          I only have the one external hard drive box (dual enclosure, one disc
                          for root, swap, and storage, the other drive for backups), but I'll
                          plug that into one of my gentoo boxes and check the partitions and
                          such carefully.


                          -Drew

                          On Tue, Aug 26, 2008 at 4:36 PM, Mike (mwester) <mwester@...> wrote:
                          > Drew Kirkpatrick wrote:
                          >> Well, my problems with my slugos 4.8 beta BE kernel has continued even
                          >> without running rtorrents. Now all the slug does is samba, ssh, and
                          >> nightly rsync backups to a second drive. I still get kernel oop's
                          >> regularly. Sometimes it's the chown process. Often it's the kswapd
                          >> process. They always start like this:
                          >>
                          >> Aug 26 07:31:55 (none) user.alert kernel: Unable to handle kernel NULL
                          >> pointer dereference at virtual address 00000204
                          >> Aug 26 07:31:55 (none) user.alert kernel: pgd = c0004000
                          >
                          > That's the first page of the kernel. Odd.
                          >
                          >> Aug 26 07:31:55 (none) user.alert kernel: [00000204] *pgd=00000000
                          >> Aug 26 07:31:55 (none) user.warn kernel: Internal error: Oops: 817 [#2]
                          >> Aug 26 07:31:55 (none) user.warn kernel: Modules linked in: ixp4xx_mac
                          >> ixp4xx_qmgr mii ext3 jbd mbcache ohci_hcd ehci_hcd
                          >> Aug 26 07:31:55 (none) user.warn kernel: CPU: 0
                          >> Aug 26 07:31:55 (none) user.warn kernel: pc : [<c0092da4>] lr :
                          >> [<bf02c5e0>] Not tainted
                          >
                          > c0092d88 t shrink_icache_memory
                          >
                          > This is code in the middle of a bunch of inode-management code.
                          >
                          >> Aug 26 07:31:55 (none) user.warn kernel: sp : c02d1eb8 ip : c02d1e88
                          >> fp : c02d1ed8
                          >> Aug 26 07:31:55 (none) user.warn kernel: r10: 00000080 r9 : 00000080
                          >> r8 : c02d1ee0
                          >> Aug 26 07:31:55 (none) user.warn kernel: r7 : 0000003c r6 : c02d0000
                          >> r5 : c0683b50 r4 : c0683b58
                          >> Aug 26 07:31:55 (none) user.warn kernel: r3 : 00000200 r2 : c070bd00
                          >> r1 : c0683b60 r0 : 00000000
                          >> Aug 26 07:31:55 (none) user.warn kernel: Flags: nzCv IRQs on FIQs on
                          >> Mode SVC_32 Segment kernel
                          >> Aug 26 07:31:55 (none) user.warn kernel: Control: 39FF
                          >> Aug 26 07:31:55 (none) user.warn kernel: Table: 00B24000 DAC: 00000017
                          >> Aug 26 07:31:55 (none) user.warn kernel: Process kswapd0 (pid: 58,
                          >> stack limit = 0xc02d0250)
                          >> Aug 26 07:31:55 (none) user.warn kernel: Stack: (0xc02d1eb8 to 0xc02d2000)
                          >> Aug 26 07:31:55 (none) user.warn kernel: 1ea0:
                          >> c19db980 c19db988
                          >> Aug 26 07:31:55 (none) user.warn kernel: 1ec0: c02d0000 c0206df0
                          >> 00000000 c02d1f10 c02d1edc c0093054 c0092d30 00000080
                          >> ...
                          >>
                          >> As you can see, this particular kernel oops was from kswapd. This is
                          >> with kernel 2.6.21.7, which I'm guessing is the only option in slugos
                          >> 4.8.
                          >
                          > Yes. (But of course you build a more recent SlugOS with newer kernel,
                          > if you wished).
                          >
                          > There was nothing high memory use going on at this time. The
                          >> crashes on process chown look exactly the same (I have some chown's in
                          >> my backup scripts). This is a new slug. I bricked my old one, and just
                          >> put this one in about 2 weeks ago. The slug is on a ups, so it should
                          >> be getting clean power.
                          >>
                          >
                          > What about the USB devices? Have you swapped any of those? Assuming
                          > that there was indeed nothing going on (which isn't really true, of
                          > coure, otherwise the slug wouldn't be doing I/O, but I know what you
                          > mean...), anyway, assuming there was nothing other than normal
                          > background and housekeeping stuff happening, then I'm beginning to
                          > wonder if perhaps you have a USB device that's going to sleep, and
                          > something goes terribly wrong when SlugOS attempts I/O, which presumably
                          > usually wakes up the device.
                          >
                          > The stack dump is quite clear that this is I/O-related. Corruption such
                          > as this may also be an event that happened some time earlier; the stack
                          > dump then would reflect nothing unusual. So another question would be
                          > if there *was* some I/O activity, possibly intensive, that may have
                          > caused swapping or some other memory shortfall, perhaps some time in the
                          > past -- and the system just had the misfortune to finally get around to
                          > touching the damaged data structure.
                          >
                          > I'd run an fsck on all partitions, testing for bad-blocks as well. I'd
                          > also format and test the swap partitions thoroughly before returning
                          > them to being swap space. If you have done a turnup to disk, you'll
                          > have a syslog that survives reboots -- it won't have data from close to
                          > that crash obviously, but it might have other events from earlier -
                          > paging errors, disk errors, etc.
                          >
                          >> I've updated and upgraded both ipkg, and ipkg-opt feeds hoping that
                          >> newer versions of packages would help. It hasn't.
                          >>
                          >
                          > That's good; that's expected -- userspace shouldn't be able to cause a
                          > kernel oops ever.
                          >
                          >> I'm about at the point that I think I might need to try another
                          >> firmware. Is slugos commonly this unstable, or do I have something
                          >> screwy going on (most likely)? Should I be trying debian instead? All
                          >> I need is openssh (not dropbear, I do a lot of ssh tunneling and
                          >> such), samba, rsync, and a system that is stable. Crashing every other
                          >> day just doesn't cut it for what I need this little server to do.
                          >>
                          >
                          > SlugOS is very stable. There are known issues with the OOM stuff in
                          > ARM, and have been for a very loooong time -- other than that there are
                          > no known problems with the stock-memory NSLU2s.
                          >
                          > Looking at your software list, though, you are certainly stressing the
                          > device, so I think looking more closely at the swap and OOM behavior is
                          > the most likely direction to find a solution. OpenSSH takes a lot of
                          > CPU, Samba is just I/O intensive -- the combination can stress a system.
                          > But I suspect rsync may be the issue. It builds its entire comparison
                          > list in memory, and if you are rsyncing anything like a typical rootfs
                          > for a backup, or such, you'll certainly be at or beyond the amount of
                          > RAM the little box has before you even transfer a single byte of file data.
                          >
                          > It would be interesting to note if Debian behaves differently, but I
                          > certainly have no expectation of that based on the stack trace.
                          >
                          >> I was going to try setting the overcommit_memory to -1 as David Given
                          >> suggested, but after googling around I don't think -1 turns off the
                          >> overcommit. I think 0 (the default) is the off setting for overcommit.
                          >> I see a number of references to setting this to 1 or 2, usually in
                          >> regards to problems to the OOM killer. But I have yet to have an out
                          >> of memory/process killed event. I just get these damn crashes.
                          >>
                          >> Any tips out there?
                          >
                          > I've attached the System.map file for that kernel for your reference.
                          > I'd keep a close watch on my I/O for errors, fsck and test *all* disk
                          > and flash memory space, do whatever is necessary to make sure that
                          > *nothing* is going to sleep or spinning down on me, and I'd persue the
                          > OOM settings -- and I'd start to monitor the rsync stuff closely to see
                          > if there is a correlation that can be drawn between the rsync activity
                          > and future crashes.
                          >
                          > Personally, just to perturb things, I'd turn off my swap space and see
                          > what happened. That'll tell you in a hurry if you're swapping. If I
                          > was, I'd try a swap file -- and move the file about on the various
                          > partitions just for fun.
                          >
                          > A serial console is a must, but netconsole might be acceptable.
                          >
                          > And I'd try a different set of USB devices if available.
                          >
                          > Mike (mwester)
                          >
                          > c0004000 A swapper_pg_dir
                          > c0008000 T __init_begin
                          > c0008000 T _sinittext
                          > c0008000 T _stext
                          > c0008000 T stext
                          > c0008030 t __enable_mmu
                          > c0008060 t __turn_mmu_on
                          > c0008078 t __create_page_tables
                          > c00080ec t __switch_data
                          > c0008110 t __mmap_switched
                          > c0008154 t __error
                          > c0008154 t __error_a
                          > c0008154 t __error_p
                          > c000815c t __lookup_processor_type
                          > c0008194 T lookup_processor_type
                          > c00081bc t __lookup_machine_type
                          > c00081f0 T lookup_machine_type
                          > c0008220 t nosmp
                          > c0008244 t set_reset_devices
                          > c0008268 t debug_kernel
                          > c0008298 t quiet_kernel
                          > c00082c8 t init_setup
                          > c0008308 t rdinit_setup
                          > c0008348 W smp_setup_processor_id
                          > c0008358 t initcall_debug_setup
                          > c000837c t loglevel
                          > c00083ac t maxcpus
                          > c00083dc t unknown_bootoption
                          > c00085f8 T parse_early_param
                          > c0008664 t do_early_param
                          > c00086e4 t init
                          > c000887c T start_kernel
                          > c0008b24 t readonly
                          > c0008b58 t readwrite
                          > c0008b8c t root_data_setup
                          > c0008bac t fs_names_setup
                          > c0008bcc t load_ramdisk
                          > c0008bfc t root_dev_setup
                          > c0008c24 t root_delay_setup
                          > c0008c50 T change_floppy
                          > c0008d54 T mount_block_root
                          > c0008fdc T mount_root
                          > c0009048 T prepare_namespace
                          > c0009194 t ramdisk_start_setup
                          > c00091c0 t prompt_ramdisk
                          > c00091f0 t error
                          > c000922c t fill_inbuf
                          > c00092b8 t flush_window
                          > c00093a4 t huft_free
                          > c00093d4 t inflate_codes
                          > c0009878 t huft_build
                          > c0009e38 t inflate_dynamic
                          > c000a490 t inflate_fixed
                          > c000a624 T rd_load_image
                          > c000b7ec T rd_load_disk
                          > c000b89c t default_rootfs
                          > c000b920 t lpj_setup
                          > c000b960 T squash_mem_tags
                          > c000b9a0 T convert_to_tag_list
                          > c000bb40 T __kuser_helper_start
                          > c000bb40 t __kuser_memory_barrier
                          > c000bb60 t __kuser_cmpxchg
                          > c000bb80 t __kuser_get_tls
                          > c000bb9c t __kuser_helper_version
                          > c000bba0 T __kuser_helper_end
                          > c000bba0 T __stubs_start
                          > c000bba0 t vector_irq
                          > c000bc20 t vector_dabt
                          > c000bca0 t vector_pabt
                          > c000bd20 t vector_und
                          > c000bda0 t vector_fiq
                          > c000bda4 t vector_addrexcptn
                          > c000bdc4 T __stubs_end
                          > c000bdc4 T __vectors_start
                          > c000bde4 T __vectors_end
                          > c000be00 T init_IRQ
                          > c000be44 t nohlt_setup
                          > c000be68 t hlt_setup
                          > c000be8c T reboot_setup
                          > c000beb0 t ptrace_break_init
                          > c000bedc t arm_add_memory
                          > c000bf40 t parse_tag_core
                          > c000bfa4 t parse_tag_ramdisk
                          > c000c00c t parse_tag_initrd2
                          > c000c044 t parse_tag_serialnr
                          > c000c07c t parse_tag_revision
                          > c000c0a0 t customize_machine
                          > c000c0cc t fpe_setup
                          > c000c0f4 t early_mem
                          > c000c170 t early_initrd
                          > c000c1cc t parse_tag_initrd
                          > c000c214 t parse_tag_mem32
                          > c000c26c t parse_tag_cmdline
                          > c000c294 t topology_init
                          > c000c2c4 T setup_arch
                          > c000c870 T time_init
                          > c000c894 t timer_init_sysfs
                          > c000c8d4 T trap_init
                          > c000c994 T pcibios_setup
                          > c000c9fc T pci_common_init
                          > c000cb78 T register_isa_ports
                          > c000cbc4 t xscale_cp0_init
                          > c000cc50 t consistent_init
                          > c000cd20 T hook_fault_code
                          > c000cd4c T mem_init
                          > c000cf5c T bootmem_init
                          > c000d248 T check_writebuffer_bugs
                          > c000d37c t early_ecc
                          > c000d3f4 t early_cachepolicy
                          > c000d4c4 t early_nowrite
                          > c000d4fc t early_nocache
                          > c000d534 T reserve_node_zero
                          > c000d578 T create_mapping
                          > c000da34 T paging_init
                          > c000de68 T iotable_init
                          > c000dea0 t noalign_setup
                          > c000dee0 t alignment_init
                          > c000df74 t __xscale_setup
                          > c000dfa4 t xscale_crval
                          > c000dfac T ixp4xx_set_udc_info
                          > c000dfe4 T ixp4xx_timer_init
                          > c000e078 T ixp4xx_map_io
                          > c000e098 T ixp4xx_init_irq
                          > c000e100 T ixp4xx_sys_init
                          > c000e1a0 t ixdp425_init
                          > c000e1e8 t avila_init
                          > c000e298 t coyote_init
                          > c000e328 t nslu2_timer_init
                          > c000e350 t nslu2_init
                          > c000e3c4 t nslu2_fixup
                          > c000e478 t nslu2_power_init
                          > c000e548 t nas100d_init
                          > c000e5c8 t nas100d_fixup
                          > c000e67c t nas100d_power_init
                          > c000e700 t dsmg600_timer_init
                          > c000e728 t dsmg600_init
                          > c000e7a8 t dsmg600_fixup
                          > c000e85c t dsmg600_power_init
                          > c000e91c t fsg_init
                          > c000e9a4 t fsg_buttons_init
                          > c000ead4 t ixdp425_map_irq
                          > c000eb44 T ixdp425_pci_init
                          > c000eb7c T ixdp425_pci_preinit
                          > c000ebc0 t avila_map_irq
                          > c000ec50 T avila_pci_init
                          > c000ec98 T avila_pci_preinit
                          > c000ecdc t ixdpg425_map_irq
                          > c000ed0c T ixdpg425_pci_init
                          > c000ed40 T ixdpg425_pci_preinit
                          > c000ed6c t nslu2_map_irq
                          > c000edd0 T nslu2_pci_init
                          > c000ee0c T nslu2_pci_preinit
                          > c000ee44 t nas100d_map_irq
                          > c000eea4 T nas100d_pci_init
                          > c000eee0 T nas100d_pci_preinit
                          > c000ef30 t dsmg600_map_irq
                          > c000ef90 T dsmg600_pci_init
                          > c000efc4 T dsmg600_pci_preinit
                          > c000f020 t fsg_map_irq
                          > c000f0a0 T fsg_pci_init
                          > c000f0dc T fsg_pci_preinit
                          > c000f114 T ixp4xx_adjust_zones
                          > c000f14c T ixp4xx_pci_preinit
                          > c000f284 t fpe_init
                          > c000f308 T sched_init_smp
                          > c000f318 T init_idle
                          > c000f3d8 T sched_init
                          > c000f4f8 T proc_caches_init
                          > c000f628 T fork_init
                          > c000f6d4 T fork_idle
                          > c000f73c t pause_on_oops_setup
                          > c000f768 t panic_setup
                          > c000f794 t printk_time_setup
                          > c000f7c4 T add_preferred_console
                          > c000f884 t console_setup
                          > c000f950 t ignore_loglevel_setup
                          > c000f980 t log_buf_len_setup
                          > c000faf0 t profile_setup
                          > c000fcbc t create_proc_profile
                          > c000fd2c T profile_init
                          > c000fd9c t cpu_callback
                          > c000fe3c T spawn_ksoftirqd
                          > c000fe84 T softirq_init
                          > c000fec0 t ioresources_init
                          > c000ff1c t reserve_setup
                          > c000fff4 t sysctl_init
                          > c0010018 t timekeeping_init_device
                          > c0010050 t timer_cpu_notify
                          > c00101b0 T init_timers
                          > c00101f8 T timekeeping_init
                          > c0010330 t uid_cache_init
                          > c0010410 T signals_init
                          > c0010458 T usermodehelper_init
                          > c0010490 T pidmap_init
                          > c0010530 T pidhash_init
                          > c00105e4 t rcu_cpu_notify
                          > c001064c T rcu_init
                          > c0010670 T sort_main_extable
                          > c0010694 t kernel_param_sysfs_setup
                          > c0010724 t param_sysfs_init
                          > c00108b8 t init_posix_timers
                          > c001097c t helper_init
                          > c00109bc t init_posix_cpu_timers
                          > c0010aa4 t hrtimer_cpu_notify
                          > c0010acc T hrtimers_init
                          > c0010af0 t latency_init
                          > c0010b2c t clocksource_done_booting
                          > c0010b50 t init_clocksource_sysfs
                          > c0010bb8 t boot_override_clocksource
                          > c0010c30 t boot_override_clock
                          > c0010c88 t init_jiffies_clocksource
                          > c0010ca4 t init_timer_list_procfs
                          > c0010ce4 t init
                          > c0010d60 t crash_notes_memory_init
                          > c0010db4 t ksysfs_init
                          > c0010df4 t irqpoll_setup
                          > c0010e30 t irqfixup_setup
                          > c0010e6c t utsname_sysctl_init
                          > c0010e8c T bootmem_bootmap_pages
                          > c0010eb0 t get_mapsize
                          > c0010ee0 t free_all_bootmem_core
                          > c0011080 T free_all_bootmem
                          > c001109c T free_all_bootmem_node
                          > c00110b0 t free_bootmem_core
                          > c001112c T free_bootmem
                          > c0011158 T free_bootmem_node
                          > c0011170 t reserve_bootmem_core
                          > c00111f0 T reserve_bootmem
                          > c001121c T reserve_bootmem_node
                          > c0011234 t init_bootmem_core
                          > c00112f0 T init_bootmem
                          > c0011338 T init_bootmem_node
                          > c001134c T __alloc_bootmem_core
                          > c0011688 T __alloc_bootmem_low_node
                          > c00116ac T __alloc_bootmem_low
                          > c001173c T __alloc_bootmem_nopanic
                          > c00117b4 T __alloc_bootmem
                          > c00117f0 T __alloc_bootmem_node
                          > c0011840 T memmap_init_zone
                          > c00118cc T set_dma_reserve
                          > c00118e8 T page_alloc_init
                          > c00118f8 t build_zonelists_node
                          > c0011950 t init_per_zone_pages_min
                          > c00119a0 T init_currently_empty_zone
                          > c0011ad8 T free_area_init_node
                          > c0011e20 T free_area_init
                          > c0011e54 t __build_all_zonelists
                          > c0011f38 T build_all_zonelists
                          > c0011fac T __free_pages_bootmem
                          > c0012044 T alloc_large_system_hash
                          > c0012218 t ratelimit_handler
                          > c0012230 T page_writeback_init
                          > c001226c t pdflush_init
                          > c0012294 T swap_setup
                          > c00122cc t kswapd_init
                          > c0012300 t disable_randmaps
                          > c0012324 T anon_vma_init
                          > c0012374 t procswaps_init
                          > c00123ac t init_tmpfs
                          > c0012498 t noaliencache_setup
                          > c00124bc t cpucache_init
                          > c00124d8 t cpuup_callback
                          > c0012870 T kmem_cache_init
                          > c0012c0c T files_init
                          > c0012c58 T unnamed_dev_init
                          > c0012c74 T chrdev_init
                          > c0012ca4 t init_pipe_fs
                          > c0012cf8 t fasync_init
                          > c0012d44 t filelock_init
                          > c0012d98 t set_dhash_entries
                          > c0012ddc T vfs_caches_init
                          > c0012f64 T vfs_caches_init_early
                          > c001300c t set_ihash_entries
                          > c0013050 T inode_init
                          > c0013138 T inode_init_early
                          > c00131dc T files_defer_init
                          > c0013224 T mnt_init
                          > c0013468 t aio_setup
                          > c0013500 T buffer_init
                          > c0013574 t init_bio
                          > c00136bc T bdev_cache_init
                          > c0013764 t inotify_setup
                          > c0013788 t inotify_user_setup
                          > c0013878 t eventpoll_init
                          > c0013958 t init_script_binfmt
                          > c0013974 t init_elf_binfmt
                          > c0013990 t dnotify_init
                          > c00139dc T proc_init_inodecache
                          > c0013a38 T proc_root_init
                          > c0013b4c T proc_tty_init
                          > c0013c00 T proc_misc_init
                          > c0013db4 T sysfs_init
                          > c0013e78 t init_devpts_fs
                          > c0013ec0 T init_rootfs
                          > c0013edc t init_ramfs_fs
                          > c0013ef8 T jffs2_compressors_init
                          > c0013f14 T jffs2_create_slab_caches
                          > c00140c4 t init_jffs2_fs
                          > c00141a8 T jffs2_zlib_init
                          > c0014254 t ipc_init
                          > c0014274 T ipc_init_ids
                          > c0014324 T ipc_init_proc_interface
                          > c00143a4 T msg_init
                          > c001440c T sem_init
                          > c0014484 T shm_init
                          > c00144f4 t ipc_sysctl_init
                          > c0014514 t crypto_algapi_init
                          > c001452c T crypto_init_proc
                          > c0014560 t cryptomgr_init
                          > c001457c t elevator_setup
                          > c00145d0 T blk_dev_init
                          > c0014704 t genhd_device_init
                          > c0014768 t noop_init
                          > c0014784 t deadline_init
                          > c00147a0 t kobject_uevent_init
                          > c00147f4 T prio_tree_init
                          > c0014834 T radix_tree_init
                          > c00148c0 t random32_init
                          > c00148ec t random32_reseed
                          > c0014920 t pci_sort_bf_cmp
                          > c0014974 t pcibus_class_init
                          > c0014990 T pci_sort_breadthfirst
                          > c0014bac t quirk_eisa_bridge
                          > c0014bc8 t quirk_tc86c001_ide
                          > c0014bf4 t quirk_alimagik
                          > c0014c30 t asus_hides_smbus_hostbridge
                          > c0014ea8 t combined_setup
                          > c0014f10 t quirk_ide_samemode
                          > c0014f88 t pci_driver_init
                          > c0014fa4 t pci_sysfs_init
                          > c0014ff4 t pci_proc_init
                          > c0015084 T pci_assign_unassigned_resources
                          > c00150f4 T pci_fixup_irqs
                          > c0015198 t chr_dev_init
                          > c001522c t rand_initialize
                          > c0015264 t seqgen_init
                          > c0015280 t tty_class_init
                          > c00152bc t tty_init
                          > c0015424 T console_init
                          > c0015468 t pty_init
                          > c0015688 t misc_init
                          > c0015724 t ixp4xx_rng_init
                          > c001577c T uart_get_console
                          > c0015804 T uart_set_options
                          > c0015904 T uart_parse_options
                          > c0015980 t serial8250_console_setup
                          > c0015a50 t serial8250_isa_init_ports
                          > c0015adc T early_serial_setup
                          > c0015b3c t serial8250_console_init
                          > c0015b60 t serial8250_init
                          > c0015c7c T serial8250_start_console
                          > c0015d5c t serial_in
                          > c0015ddc t wait_for_xmitr
                          > c0015e0c t early_uart_console_init
                          > c0015e48 t serial_out
                          > c0015ea0 t early_uart_setup
                          > c001618c t putc
                          > c00161b8 t early_uart_write
                          > c0016228 t early_uart_console_switch
                          > c00162c0 T early_serial_console_init
                          > c0016318 T devices_init
                          > c0016334 T system_bus_init
                          > c0016360 T buses_init
                          > c001637c T classes_init
                          > c00163c0 T platform_bus_init
                          > c0016404 T cpu_dev_init
                          > c0016420 T firmware_init
                          > c001643c T driver_init
                          > c001646c T attribute_container_init
                          > c0016490 t firmware_class_init
                          > c0016500 t rd_init
                          > c00166ac t ramdisk_blocksize
                          > c00166d8 t ramdisk_size
                          > c0016704 t ramdisk_size2
                          > c0016718 t probe_list2
                          > c0016780 t net_olddevs_init
                          > c001684c t loopback_init
                          > c0016868 t init_npedriver
                          > c00168d0 t npe_ucode_init
                          > c0016900 t init_scsi
                          > c0016990 T scsi_init_queue
                          > c0016aa0 T scsi_init_devinfo
                          > c0016b50 T scsi_init_sysctl
                          > c0016b88 T scsi_init_procfs
                          > c0016c04 t init_sd
                          > c0016d80 t wait_scan_init
                          > c0016d98 t init_mtd
                          > c0016de0 t redboot_parser_init
                          > c0016dfc t init_mtdchar
                          > c0016e94 t init_mtdblock
                          > c0016eb0 t init_nftl
                          > c0016ee0 t cfi_probe_init
                          > c0016f00 t ixp4xx_flash_init
                          > c0016f1c t usb_init
                          > c0017010 T usbdev_init
                          > c00170fc T usbfs_init
                          > c0017154 t usb_stor_init
                          > c00171a0 t input_init
                          > c00172c0 t evdev_init
                          > c00172dc t ixp4xx_spkr_init
                          > c00172f8 t rtc_hctosys
                          > c0017438 t rtc_init
                          > c0017490 t rtc_sysfs_init
                          > c00174ac t rtc_proc_init
                          > c00174c8 t rtc_dev_init
                          > c0017588 t x1205_init
                          > c00175a8 t isl1208_init
                          > c00175c8 t ds1307_init
                          > c00175e8 t ds1672_init
                          > c0017608 t pcf8563_init
                          > c0017628 t i2c_init
                          > c0017674 t i2c_dev_init
                          > c0017718 t ixp4xx_i2c_init
                          > c0017734 t eeprom_init
                          > c0017754 t leds_init
                          > c0017790 t ixp4xxgpioled_init
                          > c00177ac t timer_trig_init
                          > c00177c8 t heartbeat_trig_init
                          > c00177e4 t leds_cpu_trigger_init
                          > c0017814 t sock_init
                          > c001789c T sk_init
                          > c0017918 t proto_init
                          > c0017960 T skb_init
                          > c00179d4 T netdev_boot_setup
                          > c0017ab8 t net_dev_init
                          > c0017c9c T dev_mcast_init
                          > c0017cd8 T dst_init
                          > c0017cf4 T rtnetlink_init
                          > c0017dd0 t flow_cache_init
                          > c0017f1c t wireless_nlevent_init
                          > c0017f48 T wireless_proc_init
                          > c0017f90 t netpoll_init
                          > c0017fbc t netlink_proto_init
                          > c0018130 t genl_init
                          > c00181f4 T netfilter_init
                          > c00182ac T netfilter_log_init
                          > c00182f4 T netfilter_queue_init
                          > c001833c t set_rhash_entries
                          > c0018380 T ip_rt_init
                          > c0018628 T inet_initpeers
                          > c0018708 T ip_init
                          > c0018720 t set_thash_entries
                          > c0018764 T tcp_init
                          > c0018a18 T tcp_v4_init
                          > c0018a54 T tcp4_proc_init
                          > c0018a70 t tcp_congestion_default
                          > c0018a8c T raw_proc_exit
                          > c0018ab4 T raw_proc_init
                          > c0018afc T udp4_proc_init
                          > c0018b18 T udplite4_register
                          > c0018ba8 T arp_init
                          > c0018c40 T icmp_init
                          > c0018cd4 T devinet_init
                          > c0018d3c t inet_init
                          > c00190b0 T igmp_mc_proc_init
                          > c0019114 T ip_fib_init
                          > c00191bc T fib_proc_exit
                          > c00191e4 T fib_proc_init
                          > c001922c T fib_hash_init
                          > c001932c T ip_misc_proc_init
                          > c00193e8 t cubictcp_register
                          > c0019470 T xfrm4_init
                          > c0019490 T xfrm4_state_init
                          > c00194ac T xfrm_init
                          > c00195bc T xfrm_state_init
                          > c0019678 T xfrm_input_init
                          > c00196c0 t af_unix_init
                          > c0019740 t packet_init
                          > c00197b0 t __80200_A0_A1_proc_info
                          > c00197b0 T __proc_info_begin
                          > c00197b0 T _einittext
                          > c00197e4 t __80200_proc_info
                          > c0019818 t __80219_proc_info
                          > c001984c t __8032x_proc_info
                          > c0019880 t __8033x_proc_info
                          > c00198b4 t __pxa250_proc_info
                          > c00198e8 t __pxa210_proc_info
                          > c001991c t __ixp2400_proc_info
                          > c0019950 t __ixp2800_proc_info
                          > c0019984 t __ixp42x_proc_info
                          > c00199b8 t __ixp46x_proc_info
                          > c00199ec t __pxa255_proc_info
                          > c0019a20 t __pxa270_proc_info
                          > c0019a54 T __arch_info_begin
                          > c0019a54 t __mach_desc_IXDP425
                          > c0019a54 T __proc_info_end
                          > c0019a88 t __mach_desc_LOFT
                          > c0019abc t __mach_desc_AVILA
                          > c0019af0 t __mach_desc_IXDPG425
                          > c0019b24 t __mach_desc_NSLU2
                          > c0019b58 t __mach_desc_NAS100D
                          > c0019b8c t __mach_desc_DSMG600
                          > c0019bc0 t __mach_desc_FSG
                          > c0019bf4 T __arch_info_end
                          > c0019bf4 T __tagtable_begin
                          > c0019bf4 t __tagtable_parse_tag_cmdline
                          > c0019bfc t __tagtable_parse_tag_revision
                          > c0019c04 t __tagtable_parse_tag_serialnr
                          > c0019c0c t __tagtable_parse_tag_initrd2
                          > c0019c14 t __tagtable_parse_tag_initrd
                          > c0019c1c t __tagtable_parse_tag_ramdisk
                          > c0019c24 t __tagtable_parse_tag_mem32
                          > c0019c2c t __tagtable_parse_tag_core
                          > c0019c34 T __tagtable_end
                          > c0019c40 t __setup_initcall_debug_setup
                          > c0019c40 T __setup_start
                          > c0019c4c t __setup_rdinit_setup
                          > c0019c58 t __setup_init_setup
                          > c0019c64 t __setup_loglevel
                          > c0019c70 t __setup_quiet_kernel
                          > c0019c7c t __setup_debug_kernel
                          > c0019c88 t __setup_set_reset_devices
                          > c0019c94 t __setup_maxcpus
                          > c0019ca0 t __setup_nosmp
                          > c0019cac t __setup_root_delay_setup
                          > c0019cb8 t __setup_fs_names_setup
                          > c0019cc4 t __setup_root_data_setup
                          > c0019cd0 t __setup_root_dev_setup
                          > c0019cdc t __setup_readwrite
                          > c0019ce8 t __setup_readonly
                          > c0019cf4 t __setup_load_ramdisk
                          > c0019d00 t __setup_ramdisk_start_setup
                          > c0019d0c t __setup_prompt_ramdisk
                          > c0019d18 t __setup_lpj_setup
                          > c0019d24 t __setup_reboot_setup
                          > c0019d30 t __setup_hlt_setup
                          > c0019d3c t __setup_nohlt_setup
                          > c0019d48 t __setup_fpe_setup
                          > c0019d54 t __setup_noalign_setup
                          > c0019d60 t __setup_pause_on_oops_setup
                          > c0019d6c t __setup_panic_setup
                          > c0019d78 t __setup_console_setup
                          > c0019d84 t __setup_printk_time_setup
                          > c0019d90 t __setup_ignore_loglevel_setup
                          > c0019d9c t __setup_log_buf_len_setup
                          > c0019da8 t __setup_profile_setup
                          > c0019db4 t __setup_reserve_setup
                          > c0019dc0 t __setup_boot_override_clock
                          > c0019dcc t __setup_boot_override_clocksource
                          > c0019dd8 t __setup_irqpoll_setup
                          > c0019de4 t __setup_irqfixup_setup
                          > c0019df0 t __setup_noirqdebug_setup
                          > c0019dfc t __setup_disable_randmaps
                          > c0019e08 t __setup_noaliencache_setup
                          > c0019e14 t __setup_set_dhash_entries
                          > c0019e20 t __setup_set_ihash_entries
                          > c0019e2c t __setup_elevator_setup
                          > c0019e38 t __setup_pci_setup
                          > c0019e44 t __setup_combined_setup
                          > c0019e50 t __setup_ramdisk_blocksize
                          > c0019e5c t __setup_ramdisk_size2
                          > c0019e68 t __setup_ramdisk_size
                          > c0019e74 t __setup_netdev_boot_setup
                          > c0019e80 t __setup_netdev_boot_setup
                          > c0019e8c t __setup_set_rhash_entries
                          > c0019e98 t __setup_set_thash_entries
                          > c0019ea4 T __early_begin
                          > c0019ea4 t __early_early_mem
                          > c0019ea4 T __setup_end
                          > c0019eac t __early_early_initrd
                          > c0019eb4 t __early_early_ecc
                          > c0019ebc t __early_early_nowrite
                          > c0019ec4 t __early_early_nocache
                          > c0019ecc t __early_early_cachepolicy
                          > c0019ed4 T __early_end
                          > c0019ed4 t __initcall_ptrace_break_init1
                          > c0019ed4 T __initcall_start
                          > c0019ed8 t __initcall_consistent_init1
                          > c0019edc t __initcall_sysctl_init1
                          > c0019ee0 t __initcall_helper_init1
                          > c0019ee4 t __initcall_init_jiffies_clocksource1
                          > c0019ee8 t __initcall_ksysfs_init1
                          > c0019eec t __initcall_filelock_init1
                          > c0019ef0 t __initcall_init_script_binfmt1
                          > c0019ef4 t __initcall_init_elf_binfmt1
                          > c0019ef8 t __initcall_random32_init1
                          > c0019efc t __initcall_sock_init1
                          > c0019f00 t __initcall_netpoll_init1
                          > c0019f04 t __initcall_netlink_proto_init1
                          > c0019f08 t __initcall_kobject_uevent_init2
                          > c0019f0c t __initcall_pcibus_class_init2
                          > c0019f10 t __initcall_pci_driver_init2
                          > c0019f14 t __initcall_tty_class_init2
                          > c0019f18 t __initcall_customize_machine3
                          > c0019f1c t __initcall_topology_init4
                          > c0019f20 t __initcall_ixdp425_pci_init4
                          > c0019f24 t __initcall_avila_pci_init4
                          > c0019f28 t __initcall_ixdpg425_pci_init4
                          > c0019f2c t __initcall_nslu2_pci_init4
                          > c0019f30 t __initcall_nas100d_pci_init4
                          > c0019f34 t __initcall_dsmg600_pci_init4
                          > c0019f38 t __initcall_fsg_pci_init4
                          > c0019f3c t __initcall_param_sysfs_init4
                          > c0019f40 t __initcall_init_bio4
                          > c0019f44 t __initcall_genhd_device_init4
                          > c0019f48 t __initcall_misc_init4
                          > c0019f4c t __initcall_init_scsi4
                          > c0019f50 t __initcall_usb_init4
                          > c0019f54 t __initcall_input_init4
                          > c0019f58 t __initcall_rtc_init4
                          > c0019f5c t __initcall_rtc_sysfs_init4
                          > c0019f60 t __initcall_rtc_proc_init4
                          > c0019f64 t __initcall_rtc_dev_init4
                          > c0019f68 t __initcall_i2c_init4
                          > c0019f6c t __initcall_leds_init4
                          > c0019f70 t __initcall_proto_init4
                          > c0019f74 t __initcall_net_dev_init4
                          > c0019f78 t __initcall_wireless_nlevent_init4
                          > c0019f7c t __initcall_genl_init4
                          > c0019f80 t __initcall_alignment_init5
                          > c0019f84 t __initcall_clocksource_done_booting5
                          > c0019f88 t __initcall_init_pipe_fs5
                          > c0019f8c t __initcall_chr_dev_init5
                          > c0019f90 t __initcall_firmware_class_init5
                          > c0019f94 t __initcall_inet_init5
                          > c0019f98 t __initcall_default_rootfsrootfs
                          > c0019f9c t __initcall_timer_init_sysfs6
                          > c0019fa0 t __initcall_nslu2_power_init6
                          > c0019fa4 t __initcall_nas100d_power_init6
                          > c0019fa8 t __initcall_dsmg600_power_init6
                          > c0019fac t __initcall_fsg_buttons_init6
                          > c0019fb0 t __initcall_fpe_init6
                          > c0019fb4 t __initcall_create_proc_profile6
                          > c0019fb8 t __initcall_ioresources_init6
                          > c0019fbc t __initcall_timekeeping_init_device6
                          > c0019fc0 t __initcall_uid_cache_init6
                          > c0019fc4 t __initcall_init_posix_timers6
                          > c0019fc8 t __initcall_init_posix_cpu_timers6
                          > c0019fcc t __initcall_latency_init6
                          > c0019fd0 t __initcall_init_clocksource_sysfs6
                          > c0019fd4 t __initcall_init_timer_list_procfs6
                          > c0019fd8 t __initcall_init6
                          > c0019fdc t __initcall_crash_notes_memory_init6
                          > c0019fe0 t __initcall_utsname_sysctl_init6
                          > c0019fe4 t __initcall_init_per_zone_pages_min6
                          > c0019fe8 t __initcall_pdflush_init6
                          > c0019fec t __initcall_kswapd_init6
                          > c0019ff0 t __initcall_procswaps_init6
                          > c0019ff4 t __initcall_init_tmpfs6
                          > c0019ff8 t __initcall_cpucache_init6
                          > c0019ffc t __initcall_fasync_init6
                          > c001a000 t __initcall_aio_setup6
                          > c001a004 t __initcall_inotify_setup6
                          > c001a008 t __initcall_inotify_user_setup6
                          > c001a00c t __initcall_eventpoll_init6
                          > c001a010 t __initcall_dnotify_init6
                          > c001a014 t __initcall_init_devpts_fs6
                          > c001a018 t __initcall_init_ramfs_fs6
                          > c001a01c t __initcall_init_jffs2_fs6
                          > c001a020 t __initcall_ipc_init6
                          > c001a024 t __initcall_ipc_sysctl_init6
                          > c001a028 t __initcall_crypto_algapi_init6
                          > c001a02c t __initcall_cryptomgr_init6
                          > c001a030 t __initcall_noop_init6
                          > c001a034 t __initcall_deadline_init6
                          > c001a038 t __initcall_pci_init6
                          > c001a03c t __initcall_pci_sysfs_init6
                          > c001a040 t __initcall_pci_proc_init6
                          > c001a044 t __initcall_rand_initialize6
                          > c001a048 t __initcall_tty_init6
                          > c001a04c t __initcall_pty_init6
                          > c001a050 t __initcall_ixp4xx_rng_init6
                          > c001a054 t __initcall_serial8250_init6
                          > c001a058 t __initcall_rd_init6
                          > c001a05c t __initcall_net_olddevs_init6
                          > c001a060 t __initcall_loopback_init6
                          > c001a064 t __initcall_init_npedriver6
                          > c001a068 t __initcall_npe_ucode_init6
                          > c001a06c t __initcall_init_sd6
                          > c001a070 t __initcall_init_mtd6
                          > c001a074 t __initcall_redboot_parser_init6
                          > c001a078 t __initcall_init_mtdchar6
                          > c001a07c t __initcall_init_mtdblock6
                          > c001a080 t __initcall_init_nftl6
                          > c001a084 t __initcall_cfi_probe_init6
                          > c001a088 t __initcall_ixp4xx_flash_init6
                          > c001a08c t __initcall_usb_stor_init6
                          > c001a090 t __initcall_evdev_init6
                          > c001a094 t __initcall_ixp4xx_spkr_init6
                          > c001a098 t __initcall_x1205_init6
                          > c001a09c t __initcall_isl1208_init6
                          > c001a0a0 t __initcall_ds1307_init6
                          > c001a0a4 t __initcall_ds1672_init6
                          > c001a0a8 t __initcall_pcf8563_init6
                          > c001a0ac t __initcall_i2c_dev_init6
                          > c001a0b0 t __initcall_ixp4xx_i2c_init6
                          > c001a0b4 t __initcall_eeprom_init6
                          > c001a0b8 t __initcall_ixp4xxgpioled_init6
                          > c001a0bc t __initcall_timer_trig_init6
                          > c001a0c0 t __initcall_heartbeat_trig_init6
                          > c001a0c4 t __initcall_leds_cpu_trigger_init6
                          > c001a0c8 t __initcall_flow_cache_init6
                          > c001a0cc t __initcall_ipv4_netfilter_init6
                          > c001a0d0 t __initcall_cubictcp_register6
                          > c001a0d4 t __initcall_af_unix_init6
                          > c001a0d8 t __initcall_packet_init6
                          > c001a0dc t __initcall_xscale_cp0_init7
                          > c001a0e0 t __initcall_random32_reseed7
                          > c001a0e4 t __initcall_seqgen_init7
                          > c001a0e8 t __initcall_early_uart_console_switch7
                          > c001a0ec t __initcall_wait_scan_init7
                          > c001a0f0 t __initcall_rtc_hctosys7
                          > c001a0f4 t __initcall_tcp_congestion_default7
                          > c001a0f8 T __con_initcall_start
                          > c001a0f8 T __initcall_end
                          > c001a0f8 t __initcall_serial8250_console_init
                          > c001a0fc t __initcall_early_uart_console_init
                          > c001a100 T __con_initcall_end
                          > c001a100 T __per_cpu_end
                          > c001a100 T __per_cpu_start
                          > c001a100 T __security_initcall_end
                          > c001a100 T __security_initcall_start
                          > c001a100 t tmp_cmdline.24931
                          > c001a500 t done.24930
                          > c001a504 t initcall_debug
                          > c001a508 T boot_command_line
                          > c001a908 t __setup_str_initcall_debug_setup
                          > c001a918 t __setup_str_rdinit_setup
                          > c001a920 t __setup_str_init_setup
                          > c001a928 t __setup_str_loglevel
                          > c001a938 t __setup_str_quiet_kernel
                          > c001a940 t __setup_str_debug_kernel
                          > c001a948 t __setup_str_set_reset_devices
                          > c001a958 t __setup_str_maxcpus
                          > c001a968 t __setup_str_nosmp
                          > c001a970 t saved_root_name
                          > c001a9b0 t root_mount_data
                          > c001a9b4 t root_fs_names
                          > c001a9b8 t root_delay
                          > c001a9bc T root_device_name
                          > c001a9c0 T rd_doload
                          > c001a9c8 t __setup_str_root_delay_setup
                          > c001a9d8 t __setup_str_fs_names_setup
                          > c001a9e8 t __setup_str_root_data_setup
                          > c001a9f8 t __setup_str_root_dev_setup
                          > c001aa00 t __setup_str_readwrite
                          > c001aa08 t __setup_str_readonly
                          > c001aa10 t __setup_str_load_ramdisk
                          > c001aa20 T rd_prompt
                          > c001aa24 T rd_image_start
                          > c001aa28 t __setup_str_ramdisk_start_setup
                          > c001aa38 t __setup_str_prompt_ramdisk
                          > c001aa48 t __setup_str_lpj_setup
                          > c001aa50 T init_arch_irq
                          > c001aa58 t __setup_str_reboot_setup
                          > c001aa60 t __setup_str_hlt_setup
                          > c001aa68 t __setup_str_nohlt_setup
                          > c001aa70 T phys_initrd_start
                          > c001aa74 T phys_initrd_size
                          > c001aa78 t meminfo
                          > c001aadc t usermem.16395
                          > c001aae0 t default_command_line
                          > c001aee0 t init_tags
                          > c001af0c t endian_test
                          > c001af10 t command_line
                          > c001b310 t init_machine
                          > c001b318 t __setup_str_fpe_setup
                          > c001b320 t cache_policies
                          > c001b3ac t cachepolicy
                          > c001b3b0 t ecc_mask
                          > c001b3b4 t mem_types
                          > c001b448 t __setup_str_noalign_setup
                          > c001b450 T xscale_mc_user_fns
                          > c001b458 T v4wbi_tlb_fns
                          > c001b464 T xscale_processor_functions
                          > c001b488 t ixp4xx_devices
                          > c001b48c t ixp4xx_io_desc
                          > c001b4bc t ixdp425_devices
                          > c001b4d0 t avila_devices
                          > c001b4e8 t coyote_devices
                          > c001b4f0 t nslu2_devices
                          > c001b508 t nslu2_rtc_probe
                          > c001b520 t nas100d_devices
                          > c001b538 t nas100d_rtc_probe
                          > c001b558 t dsmg600_devices
                          > c001b560 t dsmg600_rtc_probe
                          > c001b580 t fsg_devices
                          > c001b594 T ixdp425_pci
                          > c001b5b8 T avila_pci
                          > c001b5dc T ixdpg425_pci
                          > c001b600 T nslu2_pci
                          > c001b624 T nas100d_pci
                          > c001b648 T dsmg600_pci
                          > c001b66c T fsg_pci
                          > c001b690 t __setup_str_pause_on_oops_setup
                          > c001b6a0 t __setup_str_panic_setup
                          > c001b6a8 t __setup_str_console_setup
                          > c001b6b8 t __setup_str_printk_time_setup
                          > c001b6c0 t __setup_str_ignore_loglevel_setup
                          > c001b6d0 t __setup_str_log_buf_len_setup
                          > c001b6e0 t kvmstr.15456
                          > c001b6e8 t sleepstr.15455
                          > c001b6f0 t schedstr.15454
                          > c001b700 t __setup_str_profile_setup
                          > c001b710 t cpu_nfb
                          > c001b720 t __setup_str_reserve_setup
                          > c001b730 t timers_nb
                          > c001b73c t rcu_nb
                          > c001b748 t hrtimers_nb
                          > c001b758 t __setup_str_boot_override_clock
                          > c001b760 t __setup_str_boot_override_clocksource
                          > c001b770 t __setup_str_irqpoll_setup
                          > c001b778 t __setup_str_irqfixup_setup
                          > c001b788 t __setup_str_noirqdebug_setup
                          > c001b798 t dma_reserve
                          > c001b79c T nr_kernel_pages
                          > c001b7a0 T nr_all_pages
                          > c001b7a8 t __setup_str_disable_randmaps
                          > c001b7b8 t initarray_cache
                          > c001b7cc t cache_names
                          > c001b84c T initkmem_list3
                          > c001b8e8 t __setup_str_noaliencache_setup
                          > c001b8f8 t dhash_entries
                          > c001b900 t __setup_str_set_dhash_entries
                          > c001b910 t ihash_entries
                          > c001b918 t __setup_str_set_ihash_entries
                          > c001b928 t __setup_str_elevator_setup
                          > c001b938 t __setup_str_pci_setup
                          > c001b940 t __setup_str_combined_setup
                          > c001b950 t early_uart_console
                          > c001b97c t early_device
                          > c001ba10 t early_uart_registered
                          > c001ba18 t __setup_str_ramdisk_blocksize
                          > c001ba30 t __setup_str_ramdisk_size2
                          > c001ba40 t __setup_str_ramdisk_size
                          > c001ba50 t m68k_probes
                          > c001ba58 t eisa_probes
                          > c001ba60 t mca_probes
                          > c001ba68 t isa_probes
                          > c001ba70 t parport_probes
                          > c001ba78 t scsi_static_device_list
                          > c001c4c8 t __setup_str_netdev_boot_setup
                          > c001c4d0 t __setup_str_netdev_boot_setup
                          > c001c4d8 t rhash_entries
                          > c001c4e0 t __setup_str_set_rhash_entries
                          > c001c4f0 t thash_entries
                          > c001c4f8 t __setup_str_set_thash_entries
                          > c001d000 T __init_end
                          > c001d000 T _text
                          > c001d000 t run_init_process
                          > c001d02c t init_post
                          > c001d114 t rest_init
                          > c001d164 t try_name
                          > c001d340 T name_to_dev_t
                          > c001d5c4 T calibrate_delay
                          > c001d760 t __pabt_invalid
                          > c001d770 t __dabt_invalid
                          > c001d780 t __irq_invalid
                          > c001d790 t __und_invalid
                          > c001d79c t common_invalid
                          > c001d7c0 t __dabt_svc
                          > c001d820 t __irq_svc
                          > c001d890 t preempt_return
                          > c001d8b0 t svc_preempt
                          > c001d900 t __und_svc
                          > c001d960 t __pabt_svc
                          > c001d9e0 t __dabt_usr
                          > c001da40 t __irq_usr
                          > c001dae0 t __und_usr
                          > c001db30 t call_fpe
                          > c001db98 t do_fpe
                          > c001dba8 t no_fp
                          > c001dbac t __und_usr_unknown
                          > c001dbc0 t __pabt_usr
                          > c001dc08 T ret_from_exception
                          > c001dc18 T __switch_to
                          > c001dc60 t ret_fast_syscall
                          > c001dc8c t fast_work_pending
                          > c001dc90 t work_pending
                          > c001dcb0 t work_resched
                          > c001dcb4 t ret_slow_syscall
                          > c001dcb4 T ret_to_user
                          > c001dcc4 t no_work_pending
                          > c001dce0 T ret_from_fork
                          > c001dd20 T vector_swi
                          > c001dda8 t __sys_trace
                          > c001ddd4 t __sys_trace_return
                          > c001de00 t __cr_alignment
                          > c001de04 T sys_call_table
                          > c001e374 t sys_syscall
                          > c001e39c t sys_fork_wrapper
                          > c001e3a4 t sys_vfork_wrapper
                          > c001e3ac t sys_execve_wrapper
                          > c001e3b4 t sys_clone_wrapper
                          > c001e3c0 t sys_sigsuspend_wrapper
                          > c001e3c8 t sys_rt_sigsuspend_wrapper
                          > c001e3d0 t sys_sigreturn_wrapper
                          > c001e3d8 t sys_rt_sigreturn_wrapper
                          > c001e3e0 t sys_sigaltstack_wrapper
                          > c001e3e8 t sys_statfs64_wrapper
                          > c001e3f4 t sys_fstatfs64_wrapper
                          > c001e400 t sys_mmap2
                          > c001e420 T set_irq_flags
                          > c001e4d8 T show_interrupts
                          > c001e664 T asm_do_IRQ
                          > c001e6c0 t kernel_thread_helper
                          > c001e6cc T disable_hlt
                          > c001e6f0 T enable_hlt
                          > c001e714 T machine_halt
                          > c001e724 T machine_power_off
                          > c001e74c T machine_restart
                          > c001e778 T exit_thread
                          > c001e788 T get_wchan
                          > c001e81c T copy_thread
                          > c001e894 T kernel_thread
                          > c001e914 T dump_fpu
                          > c001e954 T release_thread
                          > c001e978 T show_fpregs
                          > c001ea30 T __show_regs
                          > c001ebe0 T arm_machine_restart
                          > c001ec60 T cpu_idle
                          > c001ed04 T show_regs
                          > c001ed50 T dump_thread
                          > c001ee34 T flush_thread
                          > c001ee88 t default_idle
                          > c001eed4 t ptrace_getrn
                          > c001ef08 t swap_insn
                          > c001ef6c T ptrace_cancel_bpt
                          > c001f088 T ptrace_break
                          > c001f0e0 T ptrace_disable
                          > c001f100 t add_breakpoint
                          > c001f184 T ptrace_set_bpt
                          > c001f5a0 t break_trap
                          > c001f5c8 T syscall_trace
                          > c001f660 T arch_ptrace
                          > c001fa68 T __up
                          > c001fa8c T __down_trylock
                          > c001fb2c T cpu_architecture
                          > c001fb8c t c_start
                          > c001fbc0 t c_next
                          > c001fbec t c_stop
                          > c001fbfc T cpu_init
                          > c001fe00 t c_show_cache
                          > c001fe90 t c_show
                          > c0020194 t setup_return
                          > c0020288 t setup_sigframe
                          > c00203cc t do_signal
                          > c00208ac T do_notify_resume
                          > c00208dc t restore_sigframe
                          > c0020a74 T sys_sigaction
                          > c0020bb0 T sys_sigsuspend
                          > c0020c74 T sys_rt_sigsuspend
                          > c0020d98 T sys_sigreturn
                          > c0020e38 T sys_rt_sigreturn
                          > c0020eec T sys_arm_sync_file_range
                          > c0020f20 T sys_arm_fadvise64_64
                          > c0020f54 T kernel_execve
                          > c0020fdc T sys_execve
                          > c0021038 T sys_vfork
                          > c0021078 T sys_clone
                          > c00210b4 T sys_fork
                          > c00210f0 T sys_ipc
                          > c002132c T do_mmap2
                          > c00213ec T sys_pipe
                          > c002145c T sys_arm_mremap
                          > c00214e4 T old_select
                          > c002156c T old_mmap
                          > c0021698 T printk_clock
                          > c00216cc T timer_tick
                          > c00217b0 T save_time_delta
                          > c00217e0 T restore_time_delta
                          > c002181c T dump_stack
                          > c002182c T abort
                          > c002184c T __pgd_error
                          > c0021878 T __pmd_error
                          > c00218a4 T __pte_error
                          > c00218d0 T __readwrite_bug
                          > c00218f8 T __bug
                          > c0021924 T do_unexp_fiq
                          > c002194c T dump_backtrace_entry
                          > c0021970 T __bad_xchg
                          > c00219a0 T die
                          > c0021d88 T bad_mode
                          > c0021ddc T notify_die
                          > c0021e34 T baddataabort
                          > c0021e8c T arm_syscall
                          > c0022128 T register_undef_hook
                          > c002219c T unregister_undef_hook
                          > c002221c T do_undefinstr
                          > c00223b4 T show_stack
                          > c00223f4 T __div0
                          > c0022410 T module_frob_arch_sections
                          > c0022424 T module_finalize
                          > c0022438 T module_arch_cleanup
                          > c0022448 T apply_relocate_add
                          > c0022470 T apply_relocate
                          > c0022628 T module_free
                          > c0022640 T module_alloc
                          > c002269c t pci_fixup_unassign
                          > c00226c4 t pci_fixup_dec21285
                          > c0022710 t pci_fixup_prpmc1100
                          > c0022720 t pci_fixup_ide_bases
                          > c0022778 T pcibios_resource_to_bus
                          > c00227bc T pcibios_bus_to_resource
                          > c0022804 T pci_std_swizzle
                          > c0022854 T pcibios_align_resource
                          > c00228a0 T pci_mmap_page_range
                          > c0022914 t pcibios_bus_report_status
                          > c0022a10 T pcibios_report_status
                          > c0022a60 t pcibios_swizzle
                          > c0022ad0 t pcibios_map_irq
                          > c0022b48 T pcibios_enable_device
                          > c0022c48 t pci_fixup_dec21142
                          > c0022c6c t pci_fixup_cy82c693
                          > c0022d60 T pcibios_update_irq
                          > c0022db0 t pci_fixup_83c553
                          > c0022ea4 T pcibios_fixup_bus
                          > c0023104 T machine_kexec_prepare
                          > c0023118 T machine_kexec_cleanup
                          > c0023128 T machine_shutdown
                          > c0023138 T machine_crash_shutdown
                          > c0023148 T machine_kexec
                          > c00231fc T relocate_new_kernel
                          > c0023290 T kexec_start_address
                          > c0023294 T kexec_indirection_page
                          > c0023298 T kexec_mach_type
                          > c002329c T kexec_boot_params_address
                          > c00232a0 T kexec_boot_params_copy
                          > c00232a4 t relocate_new_kernel_end
                          > c00232a4 T relocate_new_kernel_size
                          > c00232a8 t dsp_do
                          > c00232fc T _memcpy_fromio
                          > c0023324 T _memcpy_toio
                          > c002334c T _memset_io
                          > c0023380 t vm_region_find
                          > c00233cc T consistent_sync
                          > c0023454 t dma_mmap
                          > c0023548 T dma_mmap_writecombine
                          > c0023574 T dma_mmap_coherent
                          > c00235a0 t __dma_alloc
                          > c00239f4 T dma_alloc_writecombine
                          > c0023a20 T dma_alloc_coherent
                          > c0023a4c T dma_free_coherent
                          > c0023ce0 T fixup_exception
                          > c0023d10 t do_bad
                          > c0023d24 T show_pte
                          > c0023dec t __do_user_fault
                          > c0023e44 t __do_kernel_fault
                          > c0023ec4 t do_page_fault
                          > c00240fc T do_bad_area
                          > c0024178 t do_sect_fault
                          > c0024190 t do_translation_fault
                          > c0024210 T do_PrefetchAbort
                          > c002422c T do_DataAbort
                          > c00242cc T free_initmem
                          > c0024384 T show_mem
                          > c002454c T pci_iounmap
                          > c0024598 T pci_iomap
                          > c0024620 t adjust_pte
                          > c0024738 T update_mmu_cache
                          > c00248f8 T __flush_dcache_page
                          > c0024928 T __flush_anon_page
                          > c0024988 T flush_dcache_page
                          > c0024af8 T __check_kvm_seq
                          > c0024b68 t unmap_area_sections
                          > c0024c40 T __ioremap_pfn
                          > c0024f14 T __ioremap
                          > c0024f80 T __iounmap
                          > c0025040 T valid_phys_addr_range
                          > c0025070 T valid_mmap_phys_addr_range
                          > c0025090 T arch_get_unmapped_area
                          > c00251c0 T get_pgd_slow
                          > c00252a4 T free_pgd_slow
                          > c0025390 T adjust_cr
                          > c00253fc T setup_mm_for_reboot
                          > c0025484 t proc_alignment_write
                          > c00254d0 t do_alignment_ldrhstrh
                          > c00255a4 t do_alignment_ldrdstrd
                          > c0025794 t do_alignment_ldrstr
                          > c002589c t do_alignment_ldmstm
                          > c0025ac8 t proc_alignment_read
                          > c0025c2c t do_alignment
                          > c00261c0 T v5t_early_abort
                          > c00261ec t not_thumb
                          > c0026210 t not_ldrd
                          > c0026220 t mc_copy_user_page
                          > c00262b0 T xscale_mc_clear_user_page
                          > c00262e4 T xscale_mc_copy_user_page
                          > c0026400 T v4wbi_flush_user_tlb_range
                          > c002644c T v4wbi_flush_kern_tlb_range
                          > c0026480 T cpu_xscale_proc_init
                          > c0026484 T cpu_xscale_proc_fin
                          > c00264c0 T cpu_xscale_reset
                          > c0026500 T cpu_xscale_do_idle
                          > c002650c T xscale_flush_kern_cache_all
                          > c002650c T xscale_flush_user_cache_all
                          > c0026514 t __flush_whole_cache
                          > c0026560 T xscale_flush_user_cache_range
                          > c002659c T xscale_coherent_kern_range
                          > c00265c0 T xscale_coherent_user_range
                          > c00265e8 T xscale_flush_kern_dcache_page
                          > c0026610 T xscale_dma_inv_range
                          > c002663c T xscale_dma_clean_range
                          > c0026658 T xscale_dma_flush_range
                          > c0026678 T xscale_cache_fns
                          > c002669c T xscale_80200_A0_A1_cache_fns
                          > c00266c0 T cpu_xscale_dcache_clean_area
                          > c00266e0 T cpu_xscale_switch_mm
                          > c0026740 T cpu_xscale_set_pte_ext
                          > c00267a0 T unregister_rtc
                          > c0026800 t rtc_fasync
                          > c002681c T register_rtc
                          > c00268a0 t rtc_read_proc
                          > c0026b54 T rtc_next_alarm_time
                          > c0026bdc t rtc_release
                          > c0026c7c T rtc_update
                          > c0026d04 t rtc_poll
                          > c0026d90 t rtc_open
                          > c0026ef4 t rtc_read
                          > c002704c t rtc_ioctl
                          > c00273ac T dmabounce_unregister_dev
                          > c002744c t dmabounce_init_pool
                          > c0027490 T dmabounce_register_dev
                          > c00275b0 T dma_sync_sg_for_device
                          > c00276ec T dma_unmap_sg
                          > c00278d8 T dma_map_sg
                          > c0027be8 T dma_sync_single_for_cpu
                          > c0027cec T dma_sync_single_for_device
                          > c0027df0 T dma_sync_sg_for_cpu
                          > c0027f2c T dma_unmap_single
                          > c00280e0 T dma_map_single
                          > c002837c t ixp4xx_set_irq_type
                          > c002849c t ixp4xx_irq_ack
                          > c00284d4 T ixp4xx_get_cycles
                          > c00284f4 t ixp4xx_timer_interrupt
                          > c00285b8 t ixp4xx_irq_mask
                          > c00285e0 t ixp4xx_irq_unmask
                          > c002862c t loft_eeprom_do
                          > c00286bc t nslu2_flash_remove
                          > c00286cc t nslu2_power_off
                          > c00286f8 t nslu2_flash_add
                          > c00287c8 t nslu2_power_handler
                          > c00287e0 t nslu2_reset_handler
                          > c00287f8 t nas100d_flash_remove
                          > c0028808 t nas100d_power_off
                          > c0028834 t nas100d_flash_add
                          > c0028904 t nas100d_reset_handler
                          > c002891c t dsmg600_power_off
                          > c0028948 t dsmg600_power_handler
                          > c00289d8 t dsmg600_reset_handler
                          > c00289f0 t fsg_flash_remove
                          > c0028a00 t fsg_flash_add
                          > c0028b28 t fsg_unplug_button_handler
                          > c0028ba8 t hotplug_button
                          > c0028c84 t fsg_sync_button_handler
                          > c0028d04 t fsg_reset_button_handler
                          > c0028d84 t ixp4xx_config_addr
                          > c0028dd8 t byte_lane_enable_bits
                          > c0028e28 T dma_needs_bounce
                          > c0028e64 T pci_set_dma_mask
                          > c0028e90 T pci_set_consistent_dma_mask
                          > c0028ebc T ixp4xx_scan_bus
                          > c0028ef4 T ixp4xx_pci_write
                          > c0028f78 t local_write_config
                          > c0029060 T ixp4xx_setup
                          > c0029164 t ixp4xx_pci_platform_notify_remove
                          > c0029190 t ixp4xx_pci_platform_notify
                          > c00291e0 T ixp4xx_pci_read_no_errata
                          > c0029268 t ixp4xx_pci_read_config
                          > c0029324 t ixp4xx_pci_write_config
                          > c00293a8 t abort_handler
                          > c0029450 T ixp4xx_pci_read_errata
                          > c00294f8 T SetRoundingMode
                          > c0029534 T SetRoundingPrecision
                          > c0029548 T EmulateAll
                          > c00295a0 T nwfpe_init_fpa
                          > c00295fc T EmulateCPDO
                          > c0029768 T PerformLDF
                          > c002988c T PerformLFM
                          > c00299dc T PerformSFM
                          > c0029b34 T PerformSTF
                          > c0029cd0 T EmulateCPDT
                          > c0029d48 T PerformFIX
                          > c0029e0c T PerformFLT
                          > c0029f20 T EmulateCPRT
                          > c002a4c0 t nwfpe_notify
                          > c002a4e4 T float_raise
                          > c002a5b0 T checkCondition
                          > c002a5e0 t countLeadingZeros32
                          > c002a628 t countLeadingZeros64
                          > c002a658 T float32_is_nan
                          > c002a678 T float32_is_signaling_nan
                          > c002a6b4 T float64_is_nan
                          > c002a6ec T float64_is_signaling_nan
                          > c002a764 t roundAndPackInt32
                          > c002a84c t normalizeFloat32Subnormal
                          > c002a888 t roundAndPackFloat32
                          > c002a9fc t normalizeRoundAndPackFloat32
                          > c002aa44 t normalizeFloat64Subnormal
                          > c002aa90 t roundAndPackFloat64
                          > c002ad3c t normalizeRoundAndPackFloat64
                          > c002adbc t propagateFloat64NaN
                          > c002ae80 t subFloat64Sigs
                          > c002b1f0 t addFloat64Sigs
                          > c002b528 t propagateFloat32NaN
                          > c002b590 t subFloat32Sigs
                          > c002b734 t addFloat32Sigs
                          > c002b8b8 t estimateSqrt32
                          > c002b96c t estimateDiv128To64
                          > c002bd0c T float32_add
                          > c002bd3c T float32_sub
                          > c002bd6c T float64_add
                          > c002bdd4 T float64_sub
                          > c002be3c T float32_to_int32
                          > c002bf5c T float32_to_int32_round_to_zero
                          > c002c020 T float32_rem
                          > c002c350 T float32_sqrt
                          > c002c4d4 T float32_eq
                          > c002c574 T float32_le
                          > c002c624 T float32_lt
                          > c002c6d4 T float32_eq_signaling
                          > c002c758 T float32_le_quiet
                          > c002c7fc T float32_lt_quiet
                          > c002c8a0 T int32_to_float32
                          > c002c8e8 T float32_round_to_int
                          > c002ca3c T float32_mul
                          > c002cbc0 T float32_div
                          > c002cd80 T float64_to_int32
                          > c002cea8 T float64_to_int32_round_to_zero
                          > c002cfe8 T float64_to_uint32
                          > c002d0d8 T float64_to_uint32_round_to_zero
                          > c002d218 T float64_to_float32
                          > c002d368 T float64_rem
                          > c002d860 T float64_sqrt
                          > c002dd34 T float64_eq
                          > c002de28 T float64_le
                          > c002df48 T float64_lt
                          > c002e070 T float64_eq_signaling
                          > c002e13c T float64_le_quiet
                          > c002e250 T float64_lt_quiet
                          > c002e36c T int32_to_float64
                          > c002e3e8 T float32_to_float64
                          > c002e570 T float64_round_to_int
                          > c002e7c4 T float64_mul
                          > c002ec00 T float64_div
                          > c002f10c t float32_mvf
                          > c002f120 t float32_mnf
                          > c002f134 t float32_abs
                          > c002f148 T SingleCPDO
                          > c002f238 t float32_rsf
                          > c002f258 t float32_rdv
                          > c002f278 t float64_mvf
                          > c002f290 t float64_mnf
                          > c002f2ac t float64_abs
                          > c002f2c8 T DoubleCPDO
                          > c002f414 t float64_rsf
                          > c002f44c t float64_rdv
                          > c002f484 T nwfpe_enter
                          > c002f490 t emulate
                          > c002f49c t next
                          > c002f4d4 W sched_clock
                          > c002f500 t static_prio_timeslice
                          > c002f540 t dequeue_task
                          > c002f5e8 t enqueue_task
                          > c002f674 t requeue_task
                          > c002f6bc t deactivate_task
                          > c002f6fc T weighted_cpuload
                          > c002f718 T nr_running
                          > c002f734 T nr_context_switches
                          > c002f754 T nr_iowait
                          > c002f770 T current_sched_time
                          > c002f7c8 T account_user_time
                          > c002f828 T account_steal_time
                          > c002f8c4 t __wake_up_common
                          > c002f938 T __wake_up_locked
                          > c002f960 T task_prio
                          > c002f978 T task_nice
                          > c002f990 T idle_cpu
                          > c002f9bc T idle_task
                          > c002f9d8 T sys_sched_get_priority_max
                          > c002fa18 T sys_sched_get_priority_min
                          > c002fa58 T in_sched_functions
                          > c002fa94 T nr_active
                          > c002fabc T nr_uninterruptible
                          > c002fae0 t __activate_task
                          > c002fb28 t effective_prio
                          > c002fba4 t recalc_task_prio
                          > c002fc98 t set_load_weight
                          > c002fd08 T can_nice
                          > c002fd44 t __cond_resched
                          > c002fd84 T sys_sched_yield
                          > c002fe20 T sched_getaffinity
                          > c002feb4 T sched_setaffinity
                          > c002fffc T sys_sched_setaffinity
                          > c003009c T sched_setscheduler
                          > c0030404 t do_sched_setscheduler
                          > c00304e8 T sys_sched_setparam
                          > c0030504 T sys_sched_setscheduler
                          > c003052c t task_running_tick
                          > c00307fc t task_rq_lock
                          > c0030838 T set_user_nice
                          > c0030998 t try_to_wake_up
                          > c0030b4c T default_wake_function
                          > c0030b64 T wake_up_state
                          > c0030b7c T wake_up_process
                          > c0030b98 T account_system_time
                          > c0030c98 T show_state_filter
                          > c0030e08 T __wake_up
                          > c0030e74 T __wake_up_sync
                          > c0030ef0 T complete
                          > c0030f70 T complete_all
                          > c0030ff0 T sys_sched_rr_get_interval
                          > c00310f4 T schedule_tail
                          > c00311c0 T scheduler_tick
                          > c0031250 T sys_nice
                          > c00312e4 T sched_fork
                          > c00313cc T task_curr
                          > c00313f4 T cond_resched_lock
                          > c0031450 T sys_sched_getscheduler
                          > c00314c4 T sys_sched_getparam
                          > c00315c0 T sys_sched_getaffinity
                          > c003163c T rt_mutex_setprio
                          > c003171c T wake_up_new_task
                          > c00318ec T sched_exit
                          > c00319a0 T nr_processes
                          > c00319bc t sighand_ctor
                          > c00319cc T __cleanup_signal
                          > c00319f0 T __cleanup_sighand
                          > c0031a38 T copy_fs_struct
                          > c0031bf4 W idle_regs
                          > c0031c14 t dup_fd
                          > c0031ed4 t copy_files
                          > c0031f58 t mm_init
                          > c0031ffc T mm_alloc
                          > c0032044 T mm_release
                          > c00320cc T __mmdrop
                          > c0032110 T mmput
                          > c0032204 T free_task
                          > c0032238 t copy_process
                          > c0033428 T __put_task_struct
                          > c003350c T unshare_files
                          > c003357c T sys_set_tid_address
                          > c00335a4 T do_fork
                          > c0033770 T sys_unshare
                          > c0033c5c T get_task_mm
                          > c0033ce0 T __set_personality
                          > c0033f68 T get_exec_domain_list
                          > c003401c T register_exec_domain
                          > c00340c8 T unregister_exec_domain
                          > c0034174 t default_handler
                          > c00341e4 T sys_personality
                          > c0034234 t no_blink
                          > c0034248 T add_taint
                          > c003427c T oops_may_print
                          > c00342a0 t spin_msec
                          > c00342d4 t do_oops_enter_exit
                          > c003440c T oops_exit
                          > c0034420 T oops_enter
                          > c0034438 T print_tainted
                          > c0034504 T panic
                          > c0034648 t __call_console_drivers
                          > c00346c4 t _call_console_drivers
                          > c0034764 t emit_log_char
                          > c00347f8 T is_console_locked
                          > c0034814 T acquire_console_sem
                          > c00348d0 T printk_timed_ratelimit
                          > c003492c T tty_write_message
                          > c0034970 T wake_up_klogd
                          > c00349b8 T release_console_sem
                          > c0034c38 T unregister_console
                          > c0034cfc T console_start
                          > c0034d24 T console_stop
                          > c0034d4c T console_device
                          > c0034dac T console_unblank
                          > c0034e70 T resume_console
                          > c0034ea8 T vprintk
                          > c003526c T printk
                          > c003529c T __printk_ratelimit
                          > c00353b8 T printk_ratelimit
                          > c00353e4 T console_print
                          > c0035404 T suspend_console
                          > c0035434 T do_syslog
                          > c0035920 T sys_syslog
                          > c0035934 T register_console
                          > c0035b8c T try_acquire_console_sem
                          > c0035bfc t profile_event_register
                          > c0035c10 t profile_event_unregister
                          > c0035c24 T profile_hits
                          > c0035cb4 T create_prof_cpu_mask
                          > c0035d04 t write_profile
                          > c0035d40 t read_profile
                          > c0035e68 t prof_cpu_mask_write_proc
                          > c0035eac t prof_cpu_mask_read_proc
                          > c0035f00 T profile_tick
                          > c0035f88 t will_become_orphaned_pgrp
                          > c0036010 t has_stopped_jobs
                          > c0036058 t eligible_child
                          > c0036128 t wait_noreap_copyout
                          > c0036224 t delayed_put_task_struct
                          > c0036264 t exit_mm
                          > c003635c T exit_fs
                          > c0036444 T put_fs_struct
                          > c00364f8 t reparent_thread
                          > c0036694 T put_files_struct
                          > c0036770 T exit_files
                          > c00367cc T reset_files_struct
                          > c003681c T __set_special_pids
                          > c00368ac T session_of_pgrp
                          > c00368f0 T release_task
                          > c0036c9c t do_wait
                          > c0037938 T sys_wait4
                          > c003797c T sys_waitid
                          > c0037a08 T do_exit
                          > c0038350 T do_group_exit
                          > c0038408 T sys_exit_group
                          > c0038424 T sys_exit
                          > c003843c T complete_and_exit
                          > c0038460 T is_current_pgrp_orphaned
                          > c00384bc T disallow_signal
                          > c003855c T daemonize
                          > c0038870 T allow_signal
                          > c003892c T get_files_struct
                          > c0038998 t fixup_timeval
                          > c0038a68 t itimer_get_remtime
                          > c0038adc T do_setitimer
                          > c0038f20 T sys_setitimer
                          > c0039018 T alarm_setitimer
                          > c0039094 T do_getitimer
                          > c00392e4 T it_real_fn
                          > c0039308 T sys_getitimer
                          > c003937c T current_kernel_time
                          > c00393c4 T timespec_trunc
                          > c0039424 T mktime
                          > c00394c4 T set_normalized_timespec
                          > c0039518 T jiffies_to_msecs
                          > c0039530 T jiffies_to_usecs
                          > c003954c T msecs_to_jiffies
                          > c003957c T timespec_to_jiffies
                          > c00395e4 T timeval_to_jiffies
                          > c0039648 T jiffies_to_clock_t
                          > c0039658 T clock_t_to_jiffies
                          > c0039668 T get_jiffies_64
                          > c00396ac T nsec_to_clock_t
                          > c0039704 T jiffies_64_to_clock_t
                          > c0039714 T jiffies_to_timeval
                          > c003979c T ns_to_timespec
                          > c00398b4 T ns_to_timeval
                          > c00398f8 T sys_adjtimex
                          > c00399b4 T sys_stime
                          > c0039a14 T do_sys_settimeofday
                          > c0039b5c T sys_time
                          > c0039ba4 T current_fs_time
                          > c0039c0c T jiffies_to_timespec
                          > c0039c84 T sys_settimeofday
                          > c0039d6c T sys_gettimeofday
                          > c0039e24 T usecs_to_jiffies
                          > c0039e68 T open_softirq
                          > c0039e8c t tasklet_hi_action
                          > c0039f58 T tasklet_init
                          > c0039f80 T raise_softirq_irqoff
                          > c0039fec T local_bh_disable
                          > c003a010 T _local_bh_enable
                          > c003a0c8 T __do_softirq
                          > c003a198 T do_softirq
                          > c003a1fc t tasklet_action
                          > c003a2c8 T __local_bh_enable
                          > c003a38c T tasklet_kill
                          > c003a41c T local_bh_enable
                          > c003a4d8 T local_bh_enable_ip
                          > c003a548 T irq_exit
                          > c003a5ac t ksoftirqd
                          > c003a6b4 T irq_enter
                          > c003a6d8 T raise_softirq
                          > c003a750 T __tasklet_schedule
                          > c003a7d8 T __tasklet_hi_schedule
                          > c003a860 t r_next
                          > c003a8ac t __request_resource
                          > c003a924 t devm_region_match
                          > c003a970 t iomem_open
                          > c003a9a8 t ioports_open
                          > c003a9e0 t r_show
                          > c003aa7c T release_resource
                          > c003ab08 T request_resource
                          > c003ab5c T __release_region
                          > c003ac60 T __devm_release_region
                          > c003acec t devm_region_release
                          > c003ad04 T __request_region
                          > c003add0 T __devm_request_region
                          > c003ae50 T __check_region
                          > c003ae8c t r_start
                          > c003af20 T adjust_resource
                          > c003b014 t r_stop
                          > c003b048 T ____request_resource
                          > c003b094 T allocate_resource
                          > c003b1e0 T insert_resource
                          > c003b334 t sysctl_set_parent
                          > c003b384 t do_proc_dointvec_conv
                          > c003b3dc t do_proc_dointvec_bset_conv
                          > c003b464 t do_proc_dointvec_minmax_conv
                          > c003b4f8 t do_proc_dointvec_jiffies_conv
                          > c003b580 T sysctl_string
                          > c003b594 T sysctl_intvec
                          > c003b5a8 T sysctl_jiffies
                          > c003b5bc T sysctl_ms_jiffies
                          > c003b5d0 t do_proc_doulongvec_minmax
                          > c003b9ac T proc_doulongvec_ms_jiffies_minmax
                          > c003b9e8 T proc_doulongvec_minmax
                          > c003ba20 t __do_proc_dointvec
                          > c003be08 t do_proc_dointvec
                          > c003be60 T proc_dointvec_ms_jiffies
                          > c003bea0 T proc_dointvec_userhz_jiffies
                          > c003bee0 T proc_dointvec_jiffies
                          > c003bf20 T proc_dointvec_minmax
                          > c003bf70 T proc_dointvec
                          > c003bfa8 T proc_dostring
                          > c003c1a8 t proc_do_cad_pid
                          > c003c258 t proc_dointvec_taint
                          > c003c2d8 T sys_sysctl
                          > c003c420 t do_proc_dointvec_ms_jiffies_conv
                          > c003c48c t do_proc_dointvec_userhz_jiffies_conv
                          > c003c4f8 T sysctl_perm
                          > c003c550 t unuse_table
                          > c003c584 T unregister_sysctl_table
                          > c003c650 T sysctl_head_finish
                          > c003c69c T sysctl_head_next
                          > c003c760 T register_sysctl_table
                          > c003c810 T proc_dointvec_bset
                          > c003c8a4 T __capable
                          > c003c8d4 T capable
                          > c003c8f8 T sys_capget
                          > c003ca58 T sys_capset
                          > c003cdd8 T __ptrace_link
                          > c003ce54 T ptrace_get_task_struct
                          > c003cedc T ptrace_check_attach
                          > c003cfe8 T ptrace_traceme
                          > c003d068 T ptrace_request
                          > c003d338 T ptrace_writedata
                          > c003d418 T ptrace_readdata
                          > c003d4e0 t may_attach
                          > c003d5b4 T ptrace_may_attach
                          > c003d604 T ptrace_attach
                          > c003d73c T sys_ptrace
                          > c003d800 T ptrace_untrace
                          > c003d874 T __ptrace_unlink
                          > c003d908 T ptrace_detach
                          > c003d9b4 T __round_jiffies
                          > c003da08 T __round_jiffies_relative
                          > c003da34 T round_jiffies
                          > c003da4c T round_jiffies_relative
                          > c003da64 t internal_add_timer
                          > c003db00 T init_timer
                          > c003db28 T timekeeping_is_continuous
                          > c003db6c W read_persistent_clock
                          > c003db80 T getnstimeofday
                          > c003dc6c T do_gettimeofday
                          > c003dd5c t cascade
                          > c003dde8 t lock_timer_base
                          > c003de64 T del_timer
                          > c003def4 T __mod_timer
                          > c003dfe8 T mod_timer
                          > c003e034 t timekeeping_suspend
                          > c003e0d0 t run_timer_softirq
                          > c003e2f4 T do_sysinfo
                          > c003e498 T sys_sysinfo
                          > c003e4fc T msleep
                          > c003e528 T msleep_interruptible
                          > c003e578 t process_timeout
                          > c003e58c T sys_alarm
                          > c003e5a0 T do_timer
                          > c003ed80 T run_local_timers
                          > c003ed98 T update_process_times
                          > c003ee08 T sys_getpid
                          > c003ee28 T sys_getppid
                          > c003ee78 T sys_getuid
                          > c003ee98 T sys_geteuid
                          > c003eeb8 T sys_getgid
                          > c003eed8 T sys_getegid
                          > c003eef8 T sys_gettid
                          > c003ef18 T add_timer_on
                          > c003efa8 T do_settimeofday
                          > c003f114 t timekeeping_resume
                          > c003f228 T free_uid
                          > c003f2e4 T switch_uid
                          > c003f34c T find_user
                          > c003f410 T alloc_uid
                          > c003f5f4 t sig_ignored
                          > c003f674 T flush_signal_handlers
                          > c003f6cc T do_no_restart_syscall
                          > c003f6e0 W arch_vma_name
                          > c003f6f4 T sys_pause
                          > c003f724 T copy_siginfo_to_user
                          > c003f97c T do_sigpending
                          > c003fa7c T sys_sigpending
                          > c003fa94 T sys_rt_sigpending
                          > c003faa8 T lock_task_sighand
                          > c003fb24 t __sigqueue_free
                          > c003fb80 t rm_from_queue
                          > c003fc08 t rm_from_queue_full
                          > c003fcb8 T flush_sigqueue
                          > c003fd08 t check_kill_permission
                          > c003fdcc T signal_wake_up
                          > c003fe0c t __sigqueue_alloc
                          > c003fecc t send_signal
                          > c0040028 t specific_send_sig_info
                          > c00400e4 T send_sig_info
                          > c004017c T send_sig
                          > c00401a0 t __group_complete_signal
                          > c0040414 T __group_send_sig_info
                          > c00404b0 t do_notify_parent_cldstop
                          > c00405f0 t finish_stop
                          > c0040670 t handle_stop_signal
                          > c0040848 t do_tkill
                          > c0040994 T sys_tkill
                          > c00409c8 T sys_tgkill
                          > c00409f4 T group_send_sig_info
                          > c0040a90 T __kill_pgrp_info
                          > c0040afc T kill_pgrp_info
                          > c0040b48 T kill_pgrp
                          > c0040b70 T kill_pid_info
                          > c0040c38 T kill_pid
                          > c0040c60 T kill_proc_info
                          > c0040cc8 T kill_proc
                          > c0040cf0 t __dequeue_signal
                          > c0040ecc T recalc_sigpending_tsk
                          > c0040f74 T do_sigaction
                          > c0041174 T sys_rt_sigaction
                          > c004126c T force_sig_info
                          > c004135c T force_sig
                          > c0041378 T force_sig_specific
                          > c0041394 T recalc_sigpending
                          > c00413b4 T sigprocmask
                          > c00414d0 t ptrace_stop
                          > c0041678 T dequeue_signal
                          > c00417e0 T sys_restart_syscall
                          > c0041804 T flush_signals
                          > c0041894 T send_group_sig_info
                          > c00418e0 T force_sigsegv
                          > c0041960 T send_group_sigqueue
                          > c0041a98 T do_notify_parent
                          > c0041c44 T block_all_signals
                          > c0041cb4 T unblock_all_signals
                          > c0041d24 T sigqueue_alloc
                          > c0041d5c T sigqueue_free
                          > c0041e28 T ptrace_notify
                          > c0041ee0 T get_signal_to_deliver
                          > c0042304 T sys_kill
                          > c0042488 T send_sigqueue
                          > c00425ec T kill_pid_info_as_uid
                          > c004272c T zap_other_threads
                          > c00427b4 T sys_rt_sigtimedwait
                          > c0042a6c T sys_rt_sigqueueinfo
                          > c0042b14 T sys_rt_sigprocmask
                          > c0042c6c T do_sigaltstack
                          > c0042dd8 T sys_sigprocmask
                          > c0042f74 t notifier_chain_register
                          > c0042fb4 t notifier_chain_unregister
                          > c0042ff8 t notifier_call_chain
                          > c0043050 T raw_notifier_chain_register
                          > c0043064 T raw_notifier_chain_unregister
                          > c0043078 T raw_notifier_call_chain
                          > c004308c T groups_search
                          > c0043100 T sys_getcpu
                          > c004318c T getrusage
                          > c00433c0 T blocking_notifier_chain_unregister
                          > c0043420 T unregister_reboot_notifier
                          > c0043440 T blocking_notifier_chain_register
                          > c00434a0 T register_reboot_notifier
                          > c00434c0 T blocking_notifier_call_chain
                          > c0043510 T groups_free
                          > c0043568 T set_current_groups
                          > c0043764 T groups_alloc
                          > c0043840 t set_user
                          > c00438fc T ctrl_alt_del
                          > c0043948 T kernel_shutdown_prepare
                          > c0043988 t kernel_restart_prepare
                          > c00439c0 T kernel_power_off
                          > c00439e8 T kernel_halt
                          > c0043a10 T kernel_restart
                          > c0043a58 t deferred_cad
                          > c0043a70 T emergency_restart
                          > c0043a88 t set_one_prio
                          > c0043b20 T srcu_init_notifier_head
                          > c0043b64 T srcu_notifier_call_chain
                          > c0043bb4 T srcu_notifier_chain_register
                          > c0043c14 T srcu_notifier_chain_unregister
                          > c0043c7c T atomic_notifier_chain_register
                          > c0043ce0 T atomic_notifier_chain_unregister
                          > c0043d48 T atomic_notifier_call_chain
                          > c0043d94 T sys_setgroups
                          > c0043ef8 T sys_getresuid
                          > c0043f50 T sys_getresgid
                          > c0043fa8 T sys_setfsgid
                          > c0044054 T sys_setsid
                          > c0044144 T sys_getgroups
                          > c0044204 T in_group_p
                          > c0044244 T in_egroup_p
                          > c0044284 T sys_setrlimit
                          > c004445c T sys_getrusage
                          > c004449c T sys_umask
                          > c00444d0 T sys_prctl
                          > c00446d8 T sys_getpgid
                          > c004475c T sys_getpgrp
                          > c0044780 T sys_getsid
                          > c0044804 T sys_setpgid
                          > c0044988 T sys_setpriority
                          > c0044b60 T sys_getpriority
                          > c0044d18 T sys_setregid
                          > c0044e48 T sys_setgid
                          > c0044f38 T sys_setresgid
                          > c0045094 T sys_setfsuid
                          > c0045154 T sys_reboot
                          > c0045368 T sys_times
                          > c0045488 T sys_newuname
                          > c00454fc T sys_gethostname
                          > c004559c T sys_getrlimit
                          > c0045648 T sys_old_getrlimit
                          > c0045710 T sys_sethostname
                          > c0045800 T sys_setdomainname
                          > c00458f4 T sys_setreuid
                          > c0045a64 T sys_setuid
                          > c0045b58 T sys_setresuid
                          > c0045d08 t __call_usermodehelper
                          > c0045d8c t wait_for_helper
                          > c0045e3c T call_usermodehelper_keys
                          > c0045f40 T request_module
                          > c0046090 t ____call_usermodehelper
                          > c00461cc T call_usermodehelper_pipe
                          > c00462cc T keventd_up
                          > c00462f0 T current_is_keventd
                          > c0046334 t __queue_work
                          > c00463f0 T delayed_work_timer_fn
                          > c0046414 T queue_work
                          > c00464a8 T schedule_work
                          > c00464cc T execute_in_process_context
                          > c004653c t cleanup_workqueue_thread
                          > c00465b0 t create_workqueue_thread
                          > c004665c t run_workqueue
                          > c0046808 t flush_cpu_workqueue
                          > c0046938 T flush_workqueue
                          > c0046984 T cancel_rearming_delayed_workqueue
                          > c00469d4 T cancel_rearming_delayed_work
                          > c00469f8 T flush_scheduled_work
                          > c0046a18 T destroy_workqueue
                          > c0046ab0 T __create_workqueue
                          > c0046bcc T init_workqueues
                          > c0046c14 T queue_delayed_work_on
                          > c0046cd4 T schedule_delayed_work_on
                          > c0046d00 T queue_delayed_work
                          > c0046dd4 T schedule_delayed_work
                          > c0046e00 T schedule_on_each_cpu
                          > c0046eb8 t worker_thread
                          > c0046ff8 T run_scheduled_work
                          > c004715c T find_pid
                          > c00471cc T attach_pid
                          > c0047228 T transfer_pid
                          > c0047298 T pid_task
                          > c00472d4 T find_task_by_pid_type
                          > c00472f8 T free_pid_ns
                          > c004730c T copy_pid_ns
                          > c0047334 T find_ge_pid
                          > c00473d8 T put_pid
                          > c0047434 t delayed_put_pid
                          > c004744c T free_pid
                          > c0047524 T detach_pid
                          > c0047598 T find_get_pid
                          > c0047604 T get_pid_task
                          > c0047670 T alloc_pid
                          > c004798c T get_task_pid
                          > c0047a00 T rcu_batches_completed
                          > c0047a1c T rcu_batches_completed_bh
                          > c0047a38 t __rcu_pending
                          > c0047aa8 T rcu_pending
                          > c0047af8 T rcu_needs_cpu
                          > c0047b44 t wakeme_after_rcu
                          > c0047b5c t rcu_barrier_callback
                          > c0047ba4 t rcu_init_percpu_data
                          > c0047c08 t rcu_start_batch
                          > c0047c6c t __rcu_process_callbacks
                          > c0047f18 t rcu_process_callbacks
                          > c0047f50 T call_rcu
                          > c0047fdc T synchronize_rcu
                          > c0048020 T call_rcu_bh
                          > c00480ac T rcu_check_callbacks
                          > c004815c T rcu_barrier
                          > c0048210 T core_kernel_text
                          > c0048270 T kernel_text_address
                          > c00482a8 T __kernel_text_address
                          > c00482e0 T search_exception_tables
                          > c0048324 T param_set_bool
                          > c00483a8 T param_set_invbool
                          > c004840c T param_array_get
                          > c00484b4 t param_attr_store
                          > c00484f8 t module_attr_show
                          > c0048530 t module_attr_store
                          > c0048568 t uevent_filter
                          > c00485a8 T parse_args
                          > c0048818 T param_set_charp
                          > c004887c t param_sysfs_setup
                          > c00489b0 T module_param_sysfs_setup
                          > c00489dc t param_attr_show
                          > c0048a30 T module_param_sysfs_remove
                          > c0048a68 T param_get_string
                          > c0048a88 T param_set_copystring
                          > c0048b00 T param_array_set
                          > c0048c08 T param_get_bool
                          > c0048c38 T param_get_invbool
                          > c0048c94 T param_get_charp
                          > c0048cb8 T param_get_ulong
                          > c0048cdc T param_get_long
                          > c0048d00 T param_get_uint
                          > c0048d24 T param_get_int
                          > c0048d48 T param_get_ushort
                          > c0048d6c T param_get_short
                          > c0048d90 T param_get_byte
                          > c0048db4 T param_set_ulong
                          > c0048e00 T param_set_uint
                          > c0048e4c T param_set_ushort
                          > c0048ea8 T param_set_byte
                          > c0048f00 T param_set_long
                          > c0048f4c T param_set_int
                          > c0048f98 T param_set_short
                          > c0048ff4 T do_posix_clock_nosettime
                          > c0049008 T do_posix_clock_nonanosleep
                          > c004901c t posix_ktime_get_ts
                          > c0049038 T clock_nanosleep_restart
                          > c0049084 T sys_clock_nanosleep
                          > c00491c4 t release_posix_timer
                          > c0049294 T exit_itimers
                          > c0049408 t lock_timer
                          > c0049518 t common_timer_get
                          > c0049650 T register_posix_clock
                          > c00496b0 T posix_timer_event
                          > c004978c T sys_clock_getres
                          > c0049878 T sys_clock_settime
                          > c0049970 T sys_timer_create
                          > c0049dcc T sys_clock_gettime
                          > c0049ec0 T do_schedule_next_timer
                          > c0049fc0 t posix_timer_fn
                          > c004a0a4 T sys_timer_gettime
                          > c004a178 T sys_timer_getoverrun
                          > c004a1d8 T sys_timer_settime
                          > c004a500 T sys_timer_delete
                          > c004a6a8 T kthread_should_stop
                          > c004a6dc T kthread_bind
                          > c004a708 T kthread_stop
                          > c004a7b0 t keventd_create_kthread
                          > c004a83c T kthread_create
                          > c004a908 t kthread
                          > c004aa30 T init_waitqueue_head
                          > c004aa48 T bit_waitqueue
                          > c004aaa8 T __wake_up_bit
                          > c004aae4 T wake_up_bit
                          > c004ab0c T autoremove_wake_function
                          > c004ab48 T wake_bit_function
                          > c004abbc T finish_wait
                          > c004ac6c T prepare_to_wait_exclusive
                          > c004ad08 T prepare_to_wait
                          > c004ada4 T remove_wait_queue
                          > c004ae28 T add_wait_queue
                          > c004aea4 T add_wait_queue_exclusive
                          > c004af20 T __kfifo_get
                          > c004af9c T __kfifo_put
                          > c004b018 T kfifo_free
                          > c004b03c T kfifo_init
                          > c004b0ac T kfifo_alloc
                          > c004b12c W compat_sys_futex
                          > c004b12c W compat_sys_get_mempolicy
                          > c004b12c W compat_sys_get_robust_list
                          > c004b12c W compat_sys_ipc
                          > c004b12c W compat_sys_kexec_load
                          > c004b12c W compat_sys_keyctl
                          > c004b12c W compat_sys_mbind
                          > c004b12c W compat_sys_migrate_pages
                          > c004b12c W compat_sys_move_pages
                          > c004b12c W compat_sys_mq_getsetattr
                          > c004b12c W compat_sys_mq_notify
                          > c004b12c W compat_sys_mq_open
                          > c004b12c W compat_sys_mq_timedreceive
                          > c004b12c W compat_sys_mq_timedsend
                          > c004b12c W compat_sys_set_mempolicy
                          > c004b12c W compat_sys_set_robust_list
                          > c004b12c W compat_sys_socketcall
                          > c004b12c W compat_sys_sysctl
                          > c004b12c W ppc_rtas
                          > c004b12c W sys32_ipc
                          > c004b12c W sys32_sysctl
                          > c004b12c W sys_acct
                          > c004b12c W sys_add_key
                          > c004b12c W sys_epoll_pwait
                          > c004b12c W sys_get_mempolicy
                          > c004b12c W sys_keyctl
                          > c004b12c W sys_lookup_dcookie
                          > c004b12c W sys_mbind
                          > c004b12c W sys_migrate_pages
                          > c004b12c W sys_move_pages
                          > c004b12c W sys_mq_getsetattr
                          > c004b12c W sys_mq_notify
                          > c004b12c W sys_mq_open
                          > c004b12c W sys_mq_timedreceive
                          > c004b12c W sys_mq_timedsend
                          > c004b12c W sys_mq_unlink
                          > c004b12c T sys_ni_syscall
                          > c004b12c W sys_pciconfig_iobase
                          > c004b12c W sys_quotactl
                          > c004b12c W sys_request_key
                          > c004b12c W sys_set_mempolicy
                          > c004b12c W sys_spu_create
                          > c004b12c W sys_spu_run
                          > c004b12c W sys_vm86
                          > c004b12c W sys_vm86old
                          > c004b140 t bump_cpu_timer
                          > c004b2ac t cleanup_timers
                          > c004b3d4 T posix_cpu_timers_exit
                          > c004b40c T posix_cpu_timers_exit_group
                          > c004b464 t process_cpu_nsleep_restart
                          > c004b478 t thread_cpu_nsleep
                          > c004b48c t thread_cpu_nsleep_restart
                          > c004b4a0 t process_timer_rebalance
                          > c004b634 t arm_timer
                          > c004b8c0 t sample_to_timespec
                          > c004b958 t cpu_clock_sample_group_locked
                          > c004ba80 T s<br/><br/>(Message over 64 KB, truncated)
                        Your message has been successfully submitted and would be delivered to recipients shortly.