Loading ...
Sorry, an error occurred while loading the content.
 

Re: [redhat] Logical Volume crashing machine on boot

Expand Messages
  • J
    Jeff chiming in too... ... That was a thought.. and please do post /etc/fstab. Maybe there s some weirdness going on, or perhaps othervg is trying to mount
    Message 1 of 7 , Nov 17 11:34 AM
      Jeff chiming in too...

      On Tue, Nov 17, 2009 at 12:19, Rajveer Singh <torajveersingh@...> wrote:
      > Hi Dan,
      >
      > On Tue, Nov 17, 2009 at 10:25 PM, Dan <hyattdj@...> wrote:
      >
      > >
      > >
      > > We have a RHEL5.3 installation. Root is on Logical Volume (rootvg). another
      > > volume is mounted as logical volume other (othervg) # ok it is not
      > > other, but we wont go there
      > >
      > > The server boots fine before adding the logical volume othervg
      > > The server hangs on boot when the othervg /other is mounted at boot time.
      > >
      > hmm, I guess, othervg may be corrupted. try to run fsck on this filesystem.
      > Any way , can you post your /etc/fstab file.

      That was a thought.. and please do post /etc/fstab. Maybe there's
      some weirdness going on, or perhaps othervg is trying to mount BEFORE
      rootvg? I don't know, that's just a shot in the dark based on limited
      information. I've seen race conditions like that before though, so I
      wouldnt be surprised... One famous one involved a root filesystem and
      extra filesystem both stored on an iSCSI SAN, and the LUN for the
      rootFS would not always be the first one announced as ready, so
      sometimes it would boot right, sometimes not at all... weirdness, I
      tell you.

      > may be your /var/log/messages and /var/log/dmesg can give you some clue.

      Seconding this, though not always. the /var/log/dmesg log does not
      get written until after the system has booted... and /var/log/messages
      only starts recording things after syslogd starts, which is way after
      the mounting of filesystems, IIRC, so you may not see anything at all
      in those files.

      Cases like these are the reason I keep null modem cables handy, as
      well as a second system or laptop with serial port and terminal
      emulation. You can then get the dmesg buffer or any messages thrown
      to a TTY at boot time dumped to a remote machine, regardless of them
      being written to files or not.

      >
      > > Can you explain how to get around the read only mounted root filesystem
      > > while in single user mode. It is in my RHEL300 book which is thousands of
      > > miles away.
      > >
      > if your file system got read only, and you want to modify some file on that
      > file system, your try remount that file system in rw mode. to do so try the
      > following command:-
      >
      > #mount -o remount,rw / (where / is your mount point)
      > so it will remount your file system in rw mode and you can modily any file.

      This trick "usually" works, but it also can just not work. I've seen
      several instances where this failed to remount with RW permissions, or
      more importantly, it remounts RW then remounts again RO due to the
      failing condition. It may also be useful to burn a CD of the first
      RHEL iso, or even a Fedora iso, and try rescue mode which will mount
      your root FS as just another partition, letting you go in and change
      things...

      Now for my union sanctioned coffee break.

      jeff

      --

      Ted Turner  - "Sports is like a war without the killing."
    • Dan
      ... as it turned out the fstab last two numbers needed to be 0 0 and another arguement starting with an underscore to get the mount to wait until the SAN LUNS
      Message 2 of 7 , Nov 17 9:38 PM
        J wrote:
        >
        > Jeff chiming in too...
        >
        > On Tue, Nov 17, 2009 at 12:19, Rajveer Singh <torajveersingh@
        > gmail.com <mailto:torajveersingh%40gmail.com>> wrote:
        > > Hi Dan,
        > >
        > > On Tue, Nov 17, 2009 at 10:25 PM, Dan <hyattdj@yahoo. com
        > <mailto:hyattdj%40yahoo.com>> wrote:
        > >
        > > >
        > > >
        > > > We have a RHEL5.3 installation. Root is on Logical Volume
        > (rootvg). another
        > > > volume is mounted as logical volume other (othervg) # ok it is not
        > > > other, but we wont go there
        > > >
        > > > The server boots fine before adding the logical volume othervg
        > > > The server hangs on boot when the othervg /other is mounted at
        > boot time.
        > > >
        > > hmm, I guess, othervg may be corrupted. try to run fsck on this
        > filesystem.
        > > Any way , can you post your /etc/fstab file.
        >
        > That was a thought.. and please do post /etc/fstab. Maybe there's
        > some weirdness going on, or perhaps othervg is trying to mount BEFORE
        > rootvg? I don't know, that's just a shot in the dark based on limited
        > information. I've seen race conditions like that before though, so I
        > wouldnt be surprised... One famous one involved a root filesystem and
        > extra filesystem both stored on an iSCSI SAN, and the LUN for the
        > rootFS would not always be the first one announced as ready, so
        > sometimes it would boot right, sometimes not at all... weirdness, I
        > tell you.
        >
        as it turned out the fstab last two numbers needed to be 0 0 and another
        arguement starting with an underscore to get the mount to wait until the
        SAN LUNS were presented...
        / was local
        /other was SAN
        >
        > > may be your /var/log/messages and /var/log/dmesg can give you some clue.
        >
        > Seconding this, though not always. the /var/log/dmesg log does not
        > get written until after the system has booted... and /var/log/messages
        > only starts recording things after syslogd starts, which is way after
        > the mounting of filesystems, IIRC, so you may not see anything at all
        > in those files.
        >
        > Cases like these are the reason I keep null modem cables handy, as
        > well as a second system or laptop with serial port and terminal
        > emulation. You can then get the dmesg buffer or any messages thrown
        > to a TTY at boot time dumped to a remote machine, regardless of them
        > being written to files or not.
        >
        > >
        > > > Can you explain how to get around the read only mounted root
        > filesystem
        > > > while in single user mode. It is in my RHEL300 book which is
        > thousands of
        > > > miles away.
        > > >
        > > if your file system got read only, and you want to modify some file
        > on that
        > > file system, your try remount that file system in rw mode. to do so
        > try the
        > > following command:-
        > >
        > > #mount -o remount,rw / (where / is your mount point)
        > > so it will remount your file system in rw mode and you can modily
        > any file.
        >
        > This trick "usually" works, but it also can just not work. I've seen
        > several instances where this failed to remount with RW permissions, or
        > more importantly, it remounts RW then remounts again RO due to the
        > failing condition. It may also be useful to burn a CD of the first
        > RHEL iso, or even a Fedora iso, and try rescue mode which will mount
        > your root FS as just another partition, letting you go in and change
        > things...
        >
        > Now for my union sanctioned coffee break.
        >
        > jeff
        >
        > --
        >
        > Ted Turner - "Sports is like a war without the killing."
        >
        >



        [Non-text portions of this message have been removed]
      • Dan
        It is a bit harder when the system is 1000 miles away.....long modem cables are hard to run and there is a timeout issue. ... [Non-text portions of this
        Message 3 of 7 , Nov 17 10:30 PM
          It is a bit harder when the system is 1000 miles away.....long modem
          cables are hard to run and there is a timeout issue.

          J wrote:
          >
          > Jeff chiming in too...
          >
          > On Tue, Nov 17, 2009 at 12:19, Rajveer Singh <torajveersingh@
          > gmail.com <mailto:torajveersingh%40gmail.com>> wrote:
          > > Hi Dan,
          > >
          > > On Tue, Nov 17, 2009 at 10:25 PM, Dan <hyattdj@yahoo. com
          > <mailto:hyattdj%40yahoo.com>> wrote:
          > >
          > > >
          > > >
          > > > We have a RHEL5.3 installation. Root is on Logical Volume
          > (rootvg). another
          > > > volume is mounted as logical volume other (othervg) # ok it is not
          > > > other, but we wont go there
          > > >
          > > > The server boots fine before adding the logical volume othervg
          > > > The server hangs on boot when the othervg /other is mounted at
          > boot time.
          > > >
          > > hmm, I guess, othervg may be corrupted. try to run fsck on this
          > filesystem.
          > > Any way , can you post your /etc/fstab file.
          >
          > That was a thought.. and please do post /etc/fstab. Maybe there's
          > some weirdness going on, or perhaps othervg is trying to mount BEFORE
          > rootvg? I don't know, that's just a shot in the dark based on limited
          > information. I've seen race conditions like that before though, so I
          > wouldnt be surprised... One famous one involved a root filesystem and
          > extra filesystem both stored on an iSCSI SAN, and the LUN for the
          > rootFS would not always be the first one announced as ready, so
          > sometimes it would boot right, sometimes not at all... weirdness, I
          > tell you.
          >
          > > may be your /var/log/messages and /var/log/dmesg can give you some clue.
          >
          > Seconding this, though not always. the /var/log/dmesg log does not
          > get written until after the system has booted... and /var/log/messages
          > only starts recording things after syslogd starts, which is way after
          > the mounting of filesystems, IIRC, so you may not see anything at all
          > in those files.
          >
          > Cases like these are the reason I keep null modem cables handy, as
          > well as a second system or laptop with serial port and terminal
          > emulation. You can then get the dmesg buffer or any messages thrown
          > to a TTY at boot time dumped to a remote machine, regardless of them
          > being written to files or not.
          >
          > >
          > > > Can you explain how to get around the read only mounted root
          > filesystem
          > > > while in single user mode. It is in my RHEL300 book which is
          > thousands of
          > > > miles away.
          > > >
          > > if your file system got read only, and you want to modify some file
          > on that
          > > file system, your try remount that file system in rw mode. to do so
          > try the
          > > following command:-
          > >
          > > #mount -o remount,rw / (where / is your mount point)
          > > so it will remount your file system in rw mode and you can modily
          > any file.
          >
          > This trick "usually" works, but it also can just not work. I've seen
          > several instances where this failed to remount with RW permissions, or
          > more importantly, it remounts RW then remounts again RO due to the
          > failing condition. It may also be useful to burn a CD of the first
          > RHEL iso, or even a Fedora iso, and try rescue mode which will mount
          > your root FS as just another partition, letting you go in and change
          > things...
          >
          > Now for my union sanctioned coffee break.
          >
          > jeff
          >
          > --
          >
          > Ted Turner - "Sports is like a war without the killing."
          >
          >



          [Non-text portions of this message have been removed]
        • J
          ... The last two numbers shouldn t have anything to do with it, AFAIK. All they do is tell Dump whether or not to back up that disk, and tell FSCK which order
          Message 4 of 7 , Nov 18 6:48 AM
            On Wed, Nov 18, 2009 at 00:38, Dan <hyattdj@...> wrote:

            > as it turned out the fstab last two numbers needed to be 0 0 and another
            > arguement starting with an underscore to get the mount to wait until the
            > SAN LUNS were presented...
            > / was local
            > /other was SAN

            The last two numbers shouldn't have anything to do with it, AFAIK.
            All they do is tell Dump whether or not to back up that disk, and tell
            FSCK which order to check the partitions in... And it shouldn't even
            create some sort of race condition since the disk checks are done
            sequentially... either by the order assigned in the 6th field of
            fstab, or if two entries have the same number in the 6th field, then
            IIRC by the order they're listed in fstab itself...

            What is the argument that needed an underscore? Just curious what the
            fix was. Though it DOES make a lot of sense if the kernel is trying
            to mount a SAN filesystem before the SAN announces it. In that case,
            it seems as though the kernel sees the failure to mount (due to SAN
            taking a while) as a filesystem failure, thus triggering the remount
            to Read-Only as a protection measure.

            And yeah, I can imagine a 1000 mile modem cable is a bit difficult...
            I mean, where would you store it? OTOH, and I really hate to say
            this, but THIS is something that Serial-Over-Lan was made for. SOL
            can be a pain in the ass, but OTOH, it can save that ass as well in
            cases like this (Remote server needing direct hands-on attention
            instead of just VPN or VNC)

            Cheers
            Jeff


            --

            Marie von Ebner-Eschenbach - "Even a stopped clock is right twice a
            day." - http://www.brainyquote.com/quotes/authors/m/marie_von_ebnereschenbac.html
          Your message has been successfully submitted and would be delivered to recipients shortly.