Loading ...
Sorry, an error occurred while loading the content.

Re: [redhat] Something fun to do, but not on a production system...

Expand Messages
  • Jeff Lane
    ... One thing bothering me about this, though, is that I can t grasp exactly WHY this happens. Normally, there s a PID limit and once the kernel reaches that
    Message 1 of 15 , May 1, 2008
    • 0 Attachment
      On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@...> wrote:

      > > bomb(){ bomb | bomb& }; bomb
      >
      > Yes, that did help.

      One thing bothering me about this, though, is that I can't grasp
      exactly WHY this happens. Normally, there's a PID limit and once the
      kernel reaches that limit, it recycles old PIDs, and if none are
      available, it either errors and refuses to start more processes, or it
      starts killing off older processes. I've seen both of those occur.

      In this case, the machine locks up hard... which to me says either the
      processors are thoroughly chewed up with the exponential number of
      forks being created, or the RAM is being eaten up by all those
      processes being created.

      However, I don't know exactly how many forks it takes to bring down
      the system... I tried instrumenting the bash version yesterday but
      that didn't work at all.. the system locked up and I never got any
      output at all.

      I am thinking, though, that using C and fork() as it is in the
      examples will let me add a simple counter and a printf to at least
      tell me how many processes are being spawned... then maybe I can do
      some memory watching or something like that...

      Not that it really matters in my environment... I do beta testing for
      a living, so user security is not my concern.... but this has my
      curiosity piqued :-)

      Jeff
    • Scott
      ... Yes, what we have happen, on these scrapers (that scrape from various websites) is that they ll get an extremely heavy load and one will stop responding.
      Message 2 of 15 , May 1, 2008
      • 0 Attachment
        On Thu, May 01, 2008 at 10:04:19AM -0400, Jeff Lane wrote:
        > On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@...> wrote:
        >
        >
        > One thing bothering me about this, though, is that I can't grasp
        > exactly WHY this happens. Normally, there's a PID limit and once the
        > kernel reaches that limit, it recycles old PIDs, and if none are
        > available, it either errors and refuses to start more processes, or it
        > starts killing off older processes. I've seen both of those occur.

        Yes, what we have happen, on these scrapers (that "scrape" from various
        websites) is that they'll get an extremely heavy load and one will stop
        responding. After a few minutes, though it's back up, after it's begun
        to kill off the older processes.


        >
        > In this case, the machine locks up hard... which to me says either the
        > processors are thoroughly chewed up with the exponential number of
        > forks being created, or the RAM is being eaten up by all those
        > processes being created.

        Were you able to run top or similar while doing it, to see what
        happened? I didn't get a chance to do that. I did wonder, if we
        waited, if the machine would recover, but AFAICT, it doesn't seem as if
        it will.


        > Not that it really matters in my environment... I do beta testing for
        > a living, so user security is not my concern.... but this has my
        > curiosity piqued :-)
        >

        Keep posting results of your findings.


        --
        Scott Robbins
        PGP keyID EB3467D6
        ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
        gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

        Spike: I did a couple of slayers in my time. I don't like to
        brag. Who am I kidding? I love to brag. One time, during the
        Boxer Rebellion...
      • Daniel Hyatt
        Didnt figure out exactly how, but what I read is that a properly configured machine will not lock up with this.  Dan The opinions expressed here are not
        Message 3 of 15 , May 1, 2008
        • 0 Attachment
          Didnt figure out exactly how, but what I read is that a properly configured machine will not lock up with this.

           Dan
          "The opinions expressed here are not necessarily my own"
           
           



          ----- Original Message ----
          From: Jeff Lane <sundowner225@...>
          To: redhat@yahoogroups.com
          Sent: Thursday, May 1, 2008 7:04:19 AM
          Subject: Re: [redhat] Something fun to do, but not on a production system...


          On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@nyc. rr.com> wrote:

          > > bomb(){ bomb | bomb& }; bomb
          >
          > Yes, that did help.

          One thing bothering me about this, though, is that I can't grasp
          exactly WHY this happens. Normally, there's a PID limit and once the
          kernel reaches that limit, it recycles old PIDs, and if none are
          available, it either errors and refuses to start more processes, or it
          starts killing off older processes. I've seen both of those occur.

          In this case, the machine locks up hard... which to me says either the
          processors are thoroughly chewed up with the exponential number of
          forks being created, or the RAM is being eaten up by all those
          processes being created.

          However, I don't know exactly how many forks it takes to bring down
          the system... I tried instrumenting the bash version yesterday but
          that didn't work at all.. the system locked up and I never got any
          output at all.

          I am thinking, though, that using C and fork() as it is in the
          examples will let me add a simple counter and a printf to at least
          tell me how many processes are being spawned... then maybe I can do
          some memory watching or something like that...

          Not that it really matters in my environment. .. I do beta testing for
          a living, so user security is not my concern.... but this has my
          curiosity piqued :-)

          Jeff


          [Non-text portions of this message have been removed]
        • Jeff Lane
          ... Yep... that is the case. The second link I provided tells how, but my problem is that I dont understand why. If the hang occurs because you are simply
          Message 4 of 15 , May 1, 2008
          • 0 Attachment
            On Thu, May 1, 2008 at 11:31 AM, Daniel Hyatt <hyattdj@...> wrote:
            >
            > Didnt figure out exactly how, but what I read is that a properly configured
            > machine will not lock up with this.
            >
            > Dan
            > "The opinions expressed here are not necessarily my own"

            Yep... that is the case. The second link I provided tells how, but my
            problem is that I dont understand why. If the hang occurs because you
            are simply opening too many processes by forking exponentially, then
            it's my experience that the kernel will simply tell you it can't open
            any new processes and error, or it will kill off existing processes to
            free up PIDs and Resources.

            It could be that this fills up any page tables though, now that I
            think about it... so while it may not be necessarily chewing up the
            processor and physical ram, it's using up all the available Linux page
            tables causing some sort of overflow, but that's just an uneducated
            guess at what's going on.

            BUT, if you limit the amount of processes a user can start, this kind
            of thing becomes avoidable because the the kernel will simply fork
            until it runs out of user allowed processes. However, I've been doing
            this as root, and limiting the number of things root can start "may"
            be a bad thing...

            Originally, it started off as an exercise demonstrating the need to
            limit what users can do... now it's something that's become a puzzle
            I want to figure out. heh...

            One thing I wonder though, is why is this even possible in the first
            place? Why does any Linux vendor allow regular users to open
            unlimited processes? I can understand doing so for system users like
            FTP or apache, but those system services generally limit the number of
            spawned processes in their configs but it still could be possible to,
            for instance, configure apache to be able to spawn enough web server
            children to bring the system down under load (what I think is a
            classic DOS attack).

            But it seems kind of silly to me to allow a normal system user that an
            admin would create to be able to do something like this by default...
            but that's just me and things that make sense to me don't necessarily
            make sense to anyone else.
          • ed
            On Thu, 1 May 2008 12:08:53 -0400 ... fork() is an expensive operation. vfork was introduced to be a lighter version of the operation. In the majority of cases
            Message 5 of 15 , May 1, 2008
            • 0 Attachment
              On Thu, 1 May 2008 12:08:53 -0400
              "Jeff Lane" <sundowner225@...> wrote:

              > On Thu, May 1, 2008 at 11:31 AM, Daniel Hyatt <hyattdj@...>
              > wrote:
              > >
              > > Didnt figure out exactly how, but what I read is that a properly
              > > configured machine will not lock up with this.
              > >
              > > Dan
              > > "The opinions expressed here are not necessarily my own"
              >
              > Yep... that is the case. The second link I provided tells how, but my
              > problem is that I dont understand why. If the hang occurs because you
              > are simply opening too many processes by forking exponentially, then
              > it's my experience that the kernel will simply tell you it can't open
              > any new processes and error, or it will kill off existing processes to
              > free up PIDs and Resources.

              fork() is an expensive operation. vfork was introduced to be a lighter
              version of the operation. In the majority of cases after fork() comes
              exec*(). So vfork creates a new process (IIRC) within the same process
              group and session but does not copy the memory foot print.

              The fork bomb DOES copy the memory foot print, so it has to take all
              that the parent has. I assume this is where things get ugly.

              From fork() the child will include:

              - environment variables
              - signal handling instructions
              - all attached shared memory segments
              - process group id
              - current working directory
              - file handles
              - file creation mask
              - real effective and saved group id
              - real effective saved user id
              - controlling terminal

              The kernel will spend a lot of time handling this. I think, like I've
              been reading here, if that is left to run away the session should kill
              the process tree, eventually. I assume here that it is a process tree.
              I will see after sending this mail with a some perl and a few print
              lines to see what goes on.

              Generally the kernel should kill stuff off with OOME errors - which
              might take a while on this system.

              > It could be that this fills up any page tables though, now that I
              > think about it... so while it may not be necessarily chewing up the
              > processor and physical ram, it's using up all the available Linux page
              > tables causing some sort of overflow, but that's just an uneducated
              > guess at what's going on.

              My guess is that it's the effort of copying memory foot prints that is
              the pain.

              > BUT, if you limit the amount of processes a user can start, this kind
              > of thing becomes avoidable because the the kernel will simply fork
              > until it runs out of user allowed processes. However, I've been doing
              > this as root, and limiting the number of things root can start "may"
              > be a bad thing...
              >
              > Originally, it started off as an exercise demonstrating the need to
              > limit what users can do... now it's something that's become a puzzle
              > I want to figure out. heh...
              >
              > One thing I wonder though, is why is this even possible in the first
              > place? Why does any Linux vendor allow regular users to open
              > unlimited processes? I can understand doing so for system users like
              > FTP or apache, but those system services generally limit the number of
              > spawned processes in their configs but it still could be possible to,
              > for instance, configure apache to be able to spawn enough web server
              > children to bring the system down under load (what I think is a
              > classic DOS attack).
              >
              > But it seems kind of silly to me to allow a normal system user that an
              > admin would create to be able to do something like this by default...
              > but that's just me and things that make sense to me don't necessarily
              > make sense to anyone else.

              I dunno, had some time to waste. Run this with ulimit, or not.
              Eventually with ulimit the error of resource unavailable will be
              returned.

              #include <stdlib.h>
              #include <stdio.h>
              #include <sys/types.h>
              #include <unistd.h>
              #include <errno.h>

              int main( int argc, char *argv[], char *env[] ) {
              int i = 0,pid;

              while( 1 == 1 ) {
              i++;
              pid = fork();

              if( errno == EAGAIN ) {
              perror( "cannot fork" );
              return( EXIT_FAILURE );
              }

              if( errno == ENOMEM ) {
              perror( "cannot fork" );
              return( EXIT_FAILURE );
              }

              if( pid == 0 ) {
              fprintf( stdout, "I am in the child eating
              lolcats\n" ); }
              fprintf( stdout, "Value of i is %d\n", i );
              }

              return( EXIT_SUCCESS );
              }


              --
              The ATM to www.brokenmachine.com is sending 1111111111 because of a
              mistake. The Rebel Alliance is ordering a pizza.
              :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


              [Non-text portions of this message have been removed]
            • Scott
              ... Ed, do you have a BSD box where you can try it? I m wondering if it will have the same effect. I ll give it a try on a FreeBSD box tonight. -- Scott
              Message 6 of 15 , May 1, 2008
              • 0 Attachment
                On Thu, May 01, 2008 at 08:50:41PM +0100, ed wrote:
                >
                >
                > fork() is an expensive operation. vfork was introduced to be a lighter
                > version of the operation. In the majority of cases after fork() comes
                > exec*(). So vfork creates a new process (IIRC) within the same process
                > group and session but does not copy the memory foot print.
                >
                > The fork bomb DOES copy the memory foot print, so it has to take all
                > that the parent has. I assume this is where things get ugly.

                Ed, do you have a BSD box where you can try it? I'm wondering if it
                will have the same effect. I'll give it a try on a FreeBSD box tonight.


                --
                Scott Robbins
                PGP keyID EB3467D6
                ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

                Xander: You were looking at my neck.
                Angel: What?
                Xander: You were checking out my neck, I saw that.
                Angel: No, I wasn't.
                Xander: Just keep your distance, pal.
                Angel: I wasn't looking at your neck.
                Xander: I told you to eat before we left.
              • ed
                On Thu, 1 May 2008 16:39:26 -0400 ... Unfortunately I don t - I ve changed jobs and we use SunOS/Linux so I had to run it here. If there s a difference it
                Message 7 of 15 , May 1, 2008
                • 0 Attachment
                  On Thu, 1 May 2008 16:39:26 -0400
                  Scott <scottro@...> wrote:

                  > On Thu, May 01, 2008 at 08:50:41PM +0100, ed wrote:
                  > >
                  > >
                  > > fork() is an expensive operation. vfork was introduced to be a
                  > > lighter version of the operation. In the majority of cases after
                  > > fork() comes exec*(). So vfork creates a new process (IIRC) within
                  > > the same process group and session but does not copy the memory
                  > > foot print.
                  > >
                  > > The fork bomb DOES copy the memory foot print, so it has to take all
                  > > that the parent has. I assume this is where things get ugly.
                  >
                  > Ed, do you have a BSD box where you can try it? I'm wondering if it
                  > will have the same effect. I'll give it a try on a FreeBSD box
                  > tonight.

                  Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                  had to run it here.

                  If there's a difference it might be due to the Linux kernel's COW fork
                  (copy memory foot print when used) rather than copying the whole lot it
                  just does virtually the same as vfork until the child process reads
                  something that the parent would have given it, such as a file
                  descriptor.

                  Most fork operations benefit from COW, since most commonly the next
                  thing after fork is exec().

                  I think BSD introduced vfork, or was it SCO/SUN, I really can't
                  remember.

                  --
                  The Ether to Dantooine is taking hits because of a crappy FE1 & CE2
                  engine. Barbarella is blasting through 30 million dollars.
                  :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


                  [Non-text portions of this message have been removed]
                • Scott
                  ... Well, I just tried it and yup, it crippled a FreeBSD box as well. It was a bit more informative, with all sorts of messages about no swap space, but it was
                  Message 8 of 15 , May 1, 2008
                  • 0 Attachment
                    On Thu, May 01, 2008 at 10:04:03PM +0100, ed wrote:
                    > On Thu, 1 May 2008 16:39:26 -0400
                    > Scott <scottro@...> wrote:
                    >
                    >
                    > Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                    > had to run it here.

                    Well, I just tried it and yup, it crippled a FreeBSD box as well. It was
                    a bit more informative, with all sorts of messages about no swap space,
                    but it was effectively crippled. :)

                    --
                    Scott Robbins
                    PGP keyID EB3467D6
                    ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                    gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

                    Buffy: Does it ever get easy?
                    Giles: You mean life?
                    Buffy: Yeah, does it ever get easy?
                    Giles: What do you want me to say?
                    Buffy: Lie to me.
                    Giles: Yes. It's terribly simple. The good guys are always s
                    talwart and true. The bad guys are easily distinguished by their
                    pointy horns or black hats, and we always defeat them and save
                    the day. No one ever dies... and everyone lives happily ever
                    after.
                    Buffy: Liar.
                  • Jeff Lane
                    Well, I tried two ways yesterday on my 8 node 512GB system... This was running beta RHEL 5.2. First I tried the c code calling fork() in a never ending while
                    Message 9 of 15 , May 2, 2008
                    • 0 Attachment
                      Well, I tried two ways yesterday on my 8 node 512GB system...

                      This was running beta RHEL 5.2.

                      First I tried the c code calling fork() in a never ending while loop.
                      The default number of processes seems to be 32768. The kernel did a
                      good job of killing off old ones and basically it just kept killing
                      them off over and over. The machine was not terribly usable, but it
                      stayed up and I was even able to ctrl-c the program and recover.

                      Then I tried the bash version... that one opened the max number of
                      processes, then the kernel said "Nope. No more resources. Screw you,
                      I'm killing them all" and it killed off ALL of them and ended my
                      script. heh...

                      On 5.0 it hangs every box I've tried. Same on SLES 9 (though I
                      haven't tried SLES10 service packs)...

                      I'm gonna wager a guess that the newer kernels may mitigate this...

                      on the other hand, I added a counter increment and a printf statement,
                      so that may even have slowed the forking down enough so the system
                      could get ahold of it. I'll have to try it without the
                      instrumetnation and see what happens.. :-)

                      Then I want to try it on a Windows XP system heh...

                      On Thu, May 1, 2008 at 7:30 PM, Scott <scottro@...> wrote:
                      >
                      >
                      >
                      >
                      >
                      >
                      > On Thu, May 01, 2008 at 10:04:03PM +0100, ed wrote:
                      > > On Thu, 1 May 2008 16:39:26 -0400
                      > > Scott <scottro@...> wrote:
                      > >
                      > >
                      > > Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                      > > had to run it here.
                      >
                      > Well, I just tried it and yup, it crippled a FreeBSD box as well. It was
                      > a bit more informative, with all sorts of messages about no swap space,
                      > but it was effectively crippled. :)
                      >
                      >
                      > --
                      > Scott Robbins
                      > PGP keyID EB3467D6
                      > ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                      > gpg --keyserver pgp.mit.edu --recv-keys EB3467D6
                      >
                      > Buffy: Does it ever get easy?
                      > Giles: You mean life?
                      > Buffy: Yeah, does it ever get easy?
                      > Giles: What do you want me to say?
                      > Buffy: Lie to me.
                      > Giles: Yes. It's terribly simple. The good guys are always s
                      > talwart and true. The bad guys are easily distinguished by their
                      > pointy horns or black hats, and we always defeat them and save
                      > the day. No one ever dies... and everyone lives happily ever
                      > after.
                      > Buffy: Liar.
                      >
                      >



                      --
                      ------------------> Jeffrey Lane - W4KDH <-------------------
                      www.jefflane.org
                      Another cog in the great
                      Corporate Wheel

                      The internet has no government, no constitution, no laws, no
                      rights, no police, no courts. Don't talk about fairness or
                      innocence, and don't talk about what should be done. Instead,
                      talk about what is being done and what will be done by the
                      amorphous unreachable undefinable blob called "the internet
                      user base." -Paul Vixie
                    • ed
                      On Fri, 2 May 2008 09:41:00 -0400 ... Does the windows kernel include a fork() - I thought they omitted that and have some other way of creating processes? --
                      Message 10 of 15 , May 2, 2008
                      • 0 Attachment
                        On Fri, 2 May 2008 09:41:00 -0400
                        "Jeff Lane" <sundowner225@...> wrote:

                        > Well, I tried two ways yesterday on my 8 node 512GB system...
                        >
                        > This was running beta RHEL 5.2.
                        >
                        > First I tried the c code calling fork() in a never ending while loop.
                        > The default number of processes seems to be 32768. The kernel did a
                        > good job of killing off old ones and basically it just kept killing
                        > them off over and over. The machine was not terribly usable, but it
                        > stayed up and I was even able to ctrl-c the program and recover.
                        >
                        > Then I tried the bash version... that one opened the max number of
                        > processes, then the kernel said "Nope. No more resources. Screw you,
                        > I'm killing them all" and it killed off ALL of them and ended my
                        > script. heh...
                        >
                        > On 5.0 it hangs every box I've tried. Same on SLES 9 (though I
                        > haven't tried SLES10 service packs)...
                        >
                        > I'm gonna wager a guess that the newer kernels may mitigate this...
                        >
                        > on the other hand, I added a counter increment and a printf statement,
                        > so that may even have slowed the forking down enough so the system
                        > could get ahold of it. I'll have to try it without the
                        > instrumetnation and see what happens.. :-)
                        >
                        > Then I want to try it on a Windows XP system heh...

                        Does the windows kernel include a fork() - I thought they omitted that
                        and have some other way of creating processes?

                        --
                        The ATM to the Ultra 5 is hosed because of a screwed up .procmailrc
                        file. AOL is playing NFL 2k2.
                        :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


                        [Non-text portions of this message have been removed]
                      Your message has been successfully submitted and would be delivered to recipients shortly.