Loading ...
Sorry, an error occurred while loading the content.

Re: [redhat] Something fun to do, but not on a production system...

Expand Messages
  • Scott
    ... Yes, that did help. ... Ah, that s by that fellow Vivek Gite, who, although English isn t his first language, has one of the best bash tutorials around,
    Message 1 of 15 , Apr 30 1:44 PM
    • 0 Attachment
      On Wed, Apr 30, 2008 at 03:07:01PM -0400, Jeff Lane wrote:
      > On Wed, Apr 30, 2008 at 2:56 PM, Scott <scottro@...> wrote:
      >
      >
      > bomb(){ bomb | bomb& }; bomb

      Yes, that did help.


      >
      > http://www.cyberciti.biz/faq/understanding-bash-fork-bomb/
      >

      Ah, that's by that fellow Vivek Gite, who, although English isn't his
      first language, has one of the best bash tutorials around, IMHO.

      >
      > It's interesting how a very simple thing can kill a system.. to
      > prevent it, all you need to do is ensure that users are limited in the
      > number of processes they can create.

      Yes, it is. Surprising how vulnerable all these things really are.

      --
      Scott Robbins
      PGP keyID EB3467D6
      ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
      gpg --keyserver pgp.mit.edu --recv-keys EB3467D6


      Xander: I have my pride. Okay, so I don't have a *lot* of my pride,
      but I have enough so that I can't do this.
    • Jeff Lane
      ... One thing bothering me about this, though, is that I can t grasp exactly WHY this happens. Normally, there s a PID limit and once the kernel reaches that
      Message 2 of 15 , May 1, 2008
      • 0 Attachment
        On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@...> wrote:

        > > bomb(){ bomb | bomb& }; bomb
        >
        > Yes, that did help.

        One thing bothering me about this, though, is that I can't grasp
        exactly WHY this happens. Normally, there's a PID limit and once the
        kernel reaches that limit, it recycles old PIDs, and if none are
        available, it either errors and refuses to start more processes, or it
        starts killing off older processes. I've seen both of those occur.

        In this case, the machine locks up hard... which to me says either the
        processors are thoroughly chewed up with the exponential number of
        forks being created, or the RAM is being eaten up by all those
        processes being created.

        However, I don't know exactly how many forks it takes to bring down
        the system... I tried instrumenting the bash version yesterday but
        that didn't work at all.. the system locked up and I never got any
        output at all.

        I am thinking, though, that using C and fork() as it is in the
        examples will let me add a simple counter and a printf to at least
        tell me how many processes are being spawned... then maybe I can do
        some memory watching or something like that...

        Not that it really matters in my environment... I do beta testing for
        a living, so user security is not my concern.... but this has my
        curiosity piqued :-)

        Jeff
      • Scott
        ... Yes, what we have happen, on these scrapers (that scrape from various websites) is that they ll get an extremely heavy load and one will stop responding.
        Message 3 of 15 , May 1, 2008
        • 0 Attachment
          On Thu, May 01, 2008 at 10:04:19AM -0400, Jeff Lane wrote:
          > On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@...> wrote:
          >
          >
          > One thing bothering me about this, though, is that I can't grasp
          > exactly WHY this happens. Normally, there's a PID limit and once the
          > kernel reaches that limit, it recycles old PIDs, and if none are
          > available, it either errors and refuses to start more processes, or it
          > starts killing off older processes. I've seen both of those occur.

          Yes, what we have happen, on these scrapers (that "scrape" from various
          websites) is that they'll get an extremely heavy load and one will stop
          responding. After a few minutes, though it's back up, after it's begun
          to kill off the older processes.


          >
          > In this case, the machine locks up hard... which to me says either the
          > processors are thoroughly chewed up with the exponential number of
          > forks being created, or the RAM is being eaten up by all those
          > processes being created.

          Were you able to run top or similar while doing it, to see what
          happened? I didn't get a chance to do that. I did wonder, if we
          waited, if the machine would recover, but AFAICT, it doesn't seem as if
          it will.


          > Not that it really matters in my environment... I do beta testing for
          > a living, so user security is not my concern.... but this has my
          > curiosity piqued :-)
          >

          Keep posting results of your findings.


          --
          Scott Robbins
          PGP keyID EB3467D6
          ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
          gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

          Spike: I did a couple of slayers in my time. I don't like to
          brag. Who am I kidding? I love to brag. One time, during the
          Boxer Rebellion...
        • Daniel Hyatt
          Didnt figure out exactly how, but what I read is that a properly configured machine will not lock up with this.  Dan The opinions expressed here are not
          Message 4 of 15 , May 1, 2008
          • 0 Attachment
            Didnt figure out exactly how, but what I read is that a properly configured machine will not lock up with this.

             Dan
            "The opinions expressed here are not necessarily my own"
             
             



            ----- Original Message ----
            From: Jeff Lane <sundowner225@...>
            To: redhat@yahoogroups.com
            Sent: Thursday, May 1, 2008 7:04:19 AM
            Subject: Re: [redhat] Something fun to do, but not on a production system...


            On Wed, Apr 30, 2008 at 4:44 PM, Scott <scottro@nyc. rr.com> wrote:

            > > bomb(){ bomb | bomb& }; bomb
            >
            > Yes, that did help.

            One thing bothering me about this, though, is that I can't grasp
            exactly WHY this happens. Normally, there's a PID limit and once the
            kernel reaches that limit, it recycles old PIDs, and if none are
            available, it either errors and refuses to start more processes, or it
            starts killing off older processes. I've seen both of those occur.

            In this case, the machine locks up hard... which to me says either the
            processors are thoroughly chewed up with the exponential number of
            forks being created, or the RAM is being eaten up by all those
            processes being created.

            However, I don't know exactly how many forks it takes to bring down
            the system... I tried instrumenting the bash version yesterday but
            that didn't work at all.. the system locked up and I never got any
            output at all.

            I am thinking, though, that using C and fork() as it is in the
            examples will let me add a simple counter and a printf to at least
            tell me how many processes are being spawned... then maybe I can do
            some memory watching or something like that...

            Not that it really matters in my environment. .. I do beta testing for
            a living, so user security is not my concern.... but this has my
            curiosity piqued :-)

            Jeff


            [Non-text portions of this message have been removed]
          • Jeff Lane
            ... Yep... that is the case. The second link I provided tells how, but my problem is that I dont understand why. If the hang occurs because you are simply
            Message 5 of 15 , May 1, 2008
            • 0 Attachment
              On Thu, May 1, 2008 at 11:31 AM, Daniel Hyatt <hyattdj@...> wrote:
              >
              > Didnt figure out exactly how, but what I read is that a properly configured
              > machine will not lock up with this.
              >
              > Dan
              > "The opinions expressed here are not necessarily my own"

              Yep... that is the case. The second link I provided tells how, but my
              problem is that I dont understand why. If the hang occurs because you
              are simply opening too many processes by forking exponentially, then
              it's my experience that the kernel will simply tell you it can't open
              any new processes and error, or it will kill off existing processes to
              free up PIDs and Resources.

              It could be that this fills up any page tables though, now that I
              think about it... so while it may not be necessarily chewing up the
              processor and physical ram, it's using up all the available Linux page
              tables causing some sort of overflow, but that's just an uneducated
              guess at what's going on.

              BUT, if you limit the amount of processes a user can start, this kind
              of thing becomes avoidable because the the kernel will simply fork
              until it runs out of user allowed processes. However, I've been doing
              this as root, and limiting the number of things root can start "may"
              be a bad thing...

              Originally, it started off as an exercise demonstrating the need to
              limit what users can do... now it's something that's become a puzzle
              I want to figure out. heh...

              One thing I wonder though, is why is this even possible in the first
              place? Why does any Linux vendor allow regular users to open
              unlimited processes? I can understand doing so for system users like
              FTP or apache, but those system services generally limit the number of
              spawned processes in their configs but it still could be possible to,
              for instance, configure apache to be able to spawn enough web server
              children to bring the system down under load (what I think is a
              classic DOS attack).

              But it seems kind of silly to me to allow a normal system user that an
              admin would create to be able to do something like this by default...
              but that's just me and things that make sense to me don't necessarily
              make sense to anyone else.
            • ed
              On Thu, 1 May 2008 12:08:53 -0400 ... fork() is an expensive operation. vfork was introduced to be a lighter version of the operation. In the majority of cases
              Message 6 of 15 , May 1, 2008
              • 0 Attachment
                On Thu, 1 May 2008 12:08:53 -0400
                "Jeff Lane" <sundowner225@...> wrote:

                > On Thu, May 1, 2008 at 11:31 AM, Daniel Hyatt <hyattdj@...>
                > wrote:
                > >
                > > Didnt figure out exactly how, but what I read is that a properly
                > > configured machine will not lock up with this.
                > >
                > > Dan
                > > "The opinions expressed here are not necessarily my own"
                >
                > Yep... that is the case. The second link I provided tells how, but my
                > problem is that I dont understand why. If the hang occurs because you
                > are simply opening too many processes by forking exponentially, then
                > it's my experience that the kernel will simply tell you it can't open
                > any new processes and error, or it will kill off existing processes to
                > free up PIDs and Resources.

                fork() is an expensive operation. vfork was introduced to be a lighter
                version of the operation. In the majority of cases after fork() comes
                exec*(). So vfork creates a new process (IIRC) within the same process
                group and session but does not copy the memory foot print.

                The fork bomb DOES copy the memory foot print, so it has to take all
                that the parent has. I assume this is where things get ugly.

                From fork() the child will include:

                - environment variables
                - signal handling instructions
                - all attached shared memory segments
                - process group id
                - current working directory
                - file handles
                - file creation mask
                - real effective and saved group id
                - real effective saved user id
                - controlling terminal

                The kernel will spend a lot of time handling this. I think, like I've
                been reading here, if that is left to run away the session should kill
                the process tree, eventually. I assume here that it is a process tree.
                I will see after sending this mail with a some perl and a few print
                lines to see what goes on.

                Generally the kernel should kill stuff off with OOME errors - which
                might take a while on this system.

                > It could be that this fills up any page tables though, now that I
                > think about it... so while it may not be necessarily chewing up the
                > processor and physical ram, it's using up all the available Linux page
                > tables causing some sort of overflow, but that's just an uneducated
                > guess at what's going on.

                My guess is that it's the effort of copying memory foot prints that is
                the pain.

                > BUT, if you limit the amount of processes a user can start, this kind
                > of thing becomes avoidable because the the kernel will simply fork
                > until it runs out of user allowed processes. However, I've been doing
                > this as root, and limiting the number of things root can start "may"
                > be a bad thing...
                >
                > Originally, it started off as an exercise demonstrating the need to
                > limit what users can do... now it's something that's become a puzzle
                > I want to figure out. heh...
                >
                > One thing I wonder though, is why is this even possible in the first
                > place? Why does any Linux vendor allow regular users to open
                > unlimited processes? I can understand doing so for system users like
                > FTP or apache, but those system services generally limit the number of
                > spawned processes in their configs but it still could be possible to,
                > for instance, configure apache to be able to spawn enough web server
                > children to bring the system down under load (what I think is a
                > classic DOS attack).
                >
                > But it seems kind of silly to me to allow a normal system user that an
                > admin would create to be able to do something like this by default...
                > but that's just me and things that make sense to me don't necessarily
                > make sense to anyone else.

                I dunno, had some time to waste. Run this with ulimit, or not.
                Eventually with ulimit the error of resource unavailable will be
                returned.

                #include <stdlib.h>
                #include <stdio.h>
                #include <sys/types.h>
                #include <unistd.h>
                #include <errno.h>

                int main( int argc, char *argv[], char *env[] ) {
                int i = 0,pid;

                while( 1 == 1 ) {
                i++;
                pid = fork();

                if( errno == EAGAIN ) {
                perror( "cannot fork" );
                return( EXIT_FAILURE );
                }

                if( errno == ENOMEM ) {
                perror( "cannot fork" );
                return( EXIT_FAILURE );
                }

                if( pid == 0 ) {
                fprintf( stdout, "I am in the child eating
                lolcats\n" ); }
                fprintf( stdout, "Value of i is %d\n", i );
                }

                return( EXIT_SUCCESS );
                }


                --
                The ATM to www.brokenmachine.com is sending 1111111111 because of a
                mistake. The Rebel Alliance is ordering a pizza.
                :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


                [Non-text portions of this message have been removed]
              • Scott
                ... Ed, do you have a BSD box where you can try it? I m wondering if it will have the same effect. I ll give it a try on a FreeBSD box tonight. -- Scott
                Message 7 of 15 , May 1, 2008
                • 0 Attachment
                  On Thu, May 01, 2008 at 08:50:41PM +0100, ed wrote:
                  >
                  >
                  > fork() is an expensive operation. vfork was introduced to be a lighter
                  > version of the operation. In the majority of cases after fork() comes
                  > exec*(). So vfork creates a new process (IIRC) within the same process
                  > group and session but does not copy the memory foot print.
                  >
                  > The fork bomb DOES copy the memory foot print, so it has to take all
                  > that the parent has. I assume this is where things get ugly.

                  Ed, do you have a BSD box where you can try it? I'm wondering if it
                  will have the same effect. I'll give it a try on a FreeBSD box tonight.


                  --
                  Scott Robbins
                  PGP keyID EB3467D6
                  ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                  gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

                  Xander: You were looking at my neck.
                  Angel: What?
                  Xander: You were checking out my neck, I saw that.
                  Angel: No, I wasn't.
                  Xander: Just keep your distance, pal.
                  Angel: I wasn't looking at your neck.
                  Xander: I told you to eat before we left.
                • ed
                  On Thu, 1 May 2008 16:39:26 -0400 ... Unfortunately I don t - I ve changed jobs and we use SunOS/Linux so I had to run it here. If there s a difference it
                  Message 8 of 15 , May 1, 2008
                  • 0 Attachment
                    On Thu, 1 May 2008 16:39:26 -0400
                    Scott <scottro@...> wrote:

                    > On Thu, May 01, 2008 at 08:50:41PM +0100, ed wrote:
                    > >
                    > >
                    > > fork() is an expensive operation. vfork was introduced to be a
                    > > lighter version of the operation. In the majority of cases after
                    > > fork() comes exec*(). So vfork creates a new process (IIRC) within
                    > > the same process group and session but does not copy the memory
                    > > foot print.
                    > >
                    > > The fork bomb DOES copy the memory foot print, so it has to take all
                    > > that the parent has. I assume this is where things get ugly.
                    >
                    > Ed, do you have a BSD box where you can try it? I'm wondering if it
                    > will have the same effect. I'll give it a try on a FreeBSD box
                    > tonight.

                    Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                    had to run it here.

                    If there's a difference it might be due to the Linux kernel's COW fork
                    (copy memory foot print when used) rather than copying the whole lot it
                    just does virtually the same as vfork until the child process reads
                    something that the parent would have given it, such as a file
                    descriptor.

                    Most fork operations benefit from COW, since most commonly the next
                    thing after fork is exec().

                    I think BSD introduced vfork, or was it SCO/SUN, I really can't
                    remember.

                    --
                    The Ether to Dantooine is taking hits because of a crappy FE1 & CE2
                    engine. Barbarella is blasting through 30 million dollars.
                    :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


                    [Non-text portions of this message have been removed]
                  • Scott
                    ... Well, I just tried it and yup, it crippled a FreeBSD box as well. It was a bit more informative, with all sorts of messages about no swap space, but it was
                    Message 9 of 15 , May 1, 2008
                    • 0 Attachment
                      On Thu, May 01, 2008 at 10:04:03PM +0100, ed wrote:
                      > On Thu, 1 May 2008 16:39:26 -0400
                      > Scott <scottro@...> wrote:
                      >
                      >
                      > Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                      > had to run it here.

                      Well, I just tried it and yup, it crippled a FreeBSD box as well. It was
                      a bit more informative, with all sorts of messages about no swap space,
                      but it was effectively crippled. :)

                      --
                      Scott Robbins
                      PGP keyID EB3467D6
                      ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                      gpg --keyserver pgp.mit.edu --recv-keys EB3467D6

                      Buffy: Does it ever get easy?
                      Giles: You mean life?
                      Buffy: Yeah, does it ever get easy?
                      Giles: What do you want me to say?
                      Buffy: Lie to me.
                      Giles: Yes. It's terribly simple. The good guys are always s
                      talwart and true. The bad guys are easily distinguished by their
                      pointy horns or black hats, and we always defeat them and save
                      the day. No one ever dies... and everyone lives happily ever
                      after.
                      Buffy: Liar.
                    • Jeff Lane
                      Well, I tried two ways yesterday on my 8 node 512GB system... This was running beta RHEL 5.2. First I tried the c code calling fork() in a never ending while
                      Message 10 of 15 , May 2, 2008
                      • 0 Attachment
                        Well, I tried two ways yesterday on my 8 node 512GB system...

                        This was running beta RHEL 5.2.

                        First I tried the c code calling fork() in a never ending while loop.
                        The default number of processes seems to be 32768. The kernel did a
                        good job of killing off old ones and basically it just kept killing
                        them off over and over. The machine was not terribly usable, but it
                        stayed up and I was even able to ctrl-c the program and recover.

                        Then I tried the bash version... that one opened the max number of
                        processes, then the kernel said "Nope. No more resources. Screw you,
                        I'm killing them all" and it killed off ALL of them and ended my
                        script. heh...

                        On 5.0 it hangs every box I've tried. Same on SLES 9 (though I
                        haven't tried SLES10 service packs)...

                        I'm gonna wager a guess that the newer kernels may mitigate this...

                        on the other hand, I added a counter increment and a printf statement,
                        so that may even have slowed the forking down enough so the system
                        could get ahold of it. I'll have to try it without the
                        instrumetnation and see what happens.. :-)

                        Then I want to try it on a Windows XP system heh...

                        On Thu, May 1, 2008 at 7:30 PM, Scott <scottro@...> wrote:
                        >
                        >
                        >
                        >
                        >
                        >
                        > On Thu, May 01, 2008 at 10:04:03PM +0100, ed wrote:
                        > > On Thu, 1 May 2008 16:39:26 -0400
                        > > Scott <scottro@...> wrote:
                        > >
                        > >
                        > > Unfortunately I don't - I've changed jobs and we use SunOS/Linux so I
                        > > had to run it here.
                        >
                        > Well, I just tried it and yup, it crippled a FreeBSD box as well. It was
                        > a bit more informative, with all sorts of messages about no swap space,
                        > but it was effectively crippled. :)
                        >
                        >
                        > --
                        > Scott Robbins
                        > PGP keyID EB3467D6
                        > ( 1B48 077D 66F6 9DB0 FDC2 A409 FA54 EB34 67D6 )
                        > gpg --keyserver pgp.mit.edu --recv-keys EB3467D6
                        >
                        > Buffy: Does it ever get easy?
                        > Giles: You mean life?
                        > Buffy: Yeah, does it ever get easy?
                        > Giles: What do you want me to say?
                        > Buffy: Lie to me.
                        > Giles: Yes. It's terribly simple. The good guys are always s
                        > talwart and true. The bad guys are easily distinguished by their
                        > pointy horns or black hats, and we always defeat them and save
                        > the day. No one ever dies... and everyone lives happily ever
                        > after.
                        > Buffy: Liar.
                        >
                        >



                        --
                        ------------------> Jeffrey Lane - W4KDH <-------------------
                        www.jefflane.org
                        Another cog in the great
                        Corporate Wheel

                        The internet has no government, no constitution, no laws, no
                        rights, no police, no courts. Don't talk about fairness or
                        innocence, and don't talk about what should be done. Instead,
                        talk about what is being done and what will be done by the
                        amorphous unreachable undefinable blob called "the internet
                        user base." -Paul Vixie
                      • ed
                        On Fri, 2 May 2008 09:41:00 -0400 ... Does the windows kernel include a fork() - I thought they omitted that and have some other way of creating processes? --
                        Message 11 of 15 , May 2, 2008
                        • 0 Attachment
                          On Fri, 2 May 2008 09:41:00 -0400
                          "Jeff Lane" <sundowner225@...> wrote:

                          > Well, I tried two ways yesterday on my 8 node 512GB system...
                          >
                          > This was running beta RHEL 5.2.
                          >
                          > First I tried the c code calling fork() in a never ending while loop.
                          > The default number of processes seems to be 32768. The kernel did a
                          > good job of killing off old ones and basically it just kept killing
                          > them off over and over. The machine was not terribly usable, but it
                          > stayed up and I was even able to ctrl-c the program and recover.
                          >
                          > Then I tried the bash version... that one opened the max number of
                          > processes, then the kernel said "Nope. No more resources. Screw you,
                          > I'm killing them all" and it killed off ALL of them and ended my
                          > script. heh...
                          >
                          > On 5.0 it hangs every box I've tried. Same on SLES 9 (though I
                          > haven't tried SLES10 service packs)...
                          >
                          > I'm gonna wager a guess that the newer kernels may mitigate this...
                          >
                          > on the other hand, I added a counter increment and a printf statement,
                          > so that may even have slowed the forking down enough so the system
                          > could get ahold of it. I'll have to try it without the
                          > instrumetnation and see what happens.. :-)
                          >
                          > Then I want to try it on a Windows XP system heh...

                          Does the windows kernel include a fork() - I thought they omitted that
                          and have some other way of creating processes?

                          --
                          The ATM to the Ultra 5 is hosed because of a screwed up .procmailrc
                          file. AOL is playing NFL 2k2.
                          :: http://www.s5h.net/ :: http://www.s5h.net/gpg.html


                          [Non-text portions of this message have been removed]
                        Your message has been successfully submitted and would be delivered to recipients shortly.