Loading ...
Sorry, an error occurred while loading the content.

Cgroups for automatically limiting resource usage for certains programs

Expand Messages
  • Pascal
    Hi all, does anyone of you have experience with cgroups? Perhabs even on a desktop system? Why am I asking? Here on my laptop whith an overwhelming 512MB RAM
    Message 1 of 10 , Aug 3, 2013
    • 0 Attachment
      Hi all, does anyone of you have experience with cgroups? Perhabs even on a desktop system?

      Why am I asking? Here on my laptop whith an overwhelming 512MB RAM and an Intel
      Celeron CPU (Should be about 6 years old) I would like to put some processes
      into a cgroup as they use many resources.I would rather have them run at a
      slower speed (in the background, even they take a lot longer to finish their
      tasks). These are programs written in Python like gPodder and easytag for
      example. I do not care whether it takes 5 minutes to update my podcast feeds and
      download new episodes or 30 minutes. It just should not get in the way of my workflow. So my dream is to create a cgroup just for those programs to which they are assigned automatically when started. This group is accorded a limited percentage of CPU, RAM (maybe I/O operations is possible too, I do not know).

      This could be tricky as I was told by an admin that it is very difficult to debug cgroups so making a mistake is not easy to correct.

      Have you used cgroup so far? If so, would they be the appropriate tool for my intention?

      Thanks to all,

      Pascal
    • ed
      ... I ve not used cgroups in that way before. I ve only used them when CONFGI_SCHED_AUTOGROUP wasn t available on earlier kernels. It seems to me that my
      Message 2 of 10 , Aug 3, 2013
      • 0 Attachment
        On Sat, Aug 03, 2013 at 05:35:02PM -0000, Pascal wrote:
        > Hi all, does anyone of you have experience with cgroups? Perhabs even on a desktop system?
        >
        > Why am I asking? Here on my laptop whith an overwhelming 512MB RAM and
        > an Intel Celeron CPU (Should be about 6 years old) I would like to put
        > some processes into a cgroup as they use many resources.I would rather
        > have them run at a slower speed (in the background, even they take a
        > lot longer to finish their tasks). These are programs written in
        > Python like gPodder and easytag for example. I do not care whether it
        > takes 5 minutes to update my podcast feeds and download new episodes
        > or 30 minutes. It just should not get in the way of my workflow. So my
        > dream is to create a cgroup just for those programs to which they are
        > assigned automatically when started. This group is accorded a limited
        > percentage of CPU, RAM (maybe I/O operations is possible too, I do not
        > know).
        >
        > This could be tricky as I was told by an admin that it is very
        > difficult to debug cgroups so making a mistake is not easy to correct.
        >
        > Have you used cgroup so far? If so, would they be the appropriate tool
        > for my intention?

        I've not used cgroups in that way before. I've only used them when
        CONFGI_SCHED_AUTOGROUP wasn't available on earlier kernels.

        It seems to me that my knee-jerk reaction would be to use "ionice -c 3"
        and "nice -n 19" to give background IO and background CPU resources.
        This won't limit RAM (see bash ulimit perhaps for that). Perhaps this is
        the lazy way, but it's the way that I generally limit process resources.

        Generally speaking, these days IO is the bottleneck for most things, so
        I tend to rely on "ionice -c 3" for things that may hammer the disk. Of
        course, if you're running seti then it won't really make any odds. With
        a Celeron though, I don't know if this is still the case. When I use my
        raspberry pi, which I don't think has a lot of CPU grunt, I think the
        storage bus was the bottleneck there too, so worth keeping in mind. If
        top shows the CPU isn't >90% busy but your application is still slow,
        try stracing it and see which syscalls are taking the longest.

        --
        Best regards,
        Ed http://www.s5h.net/
      • thad_floryan
        ... Hi Pascal, Not me. :-) ... No; I didn t even know what it is until I just did some searching; see below. ... Possibly, but I don t have a definitive
        Message 3 of 10 , Aug 3, 2013
        • 0 Attachment
          --- In linux@yahoogroups.com, "Pascal" <pascal.bernhard@...> wrote:
          >
          > Hi all, does anyone of you have experience with cgroups? Perhabs
          > even on a desktop system?

          Hi Pascal,

          Not me. :-)

          > [...]
          > This could be tricky as I was told by an admin that it is very
          > difficult to debug cgroups so making a mistake is not easy to
          > correct.
          >
          > Have you used cgroup so far?

          No; I didn't even know what it is until I just did some searching;
          see below.

          > If so, would they be the appropriate tool for my intention?

          Possibly, but I don't have a definitive answer. Searching the 'Net
          found these items which appear useful to review:

          http://en.wikipedia.org/wiki/Cgroups
          cgroups (control groups) is a Linux kernel feature to limit, account
          and isolate resource usage (CPU, memory, disk I/O, etc.) of process
          groups.

          Cgroup - The Linux Kernel Archives
          https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt
          Control Groups 1.1 What are cgroups ? 1.2 Why are cgroups needed ? 1.3
          How are cgroups implemented ? 1.4 What does notify_on_release do ? 1.5
          What ...

          Cgroups - Red Hat Customer Portal
          https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html
          Red Hat Enterprise Linux 6 provides a new kernel feature: control
          groups, which are called by their shorter name cgroups in this
          guide. Cgroups allow you to ...

          Chapter 10. Kernel Control Groups - Documentation
          http://doc.opensuse.org/products/draft/SLES/SLES-tuning_sd_draft/cha.tuning.cgroups.html
          Kernel Control Groups (abbreviated known as "cgroups") are a kernel
          feature that allows aggregating or partitioning tasks (processes) and
          all their children into ...

          [systemd-devel] [HEADSUP] cgroup changes
          http://lists.freedesktop.org/archives/systemd-devel/2013-June/011521.html
          Jun 21, 2013 - However, the main reason for this is that this is
          eventually going to be the only way how containers/VMs can get a
          cgroup of their own.

          Cgroups explained: Limiting Linux Processes | János Pásztor
          http://www.janoszen.com/2013/02/06/limiting-linux-processes-cgroups-explained/
          Feb 6, 2013 - When talking about cgroups, don't just think about a
          bunch of groups where processes can be jailed into. These groups are
          hierarchical, ...
        • thad_floryan
          ... Hi Pascal, I hope you re sitting down. I was surprised to see cgroups affiliated with systemd -- the hairs on the back of my neck raised up in alarm. And
          Message 4 of 10 , Aug 3, 2013
          • 0 Attachment
            --- In linux@yahoogroups.com, "Pascal" <pascal.bernhard@...> wrote:
            > [...]
            > Have you used cgroup so far? If so, would they be the appropriate
            > tool for my intention?

            Hi Pascal,

            I hope you're sitting down. I was surprised to see cgroups affiliated
            with systemd -- the hairs on the back of my neck raised up in alarm.

            And here's why:

            http://lists.freedesktop.org/archives/systemd-devel/2013-June/011521.html

            " Lennart Poettering lennart at poettering.net
            " Fri Jun 21 10:36:03 PDT 2013
            "
            " On monday I posted this mail:
            "
            " http://lists.freedesktop.org/archives/systemd-devel/2013-June/011388.html
            "
            " Here's an update and a bit on the bigger picture:
            "
            " Half of what I mentioned there is now in place. There's now a new
            " "slice" unit type in place in git, and everything is hooked up to
            " it. logind will now also keep track of running containers/VMs. The
            " various container/VM managers have to register with logind now.
            " This serves the purpose of better integration of containers/VMs
            " everywhere (so that "ps" can show for each process where it belongs
            " to). However, the main reason for this is that this is eventually
            " going to be the only way how containers/VMs can get a cgroup of
            " their own.
            "
            " So, in that context, a bit of the bigger picture:
            "
            " It took us a while to realize the full extent how awfully unusable
            " cgroups currently are. The attributes have way more
            " interdependencies than people might think and it is trivial to
            " create non-sensical configurations...
            "
            " Of course, understanding how awful the status quo is a good first
            " step. But we really needed to figure out what we can do about this
            " to clean this up in the long run, and how we can get to something
            " useful quickly. So, after much discussion between Tejun (the
            " kernel cgroup maintainer) and various other folks here's the new
            " scheme that we want to go for:
            " [...]

            The "something useful quickly" means it's going to be yet another POS
            that wasn't thought out and planned and implemented properly. :-)
          • thad_floryan
            ... Some additional references: Linux kernel Namespaces and cgroups by Rami Rosen http://www.haifux.org/lectures/299/netLec7.pdf Balbir Singh, Vaidynathan
            Message 5 of 10 , Aug 3, 2013
            • 0 Attachment
              --- In linux@yahoogroups.com, "thad_floryan" <thad@...> wrote:
              > [...]
              > Possibly, but I don't have a definitive answer. Searching the 'Net
              > found these items which appear useful to review:
              > [...]

              Some additional references:

              Linux kernel Namespaces and cgroups by Rami Rosen
              http://www.haifux.org/lectures/299/netLec7.pdf

              Balbir Singh, Vaidynathan Srinivasan (July 2007). "Containers:
              Challenges with the memory resource controller and its performance"
              http://www.kernel.org/doc/ols/2007/ols2007v2-pages-209-222.pdf

              Kamkamezawa Hiroyu (19 November 2008). "Cgroup and Memory Resource Controller"
              http://www.linuxfoundation.jp/jp_uploads/seminar20081119/CgroupMemcgMaster.pdf

              Dave Hansen. "Resource Management"
              http://events.linuxfoundation.org/slides/lfcs09_hansen2.pdf

              Linux kernel documentation on cgroups
              http://www.kernel.org/doc/Documentation/cgroups/
            • Pascal
              ... With these Python programs the CPU is the main culprit, and RAM is a little bit of an issue too. I m asking cgroups as I would like to automate the job of
              Message 6 of 10 , Aug 4, 2013
              • 0 Attachment
                --- In linux@yahoogroups.com, ed <ed@...> wrote:
                >
                > On Sat, Aug 03, 2013 at 05:35:02PM -0000, Pascal wrote:
                > > Hi all, does anyone of you have experience with cgroups? Perhabs even on a desktop system?
                > >
                > > Why am I asking? Here on my laptop whith an overwhelming 512MB RAM and
                > > an Intel Celeron CPU (Should be about 6 years old) I would like to put
                > > some processes into a cgroup as they use many resources.I would rather
                > > have them run at a slower speed (in the background, even they take a
                > > lot longer to finish their tasks). These are programs written in
                > > Python like gPodder and easytag for example. I do not care whether it
                > > takes 5 minutes to update my podcast feeds and download new episodes
                > > or 30 minutes. It just should not get in the way of my workflow. So my
                > > dream is to create a cgroup just for those programs to which they are
                > > assigned automatically when started. This group is accorded a limited
                > > percentage of CPU, RAM (maybe I/O operations is possible too, I do not
                > > know).
                > >
                > > This could be tricky as I was told by an admin that it is very
                > > difficult to debug cgroups so making a mistake is not easy to correct.
                > >
                > > Have you used cgroup so far? If so, would they be the appropriate tool
                > > for my intention?
                >
                > I've not used cgroups in that way before. I've only used them when
                > CONFGI_SCHED_AUTOGROUP wasn't available on earlier kernels.
                >
                > It seems to me that my knee-jerk reaction would be to use "ionice -c 3"
                > and "nice -n 19" to give background IO and background CPU resources.
                > This won't limit RAM (see bash ulimit perhaps for that). Perhaps this is
                > the lazy way, but it's the way that I generally limit process resources.
                >
                > Generally speaking, these days IO is the bottleneck for most things, so
                > I tend to rely on "ionice -c 3" for things that may hammer the disk. Of
                > course, if you're running seti then it won't really make any odds. With
                > a Celeron though, I don't know if this is still the case. When I use my
                > raspberry pi, which I don't think has a lot of CPU grunt, I think the
                > storage bus was the bottleneck there too, so worth keeping in mind. If
                > top shows the CPU isn't >90% busy but your application is still slow,
                > try stracing it and see which syscalls are taking the longest.


                With these Python programs the CPU is the main culprit, and RAM is a little bit of an issue too. I'm asking cgroups as I would like to automate the job of "reigning in" these processes. But I will try ionice to see how effective that already is.

                Pascal
              • Pascal
                Hi Thad, ... Thanks for the links, Thad. I guess in future when having a question or a problem with Linux I will always post something on the list, expecting
                Message 7 of 10 , Aug 4, 2013
                • 0 Attachment
                  Hi Thad,

                  --- In linux@yahoogroups.com, "thad_floryan" <thad@...> wrote:
                  >
                  > --- In linux@yahoogroups.com, "Pascal" <pascal.bernhard@> wrote:
                  > >
                  > > Hi all, does anyone of you have experience with cgroups? Perhabs
                  > > even on a desktop system?
                  >
                  > Hi Pascal,
                  >
                  > Not me. :-)
                  >
                  > > [...]
                  > > This could be tricky as I was told by an admin that it is very
                  > > difficult to debug cgroups so making a mistake is not easy to
                  > > correct.
                  > >
                  > > Have you used cgroup so far?
                  >
                  > No; I didn't even know what it is until I just did some searching;
                  > see below.
                  >
                  > > If so, would they be the appropriate tool for my intention?
                  >
                  > Possibly, but I don't have a definitive answer. Searching the 'Net
                  > found these items which appear useful to review:
                  >
                  > http://en.wikipedia.org/wiki/Cgroups
                  > cgroups (control groups) is a Linux kernel feature to limit, account
                  > and isolate resource usage (CPU, memory, disk I/O, etc.) of process
                  > groups.
                  >
                  > Cgroup - The Linux Kernel Archives
                  > https://www.kernel.org/doc/Documentation/cgroups/cgroups.txt
                  > Control Groups 1.1 What are cgroups ? 1.2 Why are cgroups needed ? 1.3
                  > How are cgroups implemented ? 1.4 What does notify_on_release do ? 1.5
                  > What ...
                  >
                  > Cgroups - Red Hat Customer Portal
                  > https://access.redhat.com/site/documentation/en-US/Red_Hat_Enterprise_Linux/6/html/Resource_Management_Guide/ch01.html
                  > Red Hat Enterprise Linux 6 provides a new kernel feature: control
                  > groups, which are called by their shorter name cgroups in this
                  > guide. Cgroups allow you to ...
                  >
                  > Chapter 10. Kernel Control Groups - Documentation
                  > http://doc.opensuse.org/products/draft/SLES/SLES-tuning_sd_draft/cha.tuning.cgroups.html
                  > Kernel Control Groups (abbreviated known as "cgroups") are a kernel
                  > feature that allows aggregating or partitioning tasks (processes) and
                  > all their children into ...
                  >
                  > [systemd-devel] [HEADSUP] cgroup changes
                  > http://lists.freedesktop.org/archives/systemd-devel/2013-June/011521.html
                  > Jun 21, 2013 - However, the main reason for this is that this is
                  > eventually going to be the only way how containers/VMs can get a
                  > cgroup of their own.
                  >
                  > Cgroups explained: Limiting Linux Processes | J�nos P�sztor
                  > http://www.janoszen.com/2013/02/06/limiting-linux-processes-cgroups-explained/
                  > Feb 6, 2013 - When talking about cgroups, don't just think about a
                  > bunch of groups where processes can be jailed into. These groups are
                  > hierarchical, ...
                  >
                  Thanks for the links, Thad.

                  I guess in future when having a question or a problem with Linux I will always post something on the list, expecting elaborate & well-researched answers. Thus I could save myself the (admittedly rather modest) hassle of doing the search myself. :-P
                • ed
                  ... If CPU is the main culprit then you won t notice much gain from ionice. On the note of python being a CPU hog, I ve recently converted several scripts from
                  Message 8 of 10 , Aug 4, 2013
                  • 0 Attachment
                    On Sun, Aug 04, 2013 at 05:12:55PM -0000, Pascal wrote:
                    >
                    > --- In linux@yahoogroups.com, ed <ed@...> wrote:
                    > >
                    > > On Sat, Aug 03, 2013 at 05:35:02PM -0000, Pascal wrote:
                    > > [...]
                    > > Generally speaking, these days IO is the bottleneck for most things, so
                    > > I tend to rely on "ionice -c 3" for things that may hammer the disk. Of
                    > > course, if you're running seti then it won't really make any odds. With
                    > > a Celeron though, I don't know if this is still the case. When I use my
                    > > raspberry pi, which I don't think has a lot of CPU grunt, I think the
                    > > storage bus was the bottleneck there too, so worth keeping in mind. If
                    > > top shows the CPU isn't >90% busy but your application is still slow,
                    > > try stracing it and see which syscalls are taking the longest.
                    >
                    > With these Python programs the CPU is the main culprit, and RAM is a
                    > little bit of an issue too. I'm asking cgroups as I would like to
                    > automate the job of "reigning in" these processes. But I will try
                    > ionice to see how effective that already is.

                    If CPU is the main culprit then you won't notice much gain from ionice.

                    On the note of python being a CPU hog, I've recently converted several
                    scripts from perl to java as dynamically typed languages such as
                    perl/php/python/ruby don't perform well when doing numerics compared to
                    statically typed languages such as java/c#/c/c++ etc.

                    If your scripts are simple to convert then that could be a quick win.

                    --
                    Best regards,
                    Ed http://www.s5h.net/
                  • thad_floryan
                    ... You re very welcome! ... Only if the subject interests me and/or other people. :-) In this case, I had never heard of cgroups before and my curiosity was
                    Message 9 of 10 , Aug 5, 2013
                    • 0 Attachment
                      --- In linux@yahoogroups.com, "Pascal" <pascal.bernhard@...> wrote:
                      > Hi Thad,
                      >
                      > --- In linux@yahoogroups.com, "thad_floryan" <thad@> wrote:
                      > >
                      > > --- In linux@yahoogroups.com, "Pascal" <pascal.bernhard@> wrote:
                      > > >
                      > > > Hi all, does anyone of you have experience with cgroups? Perhabs
                      > > > even on a desktop system?
                      > >
                      > > Hi Pascal,
                      > >
                      > > Not me. :-)
                      > > [...]
                      > > I didn't even know what it is until I just did some searching;
                      > > see below.
                      > > [...]
                      >
                      > Thanks for the links, Thad.

                      You're very welcome!

                      > I guess in future when having a question or a problem with Linux I
                      > will always post something on the list, expecting elaborate &
                      > well-researched answers. Thus I could save myself the (admittedly
                      > rather modest) hassle of doing the search myself. :-P

                      Only if the subject interests me and/or other people. :-)

                      In this case, I had never heard of cgroups before and my curiosity was
                      greatly piqued so I did the search and also aggregated a bunch of info
                      as PDFs in one of my Tech Reference directories and created a PDF of
                      what appears to be a clear overview of cgroups and uploaded it into
                      this group's files section several days ago:

                      http://tech.groups.yahoo.com/group/linux/message/63413
                      Sat Aug 3, 2013 4:42 pm
                      New file uploaded to linux

                      http://groups.yahoo.com/group/linux/files/Cgroups_explained.pdf

                      Searching using the better search engines should NEVER be considered
                      any kind of a hassle, especially if you construct your search query
                      as if was something you would verbally ask someone face-to-face; that
                      is how I do most of my most successful searches with only "good" hits.

                      Problem with using Google is the URL results are an unholy mess due to
                      Google wrapping the real URL within a rat's nest of tracking crapola,
                      and to bypass that you need the "Google/Yandex search links fix 1.4"
                      as a Firefox addon as I've written here before. Or you could write a
                      program to strip the Google Garbage from the results entered into the
                      past buffer from using the "copy link location" option; see my articles
                      here:

                      http://tech.groups.yahoo.com/group/linux/message/63106
                      Sun Jun 23, 2013 5:54 am
                      HOWTO fix Google's tracking URLs returned after a search

                      http://tech.groups.yahoo.com/group/linux/message/63309
                      Mon Jul 22, 2013 6:12 pm
                      Don't be pwned by Google (was: Re: Free open source security ...

                      http://tech.groups.yahoo.com/group/linux/message/63312
                      Tue Jul 23, 2013 1:29 am
                      Don't be pwned by Google (was: Re: Free open source security ...

                      Thad
                    • thad_floryan
                      ... ^^^^ S/B paste . It s too early to be typing; I only awoke early to send an urgent email and I m heading back to sleep now.
                      Message 10 of 10 , Aug 5, 2013
                      • 0 Attachment
                        --- In linux@yahoogroups.com, "thad_floryan" <thad@...> wrote:
                        > [...]
                        > program to strip the Google Garbage from the results entered into the
                        > past buffer from using the "copy link location" option; see my ...
                        ^^^^
                        S/B "paste". It's too early to be typing; I only awoke early to send an
                        urgent email and I'm heading back to sleep now.
                      Your message has been successfully submitted and would be delivered to recipients shortly.