Loading ...
Sorry, an error occurred while loading the content.

Barrier to implementing web content compression in mod_perl

Expand Messages
  • Alex Greg
    Hi, I ve been trying for some considerable time now to implement web content compression for our application, but it seems impossible without a rewrite. Our
    Message 1 of 9 , Jan 30, 2005
    • 0 Attachment
      Hi,


      I've been trying for some considerable time now to implement web
      content compression for our application, but it seems impossible
      without a rewrite.


      Our application is a (I assume) fairly typical Apache::Registry
      application. It used to run under mod_cgi but we're now running it
      under mod_perl for the obvious performance gains. HTTP headers are
      printed via a subroutine print_content_type:

      sub print_content_type
      {
      my ($content_type) = @_;

      if (!$content_type)
      {
      print "Content-type: text/html\n\n";
      }
      else
      {
      print "Content-type: $content_type\n\n";
      }
      }

      We have a few hundred .cgi files which are entry points for the user;
      these are simple stubs that then call an MVC controller module for
      that particular cgi. After print_content_type has been called to send
      the appropriate HTTP headers, the body of the HTTP response is then
      printed to standard output.

      The problem I have found is that when I modify print_content_type as follows:

      sub print_content_type
      {
      my ($content_type) = @_;

      my $r = Apache->request();

      $r = $r->filter_register;
      $fh = $r->filter_input();

      if (!$content_type)
      {
      $r->send_http_header("text/html");
      }
      else
      {
      $r->send_http_header($content_type);
      }
      }

      in order to register our application with the Apache::RegistryFilter
      chain (which is required to use either Apache::Dynagzip or
      Apache::Compress), I get the following in the error log:

      [Sat Dec 25 01:36:32 2004] [error] Not a HASH reference at
      /usr/lib/perl5/site_perl/5.8.3/Apache/Filter.pm line 197.

      and the page throws a 500 Internal Server Error. Apparently this is
      because "Apache->request is broken in terms of subclassing - it won't
      return the subclass object."
      (http://maclux-rz.uibk.ac.at/maillists/axkit-users/msg01214.shtml)

      It seems that without using Apache->request or reworking our entire
      application (over 500,000 lines of code), I can't get Apache::Filter
      to recognise the Content-type header we're sending, and it gets
      overridden with the default text/html from Apache::Dynagzip.


      Has anyone else come across this problem, or can suggest a solution or
      alternative approach? I've discussed it with Slava Bizyayev
      (http://marc.theaimsgroup.com/?l=apache-modperl&m=110435780804427&w=2)
      before, but I can't rework our application code into a handler as
      there's far too much of it. I've also emailed Ken Williams regarding
      the "Not a HASH reference" error, but the solution he suggested (using
      "$r = Apache->request->pnotes('filterobject')" instead of "$r =
      Apache->request" produced the same error.


      Regards,


      -- Alex
    • Stas Bekman
      ... [...] ... Why not switch to modperl2, where filters work better? http://perl.apache.org/docs/2.0/user/handlers/filters.html --
      Message 2 of 9 , Jan 30, 2005
      • 0 Attachment
        Alex Greg wrote:
        > Hi,
        >
        >
        > I've been trying for some considerable time now to implement web
        > content compression for our application, but it seems impossible
        > without a rewrite.
        [...]
        > Has anyone else come across this problem, or can suggest a solution or
        > alternative approach? I've discussed it with Slava Bizyayev
        > (http://marc.theaimsgroup.com/?l=apache-modperl&m=110435780804427&w=2)
        > before, but I can't rework our application code into a handler as
        > there's far too much of it. I've also emailed Ken Williams regarding
        > the "Not a HASH reference" error, but the solution he suggested (using
        > "$r = Apache->request->pnotes('filterobject')" instead of "$r =
        > Apache->request" produced the same error.

        Why not switch to modperl2, where filters work better?
        http://perl.apache.org/docs/2.0/user/handlers/filters.html


        --
        __________________________________________________________________
        Stas Bekman JAm_pH ------> Just Another mod_perl Hacker
        http://stason.org/ mod_perl Guide ---> http://perl.apache.org
        mailto:stas@... http://use.perl.org http://apacheweek.com
        http://modperlbook.org http://apache.org http://ticketmaster.com
      • Alex Greg
        ... I m not sure we re ready to migrate to MP2 / Apache 2 just yet. Can anything be done to address the filter problems in MP1? -- Alex
        Message 3 of 9 , Jan 31, 2005
        • 0 Attachment
          On Sun, 30 Jan 2005 23:02:53 -0500, Stas Bekman <stas@...> wrote:
          > Alex Greg wrote:
          > > Hi,
          > >
          > >
          > > I've been trying for some considerable time now to implement web
          > > content compression for our application, but it seems impossible
          > > without a rewrite.
          > [...]
          > > Has anyone else come across this problem, or can suggest a solution or
          > > alternative approach? I've discussed it with Slava Bizyayev
          > > (http://marc.theaimsgroup.com/?l=apache-modperl&m=110435780804427&w=2)
          > > before, but I can't rework our application code into a handler as
          > > there's far too much of it. I've also emailed Ken Williams regarding
          > > the "Not a HASH reference" error, but the solution he suggested (using
          > > "$r = Apache->request->pnotes('filterobject')" instead of "$r =
          > > Apache->request" produced the same error.
          >
          > Why not switch to modperl2, where filters work better?
          > http://perl.apache.org/docs/2.0/user/handlers/filters.html

          I'm not sure we're ready to migrate to MP2 / Apache 2 just yet.

          Can anything be done to address the filter problems in MP1?

          -- Alex
        • Geoffrey Young
          ... not really from the perspective of the mp1 api - we ve considered changing that behavior before, and IIRC philippe even produced a patch, but we considered
          Message 4 of 9 , Jan 31, 2005
          • 0 Attachment
            > Can anything be done to address the filter problems in MP1?

            not really from the perspective of the mp1 api - we've considered changing
            that behavior before, and IIRC philippe even produced a patch, but we
            considered it too risky for the stable branch.

            that said, we can certainly try to help you, but I don't have a feel for
            exactly what's wrong - having use Apache::Filter quite a bit I can't see why
            what you're doing won't work.

            so, to that end, I'd suggest you try to boil down the breakage into as small
            a reproducable test case as possible. this is a good place to start

            http://perl.apache.org/~geoff/bug-reporting-skeleton-mp1.tar.gz

            see the README in there for details, but basically what you want to do is
            try to create a simple filter with a simple function that acts as you
            describe. for instance, it looks like you're making two calls to
            filter_register and filter_input from within the same handler, but I can't
            be sure. reproducing the error will make great strides on our part in
            understanding the issue. it may also even fix the problem for you ;)

            --Geoff
          • Perrin Harkins
            ... Have you tried using Apache::RegistryFilter instead of Registry? Or try putting something like this at the top of your scripts: my $r = shift;
            Message 5 of 9 , Jan 31, 2005
            • 0 Attachment
              On Sun, 2005-01-30 at 21:54 +0000, Alex Greg wrote:
              > The problem I have found is that when I modify print_content_type as follows:
              >
              > sub print_content_type
              > {
              > my ($content_type) = @_;
              >
              > my $r = Apache->request();
              >
              > $r = $r->filter_register;
              > $fh = $r->filter_input();
              >
              > if (!$content_type)
              > {
              > $r->send_http_header("text/html");
              > }
              > else
              > {
              > $r->send_http_header($content_type);
              > }
              > }
              >
              > in order to register our application with the Apache::RegistryFilter
              > chain (which is required to use either Apache::Dynagzip or
              > Apache::Compress), I get the following in the error log:
              >
              > [Sat Dec 25 01:36:32 2004] [error] Not a HASH reference at
              > /usr/lib/perl5/site_perl/5.8.3/Apache/Filter.pm line 197.

              Have you tried using Apache::RegistryFilter instead of Registry? Or
              try putting something like this at the top of your scripts:

              my $r = shift;
              $r->filter_register();

              I don't think there's any reason for you to call $r->filter_input().

              - Perrin
            • Alex Greg
              ... HI Perrin, I m using Apache::RegistryFilter: PerlHandler Apache::RegistryFilter Apache::Dynagzip See the attached httpd.conf for full context etc. ... This
              Message 6 of 9 , Jan 31, 2005
              • 0 Attachment
                On Mon, 31 Jan 2005 15:34:05 -0500, Perrin Harkins <perrin@...> wrote:
                > On Sun, 2005-01-30 at 21:54 +0000, Alex Greg wrote:
                > > The problem I have found is that when I modify print_content_type as follows:
                > >
                > > sub print_content_type
                > > {
                > > my ($content_type) = @_;
                > >
                > > my $r = Apache->request();
                > >
                > > $r = $r->filter_register;
                > > $fh = $r->filter_input();
                > >
                > > if (!$content_type)
                > > {
                > > $r->send_http_header("text/html");
                > > }
                > > else
                > > {
                > > $r->send_http_header($content_type);
                > > }
                > > }
                > >
                > > in order to register our application with the Apache::RegistryFilter
                > > chain (which is required to use either Apache::Dynagzip or
                > > Apache::Compress), I get the following in the error log:
                > >
                > > [Sat Dec 25 01:36:32 2004] [error] Not a HASH reference at
                > > /usr/lib/perl5/site_perl/5.8.3/Apache/Filter.pm line 197.
                >
                > Have you tried using Apache::RegistryFilter instead of Registry?

                HI Perrin,


                I'm using Apache::RegistryFilter:

                PerlHandler Apache::RegistryFilter Apache::Dynagzip

                See the attached httpd.conf for full context etc.

                > Or try putting something like this at the top of your scripts:
                >
                > my $r = shift;
                > $r->filter_register();
                >
                > I don't think there's any reason for you to call $r->filter_input().

                This causes the same problem as:

                $r = Apache->request();


                Attached is my httpd.conf and a script test.cgi which reproduces the
                problem I'm having. I'm using Apache/1.3.33 with mod_perl statically
                linked.


                Regards,


                -- Alex
              • Perrin Harkins
                ... Does this actually work without the filitering? I see a use of ScriptAlias in there, which usually sets mod-cgi and breaks mod_perl for that directory.
                Message 7 of 9 , Jan 31, 2005
                • 0 Attachment
                  On Mon, 2005-01-31 at 21:23 +0000, Alex Greg wrote:
                  > I'm using Apache::RegistryFilter:
                  >
                  > PerlHandler Apache::RegistryFilter Apache::Dynagzip
                  >
                  > See the attached httpd.conf for full context etc.

                  Does this actually work without the filitering? I see a use of
                  ScriptAlias in there, which usually sets mod-cgi and breaks mod_perl for
                  that directory.

                  http://perl.apache.org/docs/1.0/guide/config.html#Alias_Configurations

                  > Attached is my httpd.conf and a script test.cgi which reproduces the
                  > problem I'm having. I'm using Apache/1.3.33 with mod_perl statically
                  > linked.

                  This script should not need to call register_filter at all, since it's
                  using Apache::RegistryFilter.

                  - Perrin
                • Alex Greg
                  ... Yep, it runs fine. $ENV{ mod_perl } is true. http://perl.apache.org/docs/1.0/guide/config.html#Alias_Configurations ... Apparently, if I don t call
                  Message 8 of 9 , Jan 31, 2005
                  • 0 Attachment
                    On Mon, 31 Jan 2005 17:55:27 -0500, Perrin Harkins <perrin@...> wrote:
                    > On Mon, 2005-01-31 at 21:23 +0000, Alex Greg wrote:
                    > > I'm using Apache::RegistryFilter:
                    > >
                    > > PerlHandler Apache::RegistryFilter Apache::Dynagzip
                    > >
                    > > See the attached httpd.conf for full context etc.
                    >
                    > Does this actually work without the filitering? I see a use of
                    > ScriptAlias in there, which usually sets mod-cgi and breaks mod_perl for
                    > that directory.

                    Yep, it runs fine. $ENV{'mod_perl'} is true.

                    http://perl.apache.org/docs/1.0/guide/config.html#Alias_Configurations
                    >
                    > > Attached is my httpd.conf and a script test.cgi which reproduces the
                    > > problem I'm having. I'm using Apache/1.3.33 with mod_perl statically
                    > > linked.
                    >
                    > This script should not need to call register_filter at all, since it's
                    > using Apache::RegistryFilter.

                    Apparently, if I don't call register_filter, then my script isn't
                    registered in the filter chain and Dynagzip then can't see my headers,
                    and it adds its own.

                    This would seem to be the case - if I comment out the $r =
                    $r->filter_register; line, I get this:

                    [root@server2 root]# (echo "GET /cgi-bin/test.cgi HTTP/1.1"; echo
                    "Host: 192.168.1.3"; echo) | nc 192.168.1.3 80
                    HTTP/1.1 200 OK
                    Date: Mon, 31 Jan 2005 23:03:52 GMT
                    Server: Apache/1.3.33 (Unix) mod_perl/1.29
                    X-Module-Sender: Apache::Dynagzip
                    Expires: Monday, 31-January-2005 23:08:52 GMT
                    Transfer-Encoding: chunked
                    Connection: close
                    Content-Type: text/html

                    ad
                    Content-type: text/html

                    <h1>Lorem Ipsum</h1>

                    <p>Lorem ipsum dolor sit amet, consectetuer adipiscing elit.
                    Suspendisse fringilla porttitor mi....</p>


                    0


                    The first Content-Type: text/html header is from Dynagzip, the second
                    one is the one from my script. The error log repors

                    [Mon Jan 31 23:03:52 2005] [debug]
                    /usr/lib/perl5/site_perl/5.8.3/Apache/Dynagzip.pm(917): [client
                    192.168.1.3] Apache::Dynagzip default_content_handler creates default
                    Content-Type for GET /cgi-bin/test.cgi HTTP/1.1


                    Regards,


                    -- Alex
                  • Rob Mueller
                    I m surprised nobody has mentioned this yet on the list, but one approach I d recommend is going to 2 separate apache servers. A light frontend one that does
                    Message 9 of 9 , Feb 1, 2005
                    • 0 Attachment
                      I'm surprised nobody has mentioned this yet on the list, but one approach
                      I'd recommend is going to 2 separate apache servers. A light frontend one
                      that does all the SSL/compression work, and a heavy backend mod_perl one.

                      http://modperlbook.org/html/ch12_01.html

                      This works fantastic for us, and avoids any of the issues of chained filters
                      in the mod_perl process. The light weight servers can easily do the
                      compression and SSL work and deal with clients on slow networks, leaving our
                      mod_perl servers to do the actual processing. Even though we process 2M+
                      requests a day, we generally don't have more than 20 mod_perl httpd procs at
                      any one time, while having 500-1000 frontend apache procs dealing with the
                      slow network clients.

                      We personally use mod_accel and mod_deflate written by Igor Sysoev. Although
                      most of the docs are in russian, we managed to get it working, and it's been
                      one of those things that has "just worked" since day 1. Anytime we've had
                      any technical issues, I've emailed Igor, and he's been fast and responsive
                      to reply and help, and he knows his stuff!

                      http://sysoev.ru/en/

                      See the link at the bottom for our success story and compilation procedure.

                      Good luck.

                      Rob
                    Your message has been successfully submitted and would be delivered to recipients shortly.