Loading ...
Sorry, an error occurred while loading the content.

Figuring out SOAP::Lite dependencies

Expand Messages
  • jtsengorg
    I maintain some scripts that read XML data from our data provider s SOAP methods. Earlier this morning I discovered my scripts on my production server weren t
    Message 1 of 6 , Sep 3, 2013
    View Source
    • 0 Attachment
      I maintain some scripts that read XML data from our data provider's SOAP methods.  Earlier this morning I discovered my scripts on my production server weren't downloading data, although the ones on my test server had no issues at all.  The errors were being generated at XML::Parser on line 187.  I suspect the code making the call to XML::Parser was mangling $arg somehow, so I made an attempt to find out what that code was and see how XML::Parser relates to SOAP::Lite.  I could see no obvious relationship.

      I know that my test and prod systems do have different versions of the modules in use, but I want to know exactly what's being affected before I propose that prod be updated through CPAN (I'm not prod's system owner; we've been updating modules via apt).  I'd greatly appreciate any ideas.

      Thx,

       - Joe
    • Michiel Beijen
      Hi Joe, Of course the version of the module could be just *one* of the differences between your production and test environment. While I would recommend you d
      Message 2 of 6 , Sep 4, 2013
      View Source
      • 0 Attachment
        Hi Joe,
         
        Of course the version of the module could be just *one* of the differences between your production and test environment.
        While I would recommend you'd sync up your modules, please also investigate the problem more deeply.
        What is the exact error message you get? What is on line 187 of XML/Parser.pm on your machine?
         
        It *could* for instance be - just as an example and not to say that it is the issue here - that the web server from which you downloaded the data provided an HTTP 500 or other kind of error causing the system not to download the data, while your test system did succeed.
        Just updating the module for the sake of updating in this case would not resolve the issue.
         
        One other factor to consider is if these problems started to happen just overnight or if there were changes before they started happening. If there were no changes (i.e. it previously worked), updating the module will really probably not be the resolution.
        --
        Mike


        On Tue, Sep 3, 2013 at 9:39 PM, <joe_tseng@...> wrote:
         

        I maintain some scripts that read XML data from our data provider's SOAP methods.  Earlier this morning I discovered my scripts on my production server weren't downloading data, although the ones on my test server had no issues at all.  The errors were being generated at XML::Parser on line 187.  I suspect the code making the call to XML::Parser was mangling $arg somehow, so I made an attempt to find out what that code was and see how XML::Parser relates to SOAP::Lite.  I could see no obvious relationship.


        I know that my test and prod systems do have different versions of the modules in use, but I want to know exactly what's being affected before I propose that prod be updated through CPAN (I'm not prod's system owner; we've been updating modules via apt).  I'd greatly appreciate any ideas.

        Thx,

         - Joe


      • Joe Tseng
        Some additional details: The first symptom of this issue was when I saw an email that was sent from my SOAP object s on_fault handler; the error said 200 OK .
        Message 3 of 6 , Sep 4, 2013
        View Source
        • 0 Attachment
          Some additional details:

          The first symptom of this issue was when I saw an email that was sent from my SOAP object's on_fault handler; the error said "200 OK".  When I looked further into it, it turns out I was still indeed pulling down XML content from the feed, but for whatever reason was being truncated.  So yes, the remote system was still up and working properly, but the problem was with one of my modules.

          Regarding symptoms, the only issue I've seen that *might* be a/the cause (and is definitely repeatable) is the amount of data is too large to be handled completely in one pass.  I get a msg saying there's an issue regarding my XML data at position 8192 - that's where the data is truncated.

          Line 187 on XML::Parser isn't really that important; what's important is that the value of $arg that's being passed on that line is already predetermined by the code calling that line.  I'm trying to figure out what is calling that, and thus, I want to know how $arg is being set, and I want to know why the value of $arg is different between systems.  I suspect it's because the two systems have different module versions, but since I'm not the owner of the prod system (and I want to be careful in my approach since I don't want to break anything else), I want to make sure I have enough evidence to make sure it's worth while for us to apply changes.

          As far as it previously working on prod, I doubt that this code ever worked under the specific conditions that triggered the exception.  It's just that this had not occurred up until yesterday.


          If you type "Google" into Google, you can break the Internet.  -- Jen Barber



          From: michiel.beijen@...
          Date: Wed, 4 Sep 2013 12:15:21 +0200
          Subject: Re: [soaplite] Figuring out SOAP::Lite dependencies
          To: joe_tseng@...
          CC: soaplite@yahoogroups.com

          Hi Joe,
           
          Of course the version of the module could be just *one* of the differences between your production and test environment.
          While I would recommend you'd sync up your modules, please also investigate the problem more deeply.
          What is the exact error message you get? What is on line 187 of XML/Parser.pm on your machine?
           
          It *could* for instance be - just as an example and not to say that it is the issue here - that the web server from which you downloaded the data provided an HTTP 500 or other kind of error causing the system not to download the data, while your test system did succeed.
          Just updating the module for the sake of updating in this case would not resolve the issue.
           
          One other factor to consider is if these problems started to happen just overnight or if there were changes before they started happening. If there were no changes (i.e. it previously worked), updating the module will really probably not be the resolution.
          --
          Mike


          On Tue, Sep 3, 2013 at 9:39 PM, <joe_tseng@...> wrote:
           
          I maintain some scripts that read XML data from our data provider's SOAP methods.  Earlier this morning I discovered my scripts on my production server weren't downloading data, although the ones on my test server had no issues at all.  The errors were being generated at XML::Parser on line 187.  I suspect the code making the call to XML::Parser was mangling $arg somehow, so I made an attempt to find out what that code was and see how XML::Parser relates to SOAP::Lite.  I could see no obvious relationship.

          I know that my test and prod systems do have different versions of the modules in use, but I want to know exactly what's being affected before I propose that prod be updated through CPAN (I'm not prod's system owner; we've been updating modules via apt).  I'd greatly appreciate any ideas.

          Thx,

           - Joe



        • Joe Tseng
          Looks like upping the value might have did the trick! I just got a feed with an object of size 13178. Thank you for the tip! (I say might because I m still
          Message 4 of 6 , Sep 27, 2013
          View Source
          • 0 Attachment
            Looks like upping the value might have did the trick!  I just got a feed with an object of size 13178.  Thank you for the tip!

            (I say might because I'm still sporadically getting the "200 OK" errors, but if I manually trigger my job a minute later, it runs normally.  I suspect this time it's my feed provider; I'll be keeping an eye out on this for a while.)

             - Joe


            If you type "Google" into Google, you can break the Internet.  -- Jen Barber



            Date: Fri, 6 Sep 2013 08:20:25 -0700
            From: qglex@...
            Subject: Re: [soaplite] Figuring out SOAP::Lite dependencies
            To: joe_tseng@...

            If the length of the envelope/xml is truncating to 10000, take a look at

            $SOAP::Constants::MAX_CONTENT_SIZE

            By default it's set to 10,000.


             
            -Jeff


            From: Joe Tseng <joe_tseng@...>
            To: "soaplite@yahoogroups.com" <soaplite@yahoogroups.com>
            Sent: Wednesday, September 4, 2013 6:41 AM
            Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies

             
            Some additional details:

            The first symptom of this issue was when I saw an email that was sent from my SOAP object's on_fault handler; the error said "200 OK".  When I looked further into it, it turns out I was still indeed pulling down XML content from the feed, but for whatever reason was being truncated.  So yes, the remote system was still up and working properly, but the problem was with one of my modules.

            Regarding symptoms, the only issue I've seen that *might* be a/the cause (and is definitely repeatable) is the amount of data is too large to be handled completely in one pass.  I get a msg saying there's an issue regarding my XML data at position 8192 - that's where the data is truncated.
            [...]


          • Joe Tseng
            ...And the error came back, despite the fact I upped the length value to 99999 (and I haven t seen anything larger than that). I have a test system that NEVER
            Message 5 of 6 , Oct 7, 2013
            View Source
            • 0 Attachment
              ...And the error came back, despite the fact I upped the length value to 99999 (and I haven't seen anything larger than that).  I have a test system that NEVER errors out, even when running in parallel with prod, so I don't think the problem is with my code.  And I can't figure out what's the diff re: modules between my test and prod systems.

              Couple of observations:
              1. The people who run the feed I use tell me their data is updated once an hour.  Whenever I see a msg from my script saying I'm getting an error and I'm sitting at my console, I manually rerun my script after no more than 5 minutes and there's no problem, and I've done this numerous times.  I do plan on making my script poll repeatedly until successful, but I still want to know the source of the problem.  I did attempt to get more information by making my on_fault handler provide more details.  Turns out my error msg came from $soap->transport->status.  Are there any other details available from $soap that might prove useful?
              2. When I looked at my modules, the one immediate difference I found was that SOAP::Lite on my test system (v0.716) was different than that on my prod box (v0.714).  I updated both (v1.06) but the problem still persists.  What other dependencies might be needed to be updated as well?


              If you type "Google" into Google, you can break the Internet.  -- Jen Barber



              From: joe_tseng@...
              To: soaplite@yahoogroups.com
              Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies
              Date: Fri, 27 Sep 2013 10:42:20 -0400

              Looks like upping the value might have did the trick!  I just got a feed with an object of size 13178.  Thank you for the tip!

              (I say might because I'm still sporadically getting the "200 OK" errors, but if I manually trigger my job a minute later, it runs normally.  I suspect this time it's my feed provider; I'll be keeping an eye out on this for a while.)

               - Joe


              If you type "Google" into Google, you can break the Internet.  -- Jen Barber



              Date: Fri, 6 Sep 2013 08:20:25 -0700
              From: qglex@...
              Subject: Re: [soaplite] Figuring out SOAP::Lite dependencies
              To: joe_tseng@...

              If the length of the envelope/xml is truncating to 10000, take a look at

              $SOAP::Constants::MAX_CONTENT_SIZE

              By default it's set to 10,000.


               
              -Jeff


              From: Joe Tseng <joe_tseng@...>
              To: "soaplite@yahoogroups.com" <soaplite@yahoogroups.com>
              Sent: Wednesday, September 4, 2013 6:41 AM
              Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies

               
              Some additional details:

              The first symptom of this issue was when I saw an email that was sent from my SOAP object's on_fault handler; the error said "200 OK".  When I looked further into it, it turns out I was still indeed pulling down XML content from the feed, but for whatever reason was being truncated.  So yes, the remote system was still up and working properly, but the problem was with one of my modules.

              Regarding symptoms, the only issue I've seen that *might* be a/the cause (and is definitely repeatable) is the amount of data is too large to be handled completely in one pass.  I get a msg saying there's an issue regarding my XML data at position 8192 - that's where the data is truncated.
              [...]


            • Joe Tseng
              I think I found my problem!!! I wrote a short script to loop my SOAP call and then have the SOAP::Lite dump its contents when it used the on_fault handler.
              Message 6 of 6 , Oct 8, 2013
              View Source
              • 0 Attachment
                I think I found my problem!!!  I wrote a short script to loop my SOAP call and then have the SOAP::Lite dump its contents when it used the on_fault handler.  After perusing through the aftermath, I found the error that looked a lot like this:

                Missing newline after chunk data: <your data here> at /usr/share/perl5/Net/HTTP/Methods.pm line 481.

                http://bit.ly/1fgrj26 

                I've updated my Net::HTTP module via CPAN (v6.02 from Ubuntu v12.04/libnet-http-perl to v6.06) and restarted my daemon.  Hope this is it...


                If you type "Google" into Google, you can break the Internet.  -- Jen Barber



                From: joe_tseng@...
                To: soaplite@yahoogroups.com
                Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies
                Date: Mon, 7 Oct 2013 09:50:15 -0400

                ...And the error came back, despite the fact I upped the length value to 99999 (and I haven't seen anything larger than that).  I have a test system that NEVER errors out, even when running in parallel with prod, so I don't think the problem is with my code.  And I can't figure out what's the diff re: modules between my test and prod systems.

                Couple of observations:
                1. The people who run the feed I use tell me their data is updated once an hour.  Whenever I see a msg from my script saying I'm getting an error and I'm sitting at my console, I manually rerun my script after no more than 5 minutes and there's no problem, and I've done this numerous times.  I do plan on making my script poll repeatedly until successful, but I still want to know the source of the problem.  I did attempt to get more information by making my on_fault handler provide more details.  Turns out my error msg came from $soap->transport->status.  Are there any other details available from $soap that might prove useful?
                2. When I looked at my modules, the one immediate difference I found was that SOAP::Lite on my test system (v0.716) was different than that on my prod box (v0.714).  I updated both (v1.06) but the problem still persists.  What other dependencies might be needed to be updated as well?


                If you type "Google" into Google, you can break the Internet.  -- Jen Barber



                From: joe_tseng@...
                To: soaplite@yahoogroups.com
                Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies
                Date: Fri, 27 Sep 2013 10:42:20 -0400

                Looks like upping the value might have did the trick!  I just got a feed with an object of size 13178.  Thank you for the tip!

                (I say might because I'm still sporadically getting the "200 OK" errors, but if I manually trigger my job a minute later, it runs normally.  I suspect this time it's my feed provider; I'll be keeping an eye out on this for a while.)

                 - Joe


                If you type "Google" into Google, you can break the Internet.  -- Jen Barber



                Date: Fri, 6 Sep 2013 08:20:25 -0700
                From: qglex@...
                Subject: Re: [soaplite] Figuring out SOAP::Lite dependencies
                To: joe_tseng@...

                If the length of the envelope/xml is truncating to 10000, take a look at

                $SOAP::Constants::MAX_CONTENT_SIZE

                By default it's set to 10,000.


                 
                -Jeff


                From: Joe Tseng <joe_tseng@...>
                To: "soaplite@yahoogroups.com" <soaplite@yahoogroups.com>
                Sent: Wednesday, September 4, 2013 6:41 AM
                Subject: RE: [soaplite] Figuring out SOAP::Lite dependencies

                 
                Some additional details:

                The first symptom of this issue was when I saw an email that was sent from my SOAP object's on_fault handler; the error said "200 OK".  When I looked further into it, it turns out I was still indeed pulling down XML content from the feed, but for whatever reason was being truncated.  So yes, the remote system was still up and working properly, but the problem was with one of my modules.

                Regarding symptoms, the only issue I've seen that *might* be a/the cause (and is definitely repeatable) is the amount of data is too large to be handled completely in one pass.  I get a msg saying there's an issue regarding my XML data at position 8192 - that's where the data is truncated.
                [...]


              Your message has been successfully submitted and would be delivered to recipients shortly.