Loading ...
Sorry, an error occurred while loading the content.

Frontier performance and page faults

Expand Messages
  • David Gewirtz
    I ve been trying to troubleshoot some recent Frontier performance problems, after noticing that the Task Manager shows CPU usage in the 50%+ range and page
    Message 1 of 16 , Mar 2, 2009
    View Source
    • 0 Attachment
      I've been trying to troubleshoot some recent Frontier performance
      problems, after noticing that the Task Manager shows CPU usage in the
      50%+ range and page loads slowing down considerably.

      I set up the Performance MMC and compared CPU usage and page faults
      per second. I was surprised to see that Frontier is pegging the chart
      very regularly with page faults per second.

      The machine it's on has 2GB RAM, not much else running except MySQL
      (which is not page faulting), and is otherwise relatively unchanged
      from what I've run before.

      Anyone else see this page fault issue or figured out how to deal with
      it? Is it even an issue, or is it symptomatic of something else I
      haven't found yet?

      Thanks,
      -- David
    • André Radke
      ... If this is a dual processor machine and if Frontier is consistently pegged at 50% CPU utilization, it probably got stuck in a very tight (possibly
      Message 2 of 16 , Mar 3, 2009
      View Source
      • 0 Attachment
        On 3 Mar 2009, at 02:53, David Gewirtz wrote:

        > I've been trying to troubleshoot some recent Frontier performance
        > problems, after noticing that the Task Manager shows CPU usage in the
        > 50%+ range and page loads slowing down considerably.

        If this is a dual processor machine and if Frontier is consistently
        pegged at 50% CPU utilization, it probably got stuck in a very tight
        (possibly infinite) loop. If you haven't tried, it might be helpful to
        run thread.getStackDump when this occurs.

        http://docserver.userland.com/thread/getStackDump

        HTH,

        André



        --
        André Radke ++ http://spicynoodles.net/
      • David Gewirtz
        Andre, although the msg box says it s got 27 thread, the stackdump only reported two threads: thread and thread.getStackdump, both listed as (1). I m thinking
        Message 3 of 16 , Mar 3, 2009
        View Source
        • 0 Attachment
          Andre, although the msg box says it's got 27 thread, the stackdump
          only reported two threads: thread and thread.getStackdump, both listed
          as (1).

          I'm thinking that perhaps the stackdump isn't really displaying
          everything?

          Oh, well... I'm going to poke around with this and getStats and see if
          I can find anything. Is there any way to tell load (beyond
          threadcount) within Frontier?

          -- David

          P.S. I sent a private message to your spicy email address last night.
          Could you check that you got it, please?

          --- In frontierkernel@yahoogroups.com, André Radke <lists@...> wrote:
          >
          >
          > On 3 Mar 2009, at 02:53, David Gewirtz wrote:
          >
          > > I've been trying to troubleshoot some recent Frontier performance
          > > problems, after noticing that the Task Manager shows CPU usage in the
          > > 50%+ range and page loads slowing down considerably.
          >
          > If this is a dual processor machine and if Frontier is consistently
          > pegged at 50% CPU utilization, it probably got stuck in a very tight
          > (possibly infinite) loop. If you haven't tried, it might be helpful to
          > run thread.getStackDump when this occurs.
          >
          > http://docserver.userland.com/thread/getStackDump
          >
          > HTH,
          >
          > André
          >
          >
          >
          > --
          > André Radke ++ http://spicynoodles.net/
          >
        • André Radke
          ... Apparently, thread.getStackDump only reports a stack trace for the current thread, i.e. the one that s executing the verb. On the other hand,
          Message 4 of 16 , Mar 4, 2009
          View Source
          • 0 Attachment
            On 3 Mar 2009, at 21:35, David Gewirtz wrote:

            > Andre, although the msg box says it's got 27 thread, the stackdump
            > only reported two threads: thread and thread.getStackdump, both listed
            > as (1).
            >
            > I'm thinking that perhaps the stackdump isn't really displaying
            > everything?

            Apparently, thread.getStackDump only reports a stack trace for the
            current thread, i.e. the one that's executing the verb.

            On the other hand, thread.getStats might still be helpful as it
            reports information about all currently running threads.

            -André




            --
            André Radke ++ http://spicynoodles.net/
          • Steve Hooker
            David baby ;-) Try this http://www.blogfootball.com/service/2006/06/27/Thread-analyser It ll help a little. I run it when there s just too many threads going
            Message 5 of 16 , Mar 4, 2009
            View Source
            • 0 Attachment
              David baby ;-)
              Try this
              http://www.blogfootball.com/service/2006/06/27/Thread-analyser

              It'll help a little. I run it when there's just too many threads going on or Frontier is spinning it's wheels.
              Steve

              David Gewirtz wrote:

              Andre, although the msg box says it's got 27 thread, the stackdump
              only reported two threads: thread and thread.getStackdump , both listed
              as (1).

              I'm thinking that perhaps the stackdump isn't really displaying
              everything?

              Oh, well... I'm going to poke around with this and getStats and see if
              I can find anything. Is there any way to tell load (beyond
              threadcount) within Frontier?

            • David Gewirtz
              I ll give it a shot. What interesting things have you discovered when running it?
              Message 6 of 16 , Mar 4, 2009
              View Source
              • 0 Attachment
                I'll give it a shot. What interesting things have you discovered when running it?

                --- In frontierkernel@yahoogroups.com, Steve Hooker <steve@...> wrote:
                >
                > David baby ;-)
                > Try this
                > http://www.blogfootball.com/service/2006/06/27/Thread-analyser
                >
                > It'll help a little. I run it when there's just too many threads going
                > on or Frontier is spinning it's wheels.
                > Steve
              • Steve Hooker
                That s a good question. Not a lot, really. I thought I d uncover the mysteries of the universe :-) I run a Manila server. I found that getting lots of emails
                Message 7 of 16 , Mar 5, 2009
                View Source
                • 0 Attachment
                  That's a good question. Not a lot, really. I thought I'd uncover the mysteries of the universe :-)

                  I run a Manila server. I found that getting lots of emails on a slow machine can slow everything else down. Which was stating the obvious. But occasionally, it has helped me find a script that was going nuts, which was usually my fault.

                  I've recently moved to a fast quad core, with lots of fast ram. And I don't get the problems I used to. Which, again is stating the bleedin' obvious. So for the past few months, I've not needed this threadAnalyzer.

                  Steve

                  David Gewirtz wrote:

                  I'll give it a shot. What interesting things have you discovered when running it?

                • David Gewirtz
                  Did you ever see problems managing XML-RPC requests. I m thinking that something in there is wonky.
                  Message 8 of 16 , Mar 5, 2009
                  View Source
                  • 0 Attachment
                    Did you ever see problems managing XML-RPC requests. I'm thinking that something in there is wonky.

                    --- In frontierkernel@yahoogroups.com, Steve Hooker <steve@...> wrote:
                    >
                    > That's a good question. Not a lot, really. I thought I'd uncover the
                    > mysteries of the universe :-)
                    >
                    > I run a Manila server. I found that getting lots of emails on a slow
                    > machine can slow everything else down. Which was stating the obvious.
                    > But occasionally, it has helped me find a script that was going nuts,
                    > which was usually my fault.
                    >
                    > I've recently moved to a fast quad core, with lots of fast ram. And I
                    > don't get the problems I used to. Which, again is stating the bleedin'
                    > obvious. So for the past few months, I've not needed this threadAnalyzer.
                    >
                    > Steve
                    >
                    > David Gewirtz wrote:
                    > >
                    > > I'll give it a shot. What interesting things have you discovered when
                    > > running it?
                    > >
                    > > _,_._,___ <!-- #ygrp-mkp{ border: 1px solid #d8d8d8; font-family:
                    > > Arial; margin: 14px 0px; padding: 0px 14px; } #ygrp-mkp hr{ border:
                    > > 1px solid #d8d8d8; } #ygrp-mkp #hd{ color: #628c2a; font-size: 85%;
                    > > font-weight: bold; line-height: 122%; margin: 10px 0px; } #ygrp-mkp
                    > > #ads{ margin-bottom: 10px; } #ygrp-mkp .ad{ padding: 0 0; } #ygrp-mkp
                    > > .ad a{ color: #0000ff; text-decoration: none; } --> <!-- #ygrp-sponsor
                    > > #ygrp-lc{ font-family: Arial; } #ygrp-sponsor #ygrp-lc #hd{ margin:
                    > > 10px 0px; font-weight: bold; font-size: 78%; line-height: 122%; }
                    > > #ygrp-sponsor #ygrp-lc .ad{ margin-bottom: 10px; padding: 0 0; } -->
                    > > <!-- #ygrp-mlmsg {font-size:13px; font-family:
                    > > arial,helvetica,clean,sans-serif;*font-size:small;*font:x-small;}
                    > > #ygrp-mlmsg table {font-size:inherit;font:100%;} #ygrp-mlmsg select,
                    > > input, textarea {font:99% arial,helvetica,clean,sans-serif;}
                    > > #ygrp-mlmsg pre, code {font:115% monospace;*font-size:100%;}
                    > > #ygrp-mlmsg * {line-height:1.22em;} #ygrp-text{ font-family: Georgia;
                    > > } #ygrp-text p{ margin: 0 0 1em 0; } dd.last p a { font-family:
                    > > Verdana; font-weight: bold; } #ygrp-vitnav{ padding-top: 10px;
                    > > font-family: Verdana; font-size: 77%; margin: 0; } #ygrp-vitnav a{
                    > > padding: 0 1px; } #ygrp-mlmsg #logo{ padding-bottom: 10px; }
                    > > #ygrp-reco { margin-bottom: 20px; padding: 0px; } #ygrp-reco
                    > > #reco-head { font-weight: bold; color: #ff7900; } #reco-category{
                    > > font-size: 77%; } #reco-desc{ font-size: 77%; } #ygrp-vital a{
                    > > text-decoration: none; } #ygrp-vital a:hover{ text-decoration:
                    > > underline; } #ygrp-sponsor #ov ul{ padding: 0 0 0 8px; margin: 0; }
                    > > #ygrp-sponsor #ov li{ list-style-type: square; padding: 6px 0;
                    > > font-size: 77%; } #ygrp-sponsor #ov li a{ text-decoration: none;
                    > > font-size: 130%; } #ygrp-sponsor #nc{ background-color: #eee;
                    > > margin-bottom: 20px; padding: 0 8px; } #ygrp-sponsor .ad{ padding: 8px
                    > > 0; } #ygrp-sponsor .ad #hd1{ font-family: Arial; font-weight: bold;
                    > > color: #628c2a; font-size: 100%; line-height: 122%; } #ygrp-sponsor
                    > > .ad a{ text-decoration: none; } #ygrp-sponsor .ad a:hover{
                    > > text-decoration: underline; } #ygrp-sponsor .ad p{ margin: 0; }
                    > > o{font-size: 0; } .MsoNormal{ margin: 0 0 0 0; } #ygrp-text tt{
                    > > font-size: 120%; } blockquote{margin: 0 0 0 4px;} .replbq{margin:4}
                    > > dd.last p span { margin-right: 10px; font-family: Verdana;
                    > > font-weight: bold; } dd.last p span.yshortcuts { margin-right: 0; }
                    > > div.photo-title a, div.photo-title a:active, div.photo-title a:hover,
                    > > div.photo-title a:visited { text-decoration: none; } div.file-title a,
                    > > div.file-title a:active, div.file-title a:hover, div.file-title
                    > > a:visited { text-decoration: none; } #ygrp-msg p { clear: both;
                    > > padding: 15px 0 3px 0; overflow: hidden; } #ygrp-msg p span { color:
                    > > #1E66AE; font-weight: bold; } div#ygrp-mlmsg #ygrp-msg p a
                    > > span.yshortcuts { font-family: Verdana; font-size: 10px; font-weight:
                    > > normal; } #ygrp-msg p a { font-family: Verdana; font-size: 10px; }
                    > > #ygrp-mlmsg a { color: #1E66AE; } div.attach-table div div a {
                    > > text-decoration: none; } div.attach-table { width: 400px; } -->
                    >
                  • Steve Hooker
                    I don t do much XML-RPC. Hope you ve done a save as on each database. Looked through each root for big tables. All the usual things. A good test to find
                    Message 9 of 16 , Mar 6, 2009
                    View Source
                    • 0 Attachment
                      I don't do much XML-RPC.
                      Hope you've done a save as on each database. Looked through each root for big tables. All the usual things.

                      A good test to find corruption is to use the search to look for some made up word "swhujbkj56" perhaps. Do this in each root, if it can't find it all is well. If an error pops up, all is not well. The error IIRC give some clue to the location. At least this found some corruption in one script (in a glossary) that otherwise I'd have not found and was creating some weird behaviour that took weeks to pin down.

                      HTH

                      David Gewirtz wrote:

                      Did you ever see problems managing XML-RPC requests. I'm thinking that something in there is wonky.

                    • David Gewirtz
                      Hmmm... I always do Save As, and sadly, I also have some ginormous tables (that s actually why I built the MySQL extensions, but I m still coding to move
                      Message 10 of 16 , Mar 6, 2009
                      View Source
                      • 0 Attachment
                        Hmmm... I always do Save As, and sadly, I also have some ginormous tables (that's actually why I built the MySQL extensions, but I'm still coding to move everything over).

                        But I'm really curious about your search idea. I've found that searches tend to crash Frontier, so I wrote my own search routine. But do you contend that a failed/crashed search indicates database corruption that might not pass through a Save As for a database?

                        -- David

                        --- In frontierkernel@yahoogroups.com, Steve Hooker <steve@...> wrote:
                        >
                        > I don't do much XML-RPC.
                        > Hope you've done a save as on each database. Looked through each root
                        > for big tables. All the usual things.
                        >
                        > A good test to find corruption is to use the search to look for some
                        > made up word "swhujbkj56" perhaps. Do this in each root, if it can't
                        > find it all is well. If an error pops up, all is not well. The error
                        > IIRC give some clue to the location. At least this found some corruption
                        > in one script (in a glossary) that otherwise I'd have not found and was
                        > creating some weird behaviour that took weeks to pin down.
                        >
                        > HTH
                      • Steve Hooker
                        I do a save as every night. But this corruption in one script passed that. It even allowed me to export. Thus, I thought everything was fine. Only the search
                        Message 11 of 16 , Mar 7, 2009
                        View Source
                        • 0 Attachment
                          I do a save as every night. But this corruption in one script passed that. It even allowed me to export. Thus, I thought everything was fine. Only the search gave me the clue. Once I found the errant script, I couldn't open it IIRC. Every time I tried to run this errant script, I crashed. I deleted it and re wrote the script. The search passed through fine and all was well once more.
                          So, yes, I'd guess if the normal Frontier search is giving errors, there may indeed be problems with your root.

                          I too have large, many thousands of tables/items, and it is these that usually cause the corruption. I keep my eye on them.
                          Steve


                          David Gewirtz wrote:

                          Hmmm... I always do Save As, and sadly, I also have some ginormous tables (that's actually why I built the MySQL extensions, but I'm still coding to move everything over).

                          But I'm really curious about your search idea. I've found that searches tend to crash Frontier, so I wrote my own search routine. But do you contend that a failed/crashed search indicates database corruption that might not pass through a Save As for a database?

                        • David Gewirtz
                          When you did your search, was it that you were searching for the corrupted item? Or did you figure out a way to find out where it was broken through your
                          Message 12 of 16 , Mar 7, 2009
                          View Source
                          • 0 Attachment
                            When you did your search, was it that you were searching for the corrupted item? Or did you figure out a way to find out where it was broken through your search process, say working your way down the tree?

                            I'm figuring this is a good vector for me to look, but anything that'd make it quicker given how many nodes we're talking about would help. And thanks, your feedback here is enormously helpful.

                            -- David

                            --- In frontierkernel@yahoogroups.com, Steve Hooker <steve@...> wrote:
                            >
                            > I do a save as every night. But this corruption in one script passed
                            > that. It even allowed me to export. Thus, I thought everything was fine.
                            > Only the search gave me the clue. Once I found the errant script, I
                            > couldn't open it IIRC. Every time I tried to run this errant script, I
                            > crashed. I deleted it and re wrote the script. The search passed through
                            > fine and all was well once more.
                            > So, yes, I'd guess if the normal Frontier search is giving errors, there
                            > may indeed be problems with your root.
                            >
                            > I too have large, many thousands of tables/items, and it is these that
                            > usually cause the corruption. I keep my eye on them.
                            > Steve
                            >
                          • Steve Hooker
                            Hi David, It was some months ago. I think I was searching through a root for something real, and found this reported error and be come interested... I merely
                            Message 13 of 16 , Mar 8, 2009
                            View Source
                            • 0 Attachment
                              Hi David,
                              It was some months ago. I think I was searching through a root for something real, and found this reported error and be come interested...

                              I merely use the search tool, command F with the cursor at the top item in the root, nothing special nor clever. The roots were mainly Manila roots with several Manila sites at the top levels. I then tried this with all my roots, later. I did it by hand, too.

                              I can't remember how I found the corrupted script (it was an item in a large glossary). IIRC there was some clues in the error that was reported. Perhaps I suspected the glossary... I can't quite remember, and it was only this once that I found this corruption. So, I can't repeat.

                              I then started searching through other roots for made up strings "12H98YHN 489" and such. A clear pass, where the string wasn't found meant there was nothing wrong. Only in this one root, in this one place was there any reported error.

                              I wonder if there are other obscure methods for finding corruption? Anyone??
                              Steve

                              David Gewirtz wrote:

                              When you did your search, was it that you were searching for the corrupted item? Or did you figure out a way to find out where it was broken through your search process, say working your way down the tree?

                              I'm figuring this is a good vector for me to look, but anything that'd make it quicker given how many nodes we're talking about would help. And thanks, your feedback here is enormously helpful.

                            • David Gewirtz
                              Well, I haven t found any fails searching the files, but I did notice that some improvement can be had by reducing table size (yeah, that s known) and also
                              Message 14 of 16 , Mar 12, 2009
                              View Source
                              • 0 Attachment
                                Well, I haven't found any fails searching the files, but I did notice that some improvement can be had by reducing table size (yeah, that's known) and also reducing WP object size. I had a couple of huge, regularly updated WP objects and once I got rid of those, it seemed to speed up a bit. Still problematic, but I still have hundreds of big tables and WP objects to ferret out and recode.

                                Thanks for the input.

                                -- David

                                --- In frontierkernel@yahoogroups.com, Steve Hooker <steve@...> wrote:
                                >
                                > Hi David,
                                > It was some months ago. I think I was searching through a root for
                                > something real, and found this reported error and be come interested...
                                >
                                > I merely use the search tool, command F with the cursor at the top item
                                > in the root, nothing special nor clever. The roots were mainly Manila
                                > roots with several Manila sites at the top levels. I then tried this
                                > with all my roots, later. I did it by hand, too.
                                >
                                > I can't remember how I found the corrupted script (it was an item in a
                                > large glossary). IIRC there was some clues in the error that was
                                > reported. Perhaps I suspected the glossary... I can't quite remember,
                                > and it was only this once that I found this corruption. So, I can't repeat.
                                >
                                > I then started searching through other roots for made up strings
                                > "12H98YHN 489" and such. A clear pass, where the string wasn't found
                                > meant there was nothing wrong. Only in this one root, in this one place
                                > was there any reported error.
                                >
                                > I wonder if there are other obscure methods for finding corruption? Anyone??
                                > Steve
                                >
                                > David Gewirtz wrote:
                                > >
                                > > When you did your search, was it that you were searching for the
                                > > corrupted item? Or did you figure out a way to find out where it was
                                > > broken through your search process, say working your way down the tree?
                                > >
                                > > I'm figuring this is a good vector for me to look, but anything that'd
                                > > make it quicker given how many nodes we're talking about would help.
                                > > And thanks, your feedback here is enormously helpful.
                                > >
                                >
                              • Steve Hooker
                                I regularly download emails with attachments, sometimes they re 10Mb each---ouch. I write them as strings to a separate database, one that I can throw away
                                Message 15 of 16 , Mar 15, 2009
                                View Source
                                • 0 Attachment
                                  I regularly download emails with attachments, sometimes they're 10Mb each—ouch. I write them as strings to a separate database, one that I can throw away after a while. I know big 'anything' is bad, and try my best to avoid them. But, sometimes, there's no getting away from them.
                                  Good luck :-)

                                  David Gewirtz wrote:

                                  Well, I haven't found any fails searching the files, but I did notice that some improvement can be had by reducing table size (yeah, that's known) and also reducing WP object size. I had a couple of huge, regularly updated WP objects and once I got rid of those, it seemed to speed up a bit. Still problematic, but I still have hundreds of big tables and WP objects to ferret out and recode.

                                  Thanks for the input.

                                • u074036
                                  ... In my case, I ve determined that 100% of my search crashes Frontier cases are caused by the search routine finding the match string inside a wpTextType
                                  Message 16 of 16 , Dec 6, 2010
                                  View Source
                                  • 0 Attachment
                                    --- In frontierkernel@yahoogroups.com, "David Gewirtz" <david@...> wrote:

                                    > But I'm really curious about your search idea. I've found that
                                    > searches tend to crash Frontier, so I wrote my own search routine.
                                    > But do you contend that a failed/crashed search indicates database
                                    > corruption that might not pass through a Save As for a database?

                                    In my case, I've determined that 100% of my "search crashes Frontier" cases are caused by the search routine finding the match string inside a wpTextType cell. I've taken a quick look at the associated source code, but (like so much of Frontier's source code) is clearly way over my head.
                                  Your message has been successfully submitted and would be delivered to recipients shortly.