Loading ...
Sorry, an error occurred while loading the content.

Re: [webalizer] Old reports are (sometimes) flushed

Expand Messages
  • Bradford L. Barrett
    ... As do I.. actually, I run it twice a day, but _not_ in incremental mode. Data loss will occur otherwise, and the amount will be determined by the amount of
    Message 1 of 19 , Feb 6, 2003
    • 0 Attachment
      > > [Using Incremental mode on live logs]
      > > And you won't notice any 'trouble'.. the program was designed to be run
      > > once a month against a full months worth of logs.. _only_ if you need to
      > > rotate more than once a month should you use Incremental mode, and then,
      > > only against the _rotated_ log file.
      >
      > If I had to use it that way, it's complete useless for me. Our
      > customers don't want info once a month. For our production machines
      > we use a daily run around midnight.

      As do I.. actually, I run it twice a day, but _not_ in incremental mode.
      Data loss will occur otherwise, and the amount will be determined by the
      amount of traffic the individual site generates.. small sites may not
      experience any.. large sites will experience a lot.

      > There is no way to give accurate data. Whether you use log files
      > or a javascript utily on the site.

      Huh?

      > And I don't believe I'm having 5-10% data loss during the 12-15 minutes
      > webalizer runs each day.

      Probably not.. depends on size of the site. The problem is not in the
      length of time to process, but the handling of timestamps in incremental
      mode. And you will not notice the loss unless you compare the results
      to those run the correct way. I've been doing this since 1997, and have
      done test results with hundreds of sites ranging in size from small
      mom and pop shops to huge porn sites with many millions of hits a day.
      Offically, you should never, ever, run against live logs when you are
      using incremental mode.. to do so means you run the risk of producing
      inaccurate results, with a greater risk the larger the site is. If you
      run in incremental mode, you should _only_ process rotated logs.. If
      you want to run against live logs, then disable incremental mode and
      run against them as much as you like.

      > > If you want to run against live logs, disable incremental mode.
      >
      > Thanks for the tip, but that will make it completely useless
      > for me and our customers.

      I fail to see why.. thosands of ISPs do it every day without a problem.

      --
      Bradford L. Barrett brad@...
      A free electron in a sea of neutrons DoD#1750 KD4NAW

      The only thing Micro$oft has done for society, is make people
      believe that computers are inherently unreliable.
    • Kees de Keizer
      ... Sorry, but if I had to do it that way, my servers would only be busy with generating stats. Maybe unix machine can handle multiple instances of webalizer
      Message 2 of 19 , Feb 6, 2003
      • 0 Attachment
        Bradford L. Barrett wrote on 06-02-2003:

        > [Using Incremental mode on live logs]
        > > > .. the program was designed to be run once a month against
        > > > a full months worth of logs.. _only_ if you need to rotate
        > > > more than once a month should you use Incremental mode, and
        > > > then, only against the _rotated_ log file.
        > > If I had to use it that way, it's complete useless for me. Our
        > > customers don't want info once a month. For our production machines
        > > we use a daily run around midnight.
        > As do I.. actually, I run it twice a day, but _not_ in incremental mode.
        > Data loss will occur otherwise, and the amount will be determined by the
        > amount of traffic the individual site generates.. small sites may not
        > experience any.. large sites will experience a lot.

        Sorry, but if I had to do it that way, my servers would only be
        busy with generating stats. Maybe unix machine can handle multiple
        instances of webalizer better but on a windows machine the impact
        of running webalizer is quite high. That's why I run it around
        mignight on our production machines.

        > > There is no way to give accurate data. Whether you use log files
        > > or a javascript utily on the site.
        > Huh?

        In this case I was talking about actual site visitors. I.E. proxies
        do not generate a a log entry when they present the site to a visitor
        using that proxy.

        > If you run in incremental mode, you should _only_ process rotated
        > logs.. If you want to run against live logs, then disable incremental
        > mode and run against them as much as you like.

        Ask Bill if he can generate rotated logfiles with a different
        extension or any other way so they can be easily selected with
        a wildcard. Since this is not the case you have to run it against
        a live log file.

        > > > If you want to run against live logs, disable incremental mode.
        > > Thanks for the tip, but that will make it completely useless
        > > for me and our customers.
        > I fail to see why.. thosands of ISPs do it every day without a
        > problem.

        I never said I have a problem. You try to talk me into one.
        --
        Kees de Keizer NT Administrator T: +31-10-2448344
        #easynet Nederland PGP: 0x24E3770B F: +31-10-2448356
        http://www.easynet.nl/ kees.de.keizer@...
        Lovers are very special people [Funny Girl]
      • Bradford L. Barrett
        ... The limitations of your platform of choice is unfortunate.. however, the implication that it s ok to run against live logs when using incremental mode is
        Message 3 of 19 , Feb 6, 2003
        • 0 Attachment
          > Sorry, but if I had to do it that way, my servers would only be
          > busy with generating stats. Maybe unix machine can handle multiple
          > instances of webalizer better but on a windows machine the impact
          > of running webalizer is quite high. That's why I run it around
          > mignight on our production machines.

          The limitations of your platform of choice is unfortunate.. however,
          the implication that it's ok to run against live logs when using
          incremental mode is inaccurate, regardless of any justifications
          you can make.

          > > > There is no way to give accurate data. Whether you use log files
          > > > or a javascript utily on the site.
          > > Huh?
          >
          > In this case I was talking about actual site visitors. I.E. proxies
          > do not generate a a log entry when they present the site to a visitor
          > using that proxy.

          The webalizer was designed to give administrators insight into the usage
          of their servers, in order to provide insight into capacity planning and
          future direction. If a request is satisified from a proxy instead of
          hitting the server, that does not impact the analysis of server
          performance or usage from that standpoint. The program will provide
          accurate analysis of actual usage in every way with the exception of
          visit/entry/exit page analysis (as described in the documentation).

          > > If you run in incremental mode, you should _only_ process rotated
          > > logs.. If you want to run against live logs, then disable incremental
          > > mode and run against them as much as you like.
          >
          > Ask Bill if he can generate rotated logfiles with a different
          > extension or any other way so they can be easily selected with
          > a wildcard. Since this is not the case you have to run it against
          > a live log file.

          Again, the limitations of your platform of choice is not justification
          for giving people the incorrect assumption that it's ok to run against
          live logs in incremental mode. To do so is a disservice to those users.

          > > > > If you want to run against live logs, disable incremental mode.
          > > > Thanks for the tip, but that will make it completely useless
          > > > for me and our customers.
          > > I fail to see why.. thosands of ISPs do it every day without a
          > > problem.
          >
          > I never said I have a problem. You try to talk me into one.

          I took your statement to mean that it is impossible for you to use
          the program correctly (ie: to do so would "make it completely useless"),
          which I would view as a problem. Apparently, you don't think so.

          --
          Bradford L. Barrett brad@...
          A free electron in a sea of neutrons DoD#1750 KD4NAW

          How do you give Microsoft the benefit of the doubt when you
          know that if you were to throw it in a room with truth, you'd
          risk a matter/anti-matter explosion? -- Nicholas Petreley IDG
        • Kees de Keizer
          ... But not my choice. We have customers who want to write their websites in BASIC. ... If I want to know the traffic to a server, I ll ask our networking
          Message 4 of 19 , Feb 6, 2003
          • 0 Attachment
            On Thu, 6 Feb 2003, Bradford L. Barrett wrote:

            > > Sorry, but if I had to do it that way, my servers would only be
            > > busy with generating stats. Maybe unix machine can handle multiple
            > > instances of webalizer better but on a windows machine the impact
            > > of running webalizer is quite high. That's why I run it around
            > > mignight on our production machines.
            > The limitations of your platform of choice is unfortunate..

            But not my choice. We have customers who want to write their
            websites in BASIC.

            > > > > There is no way to give accurate data. Whether you use log files
            > > > > or a javascript utily on the site.
            > > > Huh?
            > > In this case I was talking about actual site visitors. I.E. proxies
            > > do not generate a a log entry when they present the site to a visitor
            > > using that proxy.
            > The webalizer was designed to give administrators insight into the usage
            > of their servers, in order to provide insight into capacity planning and
            > future direction. If a request is satisified from a proxy instead of
            > hitting the server, that does not impact the analysis of server
            > performance or usage from that standpoint. The program will provide
            > accurate analysis of actual usage in every way with the exception of
            > visit/entry/exit page analysis (as described in the documentation).

            If I want to know the traffic to a server, I'll ask our networking
            department for analysis. They can provide better traffic information
            that webalizer can, because they measure all the traffic. And nowadays
            webalizer is used very often to provide statistics to customers, so
            they can see when, what, how and with what their site has been visited.
            And they won't miss the 0,5% (if it is that much anyhow) in the stats.

            > Again, the limitations of your platform of choice is not justification
            > for giving people the incorrect assumption that it's ok to run against
            > live logs in incremental mode. To do so is a disservice to those users.

            I give our customers what they want: Site statistics. Our customers know
            that they are not 100% accurate if it concern page views and visits, due
            to the use of i.e. proxies. Webalizer is no longer a tool just for the
            administrators, but is more and more used for site statistics. Why would
            you else present referrers, user agents and countries? That information
            has nothing to do with analysis of the performance.

            > > I never said I have a problem. You try to talk me into one.
            > I took your statement to mean that it is impossible for you to use
            > the program correctly (ie: to do so would "make it completely useless"),
            > which I would view as a problem. Apparently, you don't think so.

            That's right. I give our customers what they want, and that
            is site statistics. They only complaint I'll get, is the lack
            of information about the used operating system. But that is
            something that won't be a part of webalizer.
            --
            Kees de Keizer NT Administrator T: +31-10-2448344
            #easynet Nederland PGP: 0x24E3770B F: +31-10-2448356
            http://www.easynet.nl/ kees.de.keizer@...
          • Bradford L. Barrett
            ... Because it s already there and it can be reported easily with very little additional overhead. But that is beside the point.. the point is that it is NOT
            Message 5 of 19 , Feb 6, 2003
            • 0 Attachment
              > I give our customers what they want: Site statistics. Our customers know
              > that they are not 100% accurate if it concern page views and visits, due
              > to the use of i.e. proxies. Webalizer is no longer a tool just for the
              > administrators, but is more and more used for site statistics. Why would
              > you else present referrers, user agents and countries? That information
              > has nothing to do with analysis of the performance.

              Because it's already there and it can be reported easily with very little
              additional overhead.

              But that is beside the point.. the point is that it is NOT ok to run
              against live logs when using incremental mode, regardless of your
              decision, for whatever reasons, to do so anyway.

              --
              Bradford L. Barrett brad@...
              A free electron in a sea of neutrons DoD#1750 KD4NAW

              The only thing Micro$oft has done for society, is make people
              believe that computers are inherently unreliable.
            • Kees de Keizer
              ... Maybe you should add that more clearly to the README file. The only about it now is something about preventing same timestamps . -- Kees de Keizer
              Message 6 of 19 , Feb 6, 2003
              • 0 Attachment
                On Thu, 6 Feb 2003, Bradford L. Barrett wrote:

                > the point is that it is NOT ok to run against live logs when
                > using incremental mode, regardless of your decision, for
                > whatever reasons, to do so anyway.

                Maybe you should add that more clearly to the README file. The
                only about it now is something about "preventing same timestamps".
                --
                Kees de Keizer NT Administrator T: +31-10-2448344
                #easynet Nederland PGP: 0x24E3770B F: +31-10-2448356
                http://www.easynet.nl/ kees.de.keizer@...
              • Josh Kuperman
                I m very confused about this. From my reading of the documentation, it looks like running against live logs should not be a problem as long as you have a
                Message 7 of 19 , Feb 7, 2003
                • 0 Attachment
                  I'm very confused about this.

                  From my reading of the documentation, it looks like running against
                  live logs should not be a problem as long as you have a history file
                  keeping copies of the log records so they don't get counted twice. I
                  am started to use webalizer quite a bit for keeping track of both
                  web and proxy (squid) use. In all cases I'm running against live
                  logs. I really need to know what happened yesterday - ideally I'd be
                  processing hourly.

                  So far it is still very unclear to me why the live logs are a
                  problem? I suppose I could force the logs to rotate every day, but I
                  still don't get it. A file is a file, and at least on Unix systems I
                  don't see what the difference is between a live log and an archived
                  log.


                  On Fri, Feb 07, 2003 at 07:17:08AM +0100, Kees de Keizer wrote:
                  > On Thu, 6 Feb 2003, Bradford L. Barrett wrote:
                  >
                  > > the point is that it is NOT ok to run against live logs when
                  > > using incremental mode, regardless of your decision, for
                  > > whatever reasons, to do so anyway.
                  >
                  > Maybe you should add that more clearly to the README file. The
                  > only about it now is something about "preventing same timestamps".

                  --
                  Josh Kuperman
                  josh@...
                • Bradford L. Barrett
                  ... You don t need incremental mode to run against logs hourly. ... One is being written to while processing, the other isn t. -- Bradford L. Barrett
                  Message 8 of 19 , Feb 7, 2003
                  • 0 Attachment
                    > I really need to know what happened yesterday - ideally I'd be
                    > processing hourly.

                    You don't need incremental mode to run against logs hourly.

                    > I don't see what the difference is between a live log and an archived
                    > log.

                    One is being written to while processing, the other isn't.

                    --
                    Bradford L. Barrett brad@...
                    A free electron in a sea of neutrons DoD#1750 KD4NAW

                    The only thing Micro$oft has done for society, is make people
                    believe that computers are inherently unreliable.
                  Your message has been successfully submitted and would be delivered to recipients shortly.