Loading ...
Sorry, an error occurred while loading the content.

Re: webalizer help

Expand Messages
  • Enric Naval
    ... http://www2.mstewart.net:8080/stats/usage_200404.html ... Hello: (Please next time post also your webalizer.conf, cutting thigs for privacy if you need to,
    Message 1 of 5 , Apr 16 10:10 AM
    View Source
    • 0 Attachment
      --- Michael Stewart <mstewartga@...> wrote:
      > I cant get the moderator of the yahoo group to
      > accept my membership.
      > I have been pending membership for a week.
      >
      > I have webalizer, recently installed, running on IIS
      > 6 processing 3
      > websites.
      >
      > For some reason, after each process I get only 1
      > days worth of info
      > and it deletes the previous days.
      >
      >
      http://www2.mstewart.net:8080/stats/usage_200404.html
      >
      > The history file always contains only one line and
      > the file is time
      > stamped at the time of processing.
      >
      > Any ideas?
      >
      > I appreciate your help.
      >
      > Regads,
      >
      > Michael Stewart
      >
      >
      >

      Hello:

      (Please next time post also your webalizer.conf,
      cutting thigs for privacy if you need to, so we can
      help better.)

      Let's me guess, you are rotating logs daily, and in
      your webalizer.conf you have a line saying:

      #Incremental no

      or perhaps:

      Incremental no

      (the default value is 'no').

      You are also of course processing only the logfile you
      have just rotated that day, not all past rotated logs.


      Because webalizer has not been told with "Incremental
      yes" to remember past log files then it will only
      display statistics about the single daily file it has
      just processed.

      OK, you have two solutions:

      solution 1:

      Put a line saying "Incremental yes" in your
      webalizer.conf. This will create a file called
      webalizer.current in the output directory.

      Then process all old rotated log files in order.
      You'll see how webalizer will add them to the
      statistics. If the incremental info gets corrupted,
      you'll get to delete webalizer.current and reprocess
      everything again (you can make it fastly with a
      script).

      #!/bin/bash
      for logfile in $(ls -r /var/log/httpd/access_log.*);
      do
      webalizer [some options] $logfile
      done

      You may also process all old files and then save the
      output directory, writing down which was the last file
      processed (or simply looking at what is the last day
      appearing in the stats). This way you can recover the
      directory and reprocess only from that day.

      NOTE: if you delete old logfiles, and incremental
      becomes corrupted, you'll lose the data!


      solution 2:

      cat all logfiles together in a single file and then
      add every night the rotated file to the end. Make
      webalizer re-process that file every night. Prepare to
      buy more RAM for webalizer so it doesn't resort to
      swap.




      To prevent problems:

      With incremental on, just let webalizer go on
      analysing. Before it process 1 Jan 2004, save a copy
      of the stats in a different folder, so you'll have a
      nice summary page for all 12 months of 2003. Do the
      same before it process 1 Jan 2004.



      Last: Your server is not doing reverse-resolving for
      visitors'IPs. The webalizer is not resolving either,
      that's why you have no countries stats. To solve this,
      uncoment the DNSChildren line, so it will attempt to
      resolve all IPs it finds (expect your DNS server to be
      hammered for good if you process lots of entries! You
      can limit the "children" number, "0" disables
      resolving, all sites are IPs and no countries are
      found. Of course "0" is the default value... Normally,
      people relies in Apache to resolve addresses).


      Buf, sorry for being so verbose. Hope I'm useful.

      =====
      Enric Naval
      Estudiante de Inform�tica de Gesti�n en la Udl (Lleida)




      __________________________________
      Do you Yahoo!?
      Yahoo! Tax Center - File online by April 15th
      http://taxes.yahoo.com/filing.html
    • Bradford L. Barrett
      [...] ... FYI: There is no difference in RAM usage between using one big log file or processing lots of smaller files using incremental mode. The same memory
      Message 2 of 5 , Apr 16 10:25 AM
      View Source
      • 0 Attachment
        [...]
        > solution 2:
        >
        > cat all logfiles together in a single file and then
        > add every night the rotated file to the end. Make
        > webalizer re-process that file every night. Prepare to
        > buy more RAM for webalizer so it doesn't resort to
        > swap.

        FYI: There is no difference in RAM usage between using one big log file
        or processing lots of smaller files using incremental mode. The same
        memory is consumed regardless of the method used.

        Also, Option #3: Just rotate your logs once a month. Process the log
        whenever you want to get an updated report, and once at the end of the
        month after rotation to get the final monthly report. This is the
        preferred and _recommended_ method to use.

        --
        Bradford L. Barrett brad@...
        A free electron in a sea of neutrons DoD#1750 KD4NAW

        The only thing Micro$oft has done for society, is make people
        believe that computers are inherently unreliable.
      • enventa2000
        Errr, you re using IIS, so you can t of course run the script I told you. Ejem. Try this: webalizer [your_options] c: rotated_logs access_log* If rotated logs
        Message 3 of 5 , Apr 16 10:32 AM
        View Source
        • 0 Attachment
          Errr, you're using IIS, so you can't of course run the script I told
          you. Ejem. Try this:

          webalizer [your_options] c:\rotated_logs\access_log*

          If rotated logs have proper names that get the files ordered correctly
          when listing, then webalizer will work well. For names like
          "access_log20040102" (a log rotated in 02/Jan/2004) it will work fine.

          If logs are entered not in order, then all entries should be lumped
          together in a single day (200000 visits in day x and 0 visits from day
          x to today, sometimes with days before day x appearing correctly,
          meaning that some entry from day x has appeared incorrectly after day
          x+1 has been processed)


          NOTE: Remember to delete everything in the output directory before
          trying what I told you, so the old history information is not used
          accidentally.



          Have a nice day.



          --- In webalizer@yahoogroups.com, Enric Naval <enventa2000@y...>
          wrote:
          > --- Michael Stewart <mstewartga@y...> wrote:
          > > I cant get the moderator of the yahoo group to
          > > accept my membership.
          > > I have been pending membership for a week.
          > >
          > > I have webalizer, recently installed, running on IIS
          > > 6 processing 3
          > > websites.
          > >
          > > For some reason, after each process I get only 1
          > > days worth of info
          > > and it deletes the previous days.
          > >
          > >
          > http://www2.mstewart.net:8080/stats/usage_200404.html
          > >
          > > The history file always contains only one line and
          > > the file is time
          > > stamped at the time of processing.
          > >
          > > Any ideas?
          > >
          > > I appreciate your help.
          > >
          > > Regads,
          > >
          > > Michael Stewart
          > >
          > >
          > >
          >
          > Hello:
          >
          > (Please next time post also your webalizer.conf,
          > cutting thigs for privacy if you need to, so we can
          > help better.)
          >
          > Let's me guess, you are rotating logs daily, and in
          > your webalizer.conf you have a line saying:
          >
          > #Incremental no
          >
          > or perhaps:
          >
          > Incremental no
          >
          > (the default value is 'no').
          >
          > You are also of course processing only the logfile you
          > have just rotated that day, not all past rotated logs.
          >
          >
          > Because webalizer has not been told with "Incremental
          > yes" to remember past log files then it will only
          > display statistics about the single daily file it has
          > just processed.
          >
          > OK, you have two solutions:
          >
          > solution 1:
          >
          > Put a line saying "Incremental yes" in your
          > webalizer.conf. This will create a file called
          > webalizer.current in the output directory.
          >
          > Then process all old rotated log files in order.
          > You'll see how webalizer will add them to the
          > statistics. If the incremental info gets corrupted,
          > you'll get to delete webalizer.current and reprocess
          > everything again (you can make it fastly with a
          > script).
          >
          > #!/bin/bash
          > for logfile in $(ls -r /var/log/httpd/access_log.*);
          > do
          > webalizer [some options] $logfile
          > done
          >
          > You may also process all old files and then save the
          > output directory, writing down which was the last file
          > processed (or simply looking at what is the last day
          > appearing in the stats). This way you can recover the
          > directory and reprocess only from that day.
          >
          > NOTE: if you delete old logfiles, and incremental
          > becomes corrupted, you'll lose the data!
          >
          >
          > solution 2:
          >
          > cat all logfiles together in a single file and then
          > add every night the rotated file to the end. Make
          > webalizer re-process that file every night. Prepare to
          > buy more RAM for webalizer so it doesn't resort to
          > swap.
          >
          >
          >
          >
          > To prevent problems:
          >
          > With incremental on, just let webalizer go on
          > analysing. Before it process 1 Jan 2004, save a copy
          > of the stats in a different folder, so you'll have a
          > nice summary page for all 12 months of 2003. Do the
          > same before it process 1 Jan 2004.
          >
          >
          >
          > Last: Your server is not doing reverse-resolving for
          > visitors'IPs. The webalizer is not resolving either,
          > that's why you have no countries stats. To solve this,
          > uncoment the DNSChildren line, so it will attempt to
          > resolve all IPs it finds (expect your DNS server to be
          > hammered for good if you process lots of entries! You
          > can limit the "children" number, "0" disables
          > resolving, all sites are IPs and no countries are
          > found. Of course "0" is the default value... Normally,
          > people relies in Apache to resolve addresses).
          >
          >
          > Buf, sorry for being so verbose. Hope I'm useful.
          >
          > =====
          > Enric Naval
          > Estudiante de Informática de Gestión en la Udl (Lleida)
          >
          >
          >
          >
          > __________________________________
          > Do you Yahoo!?
          > Yahoo! Tax Center - File online by April 15th
          > http://taxes.yahoo.com/filing.html
        • enventa2000
          Then webalizer only uses more RAM when you add more entries to the webalizer.conf file? Entries like SearchEngine xxx x, GroupSite/HideSite,
          Message 4 of 5 , Apr 16 10:45 AM
          View Source
          • 0 Attachment
            Then webalizer only uses more RAM when you add more entries to the
            webalizer.conf file? Entries like SearchEngine xxx x,
            GroupSite/HideSite, GroupAgent/HideAgent, etc?

            How much RAM would every extra entry add on average? 100 bytes? 64
            bytes/each plus overhead for hash tables?


            --- In webalizer@yahoogroups.com, "Bradford L. Barrett" <brad@m...>
            wrote:
            >
            > [...]
            > > solution 2:
            > >
            > > cat all logfiles together in a single file and then
            > > add every night the rotated file to the end. Make
            > > webalizer re-process that file every night. Prepare to
            > > buy more RAM for webalizer so it doesn't resort to
            > > swap.
            >
            > FYI: There is no difference in RAM usage between using one big log
            file
            > or processing lots of smaller files using incremental mode. The
            same
            > memory is consumed regardless of the method used.
            >
            > Also, Option #3: Just rotate your logs once a month. Process the
            log
            > whenever you want to get an updated report, and once at the end of
            the
            > month after rotation to get the final monthly report. This is the
            > preferred and _recommended_ method to use.
            >
            > --
            > Bradford L. Barrett brad@m...
            > A free electron in a sea of neutrons DoD#1750 KD4NAW
            >
            > The only thing Micro$oft has done for society, is make people
            > believe that computers are inherently unreliable.
          • Bradford L. Barrett
            ... The use of entries such as the above have a minimal effect on the amount of RAM used, but a huge impact on processing time since each entry must be
            Message 5 of 5 , Apr 16 12:09 PM
            View Source
            • 0 Attachment
              > Then webalizer only uses more RAM when you add more entries to the
              > webalizer.conf file? Entries like SearchEngine xxx x,
              > GroupSite/HideSite, GroupAgent/HideAgent, etc?

              The use of entries such as the above have a minimal effect on the amount
              of RAM used, but a huge impact on processing time since each entry must
              be compared against each log record. If you have lots and lots of
              PageType/IndexAlias/Hide/Group/Ignore/Include/SearchEngine entries, they
              will have a large impact on processing time since each one must be
              processed for each log record, and the impact is linear depending on the
              number of such entries.

              > How much RAM would every extra entry add on average? 100 bytes? 64
              > bytes/each plus overhead for hash tables?

              Depends. Some just keep the match string in a linked list, so it only
              takes however many bytes required for the string + linkage. Others
              will also create a hash table entry of whatever size required for the
              object being referenced. The size depends on the object, since each
              object maintains different metrics particular to the object.

              --
              Bradford L. Barrett brad@...
              A free electron in a sea of neutrons DoD#1750 KD4NAW

              The only thing Micro$oft has done for society, is make people
              believe that computers are inherently unreliable.
            Your message has been successfully submitted and would be delivered to recipients shortly.