Loading ...
Sorry, an error occurred while loading the content.
 

Re: [webalizer] Re: Getting wrong results while rotating logs

Expand Messages
  • Bradford L. Barrett
    ... [...] Right.. process the _rotated_ log. Your script will work (sometimes, see below) if you insist on using cronolog.. I prefer to rotate them myself
    Message 1 of 11 , Mar 20, 2004
      On Sun, 21 Mar 2004, enventa2000 wrote:

      > Then,use:
      >
      > #!/bin/bash
      > # this script is schudled for 00:00 in crontab
      [...]

      Right.. process the _rotated_ log. Your script will work (sometimes,
      see below) if you insist on using cronolog.. I prefer to rotate them
      myself however :) There are about a zillion different ways to do the same
      thing.

      [...]
      > # of course you won't have bash in any of your machines, so

      Huh? Doesn't _every_ *nix box out there have bash?!?

      > # by the way, is there an easier way to calculate n=n-1 in bash?

      Umm, how about "let n=n-1" ?? If you mean for caculating the previous
      days value, then just use "date -d yesterday <format-string>".. In
      your example:

      [...]
      > YEAR=$(date +%Y)
      > MONTH=$(date +%m)
      > DAY=$(date +%d)
      > YESTERDAY=$(($DAY - 1)) # n=n-1
      >
      > # # this is one line
      > # USER_LOGFILE="/path/to/users_logdir/$YEAR/$MONTH/$YESTERDAY/access_log"
      [...]

      Would fail on the first day of the month.. (ie: the day before
      january 1, 2004 in your example would use year=2004, month=01 and
      yesterday=0 instead of 2003, 12 and 31 like it should).

      > # how do you calculate it in sh?

      Dunno.. don't use it ;0

      > # what machines are you running webalizer on?

      Linux and solaris. Just an fyi, the primary development machine for
      the webalizer is a Sun ultra 30 that dual boots Solaris and Linux,
      and is where the main CVS tree is for the code.

      > # I don't use incremental mode. Every day I copy the logs (rotated
      > # or not) into /tmp, then I grep them and process them. This way
      > # any change in the cron script or the config files is
      > # propagated to all old stats pages (pity the program only
      > # runs one year, in six months more I will have to download the
      > # source and see if I can change that). The process is surpri-
      > # singly fast.

      Sounds like a lot of unnecessary work for a little gain! And as for
      the 'only runs one year', that is incorrect. The program will process
      as many days, months or years of logs that you can feed it. I have a
      test log file I use all the time that goes back 5 years.

      --
      Bradford L. Barrett brad@...
      A free electron in a sea of neutrons DoD#1750 KD4NAW

      How do you give Microsoft the benefit of the doubt when you
      know that if you were to throw it in a room with truth, you'd
      risk a matter/anti-matter explosion? -- Nicholas Petreley IDG
    • Enric Naval
      ... I d say at least one undecillion ways. I, for one, prefer making a script that will process and munch the data so that the program will find most of work
      Message 2 of 11 , Mar 20, 2004
        --- "Bradford L. Barrett" <brad@...> wrote:
        >
        > On Sun, 21 Mar 2004, enventa2000 wrote:
        >
        > > Then,use:
        > >
        > > #!/bin/bash
        > > # this script is schudled for 00:00 in crontab
        > [...]
        >
        > Right.. process the _rotated_ log. Your script will
        > work (sometimes,
        > see below) if you insist on using cronolog.. I
        > prefer to rotate them
        > myself however :) There are about a zillion
        > different ways to do the same
        > thing.
        >


        I'd say at least one undecillion ways. I, for one,
        prefer making a script that will process and munch
        the data so that the program will find most of work
        already done for it.

        > [...]
        > > # of course you won't have bash in any of your
        > machines, so
        >
        > Huh? Doesn't _every_ *nix box out there have
        > bash?!?
        >

        I don't know. But in the university we have a SunOs
        sparc box that lacks things like lynx, nmap, locate,
        and so on. Of course, no webalizer.

        Also, it doesn't have /etc/cron.daily :) Just
        /etc/cron.d and one script inside called "logchecker"
        with a 1997 copyright notice. And that script doesn't
        rotate the logs. It CUTS them when they are too long.
        Ugh. I believe they run the actual maintenance scripts
        by hand every week. Ugh again. Oh well, it's not MY
        server.


        > > # by the way, is there an easier way to calculate
        > n=n-1 in bash?
        >
        > Umm, how about "let n=n-1" ?? If you mean for
        > caculating the previous
        > days value, then just use "date -d yesterday
        > <format-string>".. In
        > your example:
        >
        > [...]
        > > YEAR=$(date +%Y)
        > > MONTH=$(date +%m)
        > > DAY=$(date +%d)
        > > YESTERDAY=$(($DAY - 1)) # n=n-1
        > >
        > > # # this is one line
        > > #
        >
        USER_LOGFILE="/path/to/users_logdir/$YEAR/$MONTH/$YESTERDAY/access_log"
        > [...]
        >
        > Would fail on the first day of the month.. (ie: the
        > day before
        > january 1, 2004 in your example would use year=2004,
        > month=01 and
        > yesterday=0 instead of 2003, 12 and 31 like it
        > should).


        Ouch! Novice mistake. Me silly. Okay, we won't run
        webalizer on day 1

        if [[ ! "$day" == "1" ]]; then
        # run webalizer;
        else
        # no stats for today
        fi;

        With a bit of luck, no customer will notice...

        :) ok, ok, I'll use:

        DATE=$( date -d yesterday +%Y%m%d )


        >
        > > # how do you calculate it in sh?
        >
        > Dunno.. don't use it ;0
        >

        I feel guilty for not being geek enough to be able to
        use it XD I believe that bash is an acronym for
        Bourne Again SHell (Born Again, the developer is
        called Bourne), and it was developed from sh, so there
        must be very old machines with just sh.

        I have always taken bash as a luxury, as the SunOs
        here logins you into sh... or maybe it is bash with an
        option set so it looks like sh. After all, there is
        also bash in the machine. I believe they keep the sh
        login to keep us awake and uncomfortable, and so that
        we don't take things like bash for granted.

        I have read a rant from one person who had many
        systems and complained that sh was the only shell that
        was available on most machines, and that there was
        subtle differences among shells. Yes, he had machines
        with neither sh nor bash! You know, csh, ksh, tcsh,
        etc. Ugh. And me here complaining about webalizer
        lacking features...


        > > # what machines are you running webalizer on?
        >
        > Linux and solaris. Just an fyi, the primary
        > development machine for
        > the webalizer is a Sun ultra 30 that dual boots
        > Solaris and Linux,
        > and is where the main CVS tree is for the code.
        >

        I use a Pentium IV with 1024 MG RAM and two 80 GB hard
        disks. Mainly it sits there all day long, with 235
        hits/hour in average (very ocasionally it gets 3550
        hits/hour). That's only 4 hits per minute!. Life is
        hard for this bored machine.

        If it wasn't for the SETI@home screensaver, the hard
        disk wouldn't even budge because everything is already
        in cache. The hard disk would only be needed for
        writing tiny entries to the database, rotating logs
        and writing webalizer stats.


        > > # I don't use incremental mode. Every day I copy
        > the logs (rotated
        > > # or not) into /tmp, then I grep them and
        > process them. This way
        > > # any change in the cron script or the config
        > files is
        > > # propagated to all old stats pages (pity the
        > program only
        > > # runs one year, in six months more I will have
        > to download the
        > > # source and see if I can change that). The
        > process is surpri-
        > > # singly fast.
        >
        > Sounds like a lot of unnecessary work for a little
        > gain!

        Yes, it is too much work! However, it is fast and
        anyways I'm still making changes to the config file
        and the cron script. If I used incremental mode, then
        the changes would only apply to the last data
        collected (that's it, the last 24 hours). That doesn't
        allow to see if the changes had a nice visual output
        or not, because the data would get fused with the
        month totals.

        Doing this, of course, if I screw the config, then all
        stats for all months and sites will be screwed also
        until I mend the config and run the program again or
        wait for cron to run it. That wouldn't be acceptable
        in a commercial site and I would have to process
        stable incremental stats for the public, and then make
        the non-incremental in a test folder. When the test
        looked nice, I would put the changes in the
        "commercial" processing.


        >And as for
        > the 'only runs one year', that is incorrect. The
        > program will process
        > as many days, months or years of logs that you can
        > feed it. I have a
        > test log file I use all the time that goes back 5
        > years.
        >

        I didn't know that. Then again, when the logfile
        reaches 2 GBs (like the one on the old server before
        it got deleted) then perhaps I'll have to archive some
        old stats and some old logs in order to keep them from
        becoming unmanageables. Meanwhile, that 5 years thing
        just made me happy :)

        I will also put the old logs in some place that the
        backup script doesn't reach. I want my gziped whole
        system backups to fit in just one DVD! I will keep
        separate backups for the gigantic logs.



        > --
        > Bradford L. Barrett
        > brad@...
        > A free electron in a sea of neutrons DoD#1750
        > KD4NAW
        >
        > How do you give Microsoft the benefit of the doubt
        > when you
        > know that if you were to throw it in a room with
        > truth, you'd
        > risk a matter/anti-matter explosion? -- Nicholas
        > Petreley IDG


        =====
        Enric Naval
        Estudiante de Inform�tica de Gesti�n en la Udl (Lleida)

        __________________________________
        Do you Yahoo!?
        Yahoo! Finance Tax Center - File online. File on time.
        http://taxes.yahoo.com/filing.html
      • waldo kitty
        ... i think the point here is that one can only see one year s worth of data in the reports... the initial page only shows one years worth... some would like
        Message 3 of 11 , Mar 21, 2004
          Bradford L. Barrett wrote:

          > Sounds like a lot of unnecessary work for a little gain! And as for
          > the 'only runs one year', that is incorrect. The program will process
          > as many days, months or years of logs that you can feed it. I have a
          > test log file I use all the time that goes back 5 years.

          i think the point here is that one can only see one year's worth of data in the reports... the initial page only shows one years
          worth... some would like to be able to compare this march with last march in the overall view... i actually have all the process
          reports since dec 1997 for my stuff... i've thought about doing something where i can load the graphic files for a side by side
          comparison of the months over the years... hummm... sounds like a little something for me to toss together today... manually, of
          course... at least for now ;)

          --
          _\/
          (@@) Waldo Kitty, Waldo's Place USA
          __ooO_( )_Ooo_____________________ telnet://bbs.wpusa.dynip.com
          _|_____|_____|_____|_____|_____|_____ http://www.wpusa.dynip.com
          ____|_____|_____|_____|_____|_____|_____ ftp://ftp.wpusa.dynip.com
          _|_Eat_SPAM_to_email_me!_YUM!__|_____|_____ wkitty42 -at- alltel.net
        Your message has been successfully submitted and would be delivered to recipients shortly.