Loading ...
Sorry, an error occurred while loading the content.
 

Re: Logs vs. Tags

Expand Messages
  • bluepenguin1980
    I have to say all the talk about 4-fold visit drops make me very nervous. I was hoping that our visits would not see a dramatic decline - mainly because we re
    Message 1 of 22 , Nov 30, 2005
      I have to say all the talk about 4-fold visit drops make me very
      nervous. I was hoping that our visits would not see a dramatic
      decline - mainly because we're quite meticulous about blocking any
      unwanted traffic (mostly the spiders). The client already
      experienced a slight shock ealier this year when the spider filters
      were implemented - and visits dropped by 30%..

      Could it be possible that in other stories posted here - there might
      have also been some filtering issues, or anything else not directly
      related to the tagging vs. logging (So software settings?). One of
      the posts mentioned total unique visitors being counted as per page,
      not the entire site.

      One of the thing I'm absolutely prepared for is our page views
      dropping - right now our Webtrends interprets flash files as pages,
      hence all the flash loads (eg. 5 flash files in a page) get recorded
      as individual page views...

      I'll definitely keep you updated as to what happens.


      --- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
      >
      > One of the things to think about is that with logging, the web
      server
      > can immediately write requests to a file. However, in a page
      tagging
      > scenario, you're relying on the client browser to pass information
      to
      > another server over the Internet. There is a likelihood that a
      small
      > percentage of requests get dropped somewhere in between. In
      addition,
      > network bandwidth and server capabilities of the ASPs play a vital
      > role in how many concurrent requests (site traffic) they can log.
      > Granted, I doubt this accounts for a large percentage and there are
      > other factors at play here.
      >
      > -Fred
      >
      > --- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
      > >
      > > Here's my cookbook of reconciling page tags with server logs.
      > >
      > > 1. Maybe you don't have all your pages tagged. Possible. Try
      > picking a
      > > given representative page you know is tagged and looking at
      just it
      > - get
      > > the log lines for the same timespan for just that page from the
      > server and
      > > page tag logs for a clean compare.
      > > 2. Make sure you have a <NOSCRIPT> option set up to get hits and
      > some info
      > > off people that have JavaScript turned off (most moble clients,
      for
      > > example). Include those lines in your analysis.
      > > 3. Put the page tag inline in the top of the <BODY> section of
      the
      > HTML.
      > > 4. Look at the user-agents in the server log and clean out all
      the
      > known
      > > spiders and whatnot.
      > > 5. Do a side by side compare in Excel or the like, identify
      matching
      > > lines, lines in the page tag log not in the server log and server
      > log lines
      > > not in the page tag log. You'll see some hits in your page tag
      log
      > not in
      > > your server logs; these will mostly be "revisits" to the same
      page.
      > >
      > > I've done this and still have a 30-50% drop rate in the page tag
      > logs (page
      > > views for the one page with legit looking user agents that do
      appear in
      > > server logs but not page tag logs). It's not just page views
      but whole
      > > visits that get dropped in our case, sounds similar to yours.
      > >
      > > We've put the same page tag code on another unrelated Web site
      and
      > see the
      > > same results. We've ruled out client issues (turning off
      JavaScript is
      > > possible in many browsers, but the NOSCRIPT catches those;
      blocking
      > cookies
      > > won't affect this analysis, and blocking third party images is
      only
      > > supported in Firefox).
      > >
      > > I'm frankly wondering if there's an unknown endemic problem with
      page
      > > tagging that no one's figured out yet - not the
      theoretical "good"
      > reasons
      > > you'd see different numbers, but a "bad" reason good hits don't
      come
      > > through.
      > >
      > > Ernest
      >
    • Jon Bovard
      Our page view numbers jumped by 30% when we moved from logs to tags The amusing part was of course this was the same vendor and same software The amount of
      Message 2 of 22 , Dec 1, 2005
        Message
        Our page view numbers jumped by 30% when we moved from logs to tags
        The amusing part was of course this was the same vendor and same software
         
        The amount of page caching occurring on our CMS/site was quite staggering
        Interestingly only page views were affected. Unique visitors was the same to within 1%
         
        jon
        -----Original Message-----
        From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com] On Behalf Of Jason Grigsby
        Sent: 2005/11/30 21:34
        To: webanalytics@yahoogroups.com
        Subject: Re: [webanalytics] Logs vs. Tags

        We recently transitioned from logs to hybrid (logs + tags), and we 
        saw a similar drop. We sat there in shock when we saw the first 
        numbers from the new system. The drop was so much more than we had 
        ever imagined.

        Being the glutton for punishment that I am, I spent several hours 
        parsing logs by hand for a single day to check to see if the number 
        looked correct.

        It did.

        It wasn't 100% accurate. It under represented some usage that I 
        thought was likely to be from human visitors. But the percentage it 
        under represented was much smaller than the over-inflated number 
        coming from the IP-based log analysis solution that we had been using.

        My estimate was that the percentage under represented was about 8% 
        for that day using the hybrid, page-tagging solution versus the 4-
        fold over representing I got from the log-only solution.

        So the four-fold decrease is probably correct. The first time you 
        look at the numbers from page-tagging it is like being dragged out of 
        Plato's cave. The reality you see is nothing like the reality you've 
        known.

        -Jason

        On Nov 30, 2005, at 12:06 PM, Jennifer Cronan wrote:

        > I am very interested in this as well.  We just switched from log 
        > files to tags and we are seeing 4 fold drop in Visits.  We have 
        > been investigating definitions because we think that log file was 
        > tracking Visits as unique sessions to a page instead of unique 
        > sessions to the website as a whole.  But could there be something 
        > else that is causing this huge drop?  I knew it would drop a lot 
        > because of more clean data, but not that much.
        > -Jen
        >
        > bluepenguin1980 <ola_zaranska@...> wrote:
        > Morning everyone,
        >
        > I've been tasked in predicting how our reporting numbers are going to
        > change once we make a switch from a log-based web analytics 
        > solution to
        > tags... The client is looking for some examples (actual numbers and
        > percentages!) - and while I realize (and communicated) that this might
        > be a stretch - has anybody else done similar research?
        >
        > Obviously the numbers will not be the same, and in the end tags will
        > provide a much more accurate state of things - but saying this is not
        > enough.
        >
        > Any input very appreciated,
        >
        > Aleksandra Zaranska
        >
        >
        >
        >
        >
        >
        > ---------------------------------------
        > Web Metrics Discussion Group
        > Moderated by Eric T. Peterson
        > Author, Web Analytics Demystified
        > http://www.webanalyticsdemystified.com
        >
        >
        >
        > SPONSORED LINKS
        > Internet business plan Start internet business Internet business
        > Internet business online Internet home business Internet business 
        > consulting
        >
        > YAHOO! GROUPS LINKS
        >
        >  Visit your group "webanalytics" on the web.
        >
        >  To unsubscribe from this group, send an email to:
        >  webanalytics-unsubscribe@yahoogroups.com
        >
        >  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
        >
        >

        --

        Jason Grigsby
        (503) 238-2014 voice
        mailto:jason@...



        _______________________________________________________
        CONFIDENTIALITY NOTICE
        The information in this email is confidential and is intended solely for the addressee. Access to this email by anyone else is unauthorised. If you are not the intended recipient, you must not read, use or disseminate the information. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Net a Porter Ltd.

        Net A Porter Ltd is a company registered in England & Wales Number: 3820604 Registered Office: The Dome, Whiteleys Centre, 151 Queensway, London, W2 4YN.
        _______________________________________________________
      • Jennifer Cronan
        You are correct there are other factors exactly that you pointed out: in our logs we don t filter out spiders, and internal traffic is also being lumped int
        Message 3 of 22 , Dec 1, 2005
          You are correct there are other factors exactly that you pointed out: in our logs we don't filter out spiders, and internal traffic is also being lumped int out data.  We are pretty confident we have a handle on the average amount of traffic we are getting in the logs that we need to ignore.

          bluepenguin1980 <ola_zaranska@...> wrote:
          I have to say all the talk about 4-fold visit drops make me very
          nervous.  I was hoping that our visits would not see a dramatic
          decline - mainly because we're quite meticulous about blocking any
          unwanted traffic (mostly the spiders).  The client already
          experienced a slight shock ealier this year when the spider filters
          were implemented - and visits dropped by 30%..

          Could it be possible that in other stories posted here - there might
          have also been some filtering issues, or anything else not directly
          related to the tagging vs. logging (So software settings?).  One of
          the posts mentioned total unique visitors being counted as per page,
          not the entire site.

          One of the thing I'm absolutely prepared for is our page views
          dropping - right now our Webtrends interprets flash files as pages,
          hence all the flash loads (eg. 5 flash files in a page) get recorded
          as individual page views...

          I'll definitely keep you updated as to what happens.


          --- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
          >
          > One of the things to think about is that with logging, the web
          server
          > can immediately write requests to a file.  However, in a page
          tagging
          > scenario, you're relying on the client browser to pass information
          to
          > another server over the Internet.  There is a likelihood that a
          small
          > percentage of requests get dropped somewhere in between.  In
          addition,
          > network bandwidth and server capabilities of the ASPs play a vital
          > role in how many concurrent requests (site traffic) they can log.
          > Granted, I doubt this accounts for a large percentage and there are
          > other factors at play here.
          >
          > -Fred
          >  
          > --- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
          > >
          > > Here's my cookbook of reconciling page tags with server logs.
          > >
          > > 1.  Maybe you don't have all your pages tagged.  Possible.  Try
          > picking a
          > > given representative page you know is tagged  and looking at
          just it
          > - get
          > > the log lines for the same timespan for just that page from the
          > server and
          > > page tag logs for a clean compare.
          > > 2.  Make sure you have a <NOSCRIPT> option set up to get hits and
          > some info
          > > off people that have JavaScript turned off (most moble clients,
          for
          > > example).  Include those lines in your analysis.
          > > 3.  Put the page tag inline in the top of the <BODY> section of
          the
          > HTML.
          > > 4.  Look at the user-agents in the server log and clean out all
          the
          > known
          > > spiders and whatnot.
          > > 5.  Do a side by side compare in Excel or the like, identify
          matching
          > > lines, lines in the page tag log not in the server log and server
          > log lines
          > > not in the page tag log.  You'll see some hits in your page tag
          log
          > not in
          > > your server logs; these will mostly be "revisits" to the same
          page.
          > >
          > > I've done this and still have a 30-50% drop rate in the page tag
          > logs (page
          > > views for the one page with legit looking user agents that do
          appear in
          > > server logs but not page tag logs).   It's not just page views
          but whole
          > > visits that get dropped in our case, sounds similar to yours.
          > >
          > > We've put the same page tag code on another unrelated Web site 
          and
          > see the
          > > same results.  We've ruled out client issues (turning off
          JavaScript is
          > > possible in many browsers, but the NOSCRIPT catches those;
          blocking
          > cookies
          > > won't affect this analysis, and blocking third party images is
          only
          > > supported in Firefox).
          > >
          > > I'm frankly wondering if there's an unknown endemic problem with
          page
          > > tagging that no one's figured out yet - not the
          theoretical "good"
          > reasons
          > > you'd see different numbers, but a "bad" reason good hits don't
          come
          > > through.
          > >
          > > Ernest
          >







          SPONSORED LINKS
          Internet business planStart internet businessInternet business
          Internet business onlineInternet home businessInternet business consulting
        • ernest.mueller@ni.com
          No filtering issues in our case. If the discrepancy visitors are spiders, they re spiders coming from a wide variety of IPs with perfenctly normal IE5/6 user
          Message 4 of 22 , Dec 1, 2005
            No filtering issues in our case. If the discrepancy visitors are spiders,
            they're spiders coming from a wide variety of IPs with perfenctly normal
            IE5/6 user agent strings. We didn't see 4x loss, though, "just" 30%.
            We're still working it because we don't like any percent unexplained. (We
            count visitors for the site, I just mentioned per page because drilling
            down on data for one page is the easiest way to get to the heart of the
            problem.)

            Ernest




            "bluepenguin1980"
            <ola_zaranska@hot
            mail.com> To
            Sent by: webanalytics@yahoogroups.com
            webanalytics@yaho cc
            ogroups.com
            Subject
            [webanalytics] Re: Logs vs. Tags
            11/30/2005 07:58
            PM


            Please respond to
            webanalytics@yaho
            ogroups.com






            I have to say all the talk about 4-fold visit drops make me very
            nervous. I was hoping that our visits would not see a dramatic
            decline - mainly because we're quite meticulous about blocking any
            unwanted traffic (mostly the spiders). The client already
            experienced a slight shock ealier this year when the spider filters
            were implemented - and visits dropped by 30%..

            Could it be possible that in other stories posted here - there might
            have also been some filtering issues, or anything else not directly
            related to the tagging vs. logging (So software settings?). One of
            the posts mentioned total unique visitors being counted as per page,
            not the entire site.

            One of the thing I'm absolutely prepared for is our page views
            dropping - right now our Webtrends interprets flash files as pages,
            hence all the flash loads (eg. 5 flash files in a page) get recorded
            as individual page views...

            I'll definitely keep you updated as to what happens.


            --- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
            >
            > One of the things to think about is that with logging, the web
            server
            > can immediately write requests to a file. However, in a page
            tagging
            > scenario, you're relying on the client browser to pass information
            to
            > another server over the Internet. There is a likelihood that a
            small
            > percentage of requests get dropped somewhere in between. In
            addition,
            > network bandwidth and server capabilities of the ASPs play a vital
            > role in how many concurrent requests (site traffic) they can log.
            > Granted, I doubt this accounts for a large percentage and there are
            > other factors at play here.
            >
            > -Fred
            >
            > --- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
            > >
            > > Here's my cookbook of reconciling page tags with server logs.
            > >
            > > 1. Maybe you don't have all your pages tagged. Possible. Try
            > picking a
            > > given representative page you know is tagged and looking at
            just it
            > - get
            > > the log lines for the same timespan for just that page from the
            > server and
            > > page tag logs for a clean compare.
            > > 2. Make sure you have a <NOSCRIPT> option set up to get hits and
            > some info
            > > off people that have JavaScript turned off (most moble clients,
            for
            > > example). Include those lines in your analysis.
            > > 3. Put the page tag inline in the top of the <BODY> section of
            the
            > HTML.
            > > 4. Look at the user-agents in the server log and clean out all
            the
            > known
            > > spiders and whatnot.
            > > 5. Do a side by side compare in Excel or the like, identify
            matching
            > > lines, lines in the page tag log not in the server log and server
            > log lines
            > > not in the page tag log. You'll see some hits in your page tag
            log
            > not in
            > > your server logs; these will mostly be "revisits" to the same
            page.
            > >
            > > I've done this and still have a 30-50% drop rate in the page tag
            > logs (page
            > > views for the one page with legit looking user agents that do
            appear in
            > > server logs but not page tag logs). It's not just page views
            but whole
            > > visits that get dropped in our case, sounds similar to yours.
            > >
            > > We've put the same page tag code on another unrelated Web site
            and
            > see the
            > > same results. We've ruled out client issues (turning off
            JavaScript is
            > > possible in many browsers, but the NOSCRIPT catches those;
            blocking
            > cookies
            > > won't affect this analysis, and blocking third party images is
            only
            > > supported in Firefox).
            > >
            > > I'm frankly wondering if there's an unknown endemic problem with
            page
            > > tagging that no one's figured out yet - not the
            theoretical "good"
            > reasons
            > > you'd see different numbers, but a "bad" reason good hits don't
            come
            > > through.
            > >
            > > Ernest
            >








            ---------------------------------------
            Web Metrics Discussion Group
            Moderated by Eric T. Peterson
            Author, Web Analytics Demystified
            http://www.webanalyticsdemystified.com



            SPONSORED LINKS

            Internet business Start internet Internet business
            plan business

            Internet business Internet home Internet business
            online business consulting



            YAHOO! GROUPS LINKS

            Visit your group "webanalytics" on the web.

            To unsubscribe from this group, send an email to:
            webanalytics-unsubscribe@yahoogroups.com

            Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
          • Andrew Edwards
            In my experience--by no means universal but based on seeing dozens of company s installations--the biggest cause of discrepancy between log and tag analysis is
            Message 5 of 22 , Dec 1, 2005
              In my experience--by no means universal but based on seeing dozens of
              company's installations--the biggest cause of discrepancy between log
              and tag analysis is that a large number of robots and spiders are not
              filterable.

              Our counsel to clients has been that if you rely on log file analysis,
              your raw traffic numbers will be prone to significant overcounts.

              --Andrew

              ernest.mueller@... wrote:

              >No filtering issues in our case. If the discrepancy visitors are spiders,
              >they're spiders coming from a wide variety of IPs with perfenctly normal
              >IE5/6 user agent strings. We didn't see 4x loss, though, "just" 30%.
              >We're still working it because we don't like any percent unexplained. (We
              >count visitors for the site, I just mentioned per page because drilling
              >down on data for one page is the easiest way to get to the heart of the
              >problem.)
              >
              >Ernest
              >
              >
              >
              >
              > "bluepenguin1980"
              > <ola_zaranska@hot
              > mail.com> To
              > Sent by: webanalytics@yahoogroups.com
              > webanalytics@yaho cc
              > ogroups.com
              > Subject
              > [webanalytics] Re: Logs vs. Tags
              > 11/30/2005 07:58
              > PM
              >
              >
              > Please respond to
              > webanalytics@yaho
              > ogroups.com
              >
              >
              >
              >
              >
              >
              >I have to say all the talk about 4-fold visit drops make me very
              >nervous. I was hoping that our visits would not see a dramatic
              >decline - mainly because we're quite meticulous about blocking any
              >unwanted traffic (mostly the spiders). The client already
              >experienced a slight shock ealier this year when the spider filters
              >were implemented - and visits dropped by 30%..
              >
              >Could it be possible that in other stories posted here - there might
              >have also been some filtering issues, or anything else not directly
              >related to the tagging vs. logging (So software settings?). One of
              >the posts mentioned total unique visitors being counted as per page,
              >not the entire site.
              >
              >One of the thing I'm absolutely prepared for is our page views
              >dropping - right now our Webtrends interprets flash files as pages,
              >hence all the flash loads (eg. 5 flash files in a page) get recorded
              >as individual page views...
              >
              >I'll definitely keep you updated as to what happens.
              >
              >
              >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
              >
              >
              >>One of the things to think about is that with logging, the web
              >>
              >>
              >server
              >
              >
              >>can immediately write requests to a file. However, in a page
              >>
              >>
              >tagging
              >
              >
              >>scenario, you're relying on the client browser to pass information
              >>
              >>
              >to
              >
              >
              >>another server over the Internet. There is a likelihood that a
              >>
              >>
              >small
              >
              >
              >>percentage of requests get dropped somewhere in between. In
              >>
              >>
              >addition,
              >
              >
              >>network bandwidth and server capabilities of the ASPs play a vital
              >>role in how many concurrent requests (site traffic) they can log.
              >>Granted, I doubt this accounts for a large percentage and there are
              >>other factors at play here.
              >>
              >>-Fred
              >>
              >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
              >>
              >>
              >>>Here's my cookbook of reconciling page tags with server logs.
              >>>
              >>>1. Maybe you don't have all your pages tagged. Possible. Try
              >>>
              >>>
              >>picking a
              >>
              >>
              >>>given representative page you know is tagged and looking at
              >>>
              >>>
              >just it
              >
              >
              >>- get
              >>
              >>
              >>>the log lines for the same timespan for just that page from the
              >>>
              >>>
              >>server and
              >>
              >>
              >>>page tag logs for a clean compare.
              >>>2. Make sure you have a <NOSCRIPT> option set up to get hits and
              >>>
              >>>
              >>some info
              >>
              >>
              >>>off people that have JavaScript turned off (most moble clients,
              >>>
              >>>
              >for
              >
              >
              >>>example). Include those lines in your analysis.
              >>>3. Put the page tag inline in the top of the <BODY> section of
              >>>
              >>>
              >the
              >
              >
              >>HTML.
              >>
              >>
              >>>4. Look at the user-agents in the server log and clean out all
              >>>
              >>>
              >the
              >
              >
              >>known
              >>
              >>
              >>>spiders and whatnot.
              >>>5. Do a side by side compare in Excel or the like, identify
              >>>
              >>>
              >matching
              >
              >
              >>>lines, lines in the page tag log not in the server log and server
              >>>
              >>>
              >>log lines
              >>
              >>
              >>>not in the page tag log. You'll see some hits in your page tag
              >>>
              >>>
              >log
              >
              >
              >>not in
              >>
              >>
              >>>your server logs; these will mostly be "revisits" to the same
              >>>
              >>>
              >page.
              >
              >
              >>>I've done this and still have a 30-50% drop rate in the page tag
              >>>
              >>>
              >>logs (page
              >>
              >>
              >>>views for the one page with legit looking user agents that do
              >>>
              >>>
              >appear in
              >
              >
              >>>server logs but not page tag logs). It's not just page views
              >>>
              >>>
              >but whole
              >
              >
              >>>visits that get dropped in our case, sounds similar to yours.
              >>>
              >>>We've put the same page tag code on another unrelated Web site
              >>>
              >>>
              >and
              >
              >
              >>see the
              >>
              >>
              >>>same results. We've ruled out client issues (turning off
              >>>
              >>>
              >JavaScript is
              >
              >
              >>>possible in many browsers, but the NOSCRIPT catches those;
              >>>
              >>>
              >blocking
              >
              >
              >>cookies
              >>
              >>
              >>>won't affect this analysis, and blocking third party images is
              >>>
              >>>
              >only
              >
              >
              >>>supported in Firefox).
              >>>
              >>>I'm frankly wondering if there's an unknown endemic problem with
              >>>
              >>>
              >page
              >
              >
              >>>tagging that no one's figured out yet - not the
              >>>
              >>>
              >theoretical "good"
              >
              >
              >>reasons
              >>
              >>
              >>>you'd see different numbers, but a "bad" reason good hits don't
              >>>
              >>>
              >come
              >
              >
              >>>through.
              >>>
              >>>Ernest
              >>>
              >>>
              >
              >
              >
              >
              >
              >
              >
              >
              >---------------------------------------
              >Web Metrics Discussion Group
              >Moderated by Eric T. Peterson
              >Author, Web Analytics Demystified
              >http://www.webanalyticsdemystified.com
              >
              >
              >
              > SPONSORED LINKS
              >
              > Internet business Start internet Internet business
              > plan business
              >
              > Internet business Internet home Internet business
              > online business consulting
              >
              >
              >
              > YAHOO! GROUPS LINKS
              >
              > Visit your group "webanalytics" on the web.
              >
              > To unsubscribe from this group, send an email to:
              > webanalytics-unsubscribe@yahoogroups.com
              >
              > Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
              >
              >
              >
              >
              >
              >
              >
              >---------------------------------------
              >Web Metrics Discussion Group
              >Moderated by Eric T. Peterson
              >Author, Web Analytics Demystified
              >http://www.webanalyticsdemystified.com
              >Yahoo! Groups Links
              >
              >
              >
              >
              >
              >
              >
              >


              --
              Andrew Edwards
              Managing Partner
              Technology Leaders
              230 Park Avenue
              New York, NY 10169
              (212) 808-3058
              Cell: (917) 602-0083
              aedwards@...
              www.technologyleaders.com
            • ernest.mueller@ni.com
              I guess I just have a hard time believing that 30% of the traffic to not just our site but other sites we ve tested the page tag on are from unfilterable
              Message 6 of 22 , Dec 1, 2005
                I guess I just have a hard time believing that 30% of the traffic to not
                just our site but other sites we've tested the page tag on are from
                unfilterable spiders. All the reputable search vendors and spider software
                vendors supply a filterable user-agent. So we're saying that 30% of all
                Web site traffic remaining *excluding* those guys is coming from a wide
                variety (geo-distributed) of cloaked spiders. Which means that like 30% of
                Internet traffic is due to that. If realistic, it calls for a crackdown
                and a half. But I'm not sure that's realistic.

                Ernest





                Andrew Edwards
                <aedwards@technol
                ogyleaders.com> To
                Sent by: webanalytics@yahoogroups.com
                webanalytics@yaho cc
                ogroups.com
                Subject
                Re: [webanalytics] Re: Logs vs.
                12/01/2005 10:16 Tags
                AM


                Please respond to
                webanalytics@yaho
                ogroups.com






                In my experience--by no means universal but based on seeing dozens of
                company's installations--the biggest cause of discrepancy between log
                and tag analysis is that a large number of robots and spiders are not
                filterable.

                Our counsel to clients has been that if you rely on log file analysis,
                your raw traffic numbers will be prone to significant overcounts.

                --Andrew

                ernest.mueller@... wrote:

                >No filtering issues in our case. If the discrepancy visitors are spiders,
                >they're spiders coming from a wide variety of IPs with perfenctly normal
                >IE5/6 user agent strings. We didn't see 4x loss, though, "just" 30%.
                >We're still working it because we don't like any percent unexplained. (We
                >count visitors for the site, I just mentioned per page because drilling
                >down on data for one page is the easiest way to get to the heart of the
                >problem.)
                >
                >Ernest
                >
                >
                >
                >

                > "bluepenguin1980"

                > <ola_zaranska@hot

                > mail.com> To

                > Sent by: webanalytics@yahoogroups.com

                > webanalytics@yaho cc

                > ogroups.com

                > Subject

                > [webanalytics] Re: Logs vs. Tags

                > 11/30/2005 07:58

                > PM

                >

                >

                > Please respond to

                > webanalytics@yaho

                > ogroups.com

                >

                >

                >
                >
                >
                >
                >I have to say all the talk about 4-fold visit drops make me very
                >nervous. I was hoping that our visits would not see a dramatic
                >decline - mainly because we're quite meticulous about blocking any
                >unwanted traffic (mostly the spiders). The client already
                >experienced a slight shock ealier this year when the spider filters
                >were implemented - and visits dropped by 30%..
                >
                >Could it be possible that in other stories posted here - there might
                >have also been some filtering issues, or anything else not directly
                >related to the tagging vs. logging (So software settings?). One of
                >the posts mentioned total unique visitors being counted as per page,
                >not the entire site.
                >
                >One of the thing I'm absolutely prepared for is our page views
                >dropping - right now our Webtrends interprets flash files as pages,
                >hence all the flash loads (eg. 5 flash files in a page) get recorded
                >as individual page views...
                >
                >I'll definitely keep you updated as to what happens.
                >
                >
                >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
                >
                >
                >>One of the things to think about is that with logging, the web
                >>
                >>
                >server
                >
                >
                >>can immediately write requests to a file. However, in a page
                >>
                >>
                >tagging
                >
                >
                >>scenario, you're relying on the client browser to pass information
                >>
                >>
                >to
                >
                >
                >>another server over the Internet. There is a likelihood that a
                >>
                >>
                >small
                >
                >
                >>percentage of requests get dropped somewhere in between. In
                >>
                >>
                >addition,
                >
                >
                >>network bandwidth and server capabilities of the ASPs play a vital
                >>role in how many concurrent requests (site traffic) they can log.
                >>Granted, I doubt this accounts for a large percentage and there are
                >>other factors at play here.
                >>
                >>-Fred
                >>
                >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                >>
                >>
                >>>Here's my cookbook of reconciling page tags with server logs.
                >>>
                >>>1. Maybe you don't have all your pages tagged. Possible. Try
                >>>
                >>>
                >>picking a
                >>
                >>
                >>>given representative page you know is tagged and looking at
                >>>
                >>>
                >just it
                >
                >
                >>- get
                >>
                >>
                >>>the log lines for the same timespan for just that page from the
                >>>
                >>>
                >>server and
                >>
                >>
                >>>page tag logs for a clean compare.
                >>>2. Make sure you have a <NOSCRIPT> option set up to get hits and
                >>>
                >>>
                >>some info
                >>
                >>
                >>>off people that have JavaScript turned off (most moble clients,
                >>>
                >>>
                >for
                >
                >
                >>>example). Include those lines in your analysis.
                >>>3. Put the page tag inline in the top of the <BODY> section of
                >>>
                >>>
                >the
                >
                >
                >>HTML.
                >>
                >>
                >>>4. Look at the user-agents in the server log and clean out all
                >>>
                >>>
                >the
                >
                >
                >>known
                >>
                >>
                >>>spiders and whatnot.
                >>>5. Do a side by side compare in Excel or the like, identify
                >>>
                >>>
                >matching
                >
                >
                >>>lines, lines in the page tag log not in the server log and server
                >>>
                >>>
                >>log lines
                >>
                >>
                >>>not in the page tag log. You'll see some hits in your page tag
                >>>
                >>>
                >log
                >
                >
                >>not in
                >>
                >>
                >>>your server logs; these will mostly be "revisits" to the same
                >>>
                >>>
                >page.
                >
                >
                >>>I've done this and still have a 30-50% drop rate in the page tag
                >>>
                >>>
                >>logs (page
                >>
                >>
                >>>views for the one page with legit looking user agents that do
                >>>
                >>>
                >appear in
                >
                >
                >>>server logs but not page tag logs). It's not just page views
                >>>
                >>>
                >but whole
                >
                >
                >>>visits that get dropped in our case, sounds similar to yours.
                >>>
                >>>We've put the same page tag code on another unrelated Web site
                >>>
                >>>
                >and
                >
                >
                >>see the
                >>
                >>
                >>>same results. We've ruled out client issues (turning off
                >>>
                >>>
                >JavaScript is
                >
                >
                >>>possible in many browsers, but the NOSCRIPT catches those;
                >>>
                >>>
                >blocking
                >
                >
                >>cookies
                >>
                >>
                >>>won't affect this analysis, and blocking third party images is
                >>>
                >>>
                >only
                >
                >
                >>>supported in Firefox).
                >>>
                >>>I'm frankly wondering if there's an unknown endemic problem with
                >>>
                >>>
                >page
                >
                >
                >>>tagging that no one's figured out yet - not the
                >>>
                >>>
                >theoretical "good"
                >
                >
                >>reasons
                >>
                >>
                >>>you'd see different numbers, but a "bad" reason good hits don't
                >>>
                >>>
                >come
                >
                >
                >>>through.
                >>>
                >>>Ernest
                >>>
                >>>
                >
                >
                >
                >
                >
                >
                >
                >
                >---------------------------------------
                >Web Metrics Discussion Group
                >Moderated by Eric T. Peterson
                >Author, Web Analytics Demystified
                >http://www.webanalyticsdemystified.com
                >
                >
                >
                > SPONSORED
                LINKS
                >
                > Internet business Start internet Internet business
                > plan business
                >
                > Internet business Internet home Internet business
                > online business consulting
                >
                >
                >
                > YAHOO! GROUPS LINKS
                >
                > Visit your group "webanalytics" on the web.
                >
                > To unsubscribe from this group, send an email to:
                > webanalytics-unsubscribe@yahoogroups.com
                >
                > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                Service.
                >
                >
                >
                >
                >
                >
                >
                >---------------------------------------
                >Web Metrics Discussion Group
                >Moderated by Eric T. Peterson
                >Author, Web Analytics Demystified
                >http://www.webanalyticsdemystified.com
                >Yahoo! Groups Links
                >
                >
                >
                >
                >
                >
                >
                >


                --
                Andrew Edwards
                Managing Partner
                Technology Leaders
                230 Park Avenue
                New York, NY 10169
                (212) 808-3058
                Cell: (917) 602-0083
                aedwards@...
                www.technologyleaders.com





                ---------------------------------------
                Web Metrics Discussion Group
                Moderated by Eric T. Peterson
                Author, Web Analytics Demystified
                http://www.webanalyticsdemystified.com



                SPONSORED LINKS

                Internet business Start internet Internet business
                plan business

                Internet business Internet home Internet business
                online business consulting



                YAHOO! GROUPS LINKS

                Visit your group "webanalytics" on the web.

                To unsubscribe from this group, send an email to:
                webanalytics-unsubscribe@yahoogroups.com

                Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.
              • Gus Kormeier
                Don t crawlers often use multiple IP addresses? So a few companies/crawlers would appear like very sizable chunks of traffic? -Gus ... From:
                Message 7 of 22 , Dec 1, 2005
                  Don't crawlers often use multiple IP addresses?
                  So a few companies/crawlers would appear like very sizable chunks of
                  traffic?
                  -Gus

                  -----Original Message-----
                  From: ernest.mueller@... [mailto:ernest.mueller@...]
                  Sent: Thursday, December 01, 2005 8:48 AM
                  To: webanalytics@yahoogroups.com
                  Cc: webanalytics@yahoogroups.com
                  Subject: Re: [webanalytics] Re: Logs vs. Tags


                  I guess I just have a hard time believing that 30% of the traffic to not
                  just our site but other sites we've tested the page tag on are from
                  unfilterable spiders. All the reputable search vendors and spider software
                  vendors supply a filterable user-agent. So we're saying that 30% of all
                  Web site traffic remaining *excluding* those guys is coming from a wide
                  variety (geo-distributed) of cloaked spiders. Which means that like 30% of
                  Internet traffic is due to that. If realistic, it calls for a crackdown
                  and a half. But I'm not sure that's realistic.

                  Ernest





                  Andrew Edwards
                  <aedwards@technol
                  ogyleaders.com> To
                  Sent by: webanalytics@yahoogroups.com
                  webanalytics@yaho cc
                  ogroups.com
                  Subject
                  Re: [webanalytics] Re: Logs vs.
                  12/01/2005 10:16 Tags
                  AM


                  Please respond to
                  webanalytics@yaho
                  ogroups.com






                  In my experience--by no means universal but based on seeing dozens of
                  company's installations--the biggest cause of discrepancy between log
                  and tag analysis is that a large number of robots and spiders are not
                  filterable.

                  Our counsel to clients has been that if you rely on log file analysis,
                  your raw traffic numbers will be prone to significant overcounts.

                  --Andrew

                  ernest.mueller@... wrote:

                  >No filtering issues in our case. If the discrepancy visitors are spiders,
                  >they're spiders coming from a wide variety of IPs with perfenctly normal
                  >IE5/6 user agent strings. We didn't see 4x loss, though, "just" 30%.
                  >We're still working it because we don't like any percent unexplained. (We
                  >count visitors for the site, I just mentioned per page because drilling
                  >down on data for one page is the easiest way to get to the heart of the
                  >problem.)
                  >
                  >Ernest
                  >
                  >
                  >
                  >

                  > "bluepenguin1980"

                  > <ola_zaranska@hot

                  > mail.com> To

                  > Sent by: webanalytics@yahoogroups.com

                  > webanalytics@yaho cc

                  > ogroups.com

                  > Subject

                  > [webanalytics] Re: Logs vs. Tags

                  > 11/30/2005 07:58

                  > PM

                  >

                  >

                  > Please respond to

                  > webanalytics@yaho

                  > ogroups.com

                  >

                  >

                  >
                  >
                  >
                  >
                  >I have to say all the talk about 4-fold visit drops make me very
                  >nervous. I was hoping that our visits would not see a dramatic
                  >decline - mainly because we're quite meticulous about blocking any
                  >unwanted traffic (mostly the spiders). The client already
                  >experienced a slight shock ealier this year when the spider filters
                  >were implemented - and visits dropped by 30%..
                  >
                  >Could it be possible that in other stories posted here - there might
                  >have also been some filtering issues, or anything else not directly
                  >related to the tagging vs. logging (So software settings?). One of
                  >the posts mentioned total unique visitors being counted as per page,
                  >not the entire site.
                  >
                  >One of the thing I'm absolutely prepared for is our page views
                  >dropping - right now our Webtrends interprets flash files as pages,
                  >hence all the flash loads (eg. 5 flash files in a page) get recorded
                  >as individual page views...
                  >
                  >I'll definitely keep you updated as to what happens.
                  >
                  >
                  >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
                  >
                  >
                  >>One of the things to think about is that with logging, the web
                  >>
                  >>
                  >server
                  >
                  >
                  >>can immediately write requests to a file. However, in a page
                  >>
                  >>
                  >tagging
                  >
                  >
                  >>scenario, you're relying on the client browser to pass information
                  >>
                  >>
                  >to
                  >
                  >
                  >>another server over the Internet. There is a likelihood that a
                  >>
                  >>
                  >small
                  >
                  >
                  >>percentage of requests get dropped somewhere in between. In
                  >>
                  >>
                  >addition,
                  >
                  >
                  >>network bandwidth and server capabilities of the ASPs play a vital
                  >>role in how many concurrent requests (site traffic) they can log.
                  >>Granted, I doubt this accounts for a large percentage and there are
                  >>other factors at play here.
                  >>
                  >>-Fred
                  >>
                  >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                  >>
                  >>
                  >>>Here's my cookbook of reconciling page tags with server logs.
                  >>>
                  >>>1. Maybe you don't have all your pages tagged. Possible. Try
                  >>>
                  >>>
                  >>picking a
                  >>
                  >>
                  >>>given representative page you know is tagged and looking at
                  >>>
                  >>>
                  >just it
                  >
                  >
                  >>- get
                  >>
                  >>
                  >>>the log lines for the same timespan for just that page from the
                  >>>
                  >>>
                  >>server and
                  >>
                  >>
                  >>>page tag logs for a clean compare.
                  >>>2. Make sure you have a <NOSCRIPT> option set up to get hits and
                  >>>
                  >>>
                  >>some info
                  >>
                  >>
                  >>>off people that have JavaScript turned off (most moble clients,
                  >>>
                  >>>
                  >for
                  >
                  >
                  >>>example). Include those lines in your analysis.
                  >>>3. Put the page tag inline in the top of the <BODY> section of
                  >>>
                  >>>
                  >the
                  >
                  >
                  >>HTML.
                  >>
                  >>
                  >>>4. Look at the user-agents in the server log and clean out all
                  >>>
                  >>>
                  >the
                  >
                  >
                  >>known
                  >>
                  >>
                  >>>spiders and whatnot.
                  >>>5. Do a side by side compare in Excel or the like, identify
                  >>>
                  >>>
                  >matching
                  >
                  >
                  >>>lines, lines in the page tag log not in the server log and server
                  >>>
                  >>>
                  >>log lines
                  >>
                  >>
                  >>>not in the page tag log. You'll see some hits in your page tag
                  >>>
                  >>>
                  >log
                  >
                  >
                  >>not in
                  >>
                  >>
                  >>>your server logs; these will mostly be "revisits" to the same
                  >>>
                  >>>
                  >page.
                  >
                  >
                  >>>I've done this and still have a 30-50% drop rate in the page tag
                  >>>
                  >>>
                  >>logs (page
                  >>
                  >>
                  >>>views for the one page with legit looking user agents that do
                  >>>
                  >>>
                  >appear in
                  >
                  >
                  >>>server logs but not page tag logs). It's not just page views
                  >>>
                  >>>
                  >but whole
                  >
                  >
                  >>>visits that get dropped in our case, sounds similar to yours.
                  >>>
                  >>>We've put the same page tag code on another unrelated Web site
                  >>>
                  >>>
                  >and
                  >
                  >
                  >>see the
                  >>
                  >>
                  >>>same results. We've ruled out client issues (turning off
                  >>>
                  >>>
                  >JavaScript is
                  >
                  >
                  >>>possible in many browsers, but the NOSCRIPT catches those;
                  >>>
                  >>>
                  >blocking
                  >
                  >
                  >>cookies
                  >>
                  >>
                  >>>won't affect this analysis, and blocking third party images is
                  >>>
                  >>>
                  >only
                  >
                  >
                  >>>supported in Firefox).
                  >>>
                  >>>I'm frankly wondering if there's an unknown endemic problem with
                  >>>
                  >>>
                  >page
                  >
                  >
                  >>>tagging that no one's figured out yet - not the
                  >>>
                  >>>
                  >theoretical "good"
                  >
                  >
                  >>reasons
                  >>
                  >>
                  >>>you'd see different numbers, but a "bad" reason good hits don't
                  >>>
                  >>>
                  >come
                  >
                  >
                  >>>through.
                  >>>
                  >>>Ernest
                  >>>
                  >>>
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >---------------------------------------
                  >Web Metrics Discussion Group
                  >Moderated by Eric T. Peterson
                  >Author, Web Analytics Demystified
                  >http://www.webanalyticsdemystified.com
                  >
                  >
                  >
                  > SPONSORED
                  LINKS
                  >
                  > Internet business Start internet Internet business
                  > plan business
                  >
                  > Internet business Internet home Internet business
                  > online business consulting
                  >
                  >
                  >
                  > YAHOO! GROUPS LINKS
                  >
                  > Visit your group "webanalytics" on the web.
                  >
                  > To unsubscribe from this group, send an email to:
                  > webanalytics-unsubscribe@yahoogroups.com
                  >
                  > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                  Service.
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >---------------------------------------
                  >Web Metrics Discussion Group
                  >Moderated by Eric T. Peterson
                  >Author, Web Analytics Demystified
                  >http://www.webanalyticsdemystified.com
                  >Yahoo! Groups Links
                  >
                  >
                  >
                  >
                  >
                  >
                  >
                  >


                  --
                  Andrew Edwards
                  Managing Partner
                  Technology Leaders
                  230 Park Avenue
                  New York, NY 10169
                  (212) 808-3058
                  Cell: (917) 602-0083
                  aedwards@...
                  www.technologyleaders.com





                  ---------------------------------------
                  Web Metrics Discussion Group
                  Moderated by Eric T. Peterson
                  Author, Web Analytics Demystified
                  http://www.webanalyticsdemystified.com



                  SPONSORED LINKS

                  Internet business Start internet Internet business
                  plan business

                  Internet business Internet home Internet business
                  online business consulting



                  YAHOO! GROUPS LINKS

                  Visit your group "webanalytics" on the web.

                  To unsubscribe from this group, send an email to:
                  webanalytics-unsubscribe@yahoogroups.com

                  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.







                  ---------------------------------------
                  Web Metrics Discussion Group
                  Moderated by Eric T. Peterson
                  Author, Web Analytics Demystified
                  http://www.webanalyticsdemystified.com
                  Yahoo! Groups Links
                • joyabillings
                  Hey All, I worked for a company that saw very large (20-25%) frops in page views and visits. There was quite an uproar about the drop. However, we emphasized
                  Message 8 of 22 , Dec 1, 2005
                    Hey All,

                    I worked for a company that saw very large (20-25%) frops in page
                    views and visits. There was quite an uproar about the drop. However,
                    we emphasized accuracy - the quality of the numbers. When, for
                    example Nielsen or an auditing company came in and reported our
                    monthly numbers as way below our internal, log-based numbers, we knew
                    something had to be changed. Although the drop may scare clients
                    (which is totaly understandable), the client has a duty to be accurate
                    in reporting.

                    More importantly, page views, visits, etc. aren't all we should focus
                    on. Those metrics are quickly becoming anntiquated to more
                    sophisticated metrics, such as click-stream analyses, visitor
                    segmentation and integrated registration data with site behavior.

                    Although I preach this all, we are still struggling with moving our
                    clients over as well to see the value of visitors, personas, etc.
                    Hopefully, we will all be able to emphasize the value of the customer,
                    and not just worry about the baisc numbers...



                    --- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                    >
                    > I guess I just have a hard time believing that 30% of the traffic to not
                    > just our site but other sites we've tested the page tag on are from
                    > unfilterable spiders. All the reputable search vendors and spider
                    software
                    > vendors supply a filterable user-agent. So we're saying that 30% of all
                    > Web site traffic remaining *excluding* those guys is coming from a wide
                    > variety (geo-distributed) of cloaked spiders. Which means that like
                    30% of
                    > Internet traffic is due to that. If realistic, it calls for a crackdown
                    > and a half. But I'm not sure that's realistic.
                    >
                    > Ernest
                    >
                    >
                    >
                    >
                    >

                    > Andrew Edwards

                    > <aedwards@technol

                    > ogyleaders.com>
                    To
                    > Sent by: webanalytics@yahoogroups.com

                    > webanalytics@yaho
                    cc
                    > ogroups.com

                    >
                    Subject
                    > Re: [webanalytics] Re: Logs
                    vs.
                    > 12/01/2005 10:16 Tags

                    > AM

                    >

                    >

                    > Please respond to

                    > webanalytics@yaho

                    > ogroups.com

                    >

                    >

                    >
                    >
                    >
                    >
                    > In my experience--by no means universal but based on seeing dozens of
                    > company's installations--the biggest cause of discrepancy between log
                    > and tag analysis is that a large number of robots and spiders are not
                    > filterable.
                    >
                    > Our counsel to clients has been that if you rely on log file analysis,
                    > your raw traffic numbers will be prone to significant overcounts.
                    >
                    > --Andrew
                    >
                    > ernest.mueller@n... wrote:
                    >
                    > >No filtering issues in our case. If the discrepancy visitors are
                    spiders,
                    > >they're spiders coming from a wide variety of IPs with perfenctly
                    normal
                    > >IE5/6 user agent strings. We didn't see 4x loss, though, "just" 30%.
                    > >We're still working it because we don't like any percent
                    unexplained. (We
                    > >count visitors for the site, I just mentioned per page because drilling
                    > >down on data for one page is the easiest way to get to the heart of the
                    > >problem.)
                    > >
                    > >Ernest
                    > >
                    > >
                    > >
                    > >
                    >
                    > > "bluepenguin1980"
                    >
                    > > <ola_zaranska@hot
                    >
                    > > mail.com>
                    To
                    >
                    > > Sent by: webanalytics@yahoogroups.com
                    >
                    > > webanalytics@yaho
                    cc
                    >
                    > > ogroups.com
                    >
                    > >
                    Subject
                    >
                    > > [webanalytics] Re: Logs vs. Tags
                    >
                    > > 11/30/2005 07:58
                    >
                    > > PM
                    >
                    > >
                    >
                    > >
                    >
                    > > Please respond to
                    >
                    > > webanalytics@yaho
                    >
                    > > ogroups.com
                    >
                    > >
                    >
                    > >
                    >
                    > >
                    > >
                    > >
                    > >
                    > >I have to say all the talk about 4-fold visit drops make me very
                    > >nervous. I was hoping that our visits would not see a dramatic
                    > >decline - mainly because we're quite meticulous about blocking any
                    > >unwanted traffic (mostly the spiders). The client already
                    > >experienced a slight shock ealier this year when the spider filters
                    > >were implemented - and visits dropped by 30%..
                    > >
                    > >Could it be possible that in other stories posted here - there might
                    > >have also been some filtering issues, or anything else not directly
                    > >related to the tagging vs. logging (So software settings?). One of
                    > >the posts mentioned total unique visitors being counted as per page,
                    > >not the entire site.
                    > >
                    > >One of the thing I'm absolutely prepared for is our page views
                    > >dropping - right now our Webtrends interprets flash files as pages,
                    > >hence all the flash loads (eg. 5 flash files in a page) get recorded
                    > >as individual page views...
                    > >
                    > >I'll definitely keep you updated as to what happens.
                    > >
                    > >
                    > >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...> wrote:
                    > >
                    > >
                    > >>One of the things to think about is that with logging, the web
                    > >>
                    > >>
                    > >server
                    > >
                    > >
                    > >>can immediately write requests to a file. However, in a page
                    > >>
                    > >>
                    > >tagging
                    > >
                    > >
                    > >>scenario, you're relying on the client browser to pass information
                    > >>
                    > >>
                    > >to
                    > >
                    > >
                    > >>another server over the Internet. There is a likelihood that a
                    > >>
                    > >>
                    > >small
                    > >
                    > >
                    > >>percentage of requests get dropped somewhere in between. In
                    > >>
                    > >>
                    > >addition,
                    > >
                    > >
                    > >>network bandwidth and server capabilities of the ASPs play a vital
                    > >>role in how many concurrent requests (site traffic) they can log.
                    > >>Granted, I doubt this accounts for a large percentage and there are
                    > >>other factors at play here.
                    > >>
                    > >>-Fred
                    > >>
                    > >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                    > >>
                    > >>
                    > >>>Here's my cookbook of reconciling page tags with server logs.
                    > >>>
                    > >>>1. Maybe you don't have all your pages tagged. Possible. Try
                    > >>>
                    > >>>
                    > >>picking a
                    > >>
                    > >>
                    > >>>given representative page you know is tagged and looking at
                    > >>>
                    > >>>
                    > >just it
                    > >
                    > >
                    > >>- get
                    > >>
                    > >>
                    > >>>the log lines for the same timespan for just that page from the
                    > >>>
                    > >>>
                    > >>server and
                    > >>
                    > >>
                    > >>>page tag logs for a clean compare.
                    > >>>2. Make sure you have a <NOSCRIPT> option set up to get hits and
                    > >>>
                    > >>>
                    > >>some info
                    > >>
                    > >>
                    > >>>off people that have JavaScript turned off (most moble clients,
                    > >>>
                    > >>>
                    > >for
                    > >
                    > >
                    > >>>example). Include those lines in your analysis.
                    > >>>3. Put the page tag inline in the top of the <BODY> section of
                    > >>>
                    > >>>
                    > >the
                    > >
                    > >
                    > >>HTML.
                    > >>
                    > >>
                    > >>>4. Look at the user-agents in the server log and clean out all
                    > >>>
                    > >>>
                    > >the
                    > >
                    > >
                    > >>known
                    > >>
                    > >>
                    > >>>spiders and whatnot.
                    > >>>5. Do a side by side compare in Excel or the like, identify
                    > >>>
                    > >>>
                    > >matching
                    > >
                    > >
                    > >>>lines, lines in the page tag log not in the server log and server
                    > >>>
                    > >>>
                    > >>log lines
                    > >>
                    > >>
                    > >>>not in the page tag log. You'll see some hits in your page tag
                    > >>>
                    > >>>
                    > >log
                    > >
                    > >
                    > >>not in
                    > >>
                    > >>
                    > >>>your server logs; these will mostly be "revisits" to the same
                    > >>>
                    > >>>
                    > >page.
                    > >
                    > >
                    > >>>I've done this and still have a 30-50% drop rate in the page tag
                    > >>>
                    > >>>
                    > >>logs (page
                    > >>
                    > >>
                    > >>>views for the one page with legit looking user agents that do
                    > >>>
                    > >>>
                    > >appear in
                    > >
                    > >
                    > >>>server logs but not page tag logs). It's not just page views
                    > >>>
                    > >>>
                    > >but whole
                    > >
                    > >
                    > >>>visits that get dropped in our case, sounds similar to yours.
                    > >>>
                    > >>>We've put the same page tag code on another unrelated Web site
                    > >>>
                    > >>>
                    > >and
                    > >
                    > >
                    > >>see the
                    > >>
                    > >>
                    > >>>same results. We've ruled out client issues (turning off
                    > >>>
                    > >>>
                    > >JavaScript is
                    > >
                    > >
                    > >>>possible in many browsers, but the NOSCRIPT catches those;
                    > >>>
                    > >>>
                    > >blocking
                    > >
                    > >
                    > >>cookies
                    > >>
                    > >>
                    > >>>won't affect this analysis, and blocking third party images is
                    > >>>
                    > >>>
                    > >only
                    > >
                    > >
                    > >>>supported in Firefox).
                    > >>>
                    > >>>I'm frankly wondering if there's an unknown endemic problem with
                    > >>>
                    > >>>
                    > >page
                    > >
                    > >
                    > >>>tagging that no one's figured out yet - not the
                    > >>>
                    > >>>
                    > >theoretical "good"
                    > >
                    > >
                    > >>reasons
                    > >>
                    > >>
                    > >>>you'd see different numbers, but a "bad" reason good hits don't
                    > >>>
                    > >>>
                    > >come
                    > >
                    > >
                    > >>>through.
                    > >>>
                    > >>>Ernest
                    > >>>
                    > >>>
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >---------------------------------------
                    > >Web Metrics Discussion Group
                    > >Moderated by Eric T. Peterson
                    > >Author, Web Analytics Demystified
                    > >http://www.webanalyticsdemystified.com
                    > >
                    > >
                    > >
                    > > SPONSORED
                    > LINKS
                    > >
                    > > Internet business Start internet Internet business
                    > > plan business
                    > >
                    > > Internet business Internet home Internet business
                    > > online business consulting
                    > >
                    > >
                    > >
                    > > YAHOO! GROUPS LINKS
                    > >
                    > > Visit your group "webanalytics" on the web.
                    > >
                    > > To unsubscribe from this group, send an email to:
                    > > webanalytics-unsubscribe@yahoogroups.com
                    > >
                    > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                    > Service.
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >---------------------------------------
                    > >Web Metrics Discussion Group
                    > >Moderated by Eric T. Peterson
                    > >Author, Web Analytics Demystified
                    > >http://www.webanalyticsdemystified.com
                    > >Yahoo! Groups Links
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    > >
                    >
                    >
                    > --
                    > Andrew Edwards
                    > Managing Partner
                    > Technology Leaders
                    > 230 Park Avenue
                    > New York, NY 10169
                    > (212) 808-3058
                    > Cell: (917) 602-0083
                    > aedwards@t...
                    > www.technologyleaders.com
                    >
                    >
                    >
                    >
                    >
                    > ---------------------------------------
                    > Web Metrics Discussion Group
                    > Moderated by Eric T. Peterson
                    > Author, Web Analytics Demystified
                    > http://www.webanalyticsdemystified.com
                    >
                    >
                    >
                    >
                    SPONSORED LINKS
                    >
                    > Internet business Start internet Internet business
                    > plan business
                    >
                    > Internet business Internet home Internet business
                    > online business consulting
                    >
                    >
                    >
                    > YAHOO! GROUPS LINKS
                    >
                    > Visit your group "webanalytics" on the web.
                    >
                    > To unsubscribe from this group, send an email to:
                    > webanalytics-unsubscribe@yahoogroups.com
                    >
                    > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                    Service.
                    >
                  • bluepenguin1980
                    Interesting comments about the spiders... One of the spider-presence tests that I ve been doing is checking our Visits By Number of Page Views report
                    Message 9 of 22 , Dec 1, 2005
                      Interesting comments about the spiders...

                      One of the spider-presence "tests" that I've been doing is checking
                      our Visits By Number of Page Views report (Webtrends)

                      We initially noticed that something was wrong when about 30% of our
                      traffic in this report registered visits with 0 page views (and by
                      definition, a visit will alwasy have at least one page view!) These
                      were all caused by spiders.. After some filtering work - this
                      percentage dropped to less than 1%. (And of course.. our overall
                      visits dropped by about 30%)

                      So I'm assuming that less than 1% of our traffic is a result of
                      spider activity - ... That's what I've been using for the "health
                      check" and hopefully I'm not in for a rude awakening.

                      --- In webanalytics@yahoogroups.com, Gus Kormeier <gusk@a...> wrote:
                      >
                      > Don't crawlers often use multiple IP addresses?
                      > So a few companies/crawlers would appear like very sizable chunks
                      of
                      > traffic?
                      > -Gus
                      >
                      > -----Original Message-----
                      > From: ernest.mueller@n... [mailto:ernest.mueller@n...]
                      > Sent: Thursday, December 01, 2005 8:48 AM
                      > To: webanalytics@yahoogroups.com
                      > Cc: webanalytics@yahoogroups.com
                      > Subject: Re: [webanalytics] Re: Logs vs. Tags
                      >
                      >
                      > I guess I just have a hard time believing that 30% of the traffic
                      to not
                      > just our site but other sites we've tested the page tag on are from
                      > unfilterable spiders. All the reputable search vendors and spider
                      software
                      > vendors supply a filterable user-agent. So we're saying that 30%
                      of all
                      > Web site traffic remaining *excluding* those guys is coming from a
                      wide
                      > variety (geo-distributed) of cloaked spiders. Which means that
                      like 30% of
                      > Internet traffic is due to that. If realistic, it calls for a
                      crackdown
                      > and a half. But I'm not sure that's realistic.
                      >
                      > Ernest
                      >
                      >
                      >
                      >
                      >

                      > Andrew
                      Edwards
                      >
                      <aedwards@technol
                      >
                      ogyleaders.com> To
                      > Sent by:
                      webanalytics@yahoogroups.com
                      >
                      webanalytics@yaho cc
                      >
                      ogroups.com
                      >
                      Subject
                      > Re: [webanalytics] Re: Logs
                      vs.
                      > 12/01/2005 10:16
                      Tags
                      >
                      AM
                      >

                      >

                      > Please respond
                      to
                      >
                      webanalytics@yaho
                      >
                      ogroups.com
                      >

                      >

                      >
                      >
                      >
                      >
                      > In my experience--by no means universal but based on seeing dozens
                      of
                      > company's installations--the biggest cause of discrepancy between
                      log
                      > and tag analysis is that a large number of robots and spiders are
                      not
                      > filterable.
                      >
                      > Our counsel to clients has been that if you rely on log file
                      analysis,
                      > your raw traffic numbers will be prone to significant overcounts.
                      >
                      > --Andrew
                      >
                      > ernest.mueller@n... wrote:
                      >
                      > >No filtering issues in our case. If the discrepancy visitors are
                      spiders,
                      > >they're spiders coming from a wide variety of IPs with perfenctly
                      normal
                      > >IE5/6 user agent strings. We didn't see 4x loss, though, "just"
                      30%.
                      > >We're still working it because we don't like any percent
                      unexplained. (We
                      > >count visitors for the site, I just mentioned per page because
                      drilling
                      > >down on data for one page is the easiest way to get to the heart
                      of the
                      > >problem.)
                      > >
                      > >Ernest
                      > >
                      > >
                      > >
                      > >
                      >
                      > > "bluepenguin1980"
                      >
                      > > <ola_zaranska@hot
                      >
                      > >
                      mail.com> To
                      >
                      > > Sent by:
                      webanalytics@yahoogroups.com
                      >
                      > >
                      webanalytics@yaho cc
                      >
                      > > ogroups.com
                      >
                      >
                      >
                      Subject
                      >
                      > > [webanalytics] Re: Logs
                      vs. Tags
                      >
                      > > 11/30/2005 07:58
                      >
                      > > PM
                      >
                      > >
                      >
                      > >
                      >
                      > > Please respond to
                      >
                      > > webanalytics@yaho
                      >
                      > > ogroups.com
                      >
                      > >
                      >
                      > >
                      >
                      > >
                      > >
                      > >
                      > >
                      > >I have to say all the talk about 4-fold visit drops make me very
                      > >nervous. I was hoping that our visits would not see a dramatic
                      > >decline - mainly because we're quite meticulous about blocking any
                      > >unwanted traffic (mostly the spiders). The client already
                      > >experienced a slight shock ealier this year when the spider
                      filters
                      > >were implemented - and visits dropped by 30%..
                      > >
                      > >Could it be possible that in other stories posted here - there
                      might
                      > >have also been some filtering issues, or anything else not
                      directly
                      > >related to the tagging vs. logging (So software settings?). One
                      of
                      > >the posts mentioned total unique visitors being counted as per
                      page,
                      > >not the entire site.
                      > >
                      > >One of the thing I'm absolutely prepared for is our page views
                      > >dropping - right now our Webtrends interprets flash files as
                      pages,
                      > >hence all the flash loads (eg. 5 flash files in a page) get
                      recorded
                      > >as individual page views...
                      > >
                      > >I'll definitely keep you updated as to what happens.
                      > >
                      > >
                      > >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...>
                      wrote:
                      > >
                      > >
                      > >>One of the things to think about is that with logging, the web
                      > >>
                      > >>
                      > >server
                      > >
                      > >
                      > >>can immediately write requests to a file. However, in a page
                      > >>
                      > >>
                      > >tagging
                      > >
                      > >
                      > >>scenario, you're relying on the client browser to pass
                      information
                      > >>
                      > >>
                      > >to
                      > >
                      > >
                      > >>another server over the Internet. There is a likelihood that a
                      > >>
                      > >>
                      > >small
                      > >
                      > >
                      > >>percentage of requests get dropped somewhere in between. In
                      > >>
                      > >>
                      > >addition,
                      > >
                      > >
                      > >>network bandwidth and server capabilities of the ASPs play a
                      vital
                      > >>role in how many concurrent requests (site traffic) they can log.
                      > >>Granted, I doubt this accounts for a large percentage and there
                      are
                      > >>other factors at play here.
                      > >>
                      > >>-Fred
                      > >>
                      > >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                      > >>
                      > >>
                      > >>>Here's my cookbook of reconciling page tags with server logs.
                      > >>>
                      > >>>1. Maybe you don't have all your pages tagged. Possible. Try
                      > >>>
                      > >>>
                      > >>picking a
                      > >>
                      > >>
                      > >>>given representative page you know is tagged and looking at
                      > >>>
                      > >>>
                      > >just it
                      > >
                      > >
                      > >>- get
                      > >>
                      > >>
                      > >>>the log lines for the same timespan for just that page from the
                      > >>>
                      > >>>
                      > >>server and
                      > >>
                      > >>
                      > >>>page tag logs for a clean compare.
                      > >>>2. Make sure you have a <NOSCRIPT> option set up to get hits
                      and
                      > >>>
                      > >>>
                      > >>some info
                      > >>
                      > >>
                      > >>>off people that have JavaScript turned off (most moble clients,
                      > >>>
                      > >>>
                      > >for
                      > >
                      > >
                      > >>>example). Include those lines in your analysis.
                      > >>>3. Put the page tag inline in the top of the <BODY> section of
                      > >>>
                      > >>>
                      > >the
                      > >
                      > >
                      > >>HTML.
                      > >>
                      > >>
                      > >>>4. Look at the user-agents in the server log and clean out all
                      > >>>
                      > >>>
                      > >the
                      > >
                      > >
                      > >>known
                      > >>
                      > >>
                      > >>>spiders and whatnot.
                      > >>>5. Do a side by side compare in Excel or the like, identify
                      > >>>
                      > >>>
                      > >matching
                      > >
                      > >
                      > >>>lines, lines in the page tag log not in the server log and
                      server
                      > >>>
                      > >>>
                      > >>log lines
                      > >>
                      > >>
                      > >>>not in the page tag log. You'll see some hits in your page tag
                      > >>>
                      > >>>
                      > >log
                      > >
                      > >
                      > >>not in
                      > >>
                      > >>
                      > >>>your server logs; these will mostly be "revisits" to the same
                      > >>>
                      > >>>
                      > >page.
                      > >
                      > >
                      > >>>I've done this and still have a 30-50% drop rate in the page tag
                      > >>>
                      > >>>
                      > >>logs (page
                      > >>
                      > >>
                      > >>>views for the one page with legit looking user agents that do
                      > >>>
                      > >>>
                      > >appear in
                      > >
                      > >
                      > >>>server logs but not page tag logs). It's not just page views
                      > >>>
                      > >>>
                      > >but whole
                      > >
                      > >
                      > >>>visits that get dropped in our case, sounds similar to yours.
                      > >>>
                      > >>>We've put the same page tag code on another unrelated Web site
                      > >>>
                      > >>>
                      > >and
                      > >
                      > >
                      > >>see the
                      > >>
                      > >>
                      > >>>same results. We've ruled out client issues (turning off
                      > >>>
                      > >>>
                      > >JavaScript is
                      > >
                      > >
                      > >>>possible in many browsers, but the NOSCRIPT catches those;
                      > >>>
                      > >>>
                      > >blocking
                      > >
                      > >
                      > >>cookies
                      > >>
                      > >>
                      > >>>won't affect this analysis, and blocking third party images is
                      > >>>
                      > >>>
                      > >only
                      > >
                      > >
                      > >>>supported in Firefox).
                      > >>>
                      > >>>I'm frankly wondering if there's an unknown endemic problem with
                      > >>>
                      > >>>
                      > >page
                      > >
                      > >
                      > >>>tagging that no one's figured out yet - not the
                      > >>>
                      > >>>
                      > >theoretical "good"
                      > >
                      > >
                      > >>reasons
                      > >>
                      > >>
                      > >>>you'd see different numbers, but a "bad" reason good hits don't
                      > >>>
                      > >>>
                      > >come
                      > >
                      > >
                      > >>>through.
                      > >>>
                      > >>>Ernest
                      > >>>
                      > >>>
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >---------------------------------------
                      > >Web Metrics Discussion Group
                      > >Moderated by Eric T. Peterson
                      > >Author, Web Analytics Demystified
                      > >http://www.webanalyticsdemystified.com
                      > >
                      > >
                      > >
                      > >
                      SPONSORED
                      > LINKS
                      > >
                      > > Internet business Start internet Internet business
                      > > plan business
                      > >
                      > > Internet business Internet home Internet business
                      > > online business consulting
                      > >
                      > >
                      > >
                      > > YAHOO! GROUPS LINKS
                      > >
                      > > Visit your group "webanalytics" on the web.
                      > >
                      > > To unsubscribe from this group, send an email to:
                      > > webanalytics-unsubscribe@yahoogroups.com
                      > >
                      > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                      > Service.
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >---------------------------------------
                      > >Web Metrics Discussion Group
                      > >Moderated by Eric T. Peterson
                      > >Author, Web Analytics Demystified
                      > >http://www.webanalyticsdemystified.com
                      > >Yahoo! Groups Links
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      > >
                      >
                      >
                      > --
                      > Andrew Edwards
                      > Managing Partner
                      > Technology Leaders
                      > 230 Park Avenue
                      > New York, NY 10169
                      > (212) 808-3058
                      > Cell: (917) 602-0083
                      > aedwards@t...
                      > www.technologyleaders.com
                      >
                      >
                      >
                      >
                      >
                      > ---------------------------------------
                      > Web Metrics Discussion Group
                      > Moderated by Eric T. Peterson
                      > Author, Web Analytics Demystified
                      > http://www.webanalyticsdemystified.com
                      >
                      >
                      >
                      >
                      SPONSORED LINKS
                      >
                      > Internet business Start internet Internet business
                      > plan business
                      >
                      > Internet business Internet home Internet business
                      > online business consulting
                      >
                      >
                      >
                      > YAHOO! GROUPS LINKS
                      >
                      > Visit your group "webanalytics" on the web.
                      >
                      > To unsubscribe from this group, send an email to:
                      > webanalytics-unsubscribe@yahoogroups.com
                      >
                      > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                      Service.
                      >
                      >
                      >
                      >
                      >
                      >
                      >
                      > ---------------------------------------
                      > Web Metrics Discussion Group
                      > Moderated by Eric T. Peterson
                      > Author, Web Analytics Demystified
                      > http://www.webanalyticsdemystified.com
                      > Yahoo! Groups Links
                      >
                    • brewsterbarclay
                      Hi Everyone, I have so far refrained from contributing to the logs vs tags debate because I do not think that this forum should be too commercial. However,
                      Message 10 of 22 , Dec 2, 2005
                        Hi Everyone,
                        I have so far refrained from contributing to the logs vs tags debate
                        because I do not think that this forum should be too commercial.
                        However, there is a third way (no not packet sniffing) which has all
                        the advantages of log files and page tags and none of the disadvantages
                        of either, which could be part of the discussion.
                        I would be happy if anyone wanted to contact me directly on this or
                        many of you may know my colleague Rufus Evison, rste@...
                        who would also be happy to talk as well.
                        Apologies again if this is too much "vendor speak".
                        Regards,
                        Brewster
                      • Brendan Butterworth
                        Just having read through the Stumbleupon question and provided some feedback, I was thinking about browser prefetching. How many users prefetch with browser
                        Message 11 of 22 , Dec 3, 2005
                          Just having read through the Stumbleupon question and provided some
                          feedback, I was thinking about browser prefetching. How many users
                          prefetch with browser integrated tools? Wouldn't this type of
                          activity show up in log analysis with a proper user agent (it is the
                          web browser requesting the information) but not in strict JavaScript
                          based tags?

                          As for the test below - 30% of the traffic with no page views! Log
                          analysis tools generally say that visits are based on hits, not page
                          views, which is why it would count. However, with a percentage this
                          high of client connections that are requesting data from your server
                          without actually requesting the pages, I'd be concerned!

                          Just my two cents.

                          /brendan.

                          On 12/1/05, bluepenguin1980 <ola_zaranska@...> wrote:
                          > Interesting comments about the spiders...
                          >
                          > One of the spider-presence "tests" that I've been doing is checking
                          > our Visits By Number of Page Views report (Webtrends)
                          >
                          > We initially noticed that something was wrong when about 30% of our
                          > traffic in this report registered visits with 0 page views (and by
                          > definition, a visit will alwasy have at least one page view!) These
                          > were all caused by spiders.. After some filtering work - this
                          > percentage dropped to less than 1%. (And of course.. our overall
                          > visits dropped by about 30%)
                          >
                          > So I'm assuming that less than 1% of our traffic is a result of
                          > spider activity - ... That's what I've been using for the "health
                          > check" and hopefully I'm not in for a rude awakening.
                          >
                          > --- In webanalytics@yahoogroups.com, Gus Kormeier <gusk@a...> wrote:
                          > >
                          > > Don't crawlers often use multiple IP addresses?
                          > > So a few companies/crawlers would appear like very sizable chunks
                          > of
                          > > traffic?
                          > > -Gus
                          > >
                          > > -----Original Message-----
                          > > From: ernest.mueller@n... [mailto:ernest.mueller@n...]
                          > > Sent: Thursday, December 01, 2005 8:48 AM
                          > > To: webanalytics@yahoogroups.com
                          > > Cc: webanalytics@yahoogroups.com
                          > > Subject: Re: [webanalytics] Re: Logs vs. Tags
                          > >
                          > >
                          > > I guess I just have a hard time believing that 30% of the traffic
                          > to not
                          > > just our site but other sites we've tested the page tag on are from
                          > > unfilterable spiders. All the reputable search vendors and spider
                          > software
                          > > vendors supply a filterable user-agent. So we're saying that 30%
                          > of all
                          > > Web site traffic remaining *excluding* those guys is coming from a
                          > wide
                          > > variety (geo-distributed) of cloaked spiders. Which means that
                          > like 30% of
                          > > Internet traffic is due to that. If realistic, it calls for a
                          > crackdown
                          > > and a half. But I'm not sure that's realistic.
                          > >
                          > > Ernest
                          > >
                          > >
                          > >
                          > >
                          > >
                          >
                          > > Andrew
                          > Edwards
                          > >
                          > <aedwards@technol
                          > >
                          > ogyleaders.com> To
                          > > Sent by:
                          > webanalytics@yahoogroups.com
                          > >
                          > webanalytics@yaho cc
                          > >
                          > ogroups.com
                          > >
                          > Subject
                          > > Re: [webanalytics] Re: Logs
                          > vs.
                          > > 12/01/2005 10:16
                          > Tags
                          > >
                          > AM
                          > >
                          >
                          > >
                          >
                          > > Please respond
                          > to
                          > >
                          > webanalytics@yaho
                          > >
                          > ogroups.com
                          > >
                          >
                          > >
                          >
                          > >
                          > >
                          > >
                          > >
                          > > In my experience--by no means universal but based on seeing dozens
                          > of
                          > > company's installations--the biggest cause of discrepancy between
                          > log
                          > > and tag analysis is that a large number of robots and spiders are
                          > not
                          > > filterable.
                          > >
                          > > Our counsel to clients has been that if you rely on log file
                          > analysis,
                          > > your raw traffic numbers will be prone to significant overcounts.
                          > >
                          > > --Andrew
                          > >
                          > > ernest.mueller@n... wrote:
                          > >
                          > > >No filtering issues in our case. If the discrepancy visitors are
                          > spiders,
                          > > >they're spiders coming from a wide variety of IPs with perfenctly
                          > normal
                          > > >IE5/6 user agent strings. We didn't see 4x loss, though, "just"
                          > 30%.
                          > > >We're still working it because we don't like any percent
                          > unexplained. (We
                          > > >count visitors for the site, I just mentioned per page because
                          > drilling
                          > > >down on data for one page is the easiest way to get to the heart
                          > of the
                          > > >problem.)
                          > > >
                          > > >Ernest
                          > > >
                          > > >
                          > > >
                          > > >
                          > >
                          > > > "bluepenguin1980"
                          > >
                          > > > <ola_zaranska@hot
                          > >
                          > > >
                          > mail.com> To
                          > >
                          > > > Sent by:
                          > webanalytics@yahoogroups.com
                          > >
                          > > >
                          > webanalytics@yaho cc
                          > >
                          > > > ogroups.com
                          > >
                          > >
                          > >
                          > Subject
                          > >
                          > > > [webanalytics] Re: Logs
                          > vs. Tags
                          > >
                          > > > 11/30/2005 07:58
                          > >
                          > > > PM
                          > >
                          > > >
                          > >
                          > > >
                          > >
                          > > > Please respond to
                          > >
                          > > > webanalytics@yaho
                          > >
                          > > > ogroups.com
                          > >
                          > > >
                          > >
                          > > >
                          > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >I have to say all the talk about 4-fold visit drops make me very
                          > > >nervous. I was hoping that our visits would not see a dramatic
                          > > >decline - mainly because we're quite meticulous about blocking any
                          > > >unwanted traffic (mostly the spiders). The client already
                          > > >experienced a slight shock ealier this year when the spider
                          > filters
                          > > >were implemented - and visits dropped by 30%..
                          > > >
                          > > >Could it be possible that in other stories posted here - there
                          > might
                          > > >have also been some filtering issues, or anything else not
                          > directly
                          > > >related to the tagging vs. logging (So software settings?). One
                          > of
                          > > >the posts mentioned total unique visitors being counted as per
                          > page,
                          > > >not the entire site.
                          > > >
                          > > >One of the thing I'm absolutely prepared for is our page views
                          > > >dropping - right now our Webtrends interprets flash files as
                          > pages,
                          > > >hence all the flash loads (eg. 5 flash files in a page) get
                          > recorded
                          > > >as individual page views...
                          > > >
                          > > >I'll definitely keep you updated as to what happens.
                          > > >
                          > > >
                          > > >--- In webanalytics@yahoogroups.com, "Fred Kuu" <fkuu325@y...>
                          > wrote:
                          > > >
                          > > >
                          > > >>One of the things to think about is that with logging, the web
                          > > >>
                          > > >>
                          > > >server
                          > > >
                          > > >
                          > > >>can immediately write requests to a file. However, in a page
                          > > >>
                          > > >>
                          > > >tagging
                          > > >
                          > > >
                          > > >>scenario, you're relying on the client browser to pass
                          > information
                          > > >>
                          > > >>
                          > > >to
                          > > >
                          > > >
                          > > >>another server over the Internet. There is a likelihood that a
                          > > >>
                          > > >>
                          > > >small
                          > > >
                          > > >
                          > > >>percentage of requests get dropped somewhere in between. In
                          > > >>
                          > > >>
                          > > >addition,
                          > > >
                          > > >
                          > > >>network bandwidth and server capabilities of the ASPs play a
                          > vital
                          > > >>role in how many concurrent requests (site traffic) they can log.
                          > > >>Granted, I doubt this accounts for a large percentage and there
                          > are
                          > > >>other factors at play here.
                          > > >>
                          > > >>-Fred
                          > > >>
                          > > >>--- In webanalytics@yahoogroups.com, ernest.mueller@n... wrote:
                          > > >>
                          > > >>
                          > > >>>Here's my cookbook of reconciling page tags with server logs.
                          > > >>>
                          > > >>>1. Maybe you don't have all your pages tagged. Possible. Try
                          > > >>>
                          > > >>>
                          > > >>picking a
                          > > >>
                          > > >>
                          > > >>>given representative page you know is tagged and looking at
                          > > >>>
                          > > >>>
                          > > >just it
                          > > >
                          > > >
                          > > >>- get
                          > > >>
                          > > >>
                          > > >>>the log lines for the same timespan for just that page from the
                          > > >>>
                          > > >>>
                          > > >>server and
                          > > >>
                          > > >>
                          > > >>>page tag logs for a clean compare.
                          > > >>>2. Make sure you have a <NOSCRIPT> option set up to get hits
                          > and
                          > > >>>
                          > > >>>
                          > > >>some info
                          > > >>
                          > > >>
                          > > >>>off people that have JavaScript turned off (most moble clients,
                          > > >>>
                          > > >>>
                          > > >for
                          > > >
                          > > >
                          > > >>>example). Include those lines in your analysis.
                          > > >>>3. Put the page tag inline in the top of the <BODY> section of
                          > > >>>
                          > > >>>
                          > > >the
                          > > >
                          > > >
                          > > >>HTML.
                          > > >>
                          > > >>
                          > > >>>4. Look at the user-agents in the server log and clean out all
                          > > >>>
                          > > >>>
                          > > >the
                          > > >
                          > > >
                          > > >>known
                          > > >>
                          > > >>
                          > > >>>spiders and whatnot.
                          > > >>>5. Do a side by side compare in Excel or the like, identify
                          > > >>>
                          > > >>>
                          > > >matching
                          > > >
                          > > >
                          > > >>>lines, lines in the page tag log not in the server log and
                          > server
                          > > >>>
                          > > >>>
                          > > >>log lines
                          > > >>
                          > > >>
                          > > >>>not in the page tag log. You'll see some hits in your page tag
                          > > >>>
                          > > >>>
                          > > >log
                          > > >
                          > > >
                          > > >>not in
                          > > >>
                          > > >>
                          > > >>>your server logs; these will mostly be "revisits" to the same
                          > > >>>
                          > > >>>
                          > > >page.
                          > > >
                          > > >
                          > > >>>I've done this and still have a 30-50% drop rate in the page tag
                          > > >>>
                          > > >>>
                          > > >>logs (page
                          > > >>
                          > > >>
                          > > >>>views for the one page with legit looking user agents that do
                          > > >>>
                          > > >>>
                          > > >appear in
                          > > >
                          > > >
                          > > >>>server logs but not page tag logs). It's not just page views
                          > > >>>
                          > > >>>
                          > > >but whole
                          > > >
                          > > >
                          > > >>>visits that get dropped in our case, sounds similar to yours.
                          > > >>>
                          > > >>>We've put the same page tag code on another unrelated Web site
                          > > >>>
                          > > >>>
                          > > >and
                          > > >
                          > > >
                          > > >>see the
                          > > >>
                          > > >>
                          > > >>>same results. We've ruled out client issues (turning off
                          > > >>>
                          > > >>>
                          > > >JavaScript is
                          > > >
                          > > >
                          > > >>>possible in many browsers, but the NOSCRIPT catches those;
                          > > >>>
                          > > >>>
                          > > >blocking
                          > > >
                          > > >
                          > > >>cookies
                          > > >>
                          > > >>
                          > > >>>won't affect this analysis, and blocking third party images is
                          > > >>>
                          > > >>>
                          > > >only
                          > > >
                          > > >
                          > > >>>supported in Firefox).
                          > > >>>
                          > > >>>I'm frankly wondering if there's an unknown endemic problem with
                          > > >>>
                          > > >>>
                          > > >page
                          > > >
                          > > >
                          > > >>>tagging that no one's figured out yet - not the
                          > > >>>
                          > > >>>
                          > > >theoretical "good"
                          > > >
                          > > >
                          > > >>reasons
                          > > >>
                          > > >>
                          > > >>>you'd see different numbers, but a "bad" reason good hits don't
                          > > >>>
                          > > >>>
                          > > >come
                          > > >
                          > > >
                          > > >>>through.
                          > > >>>
                          > > >>>Ernest
                          > > >>>
                          > > >>>
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >---------------------------------------
                          > > >Web Metrics Discussion Group
                          > > >Moderated by Eric T. Peterson
                          > > >Author, Web Analytics Demystified
                          > > >http://www.webanalyticsdemystified.com
                          > > >
                          > > >
                          > > >
                          > > >
                          > SPONSORED
                          > > LINKS
                          > > >
                          > > > Internet business Start internet Internet business
                          > > > plan business
                          > > >
                          > > > Internet business Internet home Internet business
                          > > > online business consulting
                          > > >
                          > > >
                          > > >
                          > > > YAHOO! GROUPS LINKS
                          > > >
                          > > > Visit your group "webanalytics" on the web.
                          > > >
                          > > > To unsubscribe from this group, send an email to:
                          > > > webanalytics-unsubscribe@yahoogroups.com
                          > > >
                          > > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                          > > Service.
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >---------------------------------------
                          > > >Web Metrics Discussion Group
                          > > >Moderated by Eric T. Peterson
                          > > >Author, Web Analytics Demystified
                          > > >http://www.webanalyticsdemystified.com
                          > > >Yahoo! Groups Links
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > > >
                          > >
                          > >
                          > > --
                          > > Andrew Edwards
                          > > Managing Partner
                          > > Technology Leaders
                          > > 230 Park Avenue
                          > > New York, NY 10169
                          > > (212) 808-3058
                          > > Cell: (917) 602-0083
                          > > aedwards@t...
                          > > www.technologyleaders.com
                          > >
                          > >
                          > >
                          > >
                          > >
                          > > ---------------------------------------
                          > > Web Metrics Discussion Group
                          > > Moderated by Eric T. Peterson
                          > > Author, Web Analytics Demystified
                          > > http://www.webanalyticsdemystified.com
                          > >
                          > >
                          > >
                          > >
                          > SPONSORED LINKS
                          > >
                          > > Internet business Start internet Internet business
                          > > plan business
                          > >
                          > > Internet business Internet home Internet business
                          > > online business consulting
                          > >
                          > >
                          > >
                          > > YAHOO! GROUPS LINKS
                          > >
                          > > Visit your group "webanalytics" on the web.
                          > >
                          > > To unsubscribe from this group, send an email to:
                          > > webanalytics-unsubscribe@yahoogroups.com
                          > >
                          > > Your use of Yahoo! Groups is subject to the Yahoo! Terms of
                          > Service.
                          > >
                          > >
                          > >
                          > >
                          > >
                          > >
                          > >
                          > > ---------------------------------------
                          > > Web Metrics Discussion Group
                          > > Moderated by Eric T. Peterson
                          > > Author, Web Analytics Demystified
                          > > http://www.webanalyticsdemystified.com
                          > > Yahoo! Groups Links
                          > >
                          >
                          >
                          >
                          >
                          >
                          >
                          >
                          >
                          > ---------------------------------------
                          > Web Metrics Discussion Group
                          > Moderated by Eric T. Peterson
                          > Author, Web Analytics Demystified
                          > http://www.webanalyticsdemystified.com
                          > Yahoo! Groups Links
                          >
                          >
                          >
                          >
                          >
                          >
                          >
                        Your message has been successfully submitted and would be delivered to recipients shortly.