Loading ...
Sorry, an error occurred while loading the content.

Web Trends not filtering out visits from spiders

Expand Messages
  • Dan
    Hi, We have been having an issue with a client who s using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and
    Message 1 of 7 , Nov 4, 2011
    • 0 Attachment
      Hi,

      We have been having an issue with a client who's using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and inflating the figures.

      In some cases reports are showing that spiders visits are inflating traffic by over 100%.

      Is this a known problem with WebTrends and is there anything we can do to address the problem, or do we recommend a switch to a tool that does strip out these visits?

      Any help much appreciated

      Thanks

      Dan
    • Rubix
      I m actually really surprised to hear this. I m not familiar with WebTrends, but Omniture data tends to be much more reliable than my company s log data as a
      Message 2 of 7 , Nov 6, 2011
      • 0 Attachment
        I'm actually really surprised to hear this. I'm not familiar with WebTrends, but Omniture data tends to be much more reliable than my company's log data as a reflection of actual human activity. The reason for this is that spiders/scripters/bots/etc. are most likely not using JavaScript-enabled clients so that activity simply never gets recorded in the first place.

        What is WebTrends' tracking mechanism?

        --- In webanalytics@yahoogroups.com, "Dan" <dan@...> wrote:
        >
        > Hi,
        >
        > We have been having an issue with a client who's using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and inflating the figures.
        >
        > In some cases reports are showing that spiders visits are inflating traffic by over 100%.
        >
        > Is this a known problem with WebTrends and is there anything we can do to address the problem, or do we recommend a switch to a tool that does strip out these visits?
        >
        > Any help much appreciated
        >
        > Thanks
        >
        > Dan
        >
      • gamerpatrick
        Dan, I d be really surprised if your conclusions about spider inflation are accurate. What indicators have lead you to this conclusion? - Patrick Senior Web
        Message 3 of 7 , Nov 7, 2011
        • 0 Attachment
          Dan,

          I'd be really surprised if your conclusions about spider inflation are accurate. What indicators have lead you to this conclusion?

          - Patrick
          Senior Web Analytics Consultant
          http://unilytics.com
          &
          http://pmazzotta.com



          --- In webanalytics@yahoogroups.com, "Rubix" <rubix606@...> wrote:
          >
          > I'm actually really surprised to hear this. I'm not familiar with WebTrends, but Omniture data tends to be much more reliable than my company's log data as a reflection of actual human activity. The reason for this is that spiders/scripters/bots/etc. are most likely not using JavaScript-enabled clients so that activity simply never gets recorded in the first place.
          >
          > What is WebTrends' tracking mechanism?
          >
          > --- In webanalytics@yahoogroups.com, "Dan" <dan@> wrote:
          > >
          > > Hi,
          > >
          > > We have been having an issue with a client who's using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and inflating the figures.
          > >
          > > In some cases reports are showing that spiders visits are inflating traffic by over 100%.
          > >
          > > Is this a known problem with WebTrends and is there anything we can do to address the problem, or do we recommend a switch to a tool that does strip out these visits?
          > >
          > > Any help much appreciated
          > >
          > > Thanks
          > >
          > > Dan
          > >
          >
        • Bob Willis
          they are probably analyzing web log files and not the WebTrends js created log files. I think you can just update the bot/spider definition and filter this
          Message 4 of 7 , Nov 7, 2011
          • 0 Attachment
            they are probably analyzing web log files and not the WebTrends js created log files.
            I think you can just update the bot/spider definition and filter this out.

            --- On Sun, 11/6/11, Rubix <rubix606@...> wrote:


            From: Rubix <rubix606@...>
            Subject: [webanalytics] Re: Web Trends not filtering out visits from spiders
            To: webanalytics@yahoogroups.com
            Date: Sunday, November 6, 2011, 12:34 PM



             



            I'm actually really surprised to hear this. I'm not familiar with WebTrends, but Omniture data tends to be much more reliable than my company's log data as a reflection of actual human activity. The reason for this is that spiders/scripters/bots/etc. are most likely not using JavaScript-enabled clients so that activity simply never gets recorded in the first place.

            What is WebTrends' tracking mechanism?

            --- In webanalytics@yahoogroups.com, "Dan" <dan@...> wrote:
            >
            > Hi,
            >
            > We have been having an issue with a client who's using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and inflating the figures.
            >
            > In some cases reports are showing that spiders visits are inflating traffic by over 100%.
            >
            > Is this a known problem with WebTrends and is there anything we can do to address the problem, or do we recommend a switch to a tool that does strip out these visits?
            >
            > Any help much appreciated
            >
            > Thanks
            >
            > Dan
            >








            [Non-text portions of this message have been removed]
          • Wes Contreras
            ... You can create a Webtrends filter for excluding spider and bot traffic. To make one, you ll want to create a hit filter based on Browser, then select
            Message 5 of 7 , Nov 7, 2011
            • 0 Attachment
              On Fri, Nov 4, 2011 at 3:46 AM, Dan <dan@...> wrote:
              >
              > We have been having an issue with a client who's using WebTrends. It appears
              > that WebTrends has been reporting site traffic including spider visits and inflating
              > the figures.
              >
              > In some cases reports are showing that spiders visits are inflating traffic by over 100%.
              >
              > Is this a known problem with WebTrends and is there anything we can do to address
              > the problem, or do we recommend a switch to a tool that does strip out these visits?

              You can create a Webtrends filter for excluding spider and bot
              traffic. To make one, you'll want to create a "hit filter" based on
              Browser, then select "All Spiders & Robots" from the drop-down list.
              If you're using the On Premise (software) version of the product,
              you'll want to keep your local definition file for spiders and bots
              current for this to work best:
              http://kb.webtrends.com/support/solutiondetail.aspx?Id=501400000009whvAAA

              As noted, most spiders and bots don't execute JavaScript, but a few
              do, so this filter can still be useful. This is doubly true if you're
              analyzing server logfiles.


              --
              Wes Contreras
              Disclaimer: I am a Webtrends employee
            • jing.suk@thomsonreuters.com
              To be fair to WebTrends, I don’t think what your client encountered is a vendor issue. I’m assuming your client has the software version of WebTrends
              Message 6 of 7 , Nov 7, 2011
              • 0 Attachment
                To be fair to WebTrends, I don’t think what your client encountered is a vendor issue. I’m assuming your client has the software version of WebTrends instead of using WebTrends ondemand services. When using the software version of WebTrends, your client would need to manage the bots/spiders filtering rules within the WebTrends admin console. If they’re using the ondemand version, then WebTrends’s technical team would manage this. At my previous position, we used WebTrends ondemand for 5+ years, and we audit our site traffic, WebTrends didn’t report any spider/bots traffic.

                Best,
                Jing



                From: webanalytics@yahoogroups.com [mailto:webanalytics@yahoogroups.com] On Behalf Of Bob Willis
                Sent: Monday, November 07, 2011 9:33 AM
                To: webanalytics@yahoogroups.com
                Subject: Re: [webanalytics] Re: Web Trends not filtering out visits from spiders



                they are probably analyzing web log files and not the WebTrends js created log files.
                I think you can just update the bot/spider definition and filter this out.

                --- On Sun, 11/6/11, Rubix <rubix606@...<mailto:rubix606%40yahoo.com>> wrote:

                From: Rubix <rubix606@...<mailto:rubix606%40yahoo.com>>
                Subject: [webanalytics] Re: Web Trends not filtering out visits from spiders
                To: webanalytics@yahoogroups.com<mailto:webanalytics%40yahoogroups.com>
                Date: Sunday, November 6, 2011, 12:34 PM



                I'm actually really surprised to hear this. I'm not familiar with WebTrends, but Omniture data tends to be much more reliable than my company's log data as a reflection of actual human activity. The reason for this is that spiders/scripters/bots/etc. are most likely not using JavaScript-enabled clients so that activity simply never gets recorded in the first place.

                What is WebTrends' tracking mechanism?

                --- In webanalytics@yahoogroups.com<mailto:webanalytics%40yahoogroups.com>, "Dan" <dan@...> wrote:
                >
                > Hi,
                >
                > We have been having an issue with a client who's using WebTrends. It appears that WebTrends has been reporting site traffic including spider visits and inflating the figures.
                >
                > In some cases reports are showing that spiders visits are inflating traffic by over 100%.
                >
                > Is this a known problem with WebTrends and is there anything we can do to address the problem, or do we recommend a switch to a tool that does strip out these visits?
                >
                > Any help much appreciated
                >
                > Thanks
                >
                > Dan
                >

                [Non-text portions of this message have been removed]


                This email was sent to you by Thomson Reuters, the global news and information company. Any views expressed in this message are those of the individual sender, except where the sender specifically states them to be the views of Thomson Reuters.

                [Non-text portions of this message have been removed]
              • Matthias Bettag
                Hi, Webtrends can filter spiders, it is an optional setting of the profile configuration. However, I experienced some rare issues where a not-yet blacklisted
                Message 7 of 7 , Nov 7, 2011
                • 0 Attachment
                  Hi,

                  Webtrends can filter spiders, it is an optional setting of the profile
                  configuration.
                  However, I experienced some rare issues where a not-yet blacklisted
                  spider faked it's user agent and by this passed the filter.

                  best,
                  Matthias
                Your message has been successfully submitted and would be delivered to recipients shortly.