I'm also curious about what your testing. Because you talked about
being concerned that sample sizes needed might be too small. You
might be right, if your customers are biased on how they utilize your
website. Some examples of this might be customers don't purchase on
weekdays but do purchase on weekends. Customer do one kind of
behavior at the beginning of a month and an entrirely different type
of behavior at the end of the month. You have business & individual
customers at different intervals.
Basically, testing is complicated if the behavior your trying to
influence varies over time? Website are transiantly visited unlike
mailings. This makes it harder to analyze.
My experience has shown that testing longer is more accurate than
larger sample sizes when the reason for the behavior changes with
time. Understand the behavior your trying to influence to understand
the timing for the test.
--- In email@example.com
, "nevertrustab" <patriccc@...>
> Hi Philip,
> just out of curiousity: What are you trying to test?
> --- In firstname.lastname@example.org, "Philip Shannon"
> <philip.shannon@> wrote:
> > Basic stats' problem:
> > I will be running a test version of my homepage (2,000,000 monthly
> > visits). How long (or how many visits) will I need to test to be
> > statistically significant (confidence level 95%)? I will be
> > 100% of my visitors to the test (I won't be running an a/b test -
> > using historical data as my baseline).
> > According to the online calculators and equations I have found, I
> > need a few days of testing. I find that hard to believe...
> > Thanks a lot!