Loading ...
Sorry, an error occurred while loading the content.

Re: [ARRL-LOTW] Duplicates

Expand Messages
  • Joe Subich, W4TV
    And Keith is a Luddite infected with the NIH syndrome. 73, ... Joe, W4TV
    Message 1 of 147 , Feb 19 7:06 PM
    • 0 Attachment
      And Keith is a Luddite infected with the NIH syndrome.

      73,

      ... Joe, W4TV


      On 2/19/2013 10:02 PM, Pete Ennis wrote:
      > He's full of it. His data is fake and so is he.
      >
      >
      > Keith
      >
      >
      > ________________________________
      > From: Bill Aycock <billaycock@...>
      > To: ARRL-LOTW@yahoogroups.com
      > Sent: Tuesday, February 19, 2013 7:50 PM
      > Subject: Re: [ARRL-LOTW] Duplicates
      >
      >
      >
      >
      > Joe--
      > I'm confused; where are these data found? I know of the "Status" report, but
      > that is an instantaneous sample that has no duration.
      > Bill--W4BSG
      >
      > -----Original Message-----
      > From: Joe Subich, W4TV
      > Sent: Tuesday, February 19, 2013 7:12 PM
      > To: mailto:ARRL-LOTW%40yahoogroups.com
      > Subject: Re: [ARRL-LOTW] Duplicates
      >
      >> Statistic also shows that when someone is being proved wrong they
      >> get testy.
      >
      > You've done nothing to prove anyone wrong - or prove anything other
      > than that you obviously have no understanding of statistics when it
      > comes to continuous (or periodic) processes. Your value of one per
      > second is arbitrary and bears no relationship to the processing time
      > for a given log - particularly when an individual log can be processed
      > in milliseconds (the processing time on my last 10 uploads was between
      > 6.9 and 8.9 milliseconds). A ample rate of one minute, one second,
      > 100 milliseconds, or 10 milliseconds is arbitrary. Once an hour - 168
      > samples per week - is more than sufficient for accuracy particularly
      > when a sample itself is an average of several logs. This sample size
      > actually represents more than 10% *of the logs* in a given week when
      > sample sizes smaller than 0.5% are more than large enough to provide
      > reasonable error margins.
      >
      > I have calculated the number of QSOs per log on a weekly basis back
      > through the week ending December 3, 1012 - and the recorded number
      > of new QSOs/logs for those weeks. Here are the *hard facts* for the
      > rate of previously processed (duplicates) on a weekly basis for the
      > last 12 weeks:
      >
      > Logs Average % Previously
      > Week Ending New QSOs Processed QSOs/Log Processed
      > ----------------------------------------------------------
      > 12/03/2012 1,825,662 17,225 315 66.3%
      > 12/10/2012 1,837,292 13,092 329 57.4%
      > 12/17/2012 1,416,710 15,288 317 70.8%
      > 12/24/2012 1,323,942 17,952 288 74.4%
      > 12/31/2012 1,040,081 12,815 332 75.6%
      > 01/07/2013 891,254 15,229 379 84.6%
      > 01/14/2013 4,422,746 53,411 300 72.4%
      > 01/31/2013 1,760,810 18,407 393 75.6%
      > 01/28/2013 1,367,417 22,025 576 89.2%
      > 02/04/2013 1,519,401 22,830 366 81.8%
      > 02/11/2013 1,587,253 24,483 419 84.5%
      > 02/18/2013 2,310,539 33,004 271 74.1%
      >
      > Now, unless you have some real data, you have nothing to contribute on
      > this issue. No matter how much you try to deny the facts, you have no
      > evidence that says duplicates are not a significant problem and no
      > amount of pretending that the earth is flat is going to make it so.
      >
      > 73,
      >
      > ... Joe, W4TV
      >
      > On 2/19/2013 6:13 PM, Pete Ennis wrote:
      >> Statistic also shows that when someone is being proved wrong they get
      >> testy. You are the one that came up with the 1700 samples. Over any
      >> given time LoTW can only process data so fast. So time of less than 1
      >> second on average is of little use here. So the sampling size is fixed at
      >> nearly 3600 samples. And any one with a very very basic knowledge of
      >> statistical analysis knows that the sample rate is too small be be of any
      >> use. But the big reason you have no usable data is the Queue Status page
      >> only gives data at an exact second of the day. This snap shot does not
      >> give any indication as to what in happening at any other point in time.
      >> There is no way to determine how many raw qsos were processed based on the
      >> Queue Status page. If LoTW is processing 1000 qsos a minute there could
      >> be any where from 0 to 60000 raw qsos processed during the past hour.
      >> There is NO data that shows how many raw qsos were processed. All we see
      >> is what is
      >> left at that given second. Nothing more nothing less. Don't tell me to
      >> "GET LOST" you are a nothing more than someone that thinks he "knows it
      >> all". Come back when you have some "REAL" numbers. Everyone can get
      >> upset at me if they want but I can't read your post knowing you are just
      >> making up numbers. No one in there right mind should believe 6+ out of 10
      >> qsos uploaded today are dupes.
      >>
      >> Have a great day,
      >>
      >> Keith
      >>
      >>
      >>
      >>
      >> ________________________________
      >> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
      >> To: mailto:ARRL-LOTW%40yahoogroups.com
      >> Sent: Tuesday, February 19, 2013 4:26 PM
      >> Subject: Re: [ARRL-LOTW] Duplicates
      >>
      >>
      >>
      >>
      >>> You don't have any usable data.
      >>
      >> You're absolutely wrong (as usual regarding LotW).
      >>
      >> The sample is not one based on frequency of sampling - one could define
      >> the number of possible samples as one per millisecond and the sample
      >> rate would be even lower. The real (desired) value is the number of
      >> QSOs in each upload. However, 1700 samples out of 250,000 logs since
      >> December 10, 2012 is *way more* than needed for significance. National
      >> opinion polling achieves a +/- 2% significance with a sample size of
      >> fewer than 5,000 of 310 million (less than 0.002%).
      >>
      >> Regular periodic data sampling is a staple of manufacturing quality
      >> control and those sample rates are far smaller than the 0.7% sample
      >> here. Further, the sample accuracy is enhanced because they are
      >> averages - not simply measuring the number of QSOs in a single log
      >> at fixed intervals.
      >>
      >> Now, until you care to provide any factual data and contribute in a
      >> positive way, *get lost*. Simply casting stones contributes nothing.
      >>
      >> 73,
      >>
      >> ... Joe, W4TV
      >>
      >> On 2/19/2013 4:59 PM, Pete Ennis wrote:
      >>> I do know a little something about statistical analysis. That's why I
      >>> asked. You don't have any usable data. First the 1700 hourly samples
      >>> have no value because it is impossible to know what is going on between
      >>> the hours. You get only 1 sample out of 3600 possible samples. Last I
      >>> checked that's far to few to be of any value. You know 60 times 60 is
      >>> 3600 absolute samples(seconds). Now that the system is for the most part
      >>> processing data faster than it is being sent. There is no way it
      >>> accurately tell how much raw data is being processed. From what I see,
      >>> the LoTW system is only being loaded 5-10% of the time the numbers are
      >>> showing a dupe rate of somewhere near 7% +/- 3 points.
      >>>
      >>> Thanks again,
      >>>
      >>> Keith
      >>>
      >>>
      >>> ________________________________
      >>> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
      >>> To: mailto:ARRL-LOTW%40yahoogroups.com
      >>> Sent: Tuesday, February 19, 2013 3:37 PM
      >>> Subject: Re: [ARRL-LOTW] Duplicates
      >>>
      >>>
      >>>
      >>>
      >>> I am using average number of QSOs/per upload times the number of files
      >>> processed. Simple sampling based statistics familiar to anyone who
      >>> has had a basic course in statistical analysis.
      >>>
      >>> There now are over 1700 hourly samples in the LotW queue status reports
      >>> on which to base the average number of QSOs per upload. The cumulative
      >>> number of files processed is available on the LotW home page. There
      >>> are more than enough samples to make any sampling error insignificant
      >>> in the analysis.
      >>>
      >>> 73,
      >>>
      >>> ... Joe, W4TV
      >>>
      >>> On 2/19/2013 4:08 PM, Pete Ennis wrote:
      >>>> Joe,
      >>>>
      >>>> Tell us where you are getting the exact number of raw QSO's that
      >>>> are being uploaded???? The ones that include the dupes and new/
      >>>> modified QSOs???
      >>>>
      >>>> Thanks,
      >>>>
      >>>> Keith
      >>>>
      >>>>
      >>>> ________________________________
      >>>> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
      >>>> To: mailto:ARRL-LOTW%40yahoogroups.com
      >>>> Cc: VE6LB mailto:ve6lb%40telus.net>
      >>>> Sent: Tuesday, February 19, 2013 1:45 PM
      >>>> Subject: [ARRL-LOTW] Duplicates (was: Re: User Friendly LoTW)
      >>>>
      >>>>
      >>>>
      >>>>
      >>>>
      >>>>
      >>>> On 2/19/2013 12:04 PM, VE6LB wrote:
      >>>>>
      >>>>> Now that processing is lightning fast, I wonder how many duplicate
      >>>>> uploads happen.
      >>>>>
      >>>>> Gerry VE6LB
      >>>>>
      >>>>
      >>>> The duplicate rate is, if anything, greater than it was before the new
      >>>> hardware went on-line. In the last four weeks, the number of uploads
      >>>> per day is running about 58% ahead of the rate prior to the new hardware
      >>>> while the umber of new QSO records per day is only up 15%. That means
      >>>> the rate of previously processed QSOs has increased by 28% over what it
      >>>> was during the period immediately before the new hardware went on-line
      >>>> (as the average number of QSOs/log in the queue has remained nearly
      >>>> constant).
      >>>>
      >>>> A 28% increase in the previously processed QSO rate - based on K1MK's
      >>>> statement that "half the QSOs being processed were duplicates" before
      >>>> the hardware change - would indicate that nearly 2/3 of all uploads
      >>>> are now duplicates. The exact mix will change from day to day - the
      >>>> proportion of "new QSO records" is consistently higher on Monday and
      >>>> Tuesday after a major contest weekend and is consistently highest on
      >>>> Monday/Tuesday after a bigger contest than after some of the less
      >>>> popular events.
      >>>>
      >>>> In any case, the indications are that many of those who do not use
      >>>> logging programs specifically designed to track LotW uploads and
      >>>> prevent duplicates continue to export and sign their entire log without
      >>>> using tQSL's date range feature.
      >>>>
      >>>> 73,
      >>>>
      >>>> ... Joe, W4TV
      >>>>
      >>>>
      >>>>
      >>>>
      >>>
      >>>
      >>>
      >>
      >>
      >>
      >
      > ------------------------------------
      >
      > Yahoo! Groups Links
      >
      >
      >
      >
    • David Cole
      I run ACLog... When you hit the ALL SINCE button, change the date to be something about a week prior to the LoTW failure... I believe , ACLog got very
      Message 147 of 147 , Aug 24, 2014
      • 0 Attachment
        I run ACLog... When you hit the "ALL SINCE" button, change the date to
        be something about a week prior to the LoTW failure...

        I "believe", ACLog got very confused as a result of the fail mode of
        LoTW. That corrected a very similar problem for me.
        --
        Thanks and 73's,
        For equipment, and software setups and reviews see:
        www.nk7z.net
        for MixW support see;
        http://groups.yahoo.com/neo/groups/mixw/info
        for Dopplergram information see:
        http://groups.yahoo.com/neo/groups/dopplergram/info
        for MM-SSTV see:
        http://groups.yahoo.com/neo/groups/MM-SSTV/info


        On Sun, 2014-08-24 at 09:05 -0700, reillyjf@... [ARRL-LOTW]
        wrote:
        >
        >
        > Thanks for the suggestion. I did a complete download, and beat the
        > number of duplicates down from 275 to 30. No exactly sure why the
        > N3FJP ACL is missing this information.
        > - 73, John, N0TA
        >
        >
        >
      Your message has been successfully submitted and would be delivered to recipients shortly.