Re: [ARRL-LOTW] Duplicates
- And Keith is a Luddite infected with the NIH syndrome.
... Joe, W4TV
On 2/19/2013 10:02 PM, Pete Ennis wrote:
> He's full of it. His data is fake and so is he.
> From: Bill Aycock <billaycock@...>
> To: ARRL-LOTW@yahoogroups.com
> Sent: Tuesday, February 19, 2013 7:50 PM
> Subject: Re: [ARRL-LOTW] Duplicates
> I'm confused; where are these data found? I know of the "Status" report, but
> that is an instantaneous sample that has no duration.
> -----Original Message-----
> From: Joe Subich, W4TV
> Sent: Tuesday, February 19, 2013 7:12 PM
> To: mailto:ARRL-LOTW%40yahoogroups.com
> Subject: Re: [ARRL-LOTW] Duplicates
>> Statistic also shows that when someone is being proved wrong they
>> get testy.
> You've done nothing to prove anyone wrong - or prove anything other
> than that you obviously have no understanding of statistics when it
> comes to continuous (or periodic) processes. Your value of one per
> second is arbitrary and bears no relationship to the processing time
> for a given log - particularly when an individual log can be processed
> in milliseconds (the processing time on my last 10 uploads was between
> 6.9 and 8.9 milliseconds). A ample rate of one minute, one second,
> 100 milliseconds, or 10 milliseconds is arbitrary. Once an hour - 168
> samples per week - is more than sufficient for accuracy particularly
> when a sample itself is an average of several logs. This sample size
> actually represents more than 10% *of the logs* in a given week when
> sample sizes smaller than 0.5% are more than large enough to provide
> reasonable error margins.
> I have calculated the number of QSOs per log on a weekly basis back
> through the week ending December 3, 1012 - and the recorded number
> of new QSOs/logs for those weeks. Here are the *hard facts* for the
> rate of previously processed (duplicates) on a weekly basis for the
> last 12 weeks:
> Logs Average % Previously
> Week Ending New QSOs Processed QSOs/Log Processed
> 12/03/2012 1,825,662 17,225 315 66.3%
> 12/10/2012 1,837,292 13,092 329 57.4%
> 12/17/2012 1,416,710 15,288 317 70.8%
> 12/24/2012 1,323,942 17,952 288 74.4%
> 12/31/2012 1,040,081 12,815 332 75.6%
> 01/07/2013 891,254 15,229 379 84.6%
> 01/14/2013 4,422,746 53,411 300 72.4%
> 01/31/2013 1,760,810 18,407 393 75.6%
> 01/28/2013 1,367,417 22,025 576 89.2%
> 02/04/2013 1,519,401 22,830 366 81.8%
> 02/11/2013 1,587,253 24,483 419 84.5%
> 02/18/2013 2,310,539 33,004 271 74.1%
> Now, unless you have some real data, you have nothing to contribute on
> this issue. No matter how much you try to deny the facts, you have no
> evidence that says duplicates are not a significant problem and no
> amount of pretending that the earth is flat is going to make it so.
> ... Joe, W4TV
> On 2/19/2013 6:13 PM, Pete Ennis wrote:
>> Statistic also shows that when someone is being proved wrong they get
>> testy. You are the one that came up with the 1700 samples. Over any
>> given time LoTW can only process data so fast. So time of less than 1
>> second on average is of little use here. So the sampling size is fixed at
>> nearly 3600 samples. And any one with a very very basic knowledge of
>> statistical analysis knows that the sample rate is too small be be of any
>> use. But the big reason you have no usable data is the Queue Status page
>> only gives data at an exact second of the day. This snap shot does not
>> give any indication as to what in happening at any other point in time.
>> There is no way to determine how many raw qsos were processed based on the
>> Queue Status page. If LoTW is processing 1000 qsos a minute there could
>> be any where from 0 to 60000 raw qsos processed during the past hour.
>> There is NO data that shows how many raw qsos were processed. All we see
>> is what is
>> left at that given second. Nothing more nothing less. Don't tell me to
>> "GET LOST" you are a nothing more than someone that thinks he "knows it
>> all". Come back when you have some "REAL" numbers. Everyone can get
>> upset at me if they want but I can't read your post knowing you are just
>> making up numbers. No one in there right mind should believe 6+ out of 10
>> qsos uploaded today are dupes.
>> Have a great day,
>> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
>> To: mailto:ARRL-LOTW%40yahoogroups.com
>> Sent: Tuesday, February 19, 2013 4:26 PM
>> Subject: Re: [ARRL-LOTW] Duplicates
>>> You don't have any usable data.
>> You're absolutely wrong (as usual regarding LotW).
>> The sample is not one based on frequency of sampling - one could define
>> the number of possible samples as one per millisecond and the sample
>> rate would be even lower. The real (desired) value is the number of
>> QSOs in each upload. However, 1700 samples out of 250,000 logs since
>> December 10, 2012 is *way more* than needed for significance. National
>> opinion polling achieves a +/- 2% significance with a sample size of
>> fewer than 5,000 of 310 million (less than 0.002%).
>> Regular periodic data sampling is a staple of manufacturing quality
>> control and those sample rates are far smaller than the 0.7% sample
>> here. Further, the sample accuracy is enhanced because they are
>> averages - not simply measuring the number of QSOs in a single log
>> at fixed intervals.
>> Now, until you care to provide any factual data and contribute in a
>> positive way, *get lost*. Simply casting stones contributes nothing.
>> ... Joe, W4TV
>> On 2/19/2013 4:59 PM, Pete Ennis wrote:
>>> I do know a little something about statistical analysis. That's why I
>>> asked. You don't have any usable data. First the 1700 hourly samples
>>> have no value because it is impossible to know what is going on between
>>> the hours. You get only 1 sample out of 3600 possible samples. Last I
>>> checked that's far to few to be of any value. You know 60 times 60 is
>>> 3600 absolute samples(seconds). Now that the system is for the most part
>>> processing data faster than it is being sent. There is no way it
>>> accurately tell how much raw data is being processed. From what I see,
>>> the LoTW system is only being loaded 5-10% of the time the numbers are
>>> showing a dupe rate of somewhere near 7% +/- 3 points.
>>> Thanks again,
>>> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
>>> To: mailto:ARRL-LOTW%40yahoogroups.com
>>> Sent: Tuesday, February 19, 2013 3:37 PM
>>> Subject: Re: [ARRL-LOTW] Duplicates
>>> I am using average number of QSOs/per upload times the number of files
>>> processed. Simple sampling based statistics familiar to anyone who
>>> has had a basic course in statistical analysis.
>>> There now are over 1700 hourly samples in the LotW queue status reports
>>> on which to base the average number of QSOs per upload. The cumulative
>>> number of files processed is available on the LotW home page. There
>>> are more than enough samples to make any sampling error insignificant
>>> in the analysis.
>>> ... Joe, W4TV
>>> On 2/19/2013 4:08 PM, Pete Ennis wrote:
>>>> Tell us where you are getting the exact number of raw QSO's that
>>>> are being uploaded???? The ones that include the dupes and new/
>>>> modified QSOs???
>>>> From: "Joe Subich, W4TV" mailto:lists%40subich.com>
>>>> To: mailto:ARRL-LOTW%40yahoogroups.com
>>>> Cc: VE6LB mailto:ve6lb%40telus.net>
>>>> Sent: Tuesday, February 19, 2013 1:45 PM
>>>> Subject: [ARRL-LOTW] Duplicates (was: Re: User Friendly LoTW)
>>>> On 2/19/2013 12:04 PM, VE6LB wrote:
>>>>> Now that processing is lightning fast, I wonder how many duplicate
>>>>> uploads happen.
>>>>> Gerry VE6LB
>>>> The duplicate rate is, if anything, greater than it was before the new
>>>> hardware went on-line. In the last four weeks, the number of uploads
>>>> per day is running about 58% ahead of the rate prior to the new hardware
>>>> while the umber of new QSO records per day is only up 15%. That means
>>>> the rate of previously processed QSOs has increased by 28% over what it
>>>> was during the period immediately before the new hardware went on-line
>>>> (as the average number of QSOs/log in the queue has remained nearly
>>>> A 28% increase in the previously processed QSO rate - based on K1MK's
>>>> statement that "half the QSOs being processed were duplicates" before
>>>> the hardware change - would indicate that nearly 2/3 of all uploads
>>>> are now duplicates. The exact mix will change from day to day - the
>>>> proportion of "new QSO records" is consistently higher on Monday and
>>>> Tuesday after a major contest weekend and is consistently highest on
>>>> Monday/Tuesday after a bigger contest than after some of the less
>>>> popular events.
>>>> In any case, the indications are that many of those who do not use
>>>> logging programs specifically designed to track LotW uploads and
>>>> prevent duplicates continue to export and sign their entire log without
>>>> using tQSL's date range feature.
>>>> ... Joe, W4TV
> Yahoo! Groups Links
- I run ACLog... When you hit the "ALL SINCE" button, change the date to
be something about a week prior to the LoTW failure...
I "believe", ACLog got very confused as a result of the fail mode of
LoTW. That corrected a very similar problem for me.
Thanks and 73's,
For equipment, and software setups and reviews see:
for MixW support see;
for Dopplergram information see:
for MM-SSTV see:
On Sun, 2014-08-24 at 09:05 -0700, reillyjf@... [ARRL-LOTW]
> Thanks for the suggestion. I did a complete download, and beat the
> number of duplicates down from 275 to 30. No exactly sure why the
> N3FJP ACL is missing this information.
> - 73, John, N0TA