Loading ...
Sorry, an error occurred while loading the content.

Re: [ARRL-LOTW] An odd occurrence ...

Expand Messages
  • Robert Chudek - K0RC
    Hello Gerry... You would use FILE EXPORT ADIF TO FILE EXPORT ADIF TO FILE BY DATE Then when you read the helper menu that pops up asking for the date to
    Message 1 of 25 , Sep 3, 2012
    • 0 Attachment
      Hello Gerry...

      You would use FILE > EXPORT ADIF TO FILE > EXPORT ADIF TO FILE BY DATE

      Then when you read the helper menu that pops up asking for the date to begin an export, it says it will use 01-01-1900 for the start date if you have never exported from that database before. OR the date field will auto-populate with the date of your last export.

      In general, using the Export by Date option every time is what you want to use to avoid re-uploading the entire log.

      The only thing this won't catch is if you make a correction to a QSO that was in a previous upload, that QSO will not be captured (as far as I know). In that case you can back-date the date field to catch any QSO(s) that fall into that category.

      73 de Bob - KØRC in MN


      On 9/3/2012 8:15 PM, VE6LB-Gerry wrote:
       

       
      Bob,
       
      Please point me to this selection. Looked but I can't find it.
       
      73, Gerry VE6LB

      Sent: Monday, September 03, 2012 5:03 PM
      Subject: Re: [ARRL-LOTW] An odd occurrence ...

       

      I'll pile on regarding the comment "I use N1MM when I'm in A6 and have no choice but to upload my total log."

      This is not true. The N1MM Logger has an ADIF export selection that keeps track of your last QSO exported. When you select the correct ADIF export menu item, the export will resume with the next QSO found in the log since the last export. There is no need to export the entire log every time.

      Regarding misleading users with repetitive uploads of an entire log, that is one consequence. But this strategy also wastes LoTW resources when LoTW has to exam every QSO in the file and parse the old data. There is no free lunch regarding CPU cycles. LoTW consumes CPU time examining  your repetitive data, slowing everyone's log processing.

      73 de Bob - KØRC in MN


      On 9/3/2012 3:45 PM, Vladimir Sidarau wrote:
       
      Sorry, Gerry, but...
       
      - Reloadings of logs are quite missleading for users. It's nice to see a new bandslot just confirmed (likely), but then it's not really nice to find out that the overall number did not change and the likely new bandslot confirmation is in fact just another re-upload.But OK, that's just emotions. 
       
      - Now let me quote the ARRL LoTW site.
      ---
      Aug 19, 2012: Logbook of The World Processing is Current -- Log submission guidance:  Please refrain from uploading files more than once as this only adds to the processing time.  If your TQ8 file has been queued for processing it will be processed.
      When creating TQ8 files please include new QSOs only.  Uploading previously uploaded QSOs or uploading your entire log again only adds to the processing time.
      ---
      Once ahain, it is a message posted by the LoTW administration. As long as they have decided to post such a "guidance", the multiple uploads is a problem serious enough for the LoTW system, therefore the LoTW administration tries to avoid it. 
       
      73,
      Vladimir VE3IAE
       
      ---
       
       
      ----- Original Message -----
      Sent: Monday, September 03, 2012 4:21 PM
      Subject: Re: [ARRL-LOTW] An odd occurrence ...

       
       
      Nothing wrong with reloading your log. LoTW throws away the dups so it just extra bandwidth wasted. Many logging programs do not keep track of what you've uploaded. I use N1MM when I'm in A6 and have no choice but to upload my total log. I could do a bunch of manual book keeping but that takes away from Q time.
       
      Gerry VE6LB (A6/VE6LB) 

      From: Gary AL9A
      Sent: Monday, September 03, 2012 1:14 PM
      Subject: Re: [ARRL-LOTW] An odd occurrence ...

       
      That's what he's doing all right.  I have two QSOs confirmed with him dating from 2004 and 2005.  My log shows the LOTW confirmation for both QSOs to be 8/31/12.  Today I downloaded a new LOTW report and both TI3MAO QSLs are in that file again, this time with a date of 9/3/12.
       
      He appears to be a tech savvy guy as he has a blog setup at http://ti3mao.wordpress.com/ that features pictures of his shack, antennas and his new PC just acquired in July 2012 which allows him to "...keep my logbook updated and uploaded to LOTW."  I will drop him an email and "suggest" that he not upload the entire log each time!
       
      73,
      Gary AL9A
       
       
      ----- Original Message -----
      Sent: September 03, 2012 10:29 AM
      Subject: Re: [ARRL-LOTW] An odd occurrence ...

       
       
      I think he's reloading his total log every few days. Check the date stamp in "details".
       
      Gerry VE6LB

      Sent: Monday, September 03, 2012 12:22 PM
      Subject: [ARRL-LOTW] An odd occurrence ...

       

      Hi Gang,

      Perhaps someone on the list can shed some light on this.   For about the last week, a QSL from TI3MAO for a 20M SSB QSO keeps popping to the top of my most recent QSL list.   When I check the QSL time/date, it is very current and should be at the top of the list.   And a couple of days later, it disappears from lower in my QSL list and shows up at the top again, with a newer time/date for the QSL.

      This has me puzzled and I’d sure appreciate any insights you might have.

      - 73 de Mike, K6MKF, W6NAG, VP NCDXC, Conway Reef 2012, Dachshund Rescue, Maui



  • Tim
    A couple of notes of my own, having re-uploaded at least once myself: 1: Re-uploading an identical individual QSO record, one that had already been uploaded
    Message 2 of 25 , Sep 15, 2012
    • 0 Attachment
      A couple of notes of my own, having re-uploaded at least once myself:

      1: Re-uploading an identical individual QSO record, one that had already been uploaded before, does not cause a new QSL to "go to the top" of the most recent. As far as I can tell, the robot that processes incoming logs notices that the new record is identical and just passes over it.

      I'm pretty sure (1) is true because if I re-load the same ADI, I get a message in "your activity/results" saying that so-and-so QSO's were already loaded and not reloaded.

      2: Re-uploading the same QSO, but with different ADI fields, and thus a different QSO record, may cause a previously issued QSL to be "re-issued" with a new QSL timestamp. I think that some field of the ADI has to change for this to happen. I'm not sure which fields.

      I found out (2) above by the school of hard knocks when I used two different ADI exports from two different logging programs for the same QSO set, and one of them sent full frequency info (e.g. 7.038 MHz) and the other rounded off to the band edge because the QSO set had previously been put into that form for Cabrillo contest log (e.g. same QSO was just "7000" in an intermediate Cabrillo log which had been re-imported.) Confusing enough? !!!!

      Tim N3QE

      --- In ARRL-LOTW@yahoogroups.com, "Mike Flowers" <mike.flowers@...> wrote:
      >
      > Hi Gang,
      >
      >
      >
      > Perhaps someone on the list can shed some light on this. For about the
      > last week, a QSL from TI3MAO for a 20M SSB QSO keeps popping to the top of
      > my most recent QSL list. When I check the QSL time/date, it is very
      > current and should be at the top of the list. And a couple of days later,
      > it disappears from lower in my QSL list and shows up at the top again, with
      > a newer time/date for the QSL.
      >
      >
      >
      > This has me puzzled and I'd sure appreciate any insights you might have.
      >
      >
      >
      > - 73 de Mike, <http://www.qrz.com/db/k6mkf/> K6MKF,
      > <http://www.qrz.com/db/w6nag/> W6NAG, VP <http://www.ncdxc.org/> NCDXC,
      > <http://www.yt1ad.info/3d2c/index.html> Conway Reef 2012,
      > <http://www.scdr.org/> Dachshund Rescue, <http://www.waiohuli.com/> Maui
      >
    • Rick Murphy/K1MU
      ... TrustedQSL only reads the following fields from an ADIF file: CALL BAND BAND_RX MODE FREQ FREQ_RX QSO_DATE TIME_ON SAT_NAME PROP_MODE Location information
      Message 3 of 25 , Sep 15, 2012
      • 0 Attachment
        At 07:01 AM 9/15/2012, Tim wrote:
        >
        >2: Re-uploading the same QSO, but with different ADI fields, and thus
        >a different QSO record, may cause a previously issued QSL to be
        >"re-issued" with a new QSL timestamp. I think that some field of the
        >ADI has to change for this to happen. I'm not sure which fields.

        TrustedQSL only reads the following fields from an ADIF file:

        CALL
        BAND
        BAND_RX
        MODE
        FREQ
        FREQ_RX
        QSO_DATE
        TIME_ON
        SAT_NAME
        PROP_MODE

        Location information is read from your station definition and applied
        to the entire log.

        Anything else that changes in your ADIF file won't change an existing
        record. I assume that if you leave the date and time the same that a
        new upload replaces the existing one, otherwise it creates a new QSO
        record.

        This doesn't explain the situations where people see QSLs being
        repeatedly moved to the top of their list. It's been assumed that this
        was due to stations re-uploading their entire log, but LoTW should be
        ignoring the duplicates. Unless there's something they're changing in
        the QSO data (or location info) I can't explain how new QSLs keep
        getting created.

        It's possible that the 'new QSL' behavior is now fixed by LoTW ignoring
        duplicate QSOs.
        73,
        -Rick
      • w4lde
        Rick, I still get dup s from time to time. So I do not consider it fixed, hi. 73 de Ron W4LDE
        Message 4 of 25 , Sep 15, 2012
        • 0 Attachment
          Rick,

          I still get dup's from time to time.  So I do not consider it fixed, hi.

          73 de
          Ron W4LDE


          On 9/15/2012 9:25 AM, Rick Murphy/K1MU wrote:
           

          At 07:01 AM 9/15/2012, Tim wrote:
          >
          >2: Re-uploading the same QSO, but with different ADI fields, and thus
          >a different QSO record, may cause a previously issued QSL to be
          >"re-issued" with a new QSL timestamp. I think that some field of the
          >ADI has to change for this to happen. I'm not sure which fields.

          TrustedQSL only reads the following fields from an ADIF file:

          CALL
          BAND
          BAND_RX
          MODE
          FREQ
          FREQ_RX
          QSO_DATE
          TIME_ON
          SAT_NAME
          PROP_MODE

          Location information is read from your station definition and applied
          to the entire log.

          Anything else that changes in your ADIF file won't change an existing
          record. I assume that if you leave the date and time the same that a
          new upload replaces the existing one, otherwise it creates a new QSO
          record.

          This doesn't explain the situations where people see QSLs being
          repeatedly moved to the top of their list. It's been assumed that this
          was due to stations re-uploading their entire log, but LoTW should be
          ignoring the duplicates. Unless there's something they're changing in
          the QSO data (or location info) I can't explain how new QSLs keep
          getting created.

          It's possible that the 'new QSL' behavior is now fixed by LoTW ignoring
          duplicate QSOs.
          73,
          -Rick


        • Radio K0HB
          Why would you care? 73, de Hans, K0HB -- Just a boy and his radio -- Proud Member of: A1 Operators - http://www.arrl.org/a-1-op Minnesota Wireless contesters
          Message 5 of 25 , Sep 15, 2012
          • 0 Attachment
            Why would you care?
             
             
            73, de Hans, K0HB
            --
            "Just a boy and his radio"
            --
            Proud Member of:
            A1 Operators - http://www.arrl.org/a-1-op
            Minnesota Wireless contesters - http://www.W0AA.org
            Arizona Outlaws contesters - http://www.arizonaoutlaws.net
            Twin City DX Assn - http://www.tcdxa.org
            Lake Vermilion DX Assn - http://www.lvdxa.org
            CWOps - http://www.cwops.org
            SOC - http://www.qsl.net/soc
            Twin City FM Club - http://tcfmc.org
            --
            Superstition trails --> http://oldslowhans.wordpress.com/
            Sea stories here ---> http://k0hb.wordpress.com/
            Request QSL at ---> http://www.clublog.org/logsearch/K0HB
            All valid QSL requests honored with old fashioned paper QSL!
            LoTW participant
             

             
            From: w4lde
            Sent: Saturday, September 15, 2012 6:07 PM
            Subject: Re: [ARRL-LOTW] Re: An odd occurrence ...
             

            Rick,

            I still get dup's from time to time.  So I do not consider it fixed, hi.

            73 de
            Ron W4LDE


            On 9/15/2012 9:25 AM, Rick Murphy/K1MU wrote:
             

            At 07:01 AM 9/15/2012, Tim wrote:
            >
            >2: Re-uploading the same QSO, but with different ADI fields, and thus
            >a different QSO record, may cause a previously issued QSL to be
            >"re-issued" with a new QSL timestamp. I think that some field of the
            >ADI has to change for this to happen. I'm not sure which fields.

            TrustedQSL only reads the following fields from an ADIF file:

            CALL
            BAND
            BAND_RX
            MODE
            FREQ
            FREQ_RX
            QSO_DATE
            TIME_ON
            SAT_NAME
            PROP_MODE

            Location information is read from your station definition and applied
            to the entire log.

            Anything else that changes in your ADIF file won't change an existing
            record. I assume that if you leave the date and time the same that a
            new upload replaces the existing one, otherwise it creates a new QSO
            record.

            This doesn't explain the situations where people see QSLs being
            repeatedly moved to the top of their list. It's been assumed that this
            was due to stations re-uploading their entire log, but LoTW should be
            ignoring the duplicates. Unless there's something they're changing in
            the QSO data (or location info) I can't explain how new QSLs keep
            getting created.

            It's possible that the 'new QSL' behavior is now fixed by LoTW ignoring
            duplicate QSOs.
            73,
            -Rick


          • Rick Murphy/K1MU
            ... [For context: About duplicate QSLs on LoTW.] Mostly because it would be nice to understand why those happen. If they re because someone reloads their log
            Message 6 of 25 , Sep 15, 2012
            • 0 Attachment
              At 02:30 PM 9/15/2012, Radio K0HB wrote:
              >Why would you care?

              [For context: About duplicate QSLs on LoTW.]

              Mostly because it would be nice to understand why those happen. If
              they're because someone reloads their log to add a Grid Square to their
              location, or to add county info, etc. that's fine. If not, then there's
              something else going on.

              Some stations appear to upload their entire log each time they upload.
              That's a bad thing. Understanding how bad it is can help us to guide
              them to doing it right and only uploading new or updated QSOs. Given
              the number of times folks here have said that uploading everything
              every time isn't a big problem (because for most cases LoTW ignores the
              duplicates), it would be nice to understand the reasons why it doesn't
              always work that way.
              73,
              -Rick
            • John
              ... Rick, this is not a personal attack, but a question on how you know that it is a big problem when people upload their entire log? Do you work for the
              Message 7 of 25 , Sep 16, 2012
              • 0 Attachment
                --- In ARRL-LOTW@yahoogroups.com, Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                >
                >Given the number of times folks here have said that uploading >everything every time isn't a big problem (because for most cases >LoTW ignores the duplicates), it would be nice to understand the >reasons why it doesn't always work that way.


                Rick, this is not a personal attack, but a question on how you know that it is a "big problem" when people upload their entire log?

                Do you work for the company that handles the LOTW for the ARRL? Or do you work for ARRL and see a report that shows it causes a problem?

                Just wondering why if the program overlooks duplicates, just how "big" is the problem from it?

                John, W3ML
              • N3XX
                I frequently receive LOTW qsl matches that have been received before, and I don t believe they show up at the top of the list again because there has been a
                Message 8 of 25 , Sep 16, 2012
                • 0 Attachment
                  I frequently receive LOTW qsl matches that have been received before, and I
                  don't believe they show up at the top of the list again because there has
                  been a change from what was uploaded before. If I received an earlier
                  match, everything must have matched correctly with what I uploaded. Then if
                  I receive a later match for the same qso, everything must have matched
                  again, so I can't see how anything has changed between the first and second
                  time someone uploaded their log. If there was a change, I would not have
                  received the second qsl match.

                  Is it a big problem? No, but it just gets a bit tiresome going into the my
                  log to mark the qso as lotw qsl received, only to find out it's already
                  done.

                  73,
                  Tim - N3XX

                  ----- Original Message -----
                  From: "John" <w3ml@...>
                  To: <ARRL-LOTW@yahoogroups.com>
                  Sent: Sunday, September 16, 2012 10:08 AM
                  Subject: [ARRL-LOTW] Re: An odd occurrence ...




                  --- In ARRL-LOTW@yahoogroups.com, Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                  >
                  >Given the number of times folks here have said that uploading >everything
                  >every time isn't a big problem (because for most cases >LoTW ignores the
                  >duplicates), it would be nice to understand the >reasons why it doesn't
                  >always work that way.


                  Rick, this is not a personal attack, but a question on how you know that it
                  is a "big problem" when people upload their entire log?

                  Do you work for the company that handles the LOTW for the ARRL? Or do you
                  work for ARRL and see a report that shows it causes a problem?

                  Just wondering why if the program overlooks duplicates, just how "big" is
                  the problem from it?

                  John, W3ML
                • Rick Murphy/K1MU
                  ... John, No, I don t work for ARRL or for the company that runs the LoTW servers. (Actually, I don t think ANYONE from ARRL is on this mailing list). However,
                  Message 9 of 25 , Sep 16, 2012
                  • 0 Attachment
                    At 10:08 AM 9/16/2012, John wrote:
                    >--- In <mailto:ARRL-LOTW@yahoogroups.com>ARRL-LOTW@yahoogroups.com,
                    >Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                    > >
                    > >Given the number of times folks here have said that
                    > uploading >everything every time isn't a big problem (because for
                    > most cases >LoTW ignores the duplicates), it would be nice to
                    > understand the >reasons why it doesn't always work that way.
                    >
                    >Rick, this is not a personal attack, but a question on how you know
                    >that it is a "big problem" when people upload their entire log?
                    >
                    >Do you work for the company that handles the LOTW for the ARRL? Or do
                    >you work for ARRL and see a report that shows it causes a problem?
                    >
                    >Just wondering why if the program overlooks duplicates, just how "big"
                    >is the problem from it?

                    John,
                    No, I don't work for ARRL or for the company that runs the LoTW
                    servers. (Actually, I don't think ANYONE from ARRL is on this mailing
                    list). However, I do know quite a lot about the TrustedQSL software and
                    have been around here since the beginning so I think I have a bit of
                    credibility.

                    OK, so "how do I know"?

                    The ARRL recommends not uploading logs multiple times on their webpage:
                    "Log submission guidance: Please refrain from uploading files more
                    than once as this only adds to the processing time. If your TQ8 file
                    has been queued for processing it will be processed.
                    When creating TQ8 files please include new QSOs only. Uploading
                    previously uploaded QSOs or uploading your entire log again only adds
                    to the processing time."

                    "Do not send entire logs that contain previously uploaded QSOs; please
                    use the date range option when signing logs, selecting only those QSOs
                    in a specific date range to upload to LoTW."

                    "Do not send entire logs that contain previously uploaded QSOs; please
                    use the date range option when signing logs, selecting only those QSOs
                    in a specific date range to upload to LoTW. . Do not send entire logs
                    that contain previously uploaded QSOs; please use the date range option
                    when signing logs, selecting only those QSOs in a specific date range
                    to upload to LoTW."

                    To me, these messages from ARRL indicate that there IS a downside to
                    simply re-uploading your entire log every time, causing the system to
                    run more slowly for everyone. That's the basis for my recommending that
                    stations only upload new QSOs to LoTW, because that's what ARRL is asking.

                    When you submit a log with a lot of duplicate QSOs, you'll see that the
                    duplicates are ignored by LoTW. Some on this list say that since LoTW
                    is ignoring the duplicates, then there's no reason to worry about
                    uploading your log all the time. I'm not sure that's the case, as the
                    system still has to process the QSO records to find the duplicates. Big
                    logs mean more overhead to decompress and verify the signature; then
                    each one of those QSOs then has to be searched for in the LoTW system
                    to see if it's a duplicate. The only thing that you save by discarding
                    duplicates is that the system then doesn't have to search for a
                    matching QSO so a QSL can be generated. Basically, almost all of the
                    per-QSO system overhead is the same. When you're talking about
                    thousands of QSOs, that has to slow things down, at least in my opinion.

                    While I'm not really sure how much overhead that causes, the messages
                    from ARRL above clearly indicates to me that they want to discourage
                    people from uploading their whole log repeatedly. Those two points:
                    ARRL asks you not to, and it's adding unnecessary overhead, mean to me
                    that you shouldn't upload your whole log each time.

                    After all, it's very simple to avoid repeated uploads of your whole
                    log. Just keep track of the last upload date and tell TQSL to ignore
                    anything older than that. If you can't remember, you can look at your
                    activity log on LoTW and figure out when you last uploaded. If whatever
                    software you're using automatically uploads your QSOs to LoTW but
                    doesn't remember a "last upload" date so it can avoid dupes, you
                    shouldn't use it to upload to LoTW. Export an ADIF file and sign it
                    yourself. And ask the developer to fix it.

                    In conclusion, uploading entire logs is unnecessary. It also causes
                    everyone else's processing to be slower. THAT is why it is a big problem.

                    Now, let's combine this with the magically re-appearing QSLs on the
                    system. I haven't seen one of those for quite a while myself, but they
                    do occur. It's purely curiosity for me at this point that causes me to
                    want to understand the cause. Several people on the list have said that
                    this happens when people re-upload QSOs. But if they're duplicates, why
                    aren't they ignored?

                    After all, this is a concern about how Logbook of the World works. I'm
                    asking it here because that's what the topic of this mailing list is.
                    I'm surprised that's causing you concern.
                    73,
                    -Rick
                  • Tim
                    Watching from the outside , I observe that LOTW s log processing robot can load about 10 new QSO s per second from my logfiles. I m guessing that it processes
                    Message 10 of 25 , Sep 16, 2012
                    • 0 Attachment
                      Watching "from the outside", I observe that LOTW's log processing robot can load about 10 new QSO's per second from my logfiles. I'm guessing that it processes logs sequentially. That works out to about a million QSO's that can be loaded a day. I looked over the past years of my "Your Activity/Results" logs to guesstimate these figures. Sometimes it seems to be closer to 5 new QSO's per second... other times closer to 20.

                      Imagine a fall contest with 2000 participants sending in logs and average log size of 500 QSO's. That's a million QSO's right there. We shouldn't be surprised that when all cylinders are firing that LOTW loading slows down by a day or more after a big contest. Or more days after a very big contest (keep in mind that doubling the number of participants, quadruples the number of QSO's!)

                      Now imagine that everyone instead of uploading their "new" contest log of 500 QSO's, uploaded their entire log of 10000 or 100000 QSO's It would take 20 or 200 times as long for LOTW to process all the records... 20 or 200 days! At the height of contest season LOTW would just be getting further and futher behind!!! It's a good thing the vast majority of users only upload "new" log QSO's. But if some guy accidentally does a full upload a couple times a year the rest of us probably won't notice. (Unless that guy is say OH2BH, who has more than a million QSO's in his logs!!!)

                      All the above assumes that LOTW is firing on all cylinders, too. I suspect sometimes the automatic machinery gums up and requires some manual intervention.

                      Tim N3QE

                      --- In ARRL-LOTW@yahoogroups.com, Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                      >
                      > At 10:08 AM 9/16/2012, John wrote:
                      > >--- In <mailto:ARRL-LOTW@yahoogroups.com>ARRL-LOTW@yahoogroups.com,
                      > >Rick Murphy/K1MU <k1mu-lotw@> wrote:
                      > > >
                      > > >Given the number of times folks here have said that
                      > > uploading >everything every time isn't a big problem (because for
                      > > most cases >LoTW ignores the duplicates), it would be nice to
                      > > understand the >reasons why it doesn't always work that way.
                      > >
                      > >Rick, this is not a personal attack, but a question on how you know
                      > >that it is a "big problem" when people upload their entire log?
                      > >
                      > >Do you work for the company that handles the LOTW for the ARRL? Or do
                      > >you work for ARRL and see a report that shows it causes a problem?
                      > >
                      > >Just wondering why if the program overlooks duplicates, just how "big"
                      > >is the problem from it?
                      >
                      > John,
                      > No, I don't work for ARRL or for the company that runs the LoTW
                      > servers. (Actually, I don't think ANYONE from ARRL is on this mailing
                      > list). However, I do know quite a lot about the TrustedQSL software and
                      > have been around here since the beginning so I think I have a bit of
                      > credibility.
                      >
                      > OK, so "how do I know"?
                      >
                      > The ARRL recommends not uploading logs multiple times on their webpage:
                      > "Log submission guidance: Please refrain from uploading files more
                      > than once as this only adds to the processing time. If your TQ8 file
                      > has been queued for processing it will be processed.
                      > When creating TQ8 files please include new QSOs only. Uploading
                      > previously uploaded QSOs or uploading your entire log again only adds
                      > to the processing time."
                      >
                      > "Do not send entire logs that contain previously uploaded QSOs; please
                      > use the date range option when signing logs, selecting only those QSOs
                      > in a specific date range to upload to LoTW."
                      >
                      > "Do not send entire logs that contain previously uploaded QSOs; please
                      > use the date range option when signing logs, selecting only those QSOs
                      > in a specific date range to upload to LoTW. . Do not send entire logs
                      > that contain previously uploaded QSOs; please use the date range option
                      > when signing logs, selecting only those QSOs in a specific date range
                      > to upload to LoTW."
                      >
                      > To me, these messages from ARRL indicate that there IS a downside to
                      > simply re-uploading your entire log every time, causing the system to
                      > run more slowly for everyone. That's the basis for my recommending that
                      > stations only upload new QSOs to LoTW, because that's what ARRL is asking.
                      >
                      > When you submit a log with a lot of duplicate QSOs, you'll see that the
                      > duplicates are ignored by LoTW. Some on this list say that since LoTW
                      > is ignoring the duplicates, then there's no reason to worry about
                      > uploading your log all the time. I'm not sure that's the case, as the
                      > system still has to process the QSO records to find the duplicates. Big
                      > logs mean more overhead to decompress and verify the signature; then
                      > each one of those QSOs then has to be searched for in the LoTW system
                      > to see if it's a duplicate. The only thing that you save by discarding
                      > duplicates is that the system then doesn't have to search for a
                      > matching QSO so a QSL can be generated. Basically, almost all of the
                      > per-QSO system overhead is the same. When you're talking about
                      > thousands of QSOs, that has to slow things down, at least in my opinion.
                      >
                      > While I'm not really sure how much overhead that causes, the messages
                      > from ARRL above clearly indicates to me that they want to discourage
                      > people from uploading their whole log repeatedly. Those two points:
                      > ARRL asks you not to, and it's adding unnecessary overhead, mean to me
                      > that you shouldn't upload your whole log each time.
                      >
                      > After all, it's very simple to avoid repeated uploads of your whole
                      > log. Just keep track of the last upload date and tell TQSL to ignore
                      > anything older than that. If you can't remember, you can look at your
                      > activity log on LoTW and figure out when you last uploaded. If whatever
                      > software you're using automatically uploads your QSOs to LoTW but
                      > doesn't remember a "last upload" date so it can avoid dupes, you
                      > shouldn't use it to upload to LoTW. Export an ADIF file and sign it
                      > yourself. And ask the developer to fix it.
                      >
                      > In conclusion, uploading entire logs is unnecessary. It also causes
                      > everyone else's processing to be slower. THAT is why it is a big problem.
                      >
                      > Now, let's combine this with the magically re-appearing QSLs on the
                      > system. I haven't seen one of those for quite a while myself, but they
                      > do occur. It's purely curiosity for me at this point that causes me to
                      > want to understand the cause. Several people on the list have said that
                      > this happens when people re-upload QSOs. But if they're duplicates, why
                      > aren't they ignored?
                      >
                      > After all, this is a concern about how Logbook of the World works. I'm
                      > asking it here because that's what the topic of this mailing list is.
                      > I'm surprised that's causing you concern.
                      > 73,
                      > -Rick
                      >
                    • George Hughes
                      Well when you upload contacts to LOTW I am sure it runs throught the whole dad blamed thing looking for matches starting the first record uploaded. Now lets
                      Message 11 of 25 , Sep 16, 2012
                      • 0 Attachment
                        Well when you upload contacts to LOTW I am sure it runs throught the whole dad blamed thing looking for matches starting the first record uploaded.  Now lets say you upload 25 years of contacts to LOTW its starts checking for matches and does each record.  Now the next time you upload you also upload your whole log.  Guess what it does it again.  Now imagine if 5000 amateurs uploaded there logs the same way.  All of this processing take a large amount of time.  It just is not efficient at all and costs in computer time.  Think about what happens right after a major contest weekend.
                        That said you should always "just upload your new contacts only."  Saves a lot of time.
                        de AF4GH George
                        
                        On 9/16/2012 10:08 AM, John wrote:
                         



                        --- In ARRL-LOTW@yahoogroups.com, Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                        >
                        >Given the number of times folks here have said that uploading >everything every time isn't a big problem (because for most cases >LoTW ignores the duplicates), it would be nice to understand the >reasons why it doesn't always work that way.

                        Rick, this is not a personal attack, but a question on how you know that it is a "big problem" when people upload their entire log?

                        Do you work for the company that handles the LOTW for the ARRL? Or do you work for ARRL and see a report that shows it causes a problem?

                        Just wondering why if the program overlooks duplicates, just how "big" is the problem from it?

                        John, W3ML



                      • John
                        Rick, I don t have a problem with you asking anything you want to ask about LOTW. Reading most of these posts (not yours)it seems that most people expect their
                        Message 12 of 25 , Sep 17, 2012
                        • 0 Attachment
                          Rick,

                          I don't have a problem with you asking anything you want to ask about LOTW.

                          Reading most of these posts (not yours)it seems that most people expect their contacts to be listed in LOTW immediately after uploading a file.

                          What I can not understand is, why the big hurry? If it showed your upload was accepted, it will be in the listing when it gets there.

                          There have been a lot of attacks against the LOTW program and most are uncalled for.

                          Like someone said, if millions of records are being uploaded it takes time.

                          Also, I don't believe in picking on people that are not computer literate and able to send in only recent contacts. Some may be only starting out using this system and don't know how to set up their log to do recent ones.

                          Imagine the guy that is complaining because he uploaded last night and it still doesn't show up.

                          If he worked that DX station that only uploads to LOTW once a year, why does it matter if it is there or not.

                          As far as ghost contacts or what ever you want to call them, I, too, have been using LOTW since day one and can not answer or get an answer even when I was a Section Manager for ARRL as to why they show up. It seems they will always be there.


                          Well, I am back to monitoring the message flow via digest mode.

                          73
                          John W3ML

                          --- In ARRL-LOTW@yahoogroups.com, Rick Murphy/K1MU <k1mu-lotw@...> wrote:
                          >
                          > At 10:08 AM 9/16/2012, John wrote:
                          > >--- In <mailto:ARRL-LOTW@yahoogroups.com>ARRL-LOTW@yahoogroups.com,
                          > >Rick Murphy/K1MU <k1mu-lotw@> wrote:
                          > > >
                          > > >Given the number of times folks here have said that
                          > > uploading >everything every time isn't a big problem (because for
                          > > most cases >LoTW ignores the duplicates), it would be nice to
                          > > understand the >reasons why it doesn't always work that way.
                          > >
                          > >Rick, this is not a personal attack, but a question on how you know
                          > >that it is a "big problem" when people upload their entire log?
                          > >
                          > >Do you work for the company that handles the LOTW for the ARRL? Or do
                          > >you work for ARRL and see a report that shows it causes a problem?
                          > >
                          > >Just wondering why if the program overlooks duplicates, just how "big"
                          > >is the problem from it?
                          >
                          > John,
                          > No, I don't work for ARRL or for the company that runs the LoTW
                          > servers. (Actually, I don't think ANYONE from ARRL is on this mailing
                          > list). However, I do know quite a lot about the TrustedQSL software and
                          > have been around here since the beginning so I think I have a bit of
                          > credibility.
                          >
                          > OK, so "how do I know"?
                          >
                          > The ARRL recommends not uploading logs multiple times on their webpage:
                          > "Log submission guidance: Please refrain from uploading files more
                          > than once as this only adds to the processing time. If your TQ8 file
                          > has been queued for processing it will be processed.
                          > When creating TQ8 files please include new QSOs only. Uploading
                          > previously uploaded QSOs or uploading your entire log again only adds
                          > to the processing time."
                          >
                          > "Do not send entire logs that contain previously uploaded QSOs; please
                          > use the date range option when signing logs, selecting only those QSOs
                          > in a specific date range to upload to LoTW."
                          >
                          > "Do not send entire logs that contain previously uploaded QSOs; please
                          > use the date range option when signing logs, selecting only those QSOs
                          > in a specific date range to upload to LoTW. . Do not send entire logs
                          > that contain previously uploaded QSOs; please use the date range option
                          > when signing logs, selecting only those QSOs in a specific date range
                          > to upload to LoTW."
                          >
                          > To me, these messages from ARRL indicate that there IS a downside to
                          > simply re-uploading your entire log every time, causing the system to
                          > run more slowly for everyone. That's the basis for my recommending that
                          > stations only upload new QSOs to LoTW, because that's what ARRL is asking.
                          >
                          > When you submit a log with a lot of duplicate QSOs, you'll see that the
                          > duplicates are ignored by LoTW. Some on this list say that since LoTW
                          > is ignoring the duplicates, then there's no reason to worry about
                          > uploading your log all the time. I'm not sure that's the case, as the
                          > system still has to process the QSO records to find the duplicates. Big
                          > logs mean more overhead to decompress and verify the signature; then
                          > each one of those QSOs then has to be searched for in the LoTW system
                          > to see if it's a duplicate. The only thing that you save by discarding
                          > duplicates is that the system then doesn't have to search for a
                          > matching QSO so a QSL can be generated. Basically, almost all of the
                          > per-QSO system overhead is the same. When you're talking about
                          > thousands of QSOs, that has to slow things down, at least in my opinion.
                          >
                          > While I'm not really sure how much overhead that causes, the messages
                          > from ARRL above clearly indicates to me that they want to discourage
                          > people from uploading their whole log repeatedly. Those two points:
                          > ARRL asks you not to, and it's adding unnecessary overhead, mean to me
                          > that you shouldn't upload your whole log each time.
                          >
                          > After all, it's very simple to avoid repeated uploads of your whole
                          > log. Just keep track of the last upload date and tell TQSL to ignore
                          > anything older than that. If you can't remember, you can look at your
                          > activity log on LoTW and figure out when you last uploaded. If whatever
                          > software you're using automatically uploads your QSOs to LoTW but
                          > doesn't remember a "last upload" date so it can avoid dupes, you
                          > shouldn't use it to upload to LoTW. Export an ADIF file and sign it
                          > yourself. And ask the developer to fix it.
                          >
                          > In conclusion, uploading entire logs is unnecessary. It also causes
                          > everyone else's processing to be slower. THAT is why it is a big problem.
                          >
                          > Now, let's combine this with the magically re-appearing QSLs on the
                          > system. I haven't seen one of those for quite a while myself, but they
                          > do occur. It's purely curiosity for me at this point that causes me to
                          > want to understand the cause. Several people on the list have said that
                          > this happens when people re-upload QSOs. But if they're duplicates, why
                          > aren't they ignored?
                          >
                          > After all, this is a concern about how Logbook of the World works. I'm
                          > asking it here because that's what the topic of this mailing list is.
                          > I'm surprised that's causing you concern.
                          > 73,
                          > -Rick
                          >
                        • Rick Murphy/K1MU
                          ... Whew. Seems like we re mostly in agreement, actually. ... I agree. There s a lot of complaints about LoTW slowness, mostly unwarranted. If you re expecting
                          Message 13 of 25 , Sep 17, 2012
                          • 0 Attachment
                            At 06:32 PM 9/17/2012, John wrote:
                            >Rick,
                            >
                            >I don't have a problem with you asking anything you want to ask about
                            >LOTW.

                            Whew. Seems like we're mostly in agreement, actually.

                            >Reading most of these posts (not yours)it seems that most people
                            >expect their contacts to be listed in LOTW immediately after uploading
                            >a file.
                            >
                            >What I can not understand is, why the big hurry? If it showed your
                            >upload was accepted, it will be in the listing when it gets there.

                            I agree. There's a lot of complaints about LoTW slowness, mostly
                            unwarranted. If you're expecting instant gratification, LoTW isn't
                            likely to deliver. :)

                            >Also, I don't believe in picking on people that are not computer
                            >literate and able to send in only recent contacts. Some may be only
                            >starting out using this system and don't know how to set up their log
                            >to do recent ones.

                            I do try not to pick on people and (when appropriate) point out that
                            TQSL has a place where you can say what date to start from for each
                            upload. If you're not using that, a gentle reminder is appropriate.
                            I'll admit that I'm not always as tactful as I could be, but I do try.

                            >Imagine the guy that is complaining because he uploaded last night and
                            >it still doesn't show up.

                            And there's a lot of that kind of complaining, unfortunately. I do try
                            to ignore those who just post to flame. And sometimes we get replies to
                            questions that aren't quite right.. this triggers my geek reaction. See
                            <http://xkcd.com/386/>http://xkcd.com/386/ for further info about that.

                            >As far as ghost contacts or what ever you want to call them, I, too,
                            >have been using LOTW since day one and can not answer or get an answer
                            >even when I was a Section Manager for ARRL as to why they show up. It
                            >seems they will always be there.

                            They're there because the people who designed LoTW concentrated on
                            making it just like paper QSLs and apparently didn't think about human
                            nature. If you send out a paper QSL to someone with the wrong band,
                            mode, time, etc. you can't get that card back; LoTW acts the same way.
                            Once you upload, it's there and can't be removed. Well, given how
                            complicated all this is, people will mess up. Wrong date, mode, call,
                            location, and just about everything else. Not accommodating these
                            frequent but minor mistakes is a significant weakness to LoTW.

                            I'd like to think that eventually there will be a way to delete these
                            bogus QSOs, but I'm not holding my breath waiting for it. :)

                            73,
                            -Rick
                          • Your message has been successfully submitted and would be delivered to recipients shortly.