Loading ...
Sorry, an error occurred while loading the content.

Re: [ARRL-LOTW] Creating a NEW LOTW

Expand Messages
  • Pete Ennis
    Why don’t you just put LoTW out of it misery. Just get 1,000’s of users to re-upload their entire logs several times on the same day. If the system takes
    Message 1 of 7 , Dec 6, 2012
    • 0 Attachment
      Why don’t you just put LoTW out of it misery. Just get 1,000’s of users to re-upload their entire logs several times on the same day. If the system takes a total crap something will be done.


      From: Mike Flowers <mike.flowers@...>
      To: ARRL-LOTW@yahoogroups.com
      Sent: Thursday, December 6, 2012 2:56 PM
      Subject: RE: [ARRL-LOTW] Creating a NEW LOTW
       
      >> Joe wrote: "After checking LotW further, there appears to be significant "breaks" in processing."
       
      Due to the transaction volume and the apparent number of indices involved, index reorganization may well account for the observed breaks in processing.  
       
      The higher the volume of transactions applied to index-complex database schemas, the more frequently index reorganization will be required.   Just a guess on my part.   However, I used to see similar performance pauses in such database systems when I made my living as an Oracle DBA.
       
      - 73 de Mike, K6MKF, W6NAG, VP NCDXC, Conway Reef 2012, K3-P3 Addict, Dachshund Rescue, Maui
       
      From: ARRL-LOTW@yahoogroups.com [mailto:ARRL-LOTW@yahoogroups.com] On Behalf Of Robert Chudek - K0RC
      Sent: Thursday, December 06, 2012 11:28 AM
      To: ARRL-LOTW@yahoogroups.com
      Cc: Joe Subich, W4TV
      Subject: Re: [ARRL-LOTW] Creating a NEW LOTW
       
       
      Joe wrote: "After checking LotW further, there appears to be significant "breaks" in processing."

      This observation can be verified in the LoTW analysis I just posted. The 5-hour data snapshot was started about 8:00 pm Dec 4th, 2012 so it's unlikely any office activity was taking place during this period.

      73 de Bob - KØRC in MN
      On 12/6/2012 1:05 PM, Joe Subich, W4TV wrote:
       

      >>> LotW is already processing at about 5 QSO/sec
      >
      > *So with a 5 day backlog, 5 days x 24 hours/day, x 3600 sec/hour x 5
      > qso/sec, you believe the current backlog is > 2 million qsos?
      > Probably the only reason it is that big is multiple uploads, which
      > could be easily eliminated with better front end processing. Do you
      > think we'll see 500,000 new QSLs over the next five days?.*

      After checking LotW further, there appears to be significant "breaks"
      in processing. I don't know if the lack of progress is from processing
      duplicate uploads or if the server used for LotW also runs other ARRL
      office programs and LotW processing is suspended when award program
      records are being updated (entry of "paper" DXCC, WAS, VUCC, etc.).
      In any case, the backlog continues to grow ...

      This morning I reported my upload of 2012-11-29 18:38:13 that processed
      2012-12-06 05:10:33 (6 days, 10 hours, 32 minutes). My next upload -
      2012-11-30 00:58:01 processed 2012-12-06 18:24:27 (6 days, 17 hours,
      26 minutes). In 13 hours, 18 minutes the backlog increased by nearly
      7 hours ... continuing a trajectory of increasing between 20 and 30
      minutes *per hour*!

      *Logbook of the World as it is currently managed is unsustainable*

      73,

      ... Joe, W4TV

      On 12/6/2012 12:30 PM, Mickey Baker wrote:
      > W4TV makes good points:
      >
      >>> Total Number of QSO's in a database: 1 billion (ten to the 9th)
      >
      >>> Already at 500 million ... I suspect the design target should be at
      >>> least 10 if not 20 times the current size.
      >
      > *Understood. I simply doubled the current volume. I agree that the ultimate
      > design should be scalable to 10 x 10^9th qsos, but with today's technology,
      > no reason to invest in the storage day one.*
      >
      >>> Total Number of LOTW Members: 250,000
      >
      >> That's only five times the current number or registered users.
      > *Yes. I read somewhere that there are only 200,000 active amateurs
      > exchanging qsl information today, so that's the number I chose. It would be
      > handy to know things like the number of discrete call signs in the current
      > database over the past, say 5 years. Again, scalability is a design goal.
      > Just add more processors and storage if there's wider adoption.
      > *
      >
      >>> LotW is already processing at about 5 QSO/sec
      >
      > *So with a 5 day backlog, 5 days x 24 hours/day, x 3600 sec/hour x 5
      > qso/sec, you believe the current backlog is > 2 million qsos? Probably the
      > only reason it is that big is multiple uploads, which could be easily
      > eliminated with better front end processing. Do you think we'll see 500,000
      > new QSLs over the next five days?.*
      > *
      > *
      > *No worries, if that's true, 200% of that is good design criteria. Again,
      > I'm simply trying to stimulate the conversation.*
      >
      > 73,
      >
      > Mickey N4MB
      >
      >
      >
      > On Thu, Dec 6, 2012 at 11:43 AM, Joe Subich, W4TV <mailto:lists%40subich.com> wrote:
      >
      >> **
      >>
      >>
      >>
      >>> *What would an optimum LOTW system look like? (Future State)*
      >>
      >>> Total Number of QSO's in a database: 1 billion (ten to the 9th)
      >>
      >> Already at 500 million ... I suspect the design target should be at
      >> least 10 if not 20 times the current size.
      >>
      >>
      >>> Total Number of LOTW Members: 250,000
      >>
      >> That's only five times the current number or registered users.
      >>
      >>
      >>> QSO Processing Rate: 1 second per QSO worse case
      >>> (3,600 records/hour)
      >>
      >> LotW is already processing at about 5 QSO/sec based on several cycles
      >> of refreshing the home page at 10 second intervals. Even at that rate
      >> the backlog seems to be increasing. My latest upload to be processed
      >> was uploaded 2012-11-29 18:38:13 and processed 2012-12-06 05:10:33.
      >> Instead of "5 days" as the home page states three days ago, that makes
      >> it 6 days 13+ hours as of 12 hours ago. The trend seems to be for the
      >> backlog to increase 12 hours every 24 hours ... that is unsustainable.
      >>
      >> The target for QSO processing should be on the order of 50K QSO/hr.
      >> Of course if LotW has processing logs a good statistical study of
      >> upload volumes and processing rates/loads over time would tell a lot
      >>
      >>
      >>> QSO Record Design USE ADIF as a guide?
      >>> Awards Database Design Current? Notify members of
      >>> qualification? Integration with paper?
      >>> Database type, platform type? Open Source, Free
      >>> Interface type? SQL, RESTful?
      >>> Cloud or physical host?
      >>> Documentation needed for operation, changes and LOTW Gen3?
      >>
      >>> User re-training - Gen2 MUST BE EASIER TO USE
      >>
      >> The current system is trivial to use with well designed logging software
      >> once the sign-up process is completed. There improvements are needed in
      >> the sign-up process but not at the expense of user authentication and
      >> data security.
      >>
      >> 73,
      >>
      >> ... Joe, W4TV
      >>
      >> On 12/6/2012 10:57 AM, Mickey Baker wrote:
      >>> K3KO wrote:
      >>> *"It's sort of an unfair comparison to have some IT expert having
      >>
      >>> accumulated additional knowledge over the past 10 years commenting
      >>> unfavorably on something designed and built 10 years ago with the
      >>> **available knowledge then*."
      >>
      >>>
      >>> Brian (and all),
      >>>
      >>> This is an important statement and ABSOLUTELY true. We should recognize
      >>> that the original designers of LOTW and the ARRL was doing something that
      >>> has never been done before with tools that they were familiar with at the
      >>> time and the constraints of computing power available 10 years ago. Kudos
      >>> to them for doing that. May the next generation of LOTW build on their
      >>> success and respect the original creators.
      >>>
      >>> The key is that it HAS been 10 years since this system was designed.
      >>> Processors are 50 times as powerful for the same dollars, disk drives are
      >>> 1/20th as expensive per stored MB and performance, tools for database
      >>> creation and management reflect 10 years development and millions of
      >> hours
      >>> of development.
      >>>
      >>> Data security has become reliable and pervasive - the entire certificate
      >>> management system seems as antiquated as an engine without an electric
      >>> starter.
      >>>
      >>> Brian makes A KEY POINT - *10 year old systems are ripe for a refresh*.
      >>
      >>> Competent skills are available on a volunteer basis. All that remains is
      >> to
      >>> decide to do it.
      >>>
      >>> I'd suggest we drive ARRL toward a BACK END re-design, specifically a
      >>> database re-design and revamp the upload/ingest engine as a first start.
      >>> That's the part that seems broken.
      >>>
      >>> Front end tools would need to be changed to match the new database. Not
      >>> knowing the current architecture, that could be relatively easy or a
      >>> quagmire.
      >>>
      >>> Before any project should be considered... we have to know...
      >>>
      >>> *What is the current state of LOTW?*
      >>
      >>> Is continuing with the current system viable?
      >>> If so, how long?
      >>> What is the current harware/Database/platform/Web interface? Details,
      >>> please. We know that it is not robust, so why not be open about what this
      >>> thing is?
      >>>
      >>> *What is the will of ARRL Management and How might we influence this?*
      >>
      >>> Are there funds available for a new platform for LOTW and associated
      >>> systems? What's the budget? Do we need a financial campaign to raise
      >> money?
      >>> Are there opportunities to consolidate ARRL systems for Contests/Awards
      >>> with LOTW?
      >>> Could ARRL package and sell a VIABLE LOTW system to others for managing
      >>> contact records?
      >>> Could ARRL contract with eqsl.cc to maintain LOTW and provide compliant
      >>> services rather than re-writing?
      >>> Are we willing to put political support behind director candidates that
      >>> support a "New LOTW" agenda? Mailings? Email marketing, web sites?
      >>>
      >>> *What would an optimum LOTW system look like? (Future State)*
      >>
      >>> Total Number of QSO's in a database: 1 billion (ten to the 9th)
      >>> Total Number of LOTW Members: 250,000
      >>> QSO Processing Rate: 1 second per QSO worse case
      >>> (3,600 records/hour)
      >>> QSO Record Design USE ADIF as a guide?
      >>> Awards Database Design Current? Notify members of
      >>> qualification? Integration with paper?
      >>> Database type, platform type? Open Source, Free
      >>> Interface type? SQL, RESTful?
      >>> Cloud or physical host?
      >>> Documentation needed for operation, changes and LOTW Gen3?
      >>>
      >>> *Risks*
      >>
      >>> Lossless data migration
      >>> Uncovering unmatched qso's and other problems with current database.
      >>> User re-training - Gen2 MUST BE EASIER TO USE
      >>>
      >>
      >>
      >>
      >
      >
      >
       
    Your message has been successfully submitted and would be delivered to recipients shortly.