Loading ...
Sorry, an error occurred while loading the content.

Re: [ARRL-LOTW] Re: LotW Backlog continues to grow ...

Expand Messages
  • Joe Subich, W4TV
    Bob, Based on your statistics, some of my own research, and other data points posted here, I m convinced of a couple things: 1) LotW is hitting the wall at a
    Message 1 of 53 , Dec 7, 2012
    • 0 Attachment
      Bob,

      Based on your statistics, some of my own research, and other data points
      posted here, I'm convinced of a couple things:

      1) LotW is hitting the wall at a throughput of 30 QSO records per second
      2) LotW is currently processing at that rate
      3) the observable downward deviation from 30 QSOs per second is
      entirely due to *processing of duplicate QSO records*
      4) the apparent "non progress" in "user files processed" is due to
      processing some *extremely large* files. For example, several
      large multi-multi contest logs from CQWW contain more than 20 K
      QSOs *each* and some DXpedition logs currently in the queue (the
      sponsors claim to have uploaded - I can't "see" into the queue)
      have 50K or more QSOs

      It is amazing to think that the queue might contain as much as 5%
      of the total number of QSO records uploaded to LotW since the start -
      but the duration of the backlog and processing rate would indicate
      a queue that large. It is also completely frustrating to think that
      half of those QSO records have been uploaded (and processed) at least
      once previously and that some users are so careless as to waste the
      limited throughput.

      If LotW was constructed using a virtualized system as indicated in
      the design documents you posted and the storage subsystem is old
      technology, it is entirely possible that the system is, in fact,
      I/O bound. If that is the case, it might benefit to some degree by
      a change to improved storage attachment technology (whatever that
      might be) ... if that technology is part of the change from rotating
      memory to SSD. Not being a server farm jockey, I can only relate
      that to going from 10MB/sec Ethernet to fiber channel, or going
      from shared network storage to local server storage, etc.

      73,

      ... Joe, W4TV


      On 12/7/2012 10:20 PM, Robert Chudek - K0RC wrote:
      > I have updated the Excel workbook I created to track LoTW dashboard
      > statistics. You will now find it contains 4 worksheets:
      >
      > Sheet 1: Contains the original 1-minute samples and analysis (data
      > collection completed)
      > Sheet 2: Contains the original 15-minute samples and analysis (completed)
      > Sheet 3: Contains the current (on-going) data collection of 15-minute
      > samples (in process)
      > Sheet 4: Contains the *LA4RT* daily samples that were provided from the
      > message below (completed)
      >
      > Download the workbook from here: *http://tinyurl.com/bxcmub7*
      >
      > NOTE: the data for sheet 1 was a semi-automatic collection of
      > information and requires me to transfer the numbers for each line
      > manually. I'll work on updating this sheet as time and energy permits.
      > It currently contains 5 hours of data which I think is adequate for a
      > "first look" at the stats at that resolution.
      >
      > 73 de Bob - KØRC in MN
      >
      > ------------------------------------------------------------------------
      > On 12/7/2012 4:54 PM, la4rt wrote:
      >>
      >>
      >> --- In ARRL-LOTW@yahoogroups.com <mailto:ARRL-LOTW%40yahoogroups.com>,
      >> "Joe Subich, W4TV" <lists@...> wrote:
      >> >
      >> >
      >> > > Between 250 and 300,000, actually.
      >> >
      >> > Based on what data?
      >>
      >> Here are observations each morning (EU time) starting Monday morning
      >> after CQ WW CW:
      >>
      >> 2012-11-26 07:03
      >> 457,584,847 QSO records have been entered into the system.
      >> 62,382,862 QSL records have resulted.
      >> 53,759 Users are registered in the system
      >> 80,791 Certificates are active
      >> 3,707,231 User files have been processed
      >>
      >> 2012-11-27 07:00
      >> 457,783,982 QSO records have been entered into the system.
      >> 62,428,429 QSL records have resulted.
      >> 53,817 Users are registered in the system
      >> 80,863 Certificates are active
      >> 3,710,246 User files have been processed
      >>
      >> 2012-11-28 07:01
      >> 457,963,370 QSO records have been entered into the system.
      >> 62,458,137 QSL records have resulted.
      >> 53,875 Users are registered in the system
      >> 80,931 Certificates are active
      >> 3,714,079 User files have been processed
      >>
      >> 2012-11-29 06:57
      >> 458,258,059 QSO records have been entered into the system.
      >> 62,478,994 QSL records have resulted.
      >> 53,896 Users are registered in the system
      >> 80,963 Certificates are active
      >> 3,718,264 User files have been processed
      >>
      >> 2012-11-30 07:00
      >> 458,631,454 QSO records have been entered into the system.
      >> 62,508,870 QSL records have resulted.
      >> 53,930 Users are registered in the system
      >> 81,018 Certificates are active
      >> 3,719,817 User files have been processed
      >>
      >> 2012-12-01 11:33
      >> 458,960,202 QSO records have been entered into the system.
      >> 62,559,444 QSL records have resulted.
      >> 53,980 Users are registered in the system
      >> 81,079 Certificates are active
      >> 3,721,425 User files have been processed
      >>
      >> 2012-12-01 23:59
      >> 459,108,641 QSO records have been entered into the system.
      >> 62,584,034 QSL records have resulted.
      >> 53,980 Users are registered in the system
      >> 81,079 Certificates are active
      >> 3,722,670 User files have been processed
      >>
      >> 2012-12-02 06:48
      >> 459,175,199 QSO records have been entered into the system.
      >> 62,593,944 QSL records have resulted.
      >> 53,980 Users are registered in the system
      >> 81,079 Certificates are active
      >> 3,723,243 User files have been processed
      >>
      >> 2012-12-03 07:01
      >> 459,410,509 QSO records have been entered into the system.
      >> 62,655,708 QSL records have resulted.
      >> 53,979 Users are registered in the system
      >> 81,079 Certificates are active
      >> 3,724,456 User files have been processed
      >>
      >> 2012-12-04 07:04
      >> 459,643,445 QSO records have been entered into the system.
      >> 62,704,634 QSL records have resulted.
      >> 54,020 Users are registered in the system
      >> 81,125 Certificates are active
      >> 3,726,460 User files have been processed
      >>
      >> 2012-12-05 07:05
      >> 459,890,922 QSO records have been entered into the system.
      >> 62,769,535 QSL records have resulted.
      >> 54,031 Users are registered in the system
      >> 81,148 Certificates are active
      >> 3,728,007 User files have been processed
      >>
      >> 2012-12-06 07:01
      >> 460,277,589 QSO records have been entered into the system.
      >> 62,813,185 QSL records have resulted.
      >> 54,040 Users are registered in the system
      >> 81,169 Certificates are active
      >> 3,729,885 User files have been processed
      >>
      >> 2012-12-07 07:02
      >> 460,533,429 QSO records have been entered into the system.
      >> 62,867,626 QSL records have resulted.
      >> 54,104 Users are registered in the system
      >> 81,242 Certificates are active
      >> 3,731,786 User files have been processed
      >>
      >>
      >
      >
    • Kenneth Grimm
      ... Thanks for the link, Bob. Enterprise decisions concerning SSDs are the same as I went through when deciding to use them. 8*) Speaking of enterprise...do
      Message 53 of 53 , Dec 11, 2012
      • 0 Attachment
        On Tue, Dec 11, 2012 at 2:33 AM, Robert Chudek - K0RC <k0rc@...> wrote:


        It is certainly a sign of competence and design skills when you realize the LoTW system was deployed on SAN technology 10+ years ago. That was leading edge architecture at the time, and unlikely to be found outside of corporate server farms or large data centers.

        For anyone interested in considerations the regarding the marriage of SANs and Database architecture, here's an article describing the issues facing system designers. It's a pretty decent primer on some basic database design too.

        http://tinyurl.com/7bpmt64

        73 de Bob - KØRC in MN



        Thanks for the link, Bob.  Enterprise decisions concerning SSDs are the same as I went through when deciding to use them.  8*)  

        Speaking of enterprise...do you know of any studies comparing MTBF for consumer vs enterprise SSDs?

        73,

        --
        Ken - K4XL
        BoatAnchor Manual Archive
        BAMA - http://bama.edebris.com

      Your message has been successfully submitted and would be delivered to recipients shortly.