Loading ...
Sorry, an error occurred while loading the content.

Re: [dxatlas] Skim Server and Master.dta?

Expand Messages
  • Pete Smith
    Ah, I m glad to see this topic surfacing. Working with multiple Skimmers feeding the RBN, we ve encountered a rather different problem. Here s a summary from
    Message 1 of 9 , Feb 1, 2011
    • 0 Attachment
      Ah, I'm glad to see this topic surfacing. Working with multiple
      Skimmers feeding the RBN, we've encountered a rather different problem.

      Here's a summary from the N1MM developers group, in response to a
      complaint from N2IC about the plethora of bad spots he encountered over
      the weekend in the CQ 160 CW test:

      "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
      now, and as Steve says, it depends on your definition of error rate. If
      I watch the output of one skimmer, I see lots of spots and a very low
      error rate. But if you watch the whole RBN or even a sub-set (like the
      half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate
      is much higher, because the errors make it through while the good spots
      either come up gray (if already worked) or are duped out before they
      ever make it to the bandmap. This makes it a particularly bad problem
      late in a contest, when most of the legitimate stations have been worked
      but the op must still look at every spot to make sure it is a miscopy.

      [6:17:55 AM] Pete Smith: I will bring this to the people involved who
      actually know something (Felipe, Nick and Dave) and see if we can come
      up with a filter for the RBN that attempts to identify these situations
      and filter out the junk. It'll take some doing, but should be do-able.
      One simple-minded way would be to look at spots coming through within a
      short time window on the same frequency, and if there are spots that
      differ from the callsign that the majority are spotting correctly on
      that frequency, don't send them out."

      Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
      it be possible to do the same thing in Skimmer so that folks like Lee
      who prefer to collect their Skimmer spots themselves. Or do you have a
      better idea, Alex?

      73, Pete N4ZR

      The World Contest Station Database, updated daily at www.conteststations.com
      The Reverse Beacon Network at http://reversebeacon.net, blog at reversebeacon.blogspot.com,
      spots at telnet.reversebeacon.net, port 7000



      On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
      > Hi Lee,
      >
      > I thought about this but could not find an acceptable solution. If
      > Validation is set to Normal, almost every call is posted on the network when
      > it first meets the Normal validation criteria. A few seconds later, when
      > more data are received, the call may meet a higher validation level, but
      > since it has already been posted, is will not be posted again. I have
      > received tons of complaints about the same callsign posted multiple times,
      > and had to enforce the 10-minute rule. The net result is, all spots, with a
      > few exceptions, will have the Normal validation level.
      >
      > If your goal is to filter out possible errors after the aggregation of the
      > spots from multiple Skimmers, then one simple solution is to publish only
      > the callsigns reported by two or more Skimmers.
      >
      > 73 Alex VE3NEA
      >
      >
      >
      >
      >
      >
      > ----- Original Message -----
      >
      > Alex would it be possible to include a number in the Skimmer spots comment's
      > field that lets users know what level of call verification is being used?
      > It could be just before the CQ. For instance aggressive mode could be say
      > "3" and normal "1".
      >
      > Lee VE7CC
      >
      >
      >
      > ------------------------------------
      >
      > Yahoo! Groups Links
      >
      >
      >
      >
      >
    • Alex, VE3NEA
      Hi Pete, As I have already written, a program that receives spots from multiple Skimmers can radically reduce the number of errors simply by showing only the
      Message 2 of 9 , Feb 1, 2011
      • 0 Attachment
        Hi Pete,

        As I have already written, a program that receives spots from multiple
        Skimmers can radically reduce the number of errors simply by showing only
        the spots received from two or more Skimmers. This is the same principle
        that is used for Aggressive filtering in CW Skimmer, except that instead of
        waiting for more copies of the callsign decoded by the same Skimmer, we
        would get those copies instantly from other Skimmers. In other words,
        instead of time diversity we would use spatial diversity.

        73 Alex VE3NEA





        ----- Original Message -----
        From: "Pete Smith" <n4zr@...>
        To: <dxatlas_group@yahoogroups.com>
        Cc: "Alex, VE3NEA" <alshovk@...>
        Sent: Tuesday, February 01, 2011 11:44 AM
        Subject: Re: [dxatlas] Skim Server and Master.dta?


        > Ah, I'm glad to see this topic surfacing. Working with multiple Skimmers
        > feeding the RBN, we've encountered a rather different problem.
        >
        > Here's a summary from the N1MM developers group, in response to a
        > complaint from N2IC about the plethora of bad spots he encountered over
        > the weekend in the CQ 160 CW test:
        >
        > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
        > now, and as Steve says, it depends on your definition of error rate. If I
        > watch the output of one skimmer, I see lots of spots and a very low error
        > rate. But if you watch the whole RBN or even a sub-set (like the
        > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate
        > is much higher, because the errors make it through while the good spots
        > either come up gray (if already worked) or are duped out before they ever
        > make it to the bandmap. This makes it a particularly bad problem late in
        > a contest, when most of the legitimate stations have been worked but the
        > op must still look at every spot to make sure it is a miscopy.
        >
        > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
        > actually know something (Felipe, Nick and Dave) and see if we can come up
        > with a filter for the RBN that attempts to identify these situations and
        > filter out the junk. It'll take some doing, but should be do-able. One
        > simple-minded way would be to look at spots coming through within a short
        > time window on the same frequency, and if there are spots that differ
        > from the callsign that the majority are spotting correctly on that
        > frequency, don't send them out."
        >
        > Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
        > it be possible to do the same thing in Skimmer so that folks like Lee who
        > prefer to collect their Skimmer spots themselves. Or do you have a better
        > idea, Alex?
        >
        > 73, Pete N4ZR
        >
        > The World Contest Station Database, updated daily at
        > www.conteststations.com
        > The Reverse Beacon Network at http://reversebeacon.net, blog at
        > reversebeacon.blogspot.com,
        > spots at telnet.reversebeacon.net, port 7000
        >
        >
        >
        > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
        >> Hi Lee,
        >>
        >> I thought about this but could not find an acceptable solution. If
        >> Validation is set to Normal, almost every call is posted on the network
        >> when
        >> it first meets the Normal validation criteria. A few seconds later, when
        >> more data are received, the call may meet a higher validation level, but
        >> since it has already been posted, is will not be posted again. I have
        >> received tons of complaints about the same callsign posted multiple
        >> times,
        >> and had to enforce the 10-minute rule. The net result is, all spots, with
        >> a
        >> few exceptions, will have the Normal validation level.
        >>
        >> If your goal is to filter out possible errors after the aggregation of
        >> the
        >> spots from multiple Skimmers, then one simple solution is to publish only
        >> the callsigns reported by two or more Skimmers.
        >>
        >> 73 Alex VE3NEA
        >>
        >>
      • Rich Hallman - N7TR
        Im new to the skimmer group, but at least having the option to use a master.dta or some other verification file would be nice. Not all would be forced to
        Message 3 of 9 , Feb 1, 2011
        • 0 Attachment
          Im new to the skimmer group, but at least having the option to use a
          master.dta or some other verification file would be nice. Not all
          would be forced to using it, but for those like me, could turn that
          option on when needed (Manage Error Rates). I know part of this
          functionality is in CW Skimmer.....so would be nice to have the same
          option in server.



          I have just the N7TR skimmer spots showing up on my ARC4 cluster and do
          not propagate any further then to my node. Also feed the RBN Network.
          Users so far seem to like this approach....



          Telnet to dxc.n7tr.com Port 23



          Thanks....

          Rich







          From: dxatlas_group@yahoogroups.com
          [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Pete Smith
          Sent: Tuesday, February 01, 2011 8:44 AM
          To: dxatlas_group@yahoogroups.com
          Cc: Alex, VE3NEA
          Subject: Re: [dxatlas] Skim Server and Master.dta?





          Ah, I'm glad to see this topic surfacing. Working with multiple
          Skimmers feeding the RBN, we've encountered a rather different problem.

          Here's a summary from the N1MM developers group, in response to a
          complaint from N2IC about the plethora of bad spots he encountered over
          the weekend in the CQ 160 CW test:

          "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
          now, and as Steve says, it depends on your definition of error rate. If
          I watch the output of one skimmer, I see lots of spots and a very low
          error rate. But if you watch the whole RBN or even a sub-set (like the
          half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate

          is much higher, because the errors make it through while the good spots
          either come up gray (if already worked) or are duped out before they
          ever make it to the bandmap. This makes it a particularly bad problem
          late in a contest, when most of the legitimate stations have been worked

          but the op must still look at every spot to make sure it is a miscopy.

          [6:17:55 AM] Pete Smith: I will bring this to the people involved who
          actually know something (Felipe, Nick and Dave) and see if we can come
          up with a filter for the RBN that attempts to identify these situations
          and filter out the junk. It'll take some doing, but should be do-able.
          One simple-minded way would be to look at spots coming through within a
          short time window on the same frequency, and if there are spots that
          differ from the callsign that the majority are spotting correctly on
          that frequency, don't send them out."

          Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
          it be possible to do the same thing in Skimmer so that folks like Lee
          who prefer to collect their Skimmer spots themselves. Or do you have a
          better idea, Alex?

          73, Pete N4ZR

          The World Contest Station Database, updated daily at
          www.conteststations.com
          The Reverse Beacon Network at http://reversebeacon.net, blog at
          reversebeacon.blogspot.com,
          spots at telnet.reversebeacon.net, port 7000

          On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
          > Hi Lee,
          >
          > I thought about this but could not find an acceptable solution. If
          > Validation is set to Normal, almost every call is posted on the
          network when
          > it first meets the Normal validation criteria. A few seconds later,
          when
          > more data are received, the call may meet a higher validation level,
          but
          > since it has already been posted, is will not be posted again. I have
          > received tons of complaints about the same callsign posted multiple
          times,
          > and had to enforce the 10-minute rule. The net result is, all spots,
          with a
          > few exceptions, will have the Normal validation level.
          >
          > If your goal is to filter out possible errors after the aggregation of
          the
          > spots from multiple Skimmers, then one simple solution is to publish
          only
          > the callsigns reported by two or more Skimmers.
          >
          > 73 Alex VE3NEA
          >
          >
          >
          >
          >
          >
          > ----- Original Message -----
          >
          > Alex would it be possible to include a number in the Skimmer spots
          comment's
          > field that lets users know what level of call verification is being
          used?
          > It could be just before the CQ. For instance aggressive mode could be
          say
          > "3" and normal "1".
          >
          > Lee VE7CC
          >
          >
          >
          > ------------------------------------
          >
          > Yahoo! Groups Links
          >
          >
          >
          >
          >





          [Non-text portions of this message have been removed]
        • Alex, VE3NEA
          Hi Rich, Skimmer Server is designed to skim 7 bands simultaneously, and its code is optimized for decoding as many concurrent CW signals as possible. I had to
          Message 4 of 9 , Feb 1, 2011
          • 0 Attachment
            Hi Rich,

            Skimmer Server is designed to skim 7 bands simultaneously, and its code is
            optimized for decoding as many concurrent CW signals as possible. I had to
            rewrite a number of procedures in Assembler, using the SSE instructions, in
            order to save some precious CPU cycles. It would be a shame to waste those
            hard-earned cycles on searching the master.dta file, something that is not
            part of skimming and that can be easily done elsewhere, e.g., at the cluster
            node or even in the client software.

            73 Alex VE3NEA





            ----- Original Message -----
            From: "Rich Hallman - N7TR" <rich@...>
            To: <dxatlas_group@yahoogroups.com>
            Cc: "Alex, VE3NEA" <alshovk@...>
            Sent: Tuesday, February 01, 2011 12:49 PM
            Subject: RE: [dxatlas] Skim Server and Master.dta?


            > Im new to the skimmer group, but at least having the option to use a
            > master.dta or some other verification file would be nice. Not all
            > would be forced to using it, but for those like me, could turn that
            > option on when needed (Manage Error Rates). I know part of this
            > functionality is in CW Skimmer.....so would be nice to have the same
            > option in server.
            >
            >
            >
            > I have just the N7TR skimmer spots showing up on my ARC4 cluster and do
            > not propagate any further then to my node. Also feed the RBN Network.
            > Users so far seem to like this approach....
            >
            >
            >
            > Telnet to dxc.n7tr.com Port 23
            >
            >
            >
            > Thanks....
            >
            > Rich
            >
            >
            >
            >
            >
            >
            >
            > From: dxatlas_group@yahoogroups.com
            > [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Pete Smith
            > Sent: Tuesday, February 01, 2011 8:44 AM
            > To: dxatlas_group@yahoogroups.com
            > Cc: Alex, VE3NEA
            > Subject: Re: [dxatlas] Skim Server and Master.dta?
            >
            >
            >
            >
            >
            > Ah, I'm glad to see this topic surfacing. Working with multiple
            > Skimmers feeding the RBN, we've encountered a rather different problem.
            >
            > Here's a summary from the N1MM developers group, in response to a
            > complaint from N2IC about the plethora of bad spots he encountered over
            > the weekend in the CQ 160 CW test:
            >
            > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
            > now, and as Steve says, it depends on your definition of error rate. If
            > I watch the output of one skimmer, I see lots of spots and a very low
            > error rate. But if you watch the whole RBN or even a sub-set (like the
            > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate
            >
            > is much higher, because the errors make it through while the good spots
            > either come up gray (if already worked) or are duped out before they
            > ever make it to the bandmap. This makes it a particularly bad problem
            > late in a contest, when most of the legitimate stations have been worked
            >
            > but the op must still look at every spot to make sure it is a miscopy.
            >
            > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
            > actually know something (Felipe, Nick and Dave) and see if we can come
            > up with a filter for the RBN that attempts to identify these situations
            > and filter out the junk. It'll take some doing, but should be do-able.
            > One simple-minded way would be to look at spots coming through within a
            > short time window on the same frequency, and if there are spots that
            > differ from the callsign that the majority are spotting correctly on
            > that frequency, don't send them out."
            >
            > Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
            > it be possible to do the same thing in Skimmer so that folks like Lee
            > who prefer to collect their Skimmer spots themselves. Or do you have a
            > better idea, Alex?
            >
            > 73, Pete N4ZR
            >
            > The World Contest Station Database, updated daily at
            > www.conteststations.com
            > The Reverse Beacon Network at http://reversebeacon.net, blog at
            > reversebeacon.blogspot.com,
            > spots at telnet.reversebeacon.net, port 7000
            >
            > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
            >> Hi Lee,
            >>
            >> I thought about this but could not find an acceptable solution. If
            >> Validation is set to Normal, almost every call is posted on the
            > network when
            >> it first meets the Normal validation criteria. A few seconds later,
            > when
            >> more data are received, the call may meet a higher validation level,
            > but
            >> since it has already been posted, is will not be posted again. I have
            >> received tons of complaints about the same callsign posted multiple
            > times,
            >> and had to enforce the 10-minute rule. The net result is, all spots,
            > with a
            >> few exceptions, will have the Normal validation level.
            >>
            >> If your goal is to filter out possible errors after the aggregation of
            > the
            >> spots from multiple Skimmers, then one simple solution is to publish
            > only
            >> the callsigns reported by two or more Skimmers.
            >>
            >> 73 Alex VE3NEA
            >>
            >>
            >>
            >>
            >>
            >>
            >> ----- Original Message -----
            >>
            >> Alex would it be possible to include a number in the Skimmer spots
            > comment's
            >> field that lets users know what level of call verification is being
            > used?
            >> It could be just before the CQ. For instance aggressive mode could be
            > say
            >> "3" and normal "1".
            >>
            >> Lee VE7CC
            >>
            >>
            >>
            >> ------------------------------------
            >>
            >> Yahoo! Groups Links
            >>
            >>
            >>
            >>
            >>
            >
            >
            >
            >
            >
            > [Non-text portions of this message have been removed]
            >
            >
          • Rich Hallman - N7TR
            Thanks for the update Alex and makes sense..... Rich From: dxatlas_group@yahoogroups.com [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Alex, VE3NEA Sent:
            Message 5 of 9 , Feb 1, 2011
            • 0 Attachment
              Thanks for the update Alex and makes sense.....



              Rich



              From: dxatlas_group@yahoogroups.com
              [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Alex, VE3NEA
              Sent: Tuesday, February 01, 2011 10:13 AM
              To: dxatlas_group@yahoogroups.com
              Subject: Re: [dxatlas] Skim Server and Master.dta?





              Hi Rich,

              Skimmer Server is designed to skim 7 bands simultaneously, and its code
              is
              optimized for decoding as many concurrent CW signals as possible. I had
              to
              rewrite a number of procedures in Assembler, using the SSE instructions,
              in
              order to save some precious CPU cycles. It would be a shame to waste
              those
              hard-earned cycles on searching the master.dta file, something that is
              not
              part of skimming and that can be easily done elsewhere, e.g., at the
              cluster
              node or even in the client software.

              73 Alex VE3NEA

              ----- Original Message -----
              From: "Rich Hallman - N7TR" <rich@... <mailto:rich%40n7tr.com> >
              To: <dxatlas_group@yahoogroups.com
              <mailto:dxatlas_group%40yahoogroups.com> >
              Cc: "Alex, VE3NEA" <alshovk@... <mailto:alshovk%40dxatlas.com> >
              Sent: Tuesday, February 01, 2011 12:49 PM
              Subject: RE: [dxatlas] Skim Server and Master.dta?

              > Im new to the skimmer group, but at least having the option to use a
              > master.dta or some other verification file would be nice. Not all
              > would be forced to using it, but for those like me, could turn that
              > option on when needed (Manage Error Rates). I know part of this
              > functionality is in CW Skimmer.....so would be nice to have the same
              > option in server.
              >
              >
              >
              > I have just the N7TR skimmer spots showing up on my ARC4 cluster and
              do
              > not propagate any further then to my node. Also feed the RBN Network.
              > Users so far seem to like this approach....
              >
              >
              >
              > Telnet to dxc.n7tr.com Port 23
              >
              >
              >
              > Thanks....
              >
              > Rich
              >
              >
              >
              >
              >
              >
              >
              > From: dxatlas_group@yahoogroups.com
              <mailto:dxatlas_group%40yahoogroups.com>
              > [mailto:dxatlas_group@yahoogroups.com
              <mailto:dxatlas_group%40yahoogroups.com> ] On Behalf Of Pete Smith
              > Sent: Tuesday, February 01, 2011 8:44 AM
              > To: dxatlas_group@yahoogroups.com
              <mailto:dxatlas_group%40yahoogroups.com>
              > Cc: Alex, VE3NEA
              > Subject: Re: [dxatlas] Skim Server and Master.dta?
              >
              >
              >
              >
              >
              > Ah, I'm glad to see this topic surfacing. Working with multiple
              > Skimmers feeding the RBN, we've encountered a rather different
              problem.
              >
              > Here's a summary from the N1MM developers group, in response to a
              > complaint from N2IC about the plethora of bad spots he encountered
              over
              > the weekend in the CQ 160 CW test:
              >
              > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the
              problem
              > now, and as Steve says, it depends on your definition of error rate.
              If
              > I watch the output of one skimmer, I see lots of spots and a very low
              > error rate. But if you watch the whole RBN or even a sub-set (like the
              > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error
              rate
              >
              > is much higher, because the errors make it through while the good
              spots
              > either come up gray (if already worked) or are duped out before they
              > ever make it to the bandmap. This makes it a particularly bad problem
              > late in a contest, when most of the legitimate stations have been
              worked
              >
              > but the op must still look at every spot to make sure it is a miscopy.
              >
              > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
              > actually know something (Felipe, Nick and Dave) and see if we can come
              > up with a filter for the RBN that attempts to identify these
              situations
              > and filter out the junk. It'll take some doing, but should be do-able.
              > One simple-minded way would be to look at spots coming through within
              a
              > short time window on the same frequency, and if there are spots that
              > differ from the callsign that the majority are spotting correctly on
              > that frequency, don't send them out."
              >
              > Kind of a voting system, I guess. Anyhow, the RBN solution aside.
              might
              > it be possible to do the same thing in Skimmer so that folks like Lee
              > who prefer to collect their Skimmer spots themselves. Or do you have a
              > better idea, Alex?
              >
              > 73, Pete N4ZR
              >
              > The World Contest Station Database, updated daily at
              > www.conteststations.com
              > The Reverse Beacon Network at http://reversebeacon.net, blog at
              > reversebeacon.blogspot.com,
              > spots at telnet.reversebeacon.net, port 7000
              >
              > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
              >> Hi Lee,
              >>
              >> I thought about this but could not find an acceptable solution. If
              >> Validation is set to Normal, almost every call is posted on the
              > network when
              >> it first meets the Normal validation criteria. A few seconds later,
              > when
              >> more data are received, the call may meet a higher validation level,
              > but
              >> since it has already been posted, is will not be posted again. I have
              >> received tons of complaints about the same callsign posted multiple
              > times,
              >> and had to enforce the 10-minute rule. The net result is, all spots,
              > with a
              >> few exceptions, will have the Normal validation level.
              >>
              >> If your goal is to filter out possible errors after the aggregation
              of
              > the
              >> spots from multiple Skimmers, then one simple solution is to publish
              > only
              >> the callsigns reported by two or more Skimmers.
              >>
              >> 73 Alex VE3NEA
              >>
              >>
              >>
              >>
              >>
              >>
              >> ----- Original Message -----
              >>
              >> Alex would it be possible to include a number in the Skimmer spots
              > comment's
              >> field that lets users know what level of call verification is being
              > used?
              >> It could be just before the CQ. For instance aggressive mode could be
              > say
              >> "3" and normal "1".
              >>
              >> Lee VE7CC
              >>
              >>
              >>
              >> ------------------------------------
              >>
              >> Yahoo! Groups Links
              >>
              >>
              >>
              >>
              >>
              >
              >
              >
              >
              >
              > [Non-text portions of this message have been removed]
              >
              >





              [Non-text portions of this message have been removed]
            Your message has been successfully submitted and would be delivered to recipients shortly.