Loading ...
Sorry, an error occurred while loading the content.

Re: [dxatlas] Skim Server and Master.dta?

Expand Messages
  • Alex, VE3NEA
    Hi Pete, As I have already written, a program that receives spots from multiple Skimmers can radically reduce the number of errors simply by showing only the
    Message 1 of 9 , Feb 1, 2011
    • 0 Attachment
      Hi Pete,

      As I have already written, a program that receives spots from multiple
      Skimmers can radically reduce the number of errors simply by showing only
      the spots received from two or more Skimmers. This is the same principle
      that is used for Aggressive filtering in CW Skimmer, except that instead of
      waiting for more copies of the callsign decoded by the same Skimmer, we
      would get those copies instantly from other Skimmers. In other words,
      instead of time diversity we would use spatial diversity.

      73 Alex VE3NEA





      ----- Original Message -----
      From: "Pete Smith" <n4zr@...>
      To: <dxatlas_group@yahoogroups.com>
      Cc: "Alex, VE3NEA" <alshovk@...>
      Sent: Tuesday, February 01, 2011 11:44 AM
      Subject: Re: [dxatlas] Skim Server and Master.dta?


      > Ah, I'm glad to see this topic surfacing. Working with multiple Skimmers
      > feeding the RBN, we've encountered a rather different problem.
      >
      > Here's a summary from the N1MM developers group, in response to a
      > complaint from N2IC about the plethora of bad spots he encountered over
      > the weekend in the CQ 160 CW test:
      >
      > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
      > now, and as Steve says, it depends on your definition of error rate. If I
      > watch the output of one skimmer, I see lots of spots and a very low error
      > rate. But if you watch the whole RBN or even a sub-set (like the
      > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate
      > is much higher, because the errors make it through while the good spots
      > either come up gray (if already worked) or are duped out before they ever
      > make it to the bandmap. This makes it a particularly bad problem late in
      > a contest, when most of the legitimate stations have been worked but the
      > op must still look at every spot to make sure it is a miscopy.
      >
      > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
      > actually know something (Felipe, Nick and Dave) and see if we can come up
      > with a filter for the RBN that attempts to identify these situations and
      > filter out the junk. It'll take some doing, but should be do-able. One
      > simple-minded way would be to look at spots coming through within a short
      > time window on the same frequency, and if there are spots that differ
      > from the callsign that the majority are spotting correctly on that
      > frequency, don't send them out."
      >
      > Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
      > it be possible to do the same thing in Skimmer so that folks like Lee who
      > prefer to collect their Skimmer spots themselves. Or do you have a better
      > idea, Alex?
      >
      > 73, Pete N4ZR
      >
      > The World Contest Station Database, updated daily at
      > www.conteststations.com
      > The Reverse Beacon Network at http://reversebeacon.net, blog at
      > reversebeacon.blogspot.com,
      > spots at telnet.reversebeacon.net, port 7000
      >
      >
      >
      > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
      >> Hi Lee,
      >>
      >> I thought about this but could not find an acceptable solution. If
      >> Validation is set to Normal, almost every call is posted on the network
      >> when
      >> it first meets the Normal validation criteria. A few seconds later, when
      >> more data are received, the call may meet a higher validation level, but
      >> since it has already been posted, is will not be posted again. I have
      >> received tons of complaints about the same callsign posted multiple
      >> times,
      >> and had to enforce the 10-minute rule. The net result is, all spots, with
      >> a
      >> few exceptions, will have the Normal validation level.
      >>
      >> If your goal is to filter out possible errors after the aggregation of
      >> the
      >> spots from multiple Skimmers, then one simple solution is to publish only
      >> the callsigns reported by two or more Skimmers.
      >>
      >> 73 Alex VE3NEA
      >>
      >>
    • Rich Hallman - N7TR
      Im new to the skimmer group, but at least having the option to use a master.dta or some other verification file would be nice. Not all would be forced to
      Message 2 of 9 , Feb 1, 2011
      • 0 Attachment
        Im new to the skimmer group, but at least having the option to use a
        master.dta or some other verification file would be nice. Not all
        would be forced to using it, but for those like me, could turn that
        option on when needed (Manage Error Rates). I know part of this
        functionality is in CW Skimmer.....so would be nice to have the same
        option in server.



        I have just the N7TR skimmer spots showing up on my ARC4 cluster and do
        not propagate any further then to my node. Also feed the RBN Network.
        Users so far seem to like this approach....



        Telnet to dxc.n7tr.com Port 23



        Thanks....

        Rich







        From: dxatlas_group@yahoogroups.com
        [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Pete Smith
        Sent: Tuesday, February 01, 2011 8:44 AM
        To: dxatlas_group@yahoogroups.com
        Cc: Alex, VE3NEA
        Subject: Re: [dxatlas] Skim Server and Master.dta?





        Ah, I'm glad to see this topic surfacing. Working with multiple
        Skimmers feeding the RBN, we've encountered a rather different problem.

        Here's a summary from the N1MM developers group, in response to a
        complaint from N2IC about the plethora of bad spots he encountered over
        the weekend in the CQ 160 CW test:

        "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
        now, and as Steve says, it depends on your definition of error rate. If
        I watch the output of one skimmer, I see lots of spots and a very low
        error rate. But if you watch the whole RBN or even a sub-set (like the
        half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate

        is much higher, because the errors make it through while the good spots
        either come up gray (if already worked) or are duped out before they
        ever make it to the bandmap. This makes it a particularly bad problem
        late in a contest, when most of the legitimate stations have been worked

        but the op must still look at every spot to make sure it is a miscopy.

        [6:17:55 AM] Pete Smith: I will bring this to the people involved who
        actually know something (Felipe, Nick and Dave) and see if we can come
        up with a filter for the RBN that attempts to identify these situations
        and filter out the junk. It'll take some doing, but should be do-able.
        One simple-minded way would be to look at spots coming through within a
        short time window on the same frequency, and if there are spots that
        differ from the callsign that the majority are spotting correctly on
        that frequency, don't send them out."

        Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
        it be possible to do the same thing in Skimmer so that folks like Lee
        who prefer to collect their Skimmer spots themselves. Or do you have a
        better idea, Alex?

        73, Pete N4ZR

        The World Contest Station Database, updated daily at
        www.conteststations.com
        The Reverse Beacon Network at http://reversebeacon.net, blog at
        reversebeacon.blogspot.com,
        spots at telnet.reversebeacon.net, port 7000

        On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
        > Hi Lee,
        >
        > I thought about this but could not find an acceptable solution. If
        > Validation is set to Normal, almost every call is posted on the
        network when
        > it first meets the Normal validation criteria. A few seconds later,
        when
        > more data are received, the call may meet a higher validation level,
        but
        > since it has already been posted, is will not be posted again. I have
        > received tons of complaints about the same callsign posted multiple
        times,
        > and had to enforce the 10-minute rule. The net result is, all spots,
        with a
        > few exceptions, will have the Normal validation level.
        >
        > If your goal is to filter out possible errors after the aggregation of
        the
        > spots from multiple Skimmers, then one simple solution is to publish
        only
        > the callsigns reported by two or more Skimmers.
        >
        > 73 Alex VE3NEA
        >
        >
        >
        >
        >
        >
        > ----- Original Message -----
        >
        > Alex would it be possible to include a number in the Skimmer spots
        comment's
        > field that lets users know what level of call verification is being
        used?
        > It could be just before the CQ. For instance aggressive mode could be
        say
        > "3" and normal "1".
        >
        > Lee VE7CC
        >
        >
        >
        > ------------------------------------
        >
        > Yahoo! Groups Links
        >
        >
        >
        >
        >





        [Non-text portions of this message have been removed]
      • Alex, VE3NEA
        Hi Rich, Skimmer Server is designed to skim 7 bands simultaneously, and its code is optimized for decoding as many concurrent CW signals as possible. I had to
        Message 3 of 9 , Feb 1, 2011
        • 0 Attachment
          Hi Rich,

          Skimmer Server is designed to skim 7 bands simultaneously, and its code is
          optimized for decoding as many concurrent CW signals as possible. I had to
          rewrite a number of procedures in Assembler, using the SSE instructions, in
          order to save some precious CPU cycles. It would be a shame to waste those
          hard-earned cycles on searching the master.dta file, something that is not
          part of skimming and that can be easily done elsewhere, e.g., at the cluster
          node or even in the client software.

          73 Alex VE3NEA





          ----- Original Message -----
          From: "Rich Hallman - N7TR" <rich@...>
          To: <dxatlas_group@yahoogroups.com>
          Cc: "Alex, VE3NEA" <alshovk@...>
          Sent: Tuesday, February 01, 2011 12:49 PM
          Subject: RE: [dxatlas] Skim Server and Master.dta?


          > Im new to the skimmer group, but at least having the option to use a
          > master.dta or some other verification file would be nice. Not all
          > would be forced to using it, but for those like me, could turn that
          > option on when needed (Manage Error Rates). I know part of this
          > functionality is in CW Skimmer.....so would be nice to have the same
          > option in server.
          >
          >
          >
          > I have just the N7TR skimmer spots showing up on my ARC4 cluster and do
          > not propagate any further then to my node. Also feed the RBN Network.
          > Users so far seem to like this approach....
          >
          >
          >
          > Telnet to dxc.n7tr.com Port 23
          >
          >
          >
          > Thanks....
          >
          > Rich
          >
          >
          >
          >
          >
          >
          >
          > From: dxatlas_group@yahoogroups.com
          > [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Pete Smith
          > Sent: Tuesday, February 01, 2011 8:44 AM
          > To: dxatlas_group@yahoogroups.com
          > Cc: Alex, VE3NEA
          > Subject: Re: [dxatlas] Skim Server and Master.dta?
          >
          >
          >
          >
          >
          > Ah, I'm glad to see this topic surfacing. Working with multiple
          > Skimmers feeding the RBN, we've encountered a rather different problem.
          >
          > Here's a summary from the N1MM developers group, in response to a
          > complaint from N2IC about the plethora of bad spots he encountered over
          > the weekend in the CQ 160 CW test:
          >
          > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the problem
          > now, and as Steve says, it depends on your definition of error rate. If
          > I watch the output of one skimmer, I see lots of spots and a very low
          > error rate. But if you watch the whole RBN or even a sub-set (like the
          > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error rate
          >
          > is much higher, because the errors make it through while the good spots
          > either come up gray (if already worked) or are duped out before they
          > ever make it to the bandmap. This makes it a particularly bad problem
          > late in a contest, when most of the legitimate stations have been worked
          >
          > but the op must still look at every spot to make sure it is a miscopy.
          >
          > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
          > actually know something (Felipe, Nick and Dave) and see if we can come
          > up with a filter for the RBN that attempts to identify these situations
          > and filter out the junk. It'll take some doing, but should be do-able.
          > One simple-minded way would be to look at spots coming through within a
          > short time window on the same frequency, and if there are spots that
          > differ from the callsign that the majority are spotting correctly on
          > that frequency, don't send them out."
          >
          > Kind of a voting system, I guess. Anyhow, the RBN solution aside. might
          > it be possible to do the same thing in Skimmer so that folks like Lee
          > who prefer to collect their Skimmer spots themselves. Or do you have a
          > better idea, Alex?
          >
          > 73, Pete N4ZR
          >
          > The World Contest Station Database, updated daily at
          > www.conteststations.com
          > The Reverse Beacon Network at http://reversebeacon.net, blog at
          > reversebeacon.blogspot.com,
          > spots at telnet.reversebeacon.net, port 7000
          >
          > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
          >> Hi Lee,
          >>
          >> I thought about this but could not find an acceptable solution. If
          >> Validation is set to Normal, almost every call is posted on the
          > network when
          >> it first meets the Normal validation criteria. A few seconds later,
          > when
          >> more data are received, the call may meet a higher validation level,
          > but
          >> since it has already been posted, is will not be posted again. I have
          >> received tons of complaints about the same callsign posted multiple
          > times,
          >> and had to enforce the 10-minute rule. The net result is, all spots,
          > with a
          >> few exceptions, will have the Normal validation level.
          >>
          >> If your goal is to filter out possible errors after the aggregation of
          > the
          >> spots from multiple Skimmers, then one simple solution is to publish
          > only
          >> the callsigns reported by two or more Skimmers.
          >>
          >> 73 Alex VE3NEA
          >>
          >>
          >>
          >>
          >>
          >>
          >> ----- Original Message -----
          >>
          >> Alex would it be possible to include a number in the Skimmer spots
          > comment's
          >> field that lets users know what level of call verification is being
          > used?
          >> It could be just before the CQ. For instance aggressive mode could be
          > say
          >> "3" and normal "1".
          >>
          >> Lee VE7CC
          >>
          >>
          >>
          >> ------------------------------------
          >>
          >> Yahoo! Groups Links
          >>
          >>
          >>
          >>
          >>
          >
          >
          >
          >
          >
          > [Non-text portions of this message have been removed]
          >
          >
        • Rich Hallman - N7TR
          Thanks for the update Alex and makes sense..... Rich From: dxatlas_group@yahoogroups.com [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Alex, VE3NEA Sent:
          Message 4 of 9 , Feb 1, 2011
          • 0 Attachment
            Thanks for the update Alex and makes sense.....



            Rich



            From: dxatlas_group@yahoogroups.com
            [mailto:dxatlas_group@yahoogroups.com] On Behalf Of Alex, VE3NEA
            Sent: Tuesday, February 01, 2011 10:13 AM
            To: dxatlas_group@yahoogroups.com
            Subject: Re: [dxatlas] Skim Server and Master.dta?





            Hi Rich,

            Skimmer Server is designed to skim 7 bands simultaneously, and its code
            is
            optimized for decoding as many concurrent CW signals as possible. I had
            to
            rewrite a number of procedures in Assembler, using the SSE instructions,
            in
            order to save some precious CPU cycles. It would be a shame to waste
            those
            hard-earned cycles on searching the master.dta file, something that is
            not
            part of skimming and that can be easily done elsewhere, e.g., at the
            cluster
            node or even in the client software.

            73 Alex VE3NEA

            ----- Original Message -----
            From: "Rich Hallman - N7TR" <rich@... <mailto:rich%40n7tr.com> >
            To: <dxatlas_group@yahoogroups.com
            <mailto:dxatlas_group%40yahoogroups.com> >
            Cc: "Alex, VE3NEA" <alshovk@... <mailto:alshovk%40dxatlas.com> >
            Sent: Tuesday, February 01, 2011 12:49 PM
            Subject: RE: [dxatlas] Skim Server and Master.dta?

            > Im new to the skimmer group, but at least having the option to use a
            > master.dta or some other verification file would be nice. Not all
            > would be forced to using it, but for those like me, could turn that
            > option on when needed (Manage Error Rates). I know part of this
            > functionality is in CW Skimmer.....so would be nice to have the same
            > option in server.
            >
            >
            >
            > I have just the N7TR skimmer spots showing up on my ARC4 cluster and
            do
            > not propagate any further then to my node. Also feed the RBN Network.
            > Users so far seem to like this approach....
            >
            >
            >
            > Telnet to dxc.n7tr.com Port 23
            >
            >
            >
            > Thanks....
            >
            > Rich
            >
            >
            >
            >
            >
            >
            >
            > From: dxatlas_group@yahoogroups.com
            <mailto:dxatlas_group%40yahoogroups.com>
            > [mailto:dxatlas_group@yahoogroups.com
            <mailto:dxatlas_group%40yahoogroups.com> ] On Behalf Of Pete Smith
            > Sent: Tuesday, February 01, 2011 8:44 AM
            > To: dxatlas_group@yahoogroups.com
            <mailto:dxatlas_group%40yahoogroups.com>
            > Cc: Alex, VE3NEA
            > Subject: Re: [dxatlas] Skim Server and Master.dta?
            >
            >
            >
            >
            >
            > Ah, I'm glad to see this topic surfacing. Working with multiple
            > Skimmers feeding the RBN, we've encountered a rather different
            problem.
            >
            > Here's a summary from the N1MM developers group, in response to a
            > complaint from N2IC about the plethora of bad spots he encountered
            over
            > the weekend in the CQ 160 CW test:
            >
            > "[4:07:09 PM | Edited 4:07:49 PM] Pete Smith: I think I see the
            problem
            > now, and as Steve says, it depends on your definition of error rate.
            If
            > I watch the output of one skimmer, I see lots of spots and a very low
            > error rate. But if you watch the whole RBN or even a sub-set (like the
            > half-dozen Skimmers in PA, MD, VA, and WV) then the effective error
            rate
            >
            > is much higher, because the errors make it through while the good
            spots
            > either come up gray (if already worked) or are duped out before they
            > ever make it to the bandmap. This makes it a particularly bad problem
            > late in a contest, when most of the legitimate stations have been
            worked
            >
            > but the op must still look at every spot to make sure it is a miscopy.
            >
            > [6:17:55 AM] Pete Smith: I will bring this to the people involved who
            > actually know something (Felipe, Nick and Dave) and see if we can come
            > up with a filter for the RBN that attempts to identify these
            situations
            > and filter out the junk. It'll take some doing, but should be do-able.
            > One simple-minded way would be to look at spots coming through within
            a
            > short time window on the same frequency, and if there are spots that
            > differ from the callsign that the majority are spotting correctly on
            > that frequency, don't send them out."
            >
            > Kind of a voting system, I guess. Anyhow, the RBN solution aside.
            might
            > it be possible to do the same thing in Skimmer so that folks like Lee
            > who prefer to collect their Skimmer spots themselves. Or do you have a
            > better idea, Alex?
            >
            > 73, Pete N4ZR
            >
            > The World Contest Station Database, updated daily at
            > www.conteststations.com
            > The Reverse Beacon Network at http://reversebeacon.net, blog at
            > reversebeacon.blogspot.com,
            > spots at telnet.reversebeacon.net, port 7000
            >
            > On 2/1/2011 9:43 AM, Alex, VE3NEA wrote:
            >> Hi Lee,
            >>
            >> I thought about this but could not find an acceptable solution. If
            >> Validation is set to Normal, almost every call is posted on the
            > network when
            >> it first meets the Normal validation criteria. A few seconds later,
            > when
            >> more data are received, the call may meet a higher validation level,
            > but
            >> since it has already been posted, is will not be posted again. I have
            >> received tons of complaints about the same callsign posted multiple
            > times,
            >> and had to enforce the 10-minute rule. The net result is, all spots,
            > with a
            >> few exceptions, will have the Normal validation level.
            >>
            >> If your goal is to filter out possible errors after the aggregation
            of
            > the
            >> spots from multiple Skimmers, then one simple solution is to publish
            > only
            >> the callsigns reported by two or more Skimmers.
            >>
            >> 73 Alex VE3NEA
            >>
            >>
            >>
            >>
            >>
            >>
            >> ----- Original Message -----
            >>
            >> Alex would it be possible to include a number in the Skimmer spots
            > comment's
            >> field that lets users know what level of call verification is being
            > used?
            >> It could be just before the CQ. For instance aggressive mode could be
            > say
            >> "3" and normal "1".
            >>
            >> Lee VE7CC
            >>
            >>
            >>
            >> ------------------------------------
            >>
            >> Yahoo! Groups Links
            >>
            >>
            >>
            >>
            >>
            >
            >
            >
            >
            >
            > [Non-text portions of this message have been removed]
            >
            >





            [Non-text portions of this message have been removed]
          Your message has been successfully submitted and would be delivered to recipients shortly.