Loading ...
Sorry, an error occurred while loading the content.
 

Removing duplicate lines

Expand Messages
  • Charles or Mary Friederich
    Clippers, I m looking for a clip that seems so basic, it s probably already been written. See this line. See this line. See this line. See this additional
    Message 1 of 7 , Jul 5, 2005
      Clippers, I'm looking for a clip that seems so basic, it's probably already
      been written.

      See this line.
      See this line.
      See this line.
      See this additional line.

      I'd like a clip that will delete duplicates of the same line (leaving one
      copy), so that the output is:

      See this line.
      See this additional line.

      As a bonus, this dream clip would copy the deleted duplicates to a new
      window, so that I can check what's being tossed out. This new window would
      look like this:

      See this line.
      See this line.

      Thanks for the help. Charlie.
    • Jeffery Scism
      ... The following was made to look for duplicated phrases in Websites, but may be adaptable to what you need. ^!ClearVariables ^!SetHintInfo Working...
      Message 2 of 7 , Jul 5, 2005
        Charles or Mary Friederich wrote:

        >Clippers, I'm looking for a clip that seems so basic, it's probably already
        >been written.
        >
        >See this line.
        >See this line.
        >See this line.
        >See this additional line.
        >
        >I'd like a clip that will delete duplicates of the same line (leaving one
        >copy), so that the output is:
        >
        >See this line.
        >See this additional line.
        >
        >As a bonus, this dream clip would copy the deleted duplicates to a new
        >window, so that I can check what's being tossed out. This new window would
        >look like this:
        >
        >See this line.
        >See this line.
        >
        >Thanks for the help. Charlie.
        >
        >
        >
        >
        >
        The following was made to look for duplicated "phrases" in Websites, but
        may be adaptable to what you need.

        ^!ClearVariables
        ^!SetHintInfo Working...
        ^!SetScreenUpdate Off

        ^!Set %FileList%=^?{(T=D)Search directory=C:\<ENTER YOUR DIRECTORY
        HERE>}; %DoSubs%=^?{Also check in subdirectories=_Yes^=+|No^=};
        %Search4%=^?{Search for what duplicate=_<body|^=}; %Case%=^?{Ignore
        character case=_Yes^=I|No^=}; %BoxChecked%=^?{Check boxes in duplicated
        list wizard automatically=Yes^I|_No^=}
        ^!SetListDelimiter ^p
        ^!SetArray %FileList%=^$GetFiles("^%DoSubs%^%FileList%";*.htm*)$
        ^!Set %Count%=^%FileList0%
        ^!Set %Index%=0

        :Loop1
        ^!SetScreenUpdate Off
        ^!Inc %Index%
        ^!If ^%Index% > ^%Count% ShowList
        ^!Open "^%FileList^%Index%%"
        ^!Find "^%Search4%" S^%Case%
        ^!IfError CloseFile
        ^!Find "^%Search4%" S^%Case%
        ^!IfError CloseFile
        ^!Set %FoundTwice%=^##
        ^!Append %DupList%=^%FoundTwice%|
        ^!Close "^%FileList^%Index%%"
        ^!Goto Loop1

        :CloseFile
        ^!Close "^%FileList^%Index%%"
        ^!Goto Loop1

        :ShowList
        ^!IfTrue ^%BoxChecked% Next else Skip
        ^!Set %DupList%=_^$StrReplace("|";"|_";"^%DupList%";0;0)$
        ^!Set %DupList%=^$StrDelete("^%DupList%";^$StrSize("^%DupList%")$;1)$
        ^!IfTrue ^$IsEmpty(^%DupList%)$ Error
        ^!SetWizardTitle Check boxes of files that you want to open
        ^!Set %DupList%=^?{(T=A;H=20)==^%DupList%}
        ^!SetListDelimiter ^p
        ^!SetArray %DupList%=^%DupList%
        ^!Set %Count2%=^%DupList0%
        ^!Set %Index2%=0

        :Loop2
        ^!Inc %Index2%
        ^!If ^%Index2% > ^%Count2% End
        ^!Open "^%DupList^%Index2%%"
        ^!Find "^%Search4%" S^%Case%
        ^!SetView ^$GetRow$:^$GetCol$
        ^!Jump Select_Start
        ^!Goto Loop2

        :Error
        ^!Info No files were found with duplicates in them.


        --
        ~~

        Jeffery G. Scism, IBSSG

        For those who fight for it, Freedom has a flavor the protected will never know.
        Those who have fought for freedom, and have lost it, know a sadness and loss beyond bearing.
        Freedom rings loudest for those denied Freedom.

        S250 DNA Project- http://homepages.rootsweb.com/~scismfam/s250/
        Crawley DNA Project
        Use the link below to join the project and purchase your DNA test.
        http://www.familytreedna.com/surname_join.asp?code=B82518
        International Blacksheep Society of Genealogists
        Montgomery County Indiana USGenWeb site
      • Jody
        Hi Charles, ... Try this one on a copy... H=Remove Dups-Trips ; eMailed on 07-05-2005, ntb-clips@yahoogroups.com, Jody ;
        Message 3 of 7 , Jul 5, 2005
          Hi Charles,

          >I'd like a clip that will delete duplicates of the same line (leaving one
          >copy)...


          Try this one on a copy...

          <--- Copy below this line --->
          H=Remove Dups-Trips
          ; eMailed on 07-05-2005, ntb-clips@yahoogroups.com, Jody
          ; http://www.fookes.us, http://www.sojourner.us/software

          ; Last updated on 07-05-2005 by Jody Adair <Jody@...>
          ^!ClearVariables
          ^!Jump Doc_End
          ^!InsertText ^p^p
          ^!Jump 1
          ^!SetWizardLabel Dups, Trips, Quats removed ;)
          ^!Set %Prompt%=^?{Prompt before deleting?==Yes^=1|_No^=0}; %Save%=Lines deleted:^%nl%^%nl%

          ^!IfFalse ^$IsWordWrap$ Start
          ^!ToolBar Toggle Word Wrap
          ^!Set %WrapOn%=1

          :Start
          ^!Select Eol
          ^!Set %Row%=^$GetRow$; %Data%=^$GetSelection$

          :Find
          ^!Find "^%Data%" IS
          ^!IfError NextRow
          ^!If "^$GetLineSize$" > "^$GetSelSize$" Find
          ^!Append %Save%=^%Data%^%nl%
          ^!If "^$GetRow$" = "^$GetLineCount$" Wrap
          ^!IfTrue ^%Prompt% Skip_2
          ^!DeleteLine
          ^!Goto Find
          ^!Skip Delete this line?
          ^!Goto Find
          ^!DeleteLine
          ^!If "^$GetRow$" = "^$GetLineCount$" Next Else Find

          :NextRow
          ^!If ^$GetRow$ = ^$GetLineCount$ Wrap
          ^!Inc %Row%
          ^!Jump ^%Row%
          ^!Goto Start

          :Wrap
          ^!IfTrue ^%WrapOn% Next else Skip
          ^!ToolBar Toggle Word Wrap
          ^!Jump Doc_End

          :Loop1
          ^!IfFalse ^$IsEmpty(^$GetLine$)$ Skip_2
          ^!Replace "^p" >> "" SB
          ^!Goto Loop1
          ^!InsertText ^p

          :Info
          ^!SetWizardTitle ~ Duplicates Removed ::: http://www.sojourner.us/software ~
          ^!SetClipboard ^%Save%
          ^!Toolbar Paste New

          :Loop2
          ^!IfFalse ^$IsEmpty(^$GetLine$)$ Skip_2
          ^!Replace "^p" >> "" SB
          ^!Goto Loop2
          ^!InsertText ^p

          <--- Copy above this line --->
          <--- Right click over a Library --->
          <--- Choose: Add from Clipboard --->

          I used this test text:

          See this line.
          See this line.
          See this line.
          See this additional line.
          See this additional line.
          See this line.
          See this additional line.
          See this additional line.
          See this line.
          See this additional line.
          See this line.
          See this additional line.
          See this line.
          See this line.
          See this line.
          See this additional line.


          Happy Clip'n!
          Jody

          All Lists: http://www.fookes.us/maillist.htm
          Convert your eMail programs mail to another program, archive,
          date and word filters for inclusion of certain messages. You can
          then archive say be day, week, month, year, etc. Aid4Mail:
          http://www.fookes/aid4mail, http://www.sojourner.us/software
          See ya in the funnies!-) http://www.clean-funnies.com
        • Jody
          Hi Charles, I forgot to mention that you have to have the first word(s) on line one, the first line. It drove me nuts trying to debug my Clip written way back
          Message 4 of 7 , Jul 5, 2005
            Hi Charles,

            I forgot to mention that you have to have the first word(s) on
            line one, the first line. It drove me nuts trying to debug my
            Clip written way back when. I had a blank line on top when I was
            doing that, then remembered that the top and bottom of documents
            can be very picky when running Clips that do all lines. I didn't
            allow for blank lines either. Not going to either. :-) I've been
            up all night and about brain dead now.

            >> I'd like a clip that will delete duplicates of the same line
            >> (leaving one copy)...
            >
            >Try this one on a copy...
            >
            ><--- Copy below this line --->
            >H=Remove Dups-Trips
            >; eMailed on 07-05-2005, ntb-clips@yahoogroups.com, Jody
            >; http://www.fookes.us, http://www.sojourner.us/software


            See ya in the funnies,
            Jody Adair, Prov. 15:15

            Blessed are they who can laugh at themselves
            for they shall never cease to be amused.
            http://www.clean-funnies.com
          • sleazzyweasel
            Jody, I m a novice to notetab pro. I m looking to remove duplicates in an email mailing list file. The file only has email addresses. I think that I have to
            Message 5 of 7 , Nov 30, 2005
              Jody,

              I'm a novice to notetab pro. I'm looking to remove duplicates in an
              email mailing list file. The file only has email addresses. I think
              that I have to set up a library to use your clip, but I have NO idea
              how to go about doing it or where to put the clip. I was wondering
              if you could give a novice some assistance??? Thanks. Mike

              --- In ntb-clips@yahoogroups.com, Jody <kjv-av1611@e...> wrote:
              >
              > Hi Charles,
              >
              > >I'd like a clip that will delete duplicates of the same line
              (leaving one
              > >copy)...
              >
              >
              > Try this one on a copy...
              >
              > <--- Copy below this line --->
              > H=Remove Dups-Trips
              > ; eMailed on 07-05-2005, ntb-clips@yahoogroups.com, Jody
              > ; http://www.fookes.us, http://www.sojourner.us/software
              >
              > ; Last updated on 07-05-2005 by Jody Adair <Jody@F...>
              > ^!ClearVariables
              > ^!Jump Doc_End
              > ^!InsertText ^p^p
              > ^!Jump 1
              > ^!SetWizardLabel Dups, Trips, Quats removed ;)
              > ^!Set %Prompt%=^?{Prompt before deleting?==Yes^=1|_No^=0}; %Save%
              =Lines deleted:^%nl%^%nl%
              >
              > ^!IfFalse ^$IsWordWrap$ Start
              > ^!ToolBar Toggle Word Wrap
              > ^!Set %WrapOn%=1
              >
              > :Start
              > ^!Select Eol
              > ^!Set %Row%=^$GetRow$; %Data%=^$GetSelection$
              >
              > :Find
              > ^!Find "^%Data%" IS
              > ^!IfError NextRow
              > ^!If "^$GetLineSize$" > "^$GetSelSize$" Find
              > ^!Append %Save%=^%Data%^%nl%
              > ^!If "^$GetRow$" = "^$GetLineCount$" Wrap
              > ^!IfTrue ^%Prompt% Skip_2
              > ^!DeleteLine
              > ^!Goto Find
              > ^!Skip Delete this line?
              > ^!Goto Find
              > ^!DeleteLine
              > ^!If "^$GetRow$" = "^$GetLineCount$" Next Else Find
              >
              > :NextRow
              > ^!If ^$GetRow$ = ^$GetLineCount$ Wrap
              > ^!Inc %Row%
              > ^!Jump ^%Row%
              > ^!Goto Start
              >
              > :Wrap
              > ^!IfTrue ^%WrapOn% Next else Skip
              > ^!ToolBar Toggle Word Wrap
              > ^!Jump Doc_End
              >
              > :Loop1
              > ^!IfFalse ^$IsEmpty(^$GetLine$)$ Skip_2
              > ^!Replace "^p" >> "" SB
              > ^!Goto Loop1
              > ^!InsertText ^p
              >
              > :Info
              > ^!SetWizardTitle ~ Duplicates Removed :::
              http://www.sojourner.us/software ~
              > ^!SetClipboard ^%Save%
              > ^!Toolbar Paste New
              >
              > :Loop2
              > ^!IfFalse ^$IsEmpty(^$GetLine$)$ Skip_2
              > ^!Replace "^p" >> "" SB
              > ^!Goto Loop2
              > ^!InsertText ^p
              >
              > <--- Copy above this line --->
              > <--- Right click over a Library --->
              > <--- Choose: Add from Clipboard --->
              >
              > I used this test text:
              >
              > See this line.
              > See this line.
              > See this line.
              > See this additional line.
              > See this additional line.
              > See this line.
              > See this additional line.
              > See this additional line.
              > See this line.
              > See this additional line.
              > See this line.
              > See this additional line.
              > See this line.
              > See this line.
              > See this line.
              > See this additional line.
              >
              >
              > Happy Clip'n!
              > Jody
              >
              > All Lists: http://www.fookes.us/maillist.htm
              > Convert your eMail programs mail to another program, archive,
              > date and word filters for inclusion of certain messages. You can
              > then archive say be day, week, month, year, etc. Aid4Mail:
              > http://www.fookes/aid4mail, http://www.sojourner.us/software
              > See ya in the funnies!-) http://www.clean-funnies.com
              >
            • Hugo Paulissen
              ... Mike, I m not Jody, but I can give you some assistance. The easiest for you would be to sort your file, with the option remove duplicates checked. Select
              Message 6 of 7 , Nov 30, 2005
                > Jody,
                >
                > I'm a novice to notetab pro. I'm looking to remove duplicates in an
                > email mailing list file. The file only has email addresses. I think
                > that I have to set up a library to use your clip, but I have NO idea
                > how to go about doing it or where to put the clip. I was wondering
                > if you could give a novice some assistance??? Thanks. Mike
                >

                Mike,

                I'm not Jody, but I can give you some assistance. The easiest for you
                would be to sort your file, with the option "remove duplicates" checked.

                Select Menu: Modify | Lines | Sort | Ascending/Descending
                (You may have a sort button available in the toolbar as well)

                In order to remove the duplicates you should check wether in Options
                (menu View), on the Tools tab, you have a checkmark in front of Sort
                Removes Duplicates.

                Hugo
              • Sleazzy Weasel
                Hugo, Thanks a heap. It worked llike a charm. :-) Mike ... Mike, I m not Jody, but I can give you some assistance. The easiest for you would be to sort your
                Message 7 of 7 , Nov 30, 2005
                  Hugo,

                  Thanks a heap. It worked llike a charm. :-)

                  Mike

                  Hugo Paulissen <hugopaulissen@...> wrote:
                  > Jody,
                  >
                  > I'm a novice to notetab pro. I'm looking to remove duplicates in an
                  > email mailing list file. The file only has email addresses. I think
                  > that I have to set up a library to use your clip, but I have NO idea
                  > how to go about doing it or where to put the clip. I was wondering
                  > if you could give a novice some assistance??? Thanks. Mike
                  >

                  Mike,

                  I'm not Jody, but I can give you some assistance. The easiest for you
                  would be to sort your file, with the option "remove duplicates" checked.

                  Select Menu: Modify | Lines | Sort | Ascending/Descending
                  (You may have a sort button available in the toolbar as well)

                  In order to remove the duplicates you should check wether in Options
                  (menu View), on the Tools tab, you have a checkmark in front of Sort
                  Removes Duplicates.

                  Hugo






                  Fookes Software: http://www.fookes.us, http://www.fookes.com
                  Fookes Software Mailing Lists: http://www.fookes.us/maillist.htm




                  ---------------------------------
                  YAHOO! GROUPS LINKS


                  Visit your group "ntb-clips" on the web.

                  To unsubscribe from this group, send an email to:
                  ntb-clips-unsubscribe@yahoogroups.com

                  Your use of Yahoo! Groups is subject to the Yahoo! Terms of Service.


                  ---------------------------------







                  **:::::>>:§«:*´`³¤³´`*:»§:<<:::::**
                  --------------------------------------
                  .·°~ §LëAzZÿ WëA§ëL ~°·.
                  --------------------------------------
                  **:::::>>:§«:*´`³¤³´`*:»§:<<:::::**
                  sleazzyweasel@...
                  WrOnG NuMbEr RoCoCcO, yOu .·°~ §LëAzZÿ WëA§ëL ~°·.


                  ---------------------------------
                  Yahoo! DSL Something to write home about. Just $16.99/mo. or less

                  [Non-text portions of this message have been removed]
                Your message has been successfully submitted and would be delivered to recipients shortly.