Loading ...
Sorry, an error occurred while loading the content.

Re: trouble with pattern, character collections

Expand Messages
  • Marc Weber
    I want Vim defaults to be - sane - follow the principle of least surprise. (I d like nocompatible to be set by default, but that s another story) Christian:
    Message 1 of 28 , Feb 19, 2013
    • 0 Attachment
      I want Vim defaults to be
      - sane
      - follow the principle of least surprise.
      (I'd like nocompatible to be set by default, but that's another story)

      Christian: :set cpo+=l still makes my test cases fail:
      [1] echo len(matchstr("\n",'\zs[^\n]\ze'))
      [2] echo len(matchstr("\n","\\zs[^\n]\\ze")

      You explained it by \n not being chr(10), so what is it?

      Let's try by understanding \n's behaviour:
      ==========================================

      case 1) vim buf
      To my undestanding $ matches end of line (in a vim buffer) without eating that
      end of line whereas \n does both: it matches and eats the end of line.
      Eg try /..\n.. and :set hlsearch

      Thus \n is the same as $\n when applying regex to vim buffers. dos 1310 usually
      is encoded in a ff setting, so \n does what you want if you want it.

      case 2) matchstr, matchall, substitute =~ and whatnot (?)

      So if \n is not chr(10), what is it then in this case?

      echo len(matchstr("\n",'\zs[^\n]\ze'))

      clearly indicates it matches \n and and I agree on Erik which called it this way

      Sorry, [^\n] can never match \n ; not even in pink. That is broken
      behaviour.

      So from this point of view I'd say \n behavior is broken when regex get applied
      to strings only (which your patch is supposed to fix - I'll test it
      later)

      Is there more to fix?
      =====================

      issue 1)

      docs state:
      [] (with 'nomagic': \[]) */[]* */\[]* */\_[]* */collection*
      \_[]

      Well - try /[] - it will not be treated as collection, it'll match [], because
      its empty!! So there should be a comment that collections must contain at least
      one char to be seen as one.

      issue 2)
      With "\_" prepended the collection also includes the end-of-line - why does it exist, because
      [\n] is accepted and works as expected?

      So can \_[] syntax be deprecated?


      Marc Weber

      --
      --
      You received this message from the "vim_use" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php

      ---
      You received this message because you are subscribed to the Google Groups "vim_use" group.
      To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
      For more options, visit https://groups.google.com/groups/opt_out.
    • Christian Brabandt
      Hi Marc! ... Well, you need to prevent that expr-quote (:h expr-quote) is being evaluated. You need to escape the then. ... No. Please read again what I
      Message 2 of 28 , Feb 19, 2013
      • 0 Attachment
        Hi Marc!

        On Di, 19 Feb 2013, Marc Weber wrote:

        > I want Vim defaults to be
        > - sane
        > - follow the principle of least surprise.
        > (I'd like nocompatible to be set by default, but that's another story)
        >
        > Christian: :set cpo+=l still makes my test cases fail:
        > [1] echo len(matchstr("\n",'\zs[^\n]\ze'))
        > [2] echo len(matchstr("\n","\\zs[^\n]\\ze")

        Well, you need to prevent that expr-quote (:h expr-quote) is being
        evaluated. You need to escape the \ then.

        > You explained it by \n not being chr(10), so what is it?

        No. Please read again what I wrote. I am not going to repeat myself.

        > Let's try by understanding \n's behaviour:
        > ==========================================
        >
        > case 1) vim buf
        > To my undestanding $ matches end of line (in a vim buffer) without eating that
        > end of line whereas \n does both: it matches and eats the end of line.
        > Eg try /..\n.. and :set hlsearch
        >
        > Thus \n is the same as $\n when applying regex to vim buffers. dos 1310 usually
        > is encoded in a ff setting, so \n does what you want if you want it.
        >
        > case 2) matchstr, matchall, substitute =~ and whatnot (?)
        >
        > So if \n is not chr(10), what is it then in this case?
        >
        > echo len(matchstr("\n",'\zs[^\n]\ze'))
        >
        > clearly indicates it matches \n and and I agree on Erik which called it this way
        >

        Vim internals do not distinguish between evaluating functions and
        buffers. They work on matching a regular pattern on a string of text. In
        Vim buffers, lines are distinguished in memory by NUL and that is the
        only thing, that '.' does not match and so does [^\n] so that in a
        buffer a search for /[^\n] will match any char (even control codes like
        CR or LF)

        What you want is, that in a text /[^\n] also does not match LF character
        and that is what my patch provides thus it should do what you want.

        > Is there more to fix?
        > =====================
        >
        > issue 1)
        >
        > docs state:
        > [] (with 'nomagic': \[]) */[]* */\[]* */\_[]* */collection*
        > \_[]
        >
        > Well - try /[] - it will not be treated as collection, it'll match [], because
        > its empty!! So there should be a comment that collections must contain at least
        > one char to be seen as one.

        I would call this a bug as well. I think, this should give an error.

        > issue 2)
        > With "\_" prepended the collection also includes the end-of-line - why does it exist, because
        > [\n] is accepted and works as expected?
        >
        > So can \_[] syntax be deprecated?

        Why is this an issue? I don't see a problem with \_ syntax at all.

        Mit freundlichen Grüßen
        Christian
        --
        Dem großen Publikum ist ein Buch nicht leicht zu schlecht, sehr
        leicht aber zu gut.
        -- Marie von Ebner-Eschenbach

        --
        --
        You received this message from the "vim_use" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php

        ---
        You received this message because you are subscribed to the Google Groups "vim_use" group.
        To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
        For more options, visit https://groups.google.com/groups/opt_out.
      • Marc Weber
        ... Let me tell you. People get to know Vim. Vim is a tool to serve users. They want to edit text, get their job done (At least that s what I assume). For this
        Message 3 of 28 , Feb 19, 2013
        • 0 Attachment
          > Why is this an issue? I don't see a problem with \_ syntax at all.
          Let me tell you. People get to know Vim. Vim is a tool to serve users.
          They want to edit text, get their job done (At least that's what I
          assume). For this reason every construct such as \_ which requires you
          to lookup help is going to take your time - time which should be spent
          on the wiki or the homepage (yes - there is much to improve).

          So from technical point of view there is nothing wrong: Something is
          documentented, and it works as expected.

          If you look at the ruby community there are some voices which
          dislike some aspects about ruby: That there are so many ways to do
          something ..

          If you look at the whole ecosystem it is wrong, because its
          wasting resources in many ways (human resources being the most
          expensive ones). You may disagree on this - but its true.
          The best docs are the ones you don't have to read.

          I'm not proposing dropping it (if Vim was my projcet I'd do so - showing
          an error message instead) - but maybe docs can be adjusted so that
          people use [\n] and maybe even miss the Vim only special case on the
          first glance.

          Christian, thanks for your support.

          Marc Weber

          --
          --
          You received this message from the "vim_use" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php

          ---
          You received this message because you are subscribed to the Google Groups "vim_use" group.
          To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
          For more options, visit https://groups.google.com/groups/opt_out.
        • Christian Brabandt
          ... Attached is an updated patch, that also prevents /[] matching [] (a collation cannot be empty, so I think it should return an error and other vi clones do,
          Message 4 of 28 , Feb 19, 2013
          • 0 Attachment
            On Di, 19 Feb 2013, Christian Brabandt wrote:

            > On Mo, 18 Feb 2013, Marc Weber wrote:
            > > I don't think that additional threads are going to help
            > > There is an issue, and we should find a way to fix (IMHO).
            > > Let me summarize again - and tell me if you feel differently.
            > >
            > > Test cases:
            > > [1] echo len(matchstr("\n",'\zs[^\n]\ze'))
            > > [2] echo len(matchstr("\n","\\zs[^\n]\\ze"))
            > >
            > > I expect both do the same, the difference is that the second as chr(10) in [^],
            > > while the first has \n (which should be translated to chr(10).
            > >
            > > However I obsorve that [2] returns 0 as expected , but [1] does return
            > > 1, thus it matches \n even though I told Vim that I do not want to match
            > > it. People told me this was because '.' is equal to [^\n].
            > >
            > >
            > > Current situation: at least to be fixed
            > > 1:
            > > No matter whether '.' should behave like [^\n]
            > > [1] and [2] should behave the same, right?
            > > 2:
            > > This should be documented.
            > > (Do you all at least agree these two statments?)
            >
            > Bram, here is a patch, making [^\n] not match NL within the text and
            > that also documents, that '.' matches CR and LF within the text.
            >
            > This makes both [1] and [2] behave the same and seems to better match
            > the users expectations.

            Attached is an updated patch, that also prevents /[] matching []
            (a collation cannot be empty, so I think it should return an error and
            other vi clones do, also grep and perl throw an error).

            Included are tests as well.

            regards,
            Christian
            --
            Sprachlexikon-Namen: GERRITT - gemütl. Schritt-Tempo b. Pferden

            --
            --
            You received this message from the "vim_use" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php

            ---
            You received this message because you are subscribed to the Google Groups "vim_use" group.
            To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
            For more options, visit https://groups.google.com/groups/opt_out.
          • Marc Weber
            ... We should discuss whether /[] is a bug or feature. docs state that /[abc matches [abc by purpose (which could also be treated as error because there is no
            Message 5 of 28 , Feb 19, 2013
            • 0 Attachment
              > [..] patch, [..] prevents /[] matching []
              > other vi clones do, also grep and perl throw an error).
              :) You start comparing Vim against what other tools do.

              We should discuss whether /[] is a bug or feature. docs state that /[abc
              matches [abc by purpose (which could also be treated as error because
              there is no closing ]. And I actually might agree on this being useful.

              Marc Weber

              --
              --
              You received this message from the "vim_use" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php

              ---
              You received this message because you are subscribed to the Google Groups "vim_use" group.
              To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
              For more options, visit https://groups.google.com/groups/opt_out.
            • Ben Fritz
              ... Sometimes the meaning is clearer. What if you re searching for a sequence of certain characters including newlines, where the first character is NOT a
              Message 6 of 28 , Feb 19, 2013
              • 0 Attachment
                On Monday, February 18, 2013 4:38:56 PM UTC-6, MarcWeber wrote:
                >
                > So why should anybody write [^\n] if you can use '.'? So why make [^\n]
                >
                > behave the same way?

                Sometimes the meaning is clearer.

                What if you're searching for a sequence of certain characters including newlines, where the first character is NOT a newline?

                I'd probably want to use:

                /[^\n]\&[a-f0-9\n]\+

                which is equivalent to, but clearer in meaning than:

                /.\&[a-f0-9\n]\+

                --
                --
                You received this message from the "vim_use" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php

                ---
                You received this message because you are subscribed to the Google Groups "vim_use" group.
                To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                For more options, visit https://groups.google.com/groups/opt_out.
              • Erik Christiansen
                ... Christian, many thanks for the work you have done, and for putting users first. It is heartily appreciated. The consolation may be small, but vim is now
                Message 7 of 28 , Feb 20, 2013
                • 0 Attachment
                  On 19.02.13 10:23, Christian Brabandt wrote:
                  > Bram, here is a patch, making [^\n] not match NL within the text and
                  > that also documents, that '.' matches CR and LF within the text.
                  >
                  > This makes both [1] and [2] behave the same and seems to better match
                  > the users expectations.

                  Christian, many thanks for the work you have done, and for putting users
                  first. It is heartily appreciated.

                  The consolation may be small, but vim is now consistent with awk:

                  »
                  . This matches any single character, including the newline character.

                  « - xpdf gawk.pdf # "Effective AWK Programming"

                  O'Reilly's "Mastering Regular Expressions" has it as varying between
                  tools. (OK, we knew that. :-)

                  Regards,
                  Erik

                  --
                  The only thing worse than X Windows: (X Windows) - X

                  --
                  --
                  You received this message from the "vim_use" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php

                  ---
                  You received this message because you are subscribed to the Google Groups "vim_use" group.
                  To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                  For more options, visit https://groups.google.com/groups/opt_out.
                • Bram Moolenaar
                  ... This isn t right, there are no NL characters in the text. There are NUL characters which are stored as NL characters. That s an implementation detail,
                  Message 8 of 28 , Feb 20, 2013
                  • 0 Attachment
                    Christian Brabandt wrote:

                    > Hi Marc!
                    >
                    > On Mo, 18 Feb 2013, Marc Weber wrote:
                    >
                    > > I don't think that additional threads are going to help
                    > > There is an issue, and we should find a way to fix (IMHO).
                    > > Let me summarize again - and tell me if you feel differently.
                    > >
                    > > Test cases:
                    > > [1] echo len(matchstr("\n",'\zs[^\n]\ze'))
                    > > [2] echo len(matchstr("\n","\\zs[^\n]\\ze"))
                    > >
                    > > I expect both do the same, the difference is that the second as chr(10) in [^],
                    > > while the first has \n (which should be translated to chr(10).
                    > >
                    > > However I obsorve that [2] returns 0 as expected , but [1] does return
                    > > 1, thus it matches \n even though I told Vim that I do not want to match
                    > > it. People told me this was because '.' is equal to [^\n].
                    > >
                    > >
                    > > Current situation: at least to be fixed
                    > > 1:
                    > > No matter whether '.' should behave like [^\n]
                    > > [1] and [2] should behave the same, right?
                    > > 2:
                    > > This should be documented.
                    > > (Do you all at least agree these two statments?)
                    >
                    > Bram, here is a patch, making [^\n] not match NL within the text and
                    > that also documents, that '.' matches CR and LF within the text.
                    >
                    > This makes both [1] and [2] behave the same and seems to better match
                    > the users expectations.

                    This isn't right, there are no NL characters in the text. There are NUL
                    characters which are stored as NL characters. That's an implementation
                    detail, which sometimes becomes visible to the user.

                    It's good to explain this in the docs. I'm not sure we actually should
                    change the behavior, it might not really take away much confusion.


                    --
                    DENNIS: You can't expect to wield supreme executive power just 'cause some
                    watery tart threw a sword at you!
                    "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

                    /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                    /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                    \\\ an exciting new programming language -- http://www.Zimbu.org ///
                    \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                    --
                    --
                    You received this message from the "vim_use" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php

                    ---
                    You received this message because you are subscribed to the Google Groups "vim_use" group.
                    To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                    For more options, visit https://groups.google.com/groups/opt_out.
                  • Marc Weber
                    Hi Bram, thanks for joining the discussion and participating. echo len(nr2char(0)) returns 0, where is the 0 char stored as n .. So please take care about the
                    Message 9 of 28 , Feb 20, 2013
                    • 0 Attachment
                      Hi Bram,

                      thanks for joining the discussion and participating.

                      echo len(nr2char(0))
                      returns 0, where is the 0 char stored as \n ..
                      So please take care about the use case: I'm not talking about buffers,
                      I'm talking about matchstr, substitute etc and viml strings.

                      I'm aware that both should be tested and documented which becomes clear
                      reading my later summary.

                      Anyway [^\n] matching \n is a very very unexpected behaviour.
                      Having \_[ syntax doesn't make sense, either. Because [\n] works and is
                      standard. And this should be visible in docs.

                      Marc Weber

                      --
                      --
                      You received this message from the "vim_use" maillist.
                      Do not top-post! Type your reply below the text you are replying to.
                      For more information, visit http://www.vim.org/maillist.php

                      ---
                      You received this message because you are subscribed to the Google Groups "vim_use" group.
                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                      For more options, visit https://groups.google.com/groups/opt_out.
                    • Erik Christiansen
                      ... Bram, is that meant to be There are NL characters which are stored as NUL characters ? Because if there are no NL characters in the text , then nothing
                      Message 10 of 28 , Feb 20, 2013
                      • 0 Attachment
                        On 20.02.13 13:32, Bram Moolenaar wrote:
                        > Christian Brabandt wrote:
                        > > Bram, here is a patch, making [^\n] not match NL within the text and
                        > > that also documents, that '.' matches CR and LF within the text.
                        > >
                        > > This makes both [1] and [2] behave the same and seems to better match
                        > > the users expectations.
                        >
                        > This isn't right, there are no NL characters in the text. There are NUL
                        > characters which are stored as NL characters.

                        Bram, is that meant to be "There are NL characters which are stored as
                        NUL characters"? Because if "there are no NL characters in the text",
                        then nothing can be stored as them, can it?

                        > That's an implementation detail, which sometimes becomes visible to
                        > the user.

                        OK, but the actual problem is corruption of a regex syntax snippet in
                        one quoting context. Letting "[^\n]" not be the same as '[^\n]' is the
                        mark of a broken regex implementation. Other regex engines manage \n
                        without their syntax breaking, so it is possible with Vim too.

                        Vim has '.' to represent "any character", and so does not require a
                        second representation for that. But if [^\n] is the same as '.', because
                        there are no newlines present, then that must _always_ be the case,
                        regardless of a bit of quoting flim-flam.

                        > It's good to explain this in the docs.

                        Yes, knowing that there are no newlines can help us avoid looking for
                        them.

                        Another fix is needed, though, for the problem that quote flavour is
                        allowed to corrupt the regex in one case, causing behaviour contrary to
                        the quoted regex syntax. That is an illogicality boobytrap to torment
                        the user.

                        > I'm not sure we actually should change the behavior, it might not
                        > really take away much confusion.

                        Excuse me, Bram but removing the broken behaviour of "[^\n]" not being
                        the same as '[^\n]' does remove a mind boggling logical nonsense, and so
                        does do away with major confusion.

                        Deep knowledge if Vim internals is an asset when it fosters effective
                        and robust fixes, but tends toward a liability if it blinds one to the
                        failings of a self-contradicting user interface, I think.

                        In all that we construct, it is _what_ should happen which guides _how_
                        it is made to happen. Arguments defending the current broken status
                        repeatedly refer to _how_ usurping _what_, as an implementational
                        side effect, and accepting that as a justification. That is not
                        intelligent design, and has here led to impaired functionality.

                        Vim's supremacy would be improved by removing the user interface
                        deficiency.

                        Erik

                        --
                        You have all eternity to be cautious in when you're dead.
                        - Lois Platford

                        --
                        --
                        You received this message from the "vim_use" maillist.
                        Do not top-post! Type your reply below the text you are replying to.
                        For more information, visit http://www.vim.org/maillist.php

                        ---
                        You received this message because you are subscribed to the Google Groups "vim_use" group.
                        To unsubscribe from this group and stop receiving emails from it, send an email to vim_use+unsubscribe@....
                        For more options, visit https://groups.google.com/groups/opt_out.
                      Your message has been successfully submitted and would be delivered to recipients shortly.