Loading ...
Sorry, an error occurred while loading the content.
 

Re: Spell correction ignores certain characters

Expand Messages
  • Christian Brabandt
    Hi ... While I can t say anything about it, I noticed that this crashes my vim with a slight variation of the above sentence: gdb --args ./vim -u NONE -U NONE
    Message 1 of 10 , May 2, 2012
      Hi

      On Di, 01 Mai 2012, Axel wrote:

      > When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
      >
      > Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
      >
      > In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
      >

      While I can't say anything about it, I noticed that this crashes my vim
      with a slight variation of the above sentence:

      gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de

      [Vim starts and puts that sentence in the buffer]
      :norm! fsz=

      SIGSEGV, Segmentation fault.
      0x0000000000549908 in spell_soundfold_wsal (slang=0x8908b0,
      inword=0x7fffffffbe10 "strafst", res=0x7fffffffc180 "ztre")
      at spell.c:14545
      14545 for (; ((ws = smp[n].sm_lead_w)[0] & 0xff) == (c & 0xff)
      (gdb) list
      14540 if (n >= 0)
      14541 {
      14542 /* Check all rules for the same index byte.
      14543 * If c is 0x300 need extra check for the end of the array, as
      14544 * (c & 0xff) is NUL. */
      14545 for (; ((ws = smp[n].sm_lead_w)[0] & 0xff) == (c & 0xff)
      14546 && ws[0] != NUL; ++n)
      14547 {
      14548 /* Quickly skip entries that don't match the word. Most
      14549 * entries are less then three chars, optimize for that. */

      smp seems invalid:
      (gdb) p *smp
      Cannot access memory at address 0x7a

      (gdb) bt
      #0 0x0000000000549908 in spell_soundfold_wsal (slang=0x8908b0,
      inword=0x7fffffffbe10 "strafst", res=0x7fffffffc180 "ztre")
      at spell.c:14545
      #1 0x0000000000547e72 in spell_soundfold (slang=0x8908b0,
      inword=0x12cb060 "strafst", folded=0, res=0x7fffffffc180 "ztre")
      at spell.c:14106
      #2 0x0000000000545fd3 in stp_sal_score (stp=0x14bd5e8, su=0x7fffffffd640,
      slang=0x8908b0, badsound=0x7fffffffd884 "ztra") at spell.c:13189
      #3 0x0000000000547a9c in rescore_one (su=0x7fffffffd640, stp=0x14bd5e8)
      at spell.c:13969
      #4 0x0000000000547991 in rescore_suggestions (su=0x7fffffffd640)
      at spell.c:13942
      #5 0x000000000053fc13 in spell_suggest_intern (su=0x7fffffffd640,
      interactive=1) at spell.c:10840
      #6 0x000000000053f7da in spell_find_suggest (
      badptr=0x8595f8 "straße<<<, sondern in der >>>Hauptstraße<<<",
      badlen=0, su=0x7fffffffd640, maxcount=20, banbadword=1, need_cap=0,
      interactive=1) at spell.c:10690
      #7 0x000000000053e2ae in spell_suggest (count=0) at spell.c:10224
      #8 0x00000000004d8aff in nv_zet (cap=0x7fffffffdc60) at normal.c:5216
      #9 0x00000000004d9916 in normal_cmd (oap=0x7fffffffdd00, toplevel=1)
      ---Type <return> to continue, or q <return> to quit---
      at normal.c:1193
      #10 0x00000000005915ed in main_loop (cmdwin=0, noexmode=0) at main.c:1294
      #11 0x0000000000410dba in main (argc=<optimized out>, argv=<optimized out>)
      at main.c:998


      This is a Big build of Vim 7.3.513 on linux.

      Sorry, I don't know the part of the code and don't have time to
      investigate the problem now, so I am just reporting the crash.

      regards,
      Christian

      --
      You received this message from the "vim_dev" maillist.
      Do not top-post! Type your reply below the text you are replying to.
      For more information, visit http://www.vim.org/maillist.php
    • Bram Moolenaar
      ... I see it too. Also the ß in Georgenkirchstraße is not highlighted as a spell error. I wonder if this is a problem in the spell dictionary or in the
      Message 2 of 10 , May 2, 2012
        Axel Bender wrote:

        > When using the spell correction feature ("set spelllang=de_de spell")
        > a word containing a german "ß" (0xDF) is displayed incorrectly if it
        > is found to be misspelled:
        >
        > Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der
        > >>>Hauptstraße<<<.
        >
        > In this sample "Georgenkirchstraße" is flagged as misspelled (as
        > opposed to "Hauptstraße") - which would be ok - but the "ß" in the
        > flagged word is not considered as a valid word character; hence the
        > word cannot be directly added (w/o creating a selection) to the
        > exception list with e.g. "zg".

        I see it too. Also the ß in "Georgenkirchstraße" is not highlighted as
        a spell error. I wonder if this is a problem in the spell dictionary or
        in the Vim code.

        --
        Biting someone with your natural teeth is "simple assault," while biting
        someone with your false teeth is "aggravated assault."
        [real standing law in Louisana, United States of America]

        /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
        /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
        \\\ an exciting new programming language -- http://www.Zimbu.org ///
        \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

        --
        You received this message from the "vim_dev" maillist.
        Do not top-post! Type your reply below the text you are replying to.
        For more information, visit http://www.vim.org/maillist.php
      • Dominique Pellé
        ... ...snip... I also see the German eszett not being highlighted by the spelling checker (bug). However, I don t see any crash using Linux x86 or Linux x86_64
        Message 3 of 10 , May 2, 2012
          Christian Brabandt <cblists@...> wrote:

          > Hi
          >
          > On Di, 01 Mai 2012, Axel wrote:
          >
          >> When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
          >>
          >> Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
          >>
          >> In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
          >>
          >
          > While I can't say anything about it, I noticed that this crashes my vim
          > with a slight variation of the above sentence:
          >
          > gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de
          >
          > [Vim starts and puts that sentence in the buffer]
          > :norm! fsz=
          >
          >  SIGSEGV, Segmentation fault.

          ...snip...

          I also see the German eszett not being highlighted by the
          spelling checker (bug). However, I don't see any crash using
          Linux x86 or Linux x86_64 with Vim-7.3.515. I used the latest
          German dictionary which Vim downloaded automatically
          when setting ':set spelllang=de'.

          -- Dominique

          --
          You received this message from the "vim_dev" maillist.
          Do not top-post! Type your reply below the text you are replying to.
          For more information, visit http://www.vim.org/maillist.php
        • Christian Brabandt
          Hi Dominique! ... I guess, *smp got invalid somehow. This patch prevents, that it gets overwritten: diff --git a/src/spell.c b/src/spell.c ... +++
          Message 4 of 10 , May 9, 2012
            Hi Dominique!

            On Mi, 02 Mai 2012, Dominique Pellé wrote:

            > Christian Brabandt <cblists@...> wrote:
            >
            > > Hi
            > >
            > > On Di, 01 Mai 2012, Axel wrote:
            > >
            > >> When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
            > >>
            > >> Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
            > >>
            > >> In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
            > >>
            > >
            > > While I can't say anything about it, I noticed that this crashes my vim
            > > with a slight variation of the above sentence:
            > >
            > > gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de
            > >
            > > [Vim starts and puts that sentence in the buffer]
            > > :norm! fsz=
            > >
            > >  SIGSEGV, Segmentation fault.
            >
            > ...snip...
            >
            > I also see the German eszett not being highlighted by the
            > spelling checker (bug). However, I don't see any crash using
            > Linux x86 or Linux x86_64 with Vim-7.3.515. I used the latest
            > German dictionary which Vim downloaded automatically
            > when setting ':set spelllang=de'.

            I guess, *smp got invalid somehow. This patch prevents, that it gets
            overwritten:

            diff --git a/src/spell.c b/src/spell.c
            --- a/src/spell.c
            +++ b/src/spell.c
            @@ -14766,6 +14766,9 @@
            z = 0;
            k = 0;
            }
            + /* prevent overwriting of *smp pointer */
            + if (smp != slang->sl_sal.ga_data)
            + smp = (salitem_T *)slang->sl_sal.ga_data;
            }


            Mit freundlichen Grüßen
            Christian
            --

            --
            You received this message from the "vim_dev" maillist.
            Do not top-post! Type your reply below the text you are replying to.
            For more information, visit http://www.vim.org/maillist.php
          • Axel
            Will this get fixed in some official patch? -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text you are
            Message 5 of 10 , May 18, 2012
              Will this get fixed in some official patch?

              --
              You received this message from the "vim_dev" maillist.
              Do not top-post! Type your reply below the text you are replying to.
              For more information, visit http://www.vim.org/maillist.php
            • Bram Moolenaar
              ... This patch doesn t make sense. It would suggest slang- sl_sal.ga_data is changed, or that smp is changed. I don t see how that could happen. Could this
              Message 6 of 10 , May 18, 2012
                Christian Brabandt wrote:

                > > >> When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
                > > >>
                > > >> Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
                > > >>
                > > >> In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
                > > >>
                > > >
                > > > While I can't say anything about it, I noticed that this crashes my vim
                > > > with a slight variation of the above sentence:
                > > >
                > > > gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de
                > > >
                > > > [Vim starts and puts that sentence in the buffer]
                > > > :norm! fsz=
                > > >
                > > >  SIGSEGV, Segmentation fault.
                > >
                > > ...snip...
                > >
                > > I also see the German eszett not being highlighted by the
                > > spelling checker (bug). However, I don't see any crash using
                > > Linux x86 or Linux x86_64 with Vim-7.3.515. I used the latest
                > > German dictionary which Vim downloaded automatically
                > > when setting ':set spelllang=de'.
                >
                > I guess, *smp got invalid somehow. This patch prevents, that it gets
                > overwritten:
                >
                > diff --git a/src/spell.c b/src/spell.c
                > --- a/src/spell.c
                > +++ b/src/spell.c
                > @@ -14766,6 +14766,9 @@
                > z = 0;
                > k = 0;
                > }
                > + /* prevent overwriting of *smp pointer */
                > + if (smp != slang->sl_sal.ga_data)
                > + smp = (salitem_T *)slang->sl_sal.ga_data;
                > }

                This patch doesn't make sense. It would suggest slang->sl_sal.ga_data
                is changed, or that smp is changed. I don't see how that could happen.
                Could this be a memory access error? I could not reproduce it though.

                When using valgrind I do find this:

                ==22362== Conditional jump or move depends on uninitialised value(s)
                ==22362== at 0x81BD866: spell_soundfold_wsal (spell.c:14741)
                ==22362== by 0x81BC11B: spell_soundfold (spell.c:14106)
                ==22362== by 0x81BA588: stp_sal_score (spell.c:13189)

                Since word[] is just after smp and on the stack order is reversed, I
                suspect the length of the mch_memmove is wrong.

                --
                ARTHUR: I've said I'm sorry about the old woman, but from the behind you
                looked ...
                DENNIS: What I object to is that you automatically treat me like an inferior...
                ARTHUR: Well ... I AM king.
                "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

                /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                \\\ an exciting new programming language -- http://www.Zimbu.org ///
                \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                --
                You received this message from the "vim_dev" maillist.
                Do not top-post! Type your reply below the text you are replying to.
                For more information, visit http://www.vim.org/maillist.php
              • Dominique Pellé
                ... I m not able to reproduce this Valgrind error. Nor can I reproduce the issue in this thread. For me, smp is always equal to slang- sl_sal.ga_data. Perhaps
                Message 7 of 10 , May 18, 2012
                  Bram Moolenaar <Bram@...> wrote:

                  > Christian Brabandt wrote:
                  >
                  >> > >> When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
                  >> > >>
                  >> > >> Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
                  >> > >>
                  >> > >> In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
                  >> > >>
                  >> > >
                  >> > > While I can't say anything about it, I noticed that this crashes my vim
                  >> > > with a slight variation of the above sentence:
                  >> > >
                  >> > > gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de
                  >> > >
                  >> > > [Vim starts and puts that sentence in the buffer]
                  >> > > :norm! fsz=
                  >> > >
                  >> > >  SIGSEGV, Segmentation fault.
                  >> >
                  >> > ...snip...
                  >> >
                  >> > I also see the German eszett not being highlighted by the
                  >> > spelling checker (bug). However, I don't see any crash using
                  >> > Linux x86 or Linux x86_64 with Vim-7.3.515.  I used the latest
                  >> > German dictionary which Vim downloaded automatically
                  >> > when setting ':set spelllang=de'.
                  >>
                  >> I guess, *smp got invalid somehow. This patch prevents, that it gets
                  >> overwritten:
                  >>
                  >> diff --git a/src/spell.c b/src/spell.c
                  >> --- a/src/spell.c
                  >> +++ b/src/spell.c
                  >> @@ -14766,6 +14766,9 @@
                  >>             z = 0;
                  >>             k = 0;
                  >>         }
                  >> +       /* prevent overwriting of *smp pointer */
                  >> +       if (smp != slang->sl_sal.ga_data)
                  >> +           smp = (salitem_T *)slang->sl_sal.ga_data;
                  >>      }
                  >
                  > This patch doesn't make sense.  It would suggest slang->sl_sal.ga_data
                  > is changed, or that smp is changed.  I don't see how that could happen.
                  > Could this be a memory access error?  I could not reproduce it though.
                  >
                  > When using valgrind I do find this:
                  >
                  > ==22362== Conditional jump or move depends on uninitialised value(s)
                  > ==22362==    at 0x81BD866: spell_soundfold_wsal (spell.c:14741)
                  > ==22362==    by 0x81BC11B: spell_soundfold (spell.c:14106)
                  > ==22362==    by 0x81BA588: stp_sal_score (spell.c:13189)
                  >
                  > Since word[] is just after smp and on the stack order is reversed, I
                  > suspect the length of the mch_memmove is wrong.

                  I'm not able to reproduce this Valgrind error. Nor can I reproduce
                  the issue in this thread. For me, smp is always equal to slang->sl_sal.ga_data.

                  Perhaps the bug happens only with some locale? What is your locale?

                  When valgrind reports "uninitialized value(s)", it can help to add
                  the valgrind command line option --track-origins=yes. It gives better
                  error reporting at the cost of running slightly slower (which is why
                  it's off by default).

                  Regards
                  -- Dominique

                  --
                  You received this message from the "vim_dev" maillist.
                  Do not top-post! Type your reply below the text you are replying to.
                  For more information, visit http://www.vim.org/maillist.php
                • Bram Moolenaar
                  ... I located the problem: using strlen() on an int pointer. Not a good idea! :-) This doesn t solve the highlighting, just the potential crash. -- DENNIS:
                  Message 8 of 10 , May 18, 2012
                    Dominique Pelle wrote:

                    > Bram Moolenaar <Bram@...> wrote:
                    >
                    > > Christian Brabandt wrote:
                    > >
                    > >> > >> When using the spell correction feature ("set spelllang=de_de spell") a word containing a german "ß" (0xDF) is displayed incorrectly if it is found to be misspelled:
                    > >> > >>
                    > >> > >> Wir wohnen nicht in der >>>Georgenkirchstraße<<<, sondern in der >>>Hauptstraße<<<.
                    > >> > >>
                    > >> > >> In this sample "Georgenkirchstraße" is flagged as misspelled (as opposed to "Hauptstraße") - which would be ok - but the "ß" in the flagged word is not considered as a valid word character; hence the word cannot be directly added (w/o creating a selection) to the exception list with e.g. "zg".
                    > >> > >>
                    > >> > >
                    > >> > > While I can't say anything about it, I noticed that this crashes my vim
                    > >> > > with a slight variation of the above sentence:
                    > >> > >
                    > >> > > gdb --args ./vim -u NONE -U NONE -N -c ":put ='Wir wohnen nicht in der >>>Georgenkirch straße<<<, sondern in der >>>Hauptstraße<<<'" -c 'set spell spelllang=de
                    > >> > >
                    > >> > > [Vim starts and puts that sentence in the buffer]
                    > >> > > :norm! fsz=
                    > >> > >
                    > >> > >  SIGSEGV, Segmentation fault.
                    > >> >
                    > >> > ...snip...
                    > >> >
                    > >> > I also see the German eszett not being highlighted by the
                    > >> > spelling checker (bug). However, I don't see any crash using
                    > >> > Linux x86 or Linux x86_64 with Vim-7.3.515.  I used the latest
                    > >> > German dictionary which Vim downloaded automatically
                    > >> > when setting ':set spelllang=de'.
                    > >>
                    > >> I guess, *smp got invalid somehow. This patch prevents, that it gets
                    > >> overwritten:
                    > >>
                    > >> diff --git a/src/spell.c b/src/spell.c
                    > >> --- a/src/spell.c
                    > >> +++ b/src/spell.c
                    > >> @@ -14766,6 +14766,9 @@
                    > >>             z = 0;
                    > >>             k = 0;
                    > >>         }
                    > >> +       /* prevent overwriting of *smp pointer */
                    > >> +       if (smp != slang->sl_sal.ga_data)
                    > >> +           smp = (salitem_T *)slang->sl_sal.ga_data;
                    > >>      }
                    > >
                    > > This patch doesn't make sense.  It would suggest slang->sl_sal.ga_data
                    > > is changed, or that smp is changed.  I don't see how that could happen.
                    > > Could this be a memory access error?  I could not reproduce it though.
                    > >
                    > > When using valgrind I do find this:
                    > >
                    > > ==22362== Conditional jump or move depends on uninitialised value(s)
                    > > ==22362==    at 0x81BD866: spell_soundfold_wsal (spell.c:14741)
                    > > ==22362==    by 0x81BC11B: spell_soundfold (spell.c:14106)
                    > > ==22362==    by 0x81BA588: stp_sal_score (spell.c:13189)
                    > >
                    > > Since word[] is just after smp and on the stack order is reversed, I
                    > > suspect the length of the mch_memmove is wrong.
                    >
                    > I'm not able to reproduce this Valgrind error. Nor can I reproduce
                    > the issue in this thread. For me, smp is always equal to slang->sl_sal.ga_data.
                    >
                    > Perhaps the bug happens only with some locale? What is your locale?
                    >
                    > When valgrind reports "uninitialized value(s)", it can help to add
                    > the valgrind command line option --track-origins=yes. It gives better
                    > error reporting at the cost of running slightly slower (which is why
                    > it's off by default).

                    I located the problem: using strlen() on an int pointer. Not a good
                    idea! :-)

                    This doesn't solve the highlighting, just the potential crash.

                    --
                    DENNIS: Oh, very nice. King, eh! I expect you've got a palace and fine
                    clothes and courtiers and plenty of food. And how d'you get that? By
                    exploiting the workers! By hanging on to outdated imperialist dogma
                    which perpetuates the social and economic differences in our society!
                    "Monty Python and the Holy Grail" PYTHON (MONTY) PICTURES LTD

                    /// Bram Moolenaar -- Bram@... -- http://www.Moolenaar.net \\\
                    /// sponsor Vim, vote for features -- http://www.Vim.org/sponsor/ \\\
                    \\\ an exciting new programming language -- http://www.Zimbu.org ///
                    \\\ help me help AIDS victims -- http://ICCF-Holland.org ///

                    --
                    You received this message from the "vim_dev" maillist.
                    Do not top-post! Type your reply below the text you are replying to.
                    For more information, visit http://www.vim.org/maillist.php
                  • Axel Bender
                    I d like to renew my request to solve this bug... -- -- You received this message from the vim_dev maillist. Do not top-post! Type your reply below the text
                    Message 9 of 10 , Mar 1, 2015
                      I'd like to renew my request to solve this bug...

                      --
                      --
                      You received this message from the "vim_dev" maillist.
                      Do not top-post! Type your reply below the text you are replying to.
                      For more information, visit http://www.vim.org/maillist.php

                      ---
                      You received this message because you are subscribed to the Google Groups "vim_dev" group.
                      To unsubscribe from this group and stop receiving emails from it, send an email to vim_dev+unsubscribe@....
                      For more options, visit https://groups.google.com/d/optout.
                    Your message has been successfully submitted and would be delivered to recipients shortly.