Loading ...
Sorry, an error occurred while loading the content.
 

Unit Testing for vim [Was: Vim Lib [Was: List Questions]]

Expand Messages
  • Luc Hermitte
    Hello, ... I ve finally implemented my own unit testing plugin for vim. [1] This is a first draft, and all comments are welcomed. The plugin has been strongly
    Message 1 of 7 , Feb 19, 2009
      Hello,

      "Ingo Karkat" <swdev@...> wrote :

      > On 07-Feb-09 12:39, Luc Hermitte wrote:
      > > My few requirements, in case it helps:
      > > - we should be able to test values with very simple things like:
      > > :Assert value < 42
      > > - the result should end in the quickfix window (which implies
      > > a first pass on functions/files to determine the current line
      > > where the Assert is used)
      > > - it would be nice to test mappings/commands/abbreviations/...
      > > definitions, buffer alterations, etc.
      > > - setup/teardown support would be nice also.
      >
      > Thanks for your list.

      I've finally implemented my own unit testing plugin for vim. [1]
      This is a first draft, and all comments are welcomed.

      The plugin has been strongly inspired by Tom Link's tAssert plugin
      (thanks Tom!).

      However I've made a different design choice: my plugin acts as a
      preprocessor. Thanks to that, I'm able to know the line where an
      assertion failure occurred, and I'm also able to handle expressions
      with script-local s:variables.
      My plugin is made for Unit Testing, and nothing else. We write an
      independent UT script, and give it as argument to :UTRun command.

      On the other hand, Tom's plugin permits to Design scripts by Contract,
      which is also extremely useful. i.e., it makes possible to place
      assertions in regular Vim scripts (*-plugins). However by its design
      choice, it cannot fills the quickfix window with the list of failed
      assertions -- hence my NIH script.
      [It could be possible to process regular scripts with my plugin, in
      order to do DbC by working on "debug"-enabled plugins, but I'm not sure
      it worth it]

      BTW, tAssert provides convenience functions that my script don't (yet?).
      At first, I wondered if both plugins should be merged. But as the engines
      are quite different, I'm not sure it makes any sense.


      > What prompted me to start my implementation is that though it's
      > trivial to come up with some :Assert commands, these cover the unit
      > testing part well (i.e. checking that MyFunc('foo') does return 'bar'),
      > but are of little help when testing the effects of custom mappings and
      > commands, what most scripts and tips on vim.org are about.

      That's true. However, it should be possible to provide convenience
      functions that help checking a buffer content.


      [1] http://code.google.com/p/lh-vim/wiki/UT
      NB: for once, I've written a script that depends on no other
      script -- even if it can take advantage of BuildToolsWrapper if it
      is detected.

      PS: shouldn't the discussion move to vim_use ?
      --
      Luc Hermitte
      http://lh-vim.googlecode.com/
      http://hermitte.free.fr/vim/

      --~--~---------~--~----~------------~-------~--~----~
      You received this message from the "vim_dev" maillist.
      For more information, visit http://www.vim.org/maillist.php
      -~----------~----~----~----~------~----~------~--~---
    • Tom Link
      ... Cool. ... IMHO they serve different purposes. Although tassert was initially meant to work as a unit testing framework, I then was more interested in
      Message 2 of 7 , Feb 20, 2009
        > However I've made a different design choice: my plugin acts as a
        > preprocessor. Thanks to that, I'm able to know the line where an
        > assertion failure occurred

        Cool.

        > BTW, tAssert provides convenience functions that my script don't (yet?).
        > At first, I wondered if both plugins should be merged.

        IMHO they serve different purposes. Although tassert was initially
        meant to work as a unit testing framework, I then was more interested
        in sprinkling the code with assertions that could easily be turned
        off. But with a DbC-related plugin, you'd always want to load it on
        startup during (informal) testing. A unit-testing plugin on the other
        hand, you'd probably want to load only before running the test, I
        suppose. So, a DbC plugin should load fast, a UT-plugin doesn't
        necessarily have to. This is also the reason why I'd rather prefer to
        strip down my tassert plugin and to leave only the TAssert command and
        some utility functions in it.

        There is some overlap of course since the utility functions you
        mentioned are useful in both contexts. Maybe those functions should go
        in a shared autoload file?

        One main challenge with a unit testing framework for vim scripts
        actually is, as Ingo Karkat pointed out, the utility functions it
        provides. IMHO these functions should not only be able to test for
        buffer content after a defined series of simulated key presses (via
        feedkey()) but also something like window layout etc. It would also be
        interesting to combine such a UT plugin with something like Charles
        Campbell's PluginKiller[1] to make sure that the results are the same
        with different sets of option values. I think CC's plugin is quite
        interesting in this respect but I personally don't think it's very
        practical to manually simulate all different kinds of user
        interactions repeatedly with different settings. AFAIK PluginKiller
        doesn't automate (record & replay) this process, does it?

        Regards,
        Thomas.


        [1] http://www.vim.org/scripts/script.php?script_id=1489

        --~--~---------~--~----~------------~-------~--~----~
        You received this message from the "vim_dev" maillist.
        For more information, visit http://www.vim.org/maillist.php
        -~----------~----~----~----~------~----~------~--~---
      • Tom Link
        ... Cool. ... IMHO they serve different purposes. Although tassert was initially meant to work as a unit testing framework, I then was more interested in
        Message 3 of 7 , Feb 20, 2009
          > However I've made a different design choice: my plugin acts as a
          > preprocessor. Thanks to that, I'm able to know the line where an
          > assertion failure occurred

          Cool.

          > BTW, tAssert provides convenience functions that my script don't (yet?).
          > At first, I wondered if both plugins should be merged.

          IMHO they serve different purposes. Although tassert was initially
          meant to work as a unit testing framework, I then was more interested
          in sprinkling the code with assertions that could easily be turned
          off. But with a DbC-related plugin, you'd always want to load it on
          startup during (informal) testing. A unit-testing plugin on the other
          hand, you'd probably want to load only before running the test, I
          suppose. So, a DbC plugin should load fast, a UT-plugin doesn't
          necessarily have to. This is also the reason why I'd rather prefer to
          strip down my tassert plugin and to leave only the TAssert command and
          some utility functions in it.

          There is some overlap of course since the utility functions you
          mentioned are useful in both contexts. Maybe those functions should go
          in a shared autoload file?

          One main challenge with a unit testing framework for vim scripts
          actually is, as Ingo Karkat pointed out, the utility functions it
          provides. IMHO these functions should not only be able to test for
          buffer content after a defined series of simulated key presses (via
          feedkey()) but also something like window layout etc. It would also be
          interesting to combine such a UT plugin with something like Charles
          Campbell's PluginKiller[1] to make sure that the results are the same
          with different sets of option values. I think CC's plugin is quite
          interesting in this respect but I personally don't think it's very
          practical to manually simulate all different kinds of user
          interactions repeatedly with different settings. AFAIK PluginKiller
          doesn't automate (record & replay) this process, does it?

          Regards,
          Thomas.


          [1] http://www.vim.org/scripts/script.php?script_id=1489

          --~--~---------~--~----~------------~-------~--~----~
          You received this message from the "vim_dev" maillist.
          For more information, visit http://www.vim.org/maillist.php
          -~----------~----~----~----~------~----~------~--~---
        • Charles Campbell
          ... I think that one can already do a keystroke recording and playback (qa ... q @a, for example), so I don t have PK automating that process. What d be great
          Message 4 of 7 , Feb 20, 2009
            Tom Link wrote:
            >> However I've made a different design choice: my plugin acts as a
            >> preprocessor. Thanks to that, I'm able to know the line where an
            >> assertion failure occurred
            >>
            >
            > Cool.
            >
            >
            >> BTW, tAssert provides convenience functions that my script don't (yet?).
            >> At first, I wondered if both plugins should be merged.
            >>
            >
            > IMHO they serve different purposes. Although tassert was initially
            > meant to work as a unit testing framework, I then was more interested
            > in sprinkling the code with assertions that could easily be turned
            > off. But with a DbC-related plugin, you'd always want to load it on
            > startup during (informal) testing. A unit-testing plugin on the other
            > hand, you'd probably want to load only before running the test, I
            > suppose. So, a DbC plugin should load fast, a UT-plugin doesn't
            > necessarily have to. This is also the reason why I'd rather prefer to
            > strip down my tassert plugin and to leave only the TAssert command and
            > some utility functions in it.
            >
            > There is some overlap of course since the utility functions you
            > mentioned are useful in both contexts. Maybe those functions should go
            > in a shared autoload file?
            >
            > One main challenge with a unit testing framework for vim scripts
            > actually is, as Ingo Karkat pointed out, the utility functions it
            > provides. IMHO these functions should not only be able to test for
            > buffer content after a defined series of simulated key presses (via
            > feedkey()) but also something like window layout etc. It would also be
            > interesting to combine such a UT plugin with something like Charles
            > Campbell's PluginKiller[1] to make sure that the results are the same
            > with different sets of option values. I think CC's plugin is quite
            > interesting in this respect but I personally don't think it's very
            > practical to manually simulate all different kinds of user
            > interactions repeatedly with different settings. AFAIK PluginKiller
            > doesn't automate (record & replay) this process, does it?
            >
            I think that one can already do a keystroke recording and playback (qa
            ... q @a, for example),
            so I don't have PK automating that process. What'd be great would be to
            combine that playback
            with the ability to determine if the plugin behaved correctly or not --
            but I don't see any way to
            generically accomplish that. Certainly the notion of TAssert could be
            used to give a partial
            feedback of that sort.

            Regards,
            Chip Campbell


            --~--~---------~--~----~------------~-------~--~----~
            You received this message from the "vim_dev" maillist.
            For more information, visit http://www.vim.org/maillist.php
            -~----------~----~----~----~------~----~------~--~---
          • Tony Mechelynck
            ... Well, at least most of Vim (with the exception, IIUC, of the python interface) is essentially single-threaded, so unlike developers for some other
            Message 5 of 7 , Feb 20, 2009
              On 20/02/09 16:03, Charles Campbell wrote:
              > Tom Link wrote:
              >>> However I've made a different design choice: my plugin acts as a
              >>> preprocessor. Thanks to that, I'm able to know the line where an
              >>> assertion failure occurred
              >>>
              >> Cool.
              >>
              >>
              >>> BTW, tAssert provides convenience functions that my script don't (yet?).
              >>> At first, I wondered if both plugins should be merged.
              >>>
              >> IMHO they serve different purposes. Although tassert was initially
              >> meant to work as a unit testing framework, I then was more interested
              >> in sprinkling the code with assertions that could easily be turned
              >> off. But with a DbC-related plugin, you'd always want to load it on
              >> startup during (informal) testing. A unit-testing plugin on the other
              >> hand, you'd probably want to load only before running the test, I
              >> suppose. So, a DbC plugin should load fast, a UT-plugin doesn't
              >> necessarily have to. This is also the reason why I'd rather prefer to
              >> strip down my tassert plugin and to leave only the TAssert command and
              >> some utility functions in it.
              >>
              >> There is some overlap of course since the utility functions you
              >> mentioned are useful in both contexts. Maybe those functions should go
              >> in a shared autoload file?
              >>
              >> One main challenge with a unit testing framework for vim scripts
              >> actually is, as Ingo Karkat pointed out, the utility functions it
              >> provides. IMHO these functions should not only be able to test for
              >> buffer content after a defined series of simulated key presses (via
              >> feedkey()) but also something like window layout etc. It would also be
              >> interesting to combine such a UT plugin with something like Charles
              >> Campbell's PluginKiller[1] to make sure that the results are the same
              >> with different sets of option values. I think CC's plugin is quite
              >> interesting in this respect but I personally don't think it's very
              >> practical to manually simulate all different kinds of user
              >> interactions repeatedly with different settings. AFAIK PluginKiller
              >> doesn't automate (record& replay) this process, does it?
              >>
              > I think that one can already do a keystroke recording and playback (qa
              > ... q @a, for example),
              > so I don't have PK automating that process. What'd be great would be to
              > combine that playback
              > with the ability to determine if the plugin behaved correctly or not --
              > but I don't see any way to
              > generically accomplish that. Certainly the notion of TAssert could be
              > used to give a partial
              > feedback of that sort.
              >
              > Regards,
              > Chip Campbell

              Well, at least most of Vim (with the exception, IIUC, of the python
              interface) is essentially single-threaded, so unlike developers for some
              other applications, Vim developers normally don't get headaches from
              race conditions (some of them depending on "how fast you're typing and
              how busy is the system") and the sporadic crashes that often go with
              them, and even sometimes disappear when you harness the app in a
              debugging framework. (Though I have sometimes seen X11 error crashes in
              gvim at startup, which spontaneously disappeared when started with the
              --sync switch to force synchronous interfacing with the X server.)


              Best regards,
              Tony.
              --
              "The Good Ship Enterprise" (to the tune of "The Good Ship Lollipop")

              On the good ship Enterprise
              Every week there's a new surprise
              Where the Romulans lurk
              And the Klingons often go berserk.

              Yes, the good ship Enterprise
              There's excitement anywhere it flies
              Where Tribbles play
              And Nurse Chapel never gets her way.

              See Captain Kirk standing on the bridge,
              Mr. Spock is at his side.
              The weekly menace, ooh-ooh
              It gets fried, scattered far and wide.

              It's the good ship Enterprise
              Heading out where danger lies
              And you live in dread
              If you're wearing a shirt that's red.
              -- Doris Robin and Karen Trimble of The L.A. Filkharmonics

              --~--~---------~--~----~------------~-------~--~----~
              You received this message from the "vim_dev" maillist.
              For more information, visit http://www.vim.org/maillist.php
              -~----------~----~----~----~------~----~------~--~---
            • Tom
              ... Just in case somebody cares (however unlikely that may be), I now removed the UT related stuff from tassert and created a spec plugin that allows to write
              Message 6 of 7 , Feb 24, 2009
                > > BTW, tAssert provides convenience functions that my script don't (yet?).
                > > At first, I wondered if both plugins should be merged.

                > This is also the reason why I'd rather prefer to
                > strip down my tassert plugin and to leave only the TAssert command and
                > some utility functions in it.

                Just in case somebody cares (however unlikely that may be), I now
                removed the UT related stuff from tassert and created a spec plugin
                that allows to write script specifications that could look like this:

                http://github.com/tomtom/vimtlib/blob/83976d6b1572000b9950241749371206f7e03d59/spec/spec/spec.vim
                http://github.com/tomtom/vimtlib/blob/83976d6b1572000b9950241749371206f7e03d59/spec/spec/should.vim

                Other than Luc's UT, spec scripts are normal vim scripts (they are not
                pre-processed) but they have to be run by the :Spec command. I don't
                know yet which approach is better but I plan to integrate the should
                functions with his UT since I personally prefer the more descriptive
                messages when something goes wrong. There is also a function that runs
                a command and then compares the buffer contents with a file (with the
                option to ignore changes in whitespace).

                Regards,
                Thomas.

                --~--~---------~--~----~------------~-------~--~----~
                You received this message from the "vim_dev" maillist.
                For more information, visit http://www.vim.org/maillist.php
                -~----------~----~----~----~------~----~------~--~---
              • Ingo Karkat
                ... I ve just published a first version of my testing framework for VIM plugins. Whereas Tom s plugin is for Design by Contract, and Luc has started developing
                Message 7 of 7 , Mar 2, 2009
                  On 19-Feb-09 12:02, Luc Hermitte wrote:
                  > I've finally implemented my own unit testing plugin for vim. [1]
                  > This is a first draft, and all comments are welcomed.
                  >
                  > The plugin has been strongly inspired by Tom Link's tAssert plugin
                  > (thanks Tom!).
                  >
                  > ...
                  >
                  > My plugin is made for Unit Testing, and nothing else. We write an
                  > independent UT script, and give it as argument to :UTRun command.
                  >
                  > On the other hand, Tom's plugin permits to Design scripts by Contract,
                  > which is also extremely useful. i.e., it makes possible to place
                  > assertions in regular Vim scripts (*-plugins). However by its design
                  > choice, it cannot fills the quickfix window with the list of failed
                  > assertions -- hence my NIH script.
                  >
                  > [1] http://code.google.com/p/lh-vim/wiki/UT

                  I've just published a first version of my testing framework for VIM plugins.
                  Whereas Tom's plugin is for Design by Contract, and Luc has started developing a
                  unit testing plugin, my framework allows to write succinct automated regression
                  test suites. I had been dragged down because of the immense testing effort
                  involved whenever I made a modification to one of my published plugins. It is
                  especially tedious to test for different VIM settings and to re-create failure
                  conditions for custom commands and mappings.

                  Now I can specify an expected output, e.g. in file 'test001.ok'
                  expected output
                  and more
                  expected output
                  and/or specify expected messages in file 'test001.msgok'
                  /\d\+ lines changed/
                  use an input file 'test001.in'
                  ExPeCteD OuTpUt
                  AND MORE
                  ExPeCteD OuTpUt
                  write a short test script 'test001.vim'
                  edit test001.in
                  normal! gg3guu
                  write test001.out
                  quit!
                  and my framework invokes the test(s) [suites], compares the actual with the
                  expected output, checks that the messages match, and prints any failures and a
                  test summary:
                  2 tests, 2 run: 2 OK, 0 failures, 0 errors.

                  Unit tests can also be executed; I've chosen the VimTAP plugin because the TAP
                  output is very easy to parse. However (since the quickfix buffer can be saved to
                  a file, too), Luc's UT plugin could be used for unit tests just as well.

                  It's no rocket science, just some (rather ugly) shell script driver, but it has
                  totally changed the way I maintain and develop my plugins. Here's the link:
                  http://vim.sourceforge.net/scripts/script.php?script_id=2565

                  I mention this here in the hope that this may be useful to other script writers,
                  too, or maybe just to inspire them to do more in the testing area, and
                  potentially come up with a better alternative. Feedback is welcome.

                  -- regards, ingo

                  --
                  -- Ingo Karkat -- /^-- /^-- /^-- /^-- /^-- /^-- http://ingo-karkat.de/ --
                  -- http://vim.sourceforge.net/account/profile.php?user_id=9713 --


                  --~--~---------~--~----~------------~-------~--~----~
                  You received this message from the "vim_dev" maillist.
                  For more information, visit http://www.vim.org/maillist.php
                  -~----------~----~----~----~------~----~------~--~---
                Your message has been successfully submitted and would be delivered to recipients shortly.