Loading ...
Sorry, an error occurred while loading the content.

Existing tools?

Expand Messages
  • Ron Lee
    Hi Mark, Thanx for starting this discussion. It s sorely needed. I ve been looking around for some HTTP compliance/conformance tests and haven t found
    Message 1 of 14 , Oct 14, 2000
    • 0 Attachment
      Hi Mark,

      Thanx for starting this discussion. It's sorely needed. <g>

      I've been looking around for some HTTP compliance/conformance tests and haven't found much. Do you know of anyone publishing any tools?

      best regards
      ron
    • Alex Rousskov
      Hi Ron, TMF has been approached by a vendor who is ready to seed the development of a compliance testing tool. From our past conversations we know that Novell
      Message 2 of 14 , Oct 14, 2000
      • 0 Attachment
        Hi Ron,

        TMF has been approached by a vendor who is ready to seed the
        development of a compliance testing tool. From our past conversations we
        know that Novell and others are also interested. We are now collecting
        test cases to be included in the first version. If everything goes as
        planned, we will start actual and rapid development in January.

        Clearly, we would need feedback and test cases from many vendors
        for this tool to become useful. If you or other folks have particular
        tests in mind, please let us know!

        I expect the tools itself to be a collection of well documented
        test cases rather than a pure HTTP compliance test. It probably does not
        make much sense to spend a year on checking all MUSTs in HTTP just to
        find out that half of them are not triggered or not important in
        _practice_ anyway.

        My preference would be to assemble a collection of
        useful/interesting test cases, some of which will test HTTP compliance
        and some will test "best practices" or simply "compatibility" with known
        and common behavior.

        I would suggest a simple and straightforward approach of building
        such a tool: Any interested party can submit a detailed test
        specification to be included into the test suite. The tests are included
        in a more-or-less FIFO order, weighted by the interest of the active
        contributers. The decision making will be an open, consensus based
        process. With this scheme in place, and once the underlying framework is
        implemented, we will have a live and up-to-date collection of tests. Any
        given contributer may use only a subset of the available tests, of
        course; whatever they find useful...

        The tool will not be used for competition but rather "verification"
        that everything works as expected. Thus, there should be less conflicts
        between vendors sponsoring the development and less marketing influence.
        However, if there is sufficient demand, a subset of the tests can be
        used for HTTP compliance certification. Cooperation with W3C and/or IETF
        should not be a problem.

        Once again, at this time we are collecting the test cases to be
        included into the first version, starting with a few cases already
        posted to WREC and other e-lists. Please contribute!

        Comments/feedback is very welcome, as usual.

        Thanks,

        Alex.

        P.S. The big unanswered question is the scope of the tests. IMO, testing
        a proxy cache and other intermediaries is significantly easier than
        testing origin servers. Testing clients is probably the hardest. I think
        we should postpone the scope resolution untill the first collection of
        tests is established. We will know where the demand is higher then...


        From: Ron Lee <rlee@n...>
        Date: Sat Oct 14, 2000 7:31am
        Subject: Existing tools?

        Hi Mark,

        Thanx for starting this discussion. It's sorely needed.<g>

        I've been looking around for some HTTP compliance/conformance tests
        and haven't found much. Do you know of anyone publishing any tools?

        best regards
        ron
      • ylafon@raubacapeu.net
        ... I have to disagree here, if MUSTs are not implemented well, it may forbid some uses of them, then you will never see them in practice. The typical chicken
        Message 3 of 14 , Oct 14, 2000
        • 0 Attachment
          --- In http-compliance@egroups.com, Alex Rousskov <rousskov@m...>
          wrote:

          > I expect the tools itself to be a collection of well documented
          > test cases rather than a pure HTTP compliance test. It probably does
          > not make much sense to spend a year on checking all MUSTs in HTTP
          > just to find out that half of them are not triggered or not
          > important in _practice_ anyway.

          I have to disagree here, if MUSTs are not implemented well, it
          may forbid some uses of them, then you will never see them in
          practice. The typical chicken and egg problem.
          Ex: not that many people are using HTTP/1.1 PUT (I do), and now I am
          posting behind a "transparent" proxy, that just filters the 100
          Continue replies. Of course it is not common to use PUT but if
          proxies are making PUT harder, then it won't be widely used.
          My 2 cts.

          ~~Yves
        • Alex Rousskov
          ... I would not even argue. The test acceptance procedure that I envision (see my previous post) makes my personal opinion regarding MUSTs irrelevant! If the
          Message 4 of 14 , Oct 14, 2000
          • 0 Attachment
            On Sat, 14 Oct 2000 ylafon@... wrote:

            > I have to disagree here, if MUSTs are not implemented well, it
            > may forbid some uses of them, then you will never see them in
            > practice. The typical chicken and egg problem.
            > Ex: not that many people are using HTTP/1.1 PUT (I do), and now I am
            > posting behind a "transparent" proxy, that just filters the 100
            > Continue replies. Of course it is not common to use PUT but if
            > proxies are making PUT harder, then it won't be widely used.

            I would not even argue. The test acceptance procedure that I envision
            (see my previous post) makes my personal opinion regarding MUSTs
            irrelevant! If the consensus among active contributers is that all MUSTs
            should be tested first, they will be tested first.

            IMO, the objective should be to build a tool that is immediately useful
            to the greatest number of contributers. That is, we should maximize the
            usefulness of the tool. If the latter means strict HTTP compliance
            first, then so be it. But the the process should be demand-driven, not
            final-goal-driven, especially if we want to have something usable around
            January.

            Alex.
          • Ron Lee
            That s good to hear, Alex. Have you considered posting your existing test cases to your TMF Web site so we can begin our discussion from that point? best
            Message 5 of 14 , Oct 14, 2000
            • 0 Attachment
              That's good to hear, Alex.

              Have you considered posting your existing test cases to your TMF Web site so we can begin our discussion from that point?

              best regards
              ron
            • Alex Rousskov
              ... Absolutely! We will post as soon as we have 5-7 test cases collected and properly described. If you have particular case candidate(s) in mind, please let
              Message 6 of 14 , Oct 14, 2000
              • 0 Attachment
                On Sat, 14 Oct 2000, Ron Lee wrote:

                > That's good to hear, Alex.
                >
                > Have you considered posting your existing test cases to your TMF Web
                > site so we can begin our discussion from that point?

                Absolutely! We will post as soon as we have 5-7 test cases collected and
                properly described. If you have particular case candidate(s) in mind,
                please let us know.

                Alex.
              • Mark Nottingham
                hehehe... I think I can come up with a few... *grin* ... -- Mark Nottingham http://www.mnot.net/
                Message 7 of 14 , Oct 14, 2000
                • 0 Attachment
                  hehehe... I think I can come up with a few...


                  *grin*


                  On Sat, Oct 14, 2000 at 11:36:51AM -0600, Alex Rousskov wrote:
                  > On Sat, 14 Oct 2000, Ron Lee wrote:
                  >
                  > > That's good to hear, Alex.
                  > >
                  > > Have you considered posting your existing test cases to your TMF Web
                  > > site so we can begin our discussion from that point?
                  >
                  > Absolutely! We will post as soon as we have 5-7 test cases collected and
                  > properly described. If you have particular case candidate(s) in mind,
                  > please let us know.
                  >
                  > Alex.
                  >
                  >
                  >
                  > To unsubscribe from this group, send an email to:
                  > http-compliance-unsubscribe@egroups.com
                  >
                  >
                  >

                  --
                  Mark Nottingham
                  http://www.mnot.net/
                • Mark Nottingham
                  Do you have a test submission template yet? If not, let s start getting one together. ... -- Mark Nottingham http://www.mnot.net/
                  Message 8 of 14 , Oct 14, 2000
                  • 0 Attachment
                    Do you have a test submission template yet? If not, let's start getting one
                    together.



                    On Sat, Oct 14, 2000 at 11:36:51AM -0600, Alex Rousskov wrote:
                    > On Sat, 14 Oct 2000, Ron Lee wrote:
                    >
                    > > That's good to hear, Alex.
                    > >
                    > > Have you considered posting your existing test cases to your TMF Web
                    > > site so we can begin our discussion from that point?
                    >
                    > Absolutely! We will post as soon as we have 5-7 test cases collected and
                    > properly described. If you have particular case candidate(s) in mind,
                    > please let us know.
                    >
                    > Alex.
                    >
                    >
                    >
                    > To unsubscribe from this group, send an email to:
                    > http-compliance-unsubscribe@egroups.com
                    >
                    >
                    >

                    --
                    Mark Nottingham
                    http://www.mnot.net/
                  • Alex Rousskov
                    ... How about this: 1. Test ID [ assigned after initial submission ] 2. Submitter ID [ e.g., e-mail address ] 3. One-liner [ one line test
                    Message 9 of 14 , Oct 14, 2000
                    • 0 Attachment
                      On Sat, 14 Oct 2000, Mark Nottingham wrote:

                      > Do you have a test submission template yet? If not, let's start
                      > getting one together.

                      How about this:

                      1. Test ID [ assigned after initial submission ]
                      2. Submitter ID [ e.g., e-mail address ]
                      3. One-liner [ one line test description ]
                      4. DUT [ HTTP client, intermediary, and/or server, etc. ]
                      5. Importance [ relative to other; 0.0 - 1.0 ]
                      6. Subject [ precise description of *what* needs to be tested ]
                      7. Passing criteria [ precise; if applicable ]
                      8. Cases [ optional cases with DUTs passing/failing ]
                      9. References [ optional references to HTTP or other docs ]
                      10. Implementation [ optional ideas on *how* subject can be tested ]
                      11. Comments [ anything else ]
                      -------------------
                      12. Submission date
                      13. Implementation date
                      14. Application comments [ how to apply the test ]



                      Here is an example:

                      1. 36-3908-01-23
                      2. Mark C Nottingham <mnot@...>
                      3. forwarding of hop-by-hop headers
                      4. intermediary
                      5. 0.4
                      6. when forwarding HTTP messages, a hop-by-hop header, if any,
                      must not be forwarded by an intermediary
                      7. known hop-by-hop headers are not forwarded by the DUT
                      with known kinds of HTTP messages
                      8. BrokenProxy v1.0 fails
                      MyProxy v5.3 passes
                      9. RFC 2616 Section 13.5.1
                      10. pass all kinds of HTTP messages (with variable number
                      of hop-by-hop headers) through the proxy
                      and check if at least one header is passed through in either
                      direction; check hit/miss/304 cases
                      11. I think this test is important because I spent 10 hours
                      trying to understand why our system does not work
                      with a BrokenProxy v1.0 in the way.
                      12. Thu, 22 Jun 2000 16:40:17 -0700

                      Just a suggestion for a template... Please fix/add/delete whatever is
                      needed.

                      Alex.
                    • Mark Nottingham
                      I tried playing with the egroups database function, but it s a bit too simple for what we need. So, I threw this together;
                      Message 10 of 14 , Oct 15, 2000
                      • 0 Attachment
                        I tried playing with the egroups database function, but it's a bit too
                        simple for what we need. So, I threw this together;
                        http://www.mnot.net/http_compliance/list

                        Everyone please have a play with it and give feedback; if this seems
                        like it will do the trick, we can start entering tests and sifting
                        through them. If anyone has a better idea, please bring it up (I
                        looked at bugzilla, etc., but they're a little heavy for this, IMHO).

                        BTW, there's no authentication on adding or editing tests; however,
                        the db is backed up daily. If this becomes a problem (hope not), I can
                        add something.

                        I'll also work on getting an interface to dump the db included.

                        Cheers,

                        --- In http-compliance@egroups.com, Alex Rousskov <rousskov@m...>
                        wrote:
                        > On Sat, 14 Oct 2000, Mark Nottingham wrote:
                        >
                        > > Do you have a test submission template yet? If not, let's start
                        > > getting one together.
                        >
                        > How about this:
                        >
                        > 1. Test ID [ assigned after initial submission ]
                        > 2. Submitter ID [ e.g., e-mail address ]
                        > 3. One-liner [ one line test description ]
                        > 4. DUT [ HTTP client, intermediary, and/or server,
                        etc. ]
                        > 5. Importance [ relative to other; 0.0 - 1.0 ]
                        > 6. Subject [ precise description of *what* needs to be
                        tested ]
                        > 7. Passing criteria [ precise; if applicable ]
                        > 8. Cases [ optional cases with DUTs passing/failing
                        ]
                        > 9. References [ optional references to HTTP or other docs
                        ]
                        > 10. Implementation [ optional ideas on *how* subject can be
                        tested ]
                        > 11. Comments [ anything else ]
                        > -------------------
                        > 12. Submission date
                        > 13. Implementation date
                        > 14. Application comments [ how to apply the test ]
                        >
                        >
                        >
                        > Here is an example:
                        >
                        > 1. 36-3908-01-23
                        > 2. Mark C Nottingham <mnot@a...>
                        > 3. forwarding of hop-by-hop headers
                        > 4. intermediary
                        > 5. 0.4
                        > 6. when forwarding HTTP messages, a hop-by-hop header, if any,
                        > must not be forwarded by an intermediary
                        > 7. known hop-by-hop headers are not forwarded by the DUT
                        > with known kinds of HTTP messages
                        > 8. BrokenProxy v1.0 fails
                        > MyProxy v5.3 passes
                        > 9. RFC 2616 Section 13.5.1
                        > 10. pass all kinds of HTTP messages (with variable number
                        > of hop-by-hop headers) through the proxy
                        > and check if at least one header is passed through in either
                        > direction; check hit/miss/304 cases
                        > 11. I think this test is important because I spent 10 hours
                        > trying to understand why our system does not work
                        > with a BrokenProxy v1.0 in the way.
                        > 12. Thu, 22 Jun 2000 16:40:17 -0700
                        >
                        > Just a suggestion for a template... Please fix/add/delete whatever
                        is
                        > needed.
                        >
                        > Alex.
                      • Alex Rousskov
                        ... IMHO, your simple interface is sufficient for now. When we have enough cases and actually start implementing that test suite, we will probably want to
                        Message 11 of 14 , Oct 16, 2000
                        • 0 Attachment
                          On Mon, 16 Oct 2000, Mark Nottingham wrote:

                          > I tried playing with the egroups database function, but it's a bit too
                          > simple for what we need. So, I threw this together;
                          > http://www.mnot.net/http_compliance/list
                          >
                          > Everyone please have a play with it and give feedback; if this seems
                          > like it will do the trick, we can start entering tests and sifting
                          > through them. If anyone has a better idea, please bring it up (I
                          > looked at bugzilla, etc., but they're a little heavy for this, IMHO).

                          IMHO, your simple interface is sufficient for now. When we have enough
                          cases and actually start implementing that test suite, we will probably
                          want to integrate the database with the suite and add next-stage fields
                          like "meaningful id", "status", various timestamps, and implementation
                          details/instructions. For now, your list should do the job just fine,
                          and the conversion will not be painful, I think.

                          Thanks for putting it together!

                          Alex.
                        • Mark Nottingham
                          Glad you like it. I m very aware that it probably won t be enough for displaying, manipulating the db, etc. but for gathering and light review it should do the
                          Message 12 of 14 , Oct 16, 2000
                          • 0 Attachment
                            Glad you like it. I'm very aware that it probably won't be enough for
                            displaying, manipulating the db, etc. but for gathering and light review it
                            should do the trick for now.

                            At this point, I'm more interested in seeing what kinds of tests people are
                            concerned about, etc.

                            SO, everybody start submitting!




                            On Mon, Oct 16, 2000 at 07:20:31AM -0600, Alex Rousskov wrote:
                            > On Mon, 16 Oct 2000, Mark Nottingham wrote:
                            >
                            > > I tried playing with the egroups database function, but it's a bit too
                            > > simple for what we need. So, I threw this together;
                            > > http://www.mnot.net/http_compliance/list
                            > >
                            > > Everyone please have a play with it and give feedback; if this seems
                            > > like it will do the trick, we can start entering tests and sifting
                            > > through them. If anyone has a better idea, please bring it up (I
                            > > looked at bugzilla, etc., but they're a little heavy for this, IMHO).
                            >
                            > IMHO, your simple interface is sufficient for now. When we have enough
                            > cases and actually start implementing that test suite, we will probably
                            > want to integrate the database with the suite and add next-stage fields
                            > like "meaningful id", "status", various timestamps, and implementation
                            > details/instructions. For now, your list should do the job just fine,
                            > and the conversion will not be painful, I think.
                            >
                            > Thanks for putting it together!
                            >
                            > Alex.
                            >
                            >
                            >
                            > To unsubscribe from this group, send an email to:
                            > http-compliance-unsubscribe@egroups.com
                            >
                            >
                            >

                            --
                            Mark Nottingham
                            http://www.mnot.net/
                          • Ron Lee
                            So, how about that subject line? Are we to assume there are absolutely no existing tools out there? best regards ron ... Glad you like it. I m very aware
                            Message 13 of 14 , Oct 16, 2000
                            • 0 Attachment
                              So, how about that subject line? <g> Are we to assume there are absolutely no existing tools out there?

                              best regards
                              ron

                              >>> mnot@... 10/16/00 10:08AM >>>

                              Glad you like it. I'm very aware that it probably won't be enough for
                              displaying, manipulating the db, etc. but for gathering and light review it
                              should do the trick for now.

                              At this point, I'm more interested in seeing what kinds of tests people are
                              concerned about, etc.

                              SO, everybody start submitting!




                              On Mon, Oct 16, 2000 at 07:20:31AM -0600, Alex Rousskov wrote:
                              > On Mon, 16 Oct 2000, Mark Nottingham wrote:
                              >
                              > > I tried playing with the egroups database function, but it's a bit too
                              > > simple for what we need. So, I threw this together;
                              > > http://www.mnot.net/http_compliance/list
                              > >
                              > > Everyone please have a play with it and give feedback; if this seems
                              > > like it will do the trick, we can start entering tests and sifting
                              > > through them. If anyone has a better idea, please bring it up (I
                              > > looked at bugzilla, etc., but they're a little heavy for this, IMHO).
                              >
                              > IMHO, your simple interface is sufficient for now. When we have enough
                              > cases and actually start implementing that test suite, we will probably
                              > want to integrate the database with the suite and add next-stage fields
                              > like "meaningful id", "status", various timestamps, and implementation
                              > details/instructions. For now, your list should do the job just fine,
                              > and the conversion will not be painful, I think.
                              >
                              > Thanks for putting it together!
                              >
                              > Alex.
                              >
                              >
                              >
                              > To unsubscribe from this group, send an email to:
                              > http-compliance-unsubscribe@egroups.com
                              >
                              >
                              >

                              --
                              Mark Nottingham
                              http://www.mnot.net/


                              To unsubscribe from this group, send an email to:
                              http-compliance-unsubscribe@egroups.com
                            • Mark Nottingham
                              I don t know of any publicly available test harnesses. Quite a few companies have made noises about releasing theirs under certain conditions, which is
                              Message 14 of 14 , Oct 16, 2000
                              • 0 Attachment
                                I don't know of any publicly available test harnesses. Quite a few companies
                                have made noises about releasing theirs under certain conditions, which is
                                hopefully one of the conversations we can have here.


                                The one tool that I find very useful in testing is webtee, from the blast
                                toolkit:
                                http://www.cache.dfn.de/DFN-Cache/Development/blast.html

                                Webtee acts like a proxy, and lets you see the HTTP headers (request and
                                response) as they go by.




                                On Mon, Oct 16, 2000 at 10:15:47AM -0600, Ron Lee wrote:
                                > So, how about that subject line? <g> Are we to assume there are absolutely no existing tools out there?
                                >
                                > best regards
                                > ron
                                >
                                > >>> mnot@... 10/16/00 10:08AM >>>
                                >
                                > Glad you like it. I'm very aware that it probably won't be enough for
                                > displaying, manipulating the db, etc. but for gathering and light review it
                                > should do the trick for now.
                                >
                                > At this point, I'm more interested in seeing what kinds of tests people are
                                > concerned about, etc.
                                >
                                > SO, everybody start submitting!
                                >
                                >
                                >
                                >
                                > On Mon, Oct 16, 2000 at 07:20:31AM -0600, Alex Rousskov wrote:
                                > > On Mon, 16 Oct 2000, Mark Nottingham wrote:
                                > >
                                > > > I tried playing with the egroups database function, but it's a bit too
                                > > > simple for what we need. So, I threw this together;
                                > > > http://www.mnot.net/http_compliance/list
                                > > >
                                > > > Everyone please have a play with it and give feedback; if this seems
                                > > > like it will do the trick, we can start entering tests and sifting
                                > > > through them. If anyone has a better idea, please bring it up (I
                                > > > looked at bugzilla, etc., but they're a little heavy for this, IMHO).
                                > >
                                > > IMHO, your simple interface is sufficient for now. When we have enough
                                > > cases and actually start implementing that test suite, we will probably
                                > > want to integrate the database with the suite and add next-stage fields
                                > > like "meaningful id", "status", various timestamps, and implementation
                                > > details/instructions. For now, your list should do the job just fine,
                                > > and the conversion will not be painful, I think.
                                > >
                                > > Thanks for putting it together!
                                > >
                                > > Alex.
                                > >
                                > >
                                > >
                                > > To unsubscribe from this group, send an email to:
                                > > http-compliance-unsubscribe@egroups.com
                                > >
                                > >
                                > >
                                >
                                > --
                                > Mark Nottingham
                                > http://www.mnot.net/
                                >
                                >
                                > To unsubscribe from this group, send an email to:
                                > http-compliance-unsubscribe@egroups.com
                                >
                                >
                                >
                                >
                                >
                                >
                                > To unsubscribe from this group, send an email to:
                                > http-compliance-unsubscribe@egroups.com
                                >
                                >
                                >

                                --
                                Mark Nottingham
                                http://www.mnot.net/
                              Your message has been successfully submitted and would be delivered to recipients shortly.