Loading ...
Sorry, an error occurred while loading the content.
 

Benchmarking update.

Expand Messages
  • Greg Wilkins
    The Resin folks have done a new release to fix up whatever problem my benchmark was finding in their server (muttered something about TCP_NODELAY). They now
    Message 1 of 1 , Nov 11, 2002
      The Resin folks have done a new release to fix up whatever
      problem my benchmark was finding in their server (muttered something
      about TCP_NODELAY). They now server servlets a little faster than
      Jetty, but their static file serving is beyond crap.

      So here are my updated benchmarks.

      ----------------------

      I used the stresstest.war from ftp://ftp.mortbay.org/pub which
      is both a webapp and a client war. It tests HTTP/1.1 serving
      of files and servlets with a mix of filters, forwards and authentication.

      Persistent connections are used, and there is a reasonable mix of
      304 responses expected from conditional headers. Session cookies
      are also sometime supplied - but for an invalid session.

      I tested tomcat Resin 2.1.6, Resin-2.1.5 4.0.4-LE-jdk14, 4.1.10-LE-jdk1.4, Jetty 4.1.3 and 4.2.0rc1

      All configurations were "out of the box" except:

      + request logging was turned off.
      + A sample user was added to the in memory realm
      + Max 200 threads, min 20 threads.
      + deleted all example webapps.
      + Any production recommendations in the config comments
      were accepted.

      Test server was jdk1.4.1 on debian linux on a 1.2GHz PIII.
      Test client was jdk1.4.1 on debian linux on a 0.9GHz PIII.

      Default JVM memory allocation was used for Jetty.
      Default tomcat memory allocation was used for tomcat & Resin
      The stresstest.war was just put in the webapps directory before starting.

      The test client was run with 10 threads, 30 iterations from a
      cold start of the server. The client does a single warmup iteration
      with all threads before starting the timing. I picked 10 threads as that
      maxed out cpu in tc404. The war is also a client and is run with:

      java -jar stresstest.war serverhost 8080 10 30

      And the results are:

      Resin-2.1.5:
      ============
      MODE: /plain/file CONNECTS=300(3/sec) REQUESTS=9600(127/sec) FAILURES=0(0/sec)
      MODE: /plain/servlet CONNECTS=300(7/sec) REQUESTS=9900(254/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(3/sec) REQUESTS=9600(127/sec) FAILURES=0(0/sec)
      MODE: /auth/servlet CONNECTS=300(7/sec) REQUESTS=9900(253/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(3/sec) REQUESTS=9600(127/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(7/sec) REQUESTS=9900(254/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(7/sec) REQUESTS=9900(254/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(7/sec) REQUESTS=9900(255/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(7/sec) REQUESTS=9900(254/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(5/sec) REQUESTS=117000(170/sec) FAILURES=0(0/sec)

      This is rather pathetic. So pathetic that I was sure I was doing something
      wrong. So I contacted the Resin folks and they made a new release (2.1.6)
      as a result, which goes much much faster (see below).


      Resin 2.1.6
      ===========
      MODE: /plain/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /plain/servlet CONNECTS=300(65/sec) REQUESTS=9900(2155/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /auth/servlet CONNECTS=300(56/sec) REQUESTS=9900(1854/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(65/sec) REQUESTS=9900(2174/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(65/sec) REQUESTS=9900(2167/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(65/sec) REQUESTS=9900(2157/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(3/sec) REQUESTS=9600(126/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(65/sec) REQUESTS=9900(2155/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(7/sec) REQUESTS=117000(242/sec)FAILURES=0(0/sec)

      Much better servlet serving - in fact the fastest servlet server tested (about 5% faster
      than Jetty4.2). But their serving of static content is still pathetic. So I guess they still
      have problems to be fixed.


      Tomcat 4.0.4:
      =============
      MODE: /plain/file CONNECTS=300(4/sec) REQUESTS=9592(139/sec) FAILURES=1(0/sec)
      MODE: /plain/servlet CONNECTS=300(36/sec) REQUESTS=9900(1189/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(4/sec) REQUESTS=9580(139/sec) FAILURES=1(0/sec)
      MODE: /auth/servlet CONNECTS=300(28/sec) REQUESTS=9900(953/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(4/sec) REQUESTS=9600(141/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(34/sec) REQUESTS=9900(1133/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(4/sec) REQUESTS=9600(141/sec) FAILURES=1(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(25/sec) REQUESTS=9900(828/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(4/sec) REQUESTS=9579(140/sec) FAILURES=1(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(26/sec) REQUESTS=9900(864/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(4/sec) REQUESTS=9572(140/sec) FAILURES=2(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(25/sec) REQUESTS=9900(836/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(7/sec) REQUESTS=116923(247/sec) FAILURES=6(0/sec)

      It had a bit of trouble with the test. The test uses a slightly
      non standard date format for a few requests, which tc404 does a 500 exception for.
      But it limped by enough to get an idea. Bad serving static files and OK
      serving of servlets.



      Tomcat 4.1.10:
      ==============
      MODE: /plain/file CONNECTS=300(4/sec) REQUESTS=9600(153/sec) FAILURES=0(0/sec)
      MODE: /plain/servlet CONNECTS=300(57/sec) REQUESTS=9900(1901/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(4/sec) REQUESTS=9600(154/sec) FAILURES=0(0/sec)
      MODE: /auth/servlet CONNECTS=300(44/sec) REQUESTS=9900(1461/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(4/sec) REQUESTS=9600(155/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(57/sec) REQUESTS=9900(1891/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(4/sec) REQUESTS=9600(154/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(52/sec) REQUESTS=9900(1744/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(4/sec) REQUESTS=9600(154/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(51/sec) REQUESTS=9900(1700/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(4/sec) REQUESTS=9600(154/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(50/sec) REQUESTS=9900(1654/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(8/sec) REQUESTS=117000(286/sec)FAILURES=0(0/sec)

      Let down badly by its file serving. Servlet serving is pretty good unless you
      use authentication.



      Jetty 4.1.3:
      ============
      MODE: /plain/file CONNECTS=300(21/sec) REQUESTS=9600(683/sec) FAILURES=0(0/sec)
      MODE: /plain/servlet CONNECTS=300(52/sec) REQUESTS=9900(1746/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(21/sec) REQUESTS=9600(679/sec) FAILURES=0(0/sec)
      MODE: /auth/servlet CONNECTS=300(50/sec) REQUESTS=9900(1659/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(21/sec) REQUESTS=9600(690/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(50/sec) REQUESTS=9900(1675/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(21/sec) REQUESTS=9600(678/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(46/sec) REQUESTS=9900(1525/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(21/sec) REQUESTS=9600(680/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(46/sec) REQUESTS=9900(1534/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(21/sec) REQUESTS=9600(680/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(44/sec) REQUESTS=9900(1469/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(29/sec)REQUESTS=117000(961/sec)FAILURES=0(0/sec)

      Much better file serving that tomcat, but servlets actually a little slower than tc4.1
      at this load except that authentication etc does not slow it down.



      Jetty 4.2.0rc1
      ==============
      MODE: /plain/file CONNECTS=300(21/sec) REQUESTS=9600(677/sec) FAILURES=0(0/sec)
      MODE: /plain/servlet CONNECTS=300(61/sec) REQUESTS=9900(2017/sec) FAILURES=0(0/sec)
      MODE: /auth/file CONNECTS=300(21/sec) REQUESTS=9600(684/sec) FAILURES=0(0/sec)
      MODE: /auth/servlet CONNECTS=300(61/sec) REQUESTS=9900(2034/sec) FAILURES=0(0/sec)
      MODE: /filter/file CONNECTS=300(21/sec) REQUESTS=9600(681/sec) FAILURES=0(0/sec)
      MODE: /filter/servlet CONNECTS=300(61/sec) REQUESTS=9900(2035/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/file CONNECTS=300(21/sec) REQUESTS=9600(675/sec) FAILURES=0(0/sec)
      MODE: /forward/plain/servlet CONNECTS=300(60/sec) REQUESTS=9900(2012/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/file CONNECTS=300(21/sec) REQUESTS=9600(679/sec) FAILURES=0(0/sec)
      MODE: /forward/auth/servlet CONNECTS=300(62/sec) REQUESTS=9900(2060/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/file CONNECTS=300(21/sec) REQUESTS=9600(680/sec) FAILURES=0(0/sec)
      MODE: /forward/filter/servlet CONNECTS=300(61/sec) REQUESTS=9900(2044/sec) FAILURES=0(0/sec)
      GRAND TOTAL: CONNECTS=3600(31/sec)REQUESTS=117000(1026/sec)FAILURES=0(0/sec)


      Jetty 4.2 is looking pretty cool!
      Resin is still a tad (5%) faster for servlets - so we have something to chase. But our static
      file handling still makes us king of this benchmark.


      All in all, I think that Jetty, Resin and Tomcat must all be getting pretty close to terminal
      velocity for serving servlets. We are all maxing out just under or over 2000 servlet requests
      a second on my hardware - I doubt there is much scope of going much beyond that and I don't
      think we'll see 3000 unless I get a new network :-)

      cheers





      --
      Greg Wilkins<gregw@...> Phone/fax: +44 7092063462
      Mort Bay Consulting Australia and UK. http://www.mortbay.com
    Your message has been successfully submitted and would be delivered to recipients shortly.