Gregor: XSLTMark Benchmark Results

7/02/2002

 Here are the speed results for different XSLT solutions for Java running on a variety of JVM. The bigger the number the better.

Analysis

 1.    It is very encouraging to see that Gregor translets are already over twice as fast as XSLTC's (on the average), even before "code improvement" optimizations are implemented. The efforts so far have concentrated on implementing the new compilation strategy, based on the traditional modular decomposition into syntax, semantics/IR, and code generation-as well as on implementing the XSLT 1.0 functionality within the new framework. XSLTC's compilation code has all been removed. The optimizations will be implemented following Beta testing with customers.

2.    The XSLTMark configuration has been modified for the benchmarking by setting the number of iterations for each example uniformly to 200. The default configuration uses different iteration numbers depending on a test: from 10 to 150 repetitions. For transformations taking longer the number of repetitions was set to a lower value, even by an order of magnitude. The motivation for a test with a higher constant number of repetitions has been to better model a server-side transformation scenario.

a.     XSLTMark is an excellent collection of XSLT programming techniques that are likely to be used in real life stylesheets.

b.    Some of these techniques lead to relatively trivial computations, easy for any processor.

c.     Other useful constructs truly challenge the efficiency of the implementation.

d.    Users are likely to need stylesheets with any combination of trivial/challenging XSLT constructs and some level of load on their servers: this level is not going to be lower because a stylesheet happens to consume more CPU.

e.    Relatively high, uniformly constant number of iterations is therefore believed to better model the realistic scenario where users at any given moment:

·        Have to deal with a particular load on their servers

·        Have at their disposal limited hardware resources

·        Run a set of transformations depending on a mixture of techniques.

Although the individual tests of XSLTMark each drills on just one or two specific mechanisms, collectively they model very well large real-life transformations.

3.    Gregor's speed advantage over XSLTC varies from test to test. Gregor transformations are anywhere from 2 to 8 times faster than XSLTC's. For several of these tests Gregor transformations run faster on a Zaurus SL-5500 PDA than XSLTC's on the Dell Inspiron used for the benchmarking. For interactive PDA applications using XSLT the speedup of 2-3 times makes a big difference, e.g. 2 seconds response time compared to 6 seconds.

4.    For some tests Gregor is not faster than XSLTC: after all Gregor relies on basically the same DOM/Iterator model as XSLTC and in trivial transformations not much is left for improvement. The bigger and more complicated the transforms the clearer the Gregor advantage. At the moment XSLTC serves as the lower bound for Gregor's performance: because of the common origin Gregor cannot be slower than XSLTC: it can only be faster. And it already is.

5.    Please remember that Gregor doesn't yet perform many planned optimizations. The speedup gain comes from approaching several XSLT issues with a fresh outlook. Planned optimizations will make Gregor translets still faster even for relatively simple transformations. The compiler's job is to first analyze the source program and then to synthesize an executable equivalent using an appropriate instruction set and a RuntimeLibrary. One may think about Gregor emulating a smart programmer who first makes a number of observations about a stylesheet and then carefully chooses implementations to be syntesized into the resulting program. For this vision to work a carefully chosen set of abstractions needs to be implemented to ease the tasks of high level analysis and synthesis. XSLTC lacks these abstractions opting instead for direct translation of Abstract Syntax into Java bytecodes. The scheme is error prone, leads to hardly managable tangled software, and virtually prohibits Gregor-style optimization and easy extensibility. Gregor has been born out of deep dissatisfaction with XSLTC.

6.    The work so far has indeed generated a long list of speedup ideas. The game plan is however

a.     To first enable customers to start benefiting now from up to 8 times speed advantage of Gregor

b.    To continue performance tuning over the next several months

c.     To offer free upgrades option: customers subscribing to this option will continue to receive performance upgrades for one year.

7.    Although your actual mileage will vary XSLTMark testing implies an average speedup by the factor of 2 over XSLTC. This means that

a.     Gregor users will need half the hardware resources to run the same number of transformations, or

b.    They will be able to deal with twice as big a server load on the same hardware;

c.     PDA/Wireless apps will respond faster making for a qualitatively better user experience. We will come back to this topic soon.

8.    The unquestionably fastest Java has been JRockIt from BEA. Surprisingly, Xalan differs from the other solutions by running the slowest on JRockIt. It runs the fastest on IBM's Java.

9.    Saxon clearly benefits from JRockIt, however, it too has a strong preference for IBM's over Sun's Java.

10.  The newest Java from Sun, J2SDK1.4.0_01 (Linux platform) hurts every processor to some degree, the extreme case being XSLTC which appears to be looping on the JVM hence the "N/A" entry in the results table above. This is strange because every other processor runs OK on the new Java (albeit slower), and XSLTC runs OK on every other JVM; it is only the combination of the J2SDK1.4.0_01 and XSLTC that couldn't be benchmarked. XSLTC coming from "java.sun.com" XML Pack Summer 02 exhibits the same problem. It is an unfortunate coincidence that the two new Java and XML offerings from Sun do not work together well.

Summary

Even before applying traditional compiler optimizations Gregor beats XSLTC by a factor of 2 leading to a clear economic advantage for Ambrosoft's customers. The speed advantage so far comes from Gregor's not making the shortcuts and simplifications of XSLTC, and from new compilation approaches overlooked by XSLTC. The speed will only get higher now that more energy will be directed specifically towards performance goals. Customers buying the option "with yearly upgrades" will automatically subscribe to one year of free Gregor performance upgrades. The PDA/wireless applications are, in addition to scalable server environments, the other major focus of the Gregor work. They are supported by smaller RuntimeLibrary and a new compressed DOM.

About the background color

Previously Ambrosoft's web pages were Navajo white, matching the background color I have been using for years editing programs in XEmacs. The change has been inspired by astronomy research into the "color of the universe." I find the new color subtler, yet optimistic and vibrant, and corresponding with my vision that "sky is the limit."

Jacek R. Ambroziak

For all inquiries please write to gregor-info@ambrosoft.com.