Pet Shop Boys

The current brouhaha over benchmarking of the Pet Shop Application is irrelevant to the real issues of .NET vs. J2EE performance. A technology manager who allows a single number to significantly influence their platform and language decision is incompetent; "application performance" simply does not exist out of context, just as "weight" (as opposed to "mass") does not exist out of context. And the context in which the Pet Shop Application has been developed and benchmarked is so radically different from the context in which the typical business application is developed and judged that no useful conclusions can be drawn from it.

Performance is as performance does. In a business application, performance is the cost of a delivering some quantum of user value in the common context where increasing response-time is correlated with decreasing user value. In other words, if it takes 30 seconds for you to present data to your client for evaluation and your competitors can deliver the exact same data in 15 seconds, then it's probably the case that your system is legitimately under-performing them. On the other hand, if it takes you 30 seconds, and your competitor takes 29 seconds (and thus, the time difference does not decrease user value), but it costs them twice as much to run their system, you have better business performance! This is a fundamental point that is lost on too many programmers.

To be meaningful to business, the cost of delivering a quantum of information to the user has to include two factors: the cost of running the system and the cost of developing the system. So not only is it non-performant to gain a 1% response-time advantage by running your system on overclocked processors that burn up every 12 hours, it's non-performant to gain a 1% response-time advantage if it takes you an additional 18 months to deliver the system. Even this statement is dependent on context, though: if a 1% advantage gives you an edge in pricing stock options correctly, you might be willing to replace the CPUs twice daily or engage in the R&D. It is in the cost of developing the Pet Shop Application that things fall apart irreparably.

The cost of developing a debugged line of source code varies between companies by a factor of 10. You may have good developers and you may have bad developers, but you absolutely, positively do not have the same developers who developed the Pet Shop Application. And the developers of this application, whether for the J2EE platform or the .NET platform are not typical. They are absolutely, positively not median developers. The Pet Shop Application is not the result of typical or business-like development processes. Even aside from the question of development cost, the run-time behavior of a system is highly influenced by the developers and development processes; again, the non-typical processes and developers of the Pet Shop Application make it inappropriate as the basis for any reasonable conclusion about the relative runtime behavior, lines of code, or maintainability of J2EE or .NET applications.

C# vs. Java performance: My conclusion based on 100s+ strictly comparable programs

I have spent the past year writing Thinking in C#, a book based upon Thinking in Java. In the past year, I have compared C# and Java performance on hundreds of small, focused programs. Every indication that I have seen is that, currently, the two platforms are quite comparable in run-time performance of strictly comparable code. In my experience, one of the two platforms displays a small performance advantage in the majority of programs, a performance advantage which I absolutely, positively believe to be irrelevant to the issue of business performance in an enterprise-level application.

In a context of scientific computation, a context that is quite different from the typical business application, I think there is a strong case to be made that the .NET runtime has advantages (value types, square arrays, the ability to turn off range-checking) that make it likely that in the future the .NET runtime is likely to gain an advantage over the JVM in terms of numerics-intensive computation. On the other hand, libraries such as Ninja make it possible to achieve dramatic performance increases in Java. Again, these have nothing to do with performance in a typical business situation.

If you are in the position of choosing a platform for the development of your company's applications, there is no benchmark, set of benchmarks, or sufficiently rigorous studies that provide an objective reason to prefer one platform over the other. That's the true situation on 10/31/2002.