TMC has released their “retrial” of the Petstore “case study”. I know from experience that this one was handled entirely differently, and I think the results show this. Although some people will jump through the whitepaper, looking for a statement like “X beats Y”, they shouldn’t be looking for that kind of statement in this paper. In my opinion, the value of this paper is in showing exactly what was done, giving us valuable information on tuning this particular application. You can’t jump to too many conclusions. For example, the CMP version of the PetStore (mPetstore) actually out performed JPetstore, which didn’t use CMP. Does this now mean that CMP is suddenly fantastic and that we should all use it? no. The paper explains some theories on why this result was seen.
Basically it is due to the nature of the benchmark where hitting the app very hard gives the method dispatch mechanism a chance to show a lot of the performance. JPetstore used a fair amount of Reflection, compared to next to none for the CMP side…. so this made the difference. Does this mean that reflection is evil? NO. Most of the time the performance hit doesn’t matter to use at all (especially compared to the productivity we are getting), but in certain cases, yes, you want to bypass this. Reflection is getting better and better, but even in JDK 1.4.2 (which was used in the case study) it reminds us that there IS a penalty.
If you are a J2EE (or even a .NET) architect, I would take a look through this paper, as it takes the time to detail what was done, and what came out of the experiment.
Web Services was shown to be lagging behind J2EE (even though the other tests: web app and 24hr reliability were shown to be fairly even). I would love to know if a tool like The Mind Electic’s GLUE would have made up the difference (or gone even better).
I also hope we get more feedback on other things that TMC could do with this case study. I hope we continue to get the opportunity to do more case studies like this, trying to get more experiences that can help developers in the future. If you have any questions/concerns please post on TSS, and/or email firstname.lastname@example.org.
Go to the TSS thread on the case study
Also visit the Case Study Home page
I look forward to reading what other bloggers/folks think/feel about these results