Time for SPEC JSVM? Benchmarks are bloody hard, but we need better
I have had the pleasure of spending time with members of the V8 team such as Kevin Milikin and Kasper Lund, and Brendan Eich of Mozilla and TraceMonkey. I haven’t had a chance to talk with the great folks behind SquirrelFish Extreme yet, which looks very good too.
It has been great to talk to these guys about the way their implementations work, and to look at differences too. What has been particularly exciting has been how they have loved each others work. There is a huge amount of respect between all of the teams, and they are jazzed to be making the JavaScript world scream.
One common issue that I have found has been benchmarks. They all use slightly different benchmarks and can all agree that:
The JavaScript benchmarks suck!
SunSpider has known bugs in it which are not changed because they have a baseline with IE. Are you kidding me? It is also very Regex heavy, and also focuses on other areas. The methodology of just adding up all of the timings also doesn’t help, and totally skews results (you could do the important things fast, and the obscure things slow, and still look bad). IE can’t even run the tests correctly, and thus their numbers are messed up too.
Dromeo is interesting, but has flaws too. The timing seems to be broken as only V8 gives you precision that doesn’t get rounded down, and thus they get penalized. Also, it uses libraries which do “if browserX do X elsif … else ….” which means different engines actually go down totally different paths.
This is the tip of the iceberg. Benchmarks are notoriously hard to build in a way that measures practical performance versus micro meaninglessness.
It seems like everyone acknowledges that JS benchmark life is pretty bad, so we can either keep that in mind, or we can get everyone together to have an attempt to do a better job. Doing the practical thing can be tough too, as there are too forms of practical:
- How we do things now to get past browser bugs
- How we could do things if there were those silly browser bugs
Time for SPEC JSVM?
September 25th, 2008 at 9:14 pm
I think for these microbenchmarks we should just use the computer language shootout so we know where they stand overall as well.