benchmarking the sones GraphDB
Since we’re at it – we not only took the new Mono garbage collector through it’s paces regarding linear scaling but we also made some interesting measurements when it comes to query performance on the two .NET platform alternatives.
The same data was used as in the last article about the Mono GC. It’s basically a set of 200.000 nodes which hold between 15 to 25 edges to instances of another type of nodes. One INSERT operation means that the starting node and all edges + connected nodes are inserted at once.
We did not use any bulk loading optimizations – we just fed the sones GraphDB with the INSERT queries. We tested on two platforms – on Windows x64 we used the Microsoft .NET Framework and on Linux x64 we used a current Mono 2.7 build which soon will be replaced by the 2.8 release.
After the import was done we started the benchmarking runs. Every run was given a specified time to complete it’s job. The number of queries that were executed within this time window was logged. Each run utilized 10 simultaneously querying clients. Each client executed randomly generated queries with pre-specified complexity.
The Import
Not surprisingly both platforms are almost head-to-head in average import times. While Mono starts way faster than .NET the .NET platform is faster at the end with a larger dataset. We also measured the ram consumption on each platform and it turns out that while Mono takes 17 kbyte per complex insert operation on average the Microsoft .NET Framework only seems to take 11 kbyte per complex insert operation.
The Benchmark
Let the charts speak for themselves first:
click to enlarge
click on the picture to enlarge
click on the picture to enlarge
As you can see on both platforms the sones GraphDB is able to work through more than 2.000 queries per second on average. For the longest running benchmark (1800 seconds) with all the data imported .NET allows us to answer 2.339 queries per second while Mono allows us to answer 1.980 queries per second.
The Conclusion
With the new generational garbage collector Mono surely made a great leap forward. It’s impressive to see the progress the Mono team was able to make in the last months regarding performance and memory consumption. We’re already considering Mono an important part of our platform strategy – this new garbage collector and benchmark results are showing us that it’s the right thing to do!
UPDATE: There was a mishap in the “import objects per second” row of the above table.
On 10. September 2010 at 23:17
Hey folks, you guys are doing an awesome job with those posts.
Do you mind sharing this benchmark with us (the mono team). I’m curious to profile it and see where we can improve the runtime.
By the way, did you tried the benchmark with either fixed size heap or parallel collector? Those two can make a significant diference.
On 11. September 2010 at 18:55
Sadly, the values of “import objects per second” are wrong, so how can we trust the others values? The values show are for “time to import each object (ms)”.
On 12. September 2010 at 02:44
[…] .NET at Java’s expense, totally under Microsoft’s control and at its mercy. According to this new benchmark, Mono continues to be more like “.NET express”, i.e. an introductory version to […]
On 12. September 2010 at 16:16
@PR73: you are right, the times previously shown were not the import objects per second. I pushed an update of that table. Thanks for your input.
Besides that: we will add the benchmarks sourcecode in the future so everyone can rerun it at their hardware and mono settings.
On 12. September 2010 at 16:19
@Rodrigo Kumpera: Sure! We would love to share the benchmark with you. Since I am not at the office right now I cannot say if we used either way. I know we did not add any additional parameters or settings so it would be the default settings of the sgen gc in current git trunk.
On 7. October 2010 at 20:04
[…] Oh – they even linked my benchmark article. […]
On 8. October 2010 at 15:43
[…] the new garbage collector works can be found on the Mono project’s website. There are also independent benchmarks available that demonstrate some of the performance […]
On 8. October 2010 at 18:32
[…] the new garbage collector works can be found on the Mono project’s website. There are also independent benchmarks available that demonstrate some of the performance […]
On 15. October 2010 at 13:34
[…] the new garbage collector works can be found on the Mono project’s website. There are also independent benchmarks available that demonstrate some of the performance […]