Data Provider Performance Comparison

As it provides uniform access to a variety of different data providers, CIPl offers a unique opportunity to compare the performance of these providers. The protocol for update, retrieval and delete is exactly the same across all providers. Though there are some respects in which comparisons may not fairly reflect the capability of the underlying store, it is possible to isolate these differences and make useful general statements about the performance of the stores.

The performance tests involve objects with the following type structure:

Instances of the type Person are stored with varying amounts of data in the Description property. Across all providers, the instances are represented as a blob (xml for everything except the MongoDB provider). It should be noted that if, for example, queries were required targeting Person.Tag. MongoDB would be substantially faster than everything else because MongoDB is the only store that can index multi-valued properties. Presumably if a normalized relational mapping were used for the type structure, the Sql providers would suffer accordingly.

The test runs were done saving 1000 items, then retrieving and deleting them, giving three time values.

Caveats and Comments

The statistics were generated on a system using an SSD drive, so the numbers really reflect CPU and memory utilization, not IO efficiency and could be very different from what you might see on a given system.

The mySql provider is not using a stored procedure for deletes and so is paying the query compile penalty for every call. Query compilation is very expensive.

The substantial relative slow down in Sql Server beyond Description.Length=10000, is presumably because Sql Server is being forced to swap memory and not doing it very well.

MySql below 10000 is presumably not taking advantage of the available memory.

Not surprisingly, the cloud stores (S3 and Mosso) are much slower than the others. Statistics for a smaller test run are provided at the end of the document.

The source for the application used to generate the statistics is available here.

This spreadsheet (in OpenDoc ods format)  contains some much more detailed statistics varying thread count, item count and item size. Same caveat, this is on a system using an SSD drive, your experience will vary.

This spreadsheet contains some performance numbers for providers run under the MultiThread provider. These are substantially similar to the multi-threaded statistics in the spreadsheet mentioned in the previous paragraph.  It might be of some interest as it offers some insight into how the providers perform where there is a lot of contention for CPU and memory resources. For example, Mongo does not like 4 threads on a 2 CPU system.

Usual caveats, these are very rough numbers for just one physical scenario. Your performance will vary dramatically based on the type of traffic, exact software configuration and the type of hardware. 

1000 Items, Description.Length = 1000


1000 Items, Description.Length = 10000

1000 Items, Description.Length = 100000



100 Items, Description.Length = 10

The file statistics are provided just for comparison purposes.

Last edited Nov 21, 2010 at 4:43 PM by PatrickThompson, version 12

Comments

No comments yet.