At Tecplot, Inc. we actively benchmark our products (Tecplot 360, Tecplot RS, and Tecplot Chorus). The motivation is to improve our customer experience by making our products faster, and to clarify our performance relative to competitors. For various reasons, Tecplot 360 has an undeserved reputation for being slow when visualizing large datasets. The fact is that the performance of Tecplot 360 is very competitive for the majority of use cases. Never the less, the reputation has given us great motivation to improve the performance of our software. To do this, we have developed, and continue to develop, a large variety of good benchmarks.
Creating a benchmark sounds easy. Just create a macro that reads a large data file and generates the desired plot, run Tecplot in batch mode with this macro, and time how long it takes – right? Unfortunately, it isn’t that simple. The majority of time required to generate a plot is in reading the data from the file system, and the file system is optimized to provide recently accessed data faster than “stale” data (data that hasn’t been accessed for a while). This optimization takes many forms: operating systems have page caches that keeps recently accessed pages in system memory, advanced hard disk controllers have non-volatile block caches, storage area networks and parallel file systems have additional caches. All of these caches, which I will collectively call the disk cache, work to speed up the access of data that is most frequently used.
As a result of the disk cache, any benchmark will run much faster the second time. The figure below shows Tecplot 360 performance on a benchmark I ran on my Windows desktop computer at work. This specific benchmark reads a 300 million cell finite-element dataset from my local hard disk and generates an isosurface. The result is the time to first image in seconds.
The first run takes 2.5 times longer than subsequent runs. On this particular computer, in this case, the second through sixth runs take about the same amount of time, but this is not always the case. On other systems, the second run may take a tenth the time of the first and subsequent runs may continue to get even faster. If you are going to do benchmarks of visualization software, you must take disk caching into account.
So, what do you do about disk caching? There are two basic approaches. First, you can try to purge the cache. If there is no non-volatile memory in the system, you can do this by turning off the computer every time you run the benchmark, a very slow and cumbersome process, and non-volatile block caches are becoming a common feature of the modern hard-disk controllers. The second approach is to embrace the cache–run the benchmark repeatedly until the result is no longer changing. This second approach is much easier, but it may not match the user behavior you wish to test.