Introduction

On one hand each run of rzip64 (with realistic testfile sizes) takes a considerable amount of time. While this mitigates statistical effects in the measurements the overall measurement procedure took us several weeks to complete.

On the other hand rzip64 can create an exceptional high system load. This stress applies not only to the cpu cores and caches but also to the disk I/O system. When multiple cores are involved the I/O capacity of the system may quickly become a bottleneck.

Test platform

Tyan Tempest i5400PW (S5397) Mainboard (dual-socket system)
2 x Intel Xeon Quadcore CPUs E5430 @ 2.66 GHz
4 x 4 GByte FBDIMMs
Single 1.2 TByte Seagate SATA Disk, XFS-Filesystem
openSuSE 11.3 Linux (x86_64, Kernel 2.6.34.7-0.5)

Measurement results

• User time: time spent in user space
• Sys time: time spent in kernel space
• % CPU: average cpu utilization (8 = all cores permanently)
• Wall clock: total wait time for the user
• Max. resident size: number of occupied memory bytes
• Major page faults: page faults that resulted in disk I/O
• Minor page faults: page faults without disk I/O
• Context switches, voluntary (waiting for I/O operations)
• Context switches, involuntary (time slice elapsed)
• I/O in: number of bytes read from disk
• I/O out: number of bytes written to disk

Discussion

The black line answers the question "How does it scale with an increasing number of cores?".
Adding a second core cuts the total wall clock time by apporoximately 40 %. Add a third core and your compression jobs will complete in less than half the single core time. The gain of adding more cores, however, decreases with an increasing number of cores.

As many data compression programs rzip64 operates blockwise. Each block has a size of nearly 1 GByte and is read into memory by a single mmap() operation. Immidiate access to the data in the block causes major page faults with subsequent disk I/O. An optional flag MAP_POPULATE can prevent some page faults by immediately reading ahead the file but that flag is not available on many platforms and is therefore not used currently.

The yellow line (paging activity) also gives some insight how the linux kernel distributes the compression processes over the available cores. This is critcal since that heavily influences the cache performance of the cores.

As stated above the system is equipped with two quad core CPUs. The first four cores share a single die and therefore a single cache. When more cores are added the kernel starts to distribute the processes to the other CPU socket, too. That results in an increase of page acitivity as shown by the yellow line that starts at 5 cores.

As it turns out there is no additional benefit in adding more parallel tasks than physically available cores. The 9th thread on the contrary slows down the compression significantly.

Running rzip64 in parallel on a higher number of cores imposes a high load on the disk I/O subsystem. Each core loads a chunk of nearly 1 GByte in parallel to the other cores. That quickly saturates the available I/O bandwidth of the test system.

Some tests with a large RAM disk suggest a very large impact of the underlying mass storage system. A properly configured RAID may also be used to speed up the overall compression time.