Singularity, Part IV: Performance Testing

Articles in this series
Singularity, Part I: Confessions of a Digital Pack Rat
Singularity, Part II: The Network Upgrade
Singularity, Part III: Construction
Singularity, Part IV: Performance Testing
Singularity, Part V: Conclusion

Intro

With Sin successfully built the big question on our minds was how ZFS would compare to ext3 + md software raid on our hardware. To discover this we spent the first weekend after the server was built running the benchmarks below.

Setup

Unfortunately Ubuntu out of the box had problems with the two array drives sitting on the motherboard rather than the PCI-X controller. We found a bug report for it but didn't feel like updating the kernel just to fix it. Instead we built an 8-disk array instead of a 10-disk for these tests. We're now running a full 10-disk array which obviously performs slightly better than the 8-disk benchmarks below.

What impressed me most about ZFS is that it doesn't need to construct the new disk array. On Linux, md must construct the array, usually by flagging one of the disks as faulty and rebuilding. For both the raid5 and raid6 setups below this took over nine hours under Ubuntu. ZFS, by contrast, doesn't (at least as long as you give it entire disks to use). The array was available for us immediately.

Our tests involved unpacking (not uncompressing) the Linux 2.6.23 kernel tarball, writing a 1GB dd file filled with zeros, and using a benchmarking utility called Bonnie++.


Tests

Untarring the Linux 2.6.23 kernel
Sol: 37.0s
Sin Ubuntu raid5: 3.8s
Sin Ubuntu raid6: 4.1s
Sin Solaris raidz: 3.0s
Sin Solaris raidz2 test: 3.1s

1 GiB dd test
Sol: 10.9s
Sin Ubuntu raid5: 9.9s
Sin Ubuntu raid6: 10.1s
Sin Solaris raidz: 11.9s
Sin Solaris raidz2 test: 8.6s

It should be noted here that mkfile is preferred on Solaris. Using this instead of dd provided more realistic results:

Sin Solaris raidz mkfile: 3.7s
Sin Solaris raidz2 mkfile: 3.8s

bonnie++
The results here are tabled and fairly verbose. They follow the basic trend described above, but click here for full results.


Conclusions

We gained a lot over our older fileserver, Sol, as expected. We were afraid Solaris ZFS would end up being slower than ext3 and software raid, however this turned out not to be the case (other than the dd test, which I don't think is a very accurate test). Both raid5 and raidz outperform their double-parity partners (raid6 and raidz2 respectively), however only by a trivial amount.

More surprising is that ZFS raidz2 with compression performed higher than anything else. Although a bane in the past, compressed filesystems may just make a comeback in the near future as idle CPU cycles outmatch IO bandwidth.

Since our server will be mostly idle IO is our biggest bottleneck; sacrificing CPU for IO makes sense for us. Thus we've chosen to leave compression enabled on our filesystem.


Next: Singularity, Part V: Conclusion