What was the physical network topology in this test? (Was there one, or was this test just done from a Solaris machine with some ZFS file-systems created?)

How much cache/system memory was available for ZFS to cache with?

It is handily noted that 512MB files were used, but how many?
It makes it quite difficult to know if caching boundaries are being surpassed without understanding in fact how many 512MB files were used and how much total cache was available on the system in question…

More than this…

What storage was unsed in this configuration? (Vendor, Model, etc.?) And is it safe to assume that this is going over a straight through Fibre Channel connection to the backend storage from the server/machine in question?

Thanks.

The reason for the directio setting is to avoid unneeded double caching of the blocks, because oracle manages a cache itself, and data which is read from disk wont be read again from oracle (unless memeory is low, but it is lower if you have two caches).

However I do not know – but I asume – that ZFS has a working f(data)sync() implementation.

“” So what seems to be happening is that ZFS is caching extremely aggressively – way more than UFS, for instance. Given that most of the “I/O” was actually happening straight to and from the ARC cache, this explained the phenomenal transfer rates. Watching iostat again also showed that during the read tests, I was using far less of the SAN bandwidth. While performing “read” iozone tests on UFS, I was nearly maxing out the fabric bandwidth which could have lead to resource starvation issues, both for the host running the tests and for other hosts sharing the SAN. Using ZFS, the bandwidth dropped down to 50MB/s or so at peak, and much lower for the remainder of the tests – presumably due to the caching and prefetch behaviour of ZFS.

The upshot of all this is that it seems ZFS is just as “safe” as UFS – any application that really wants to be sure data has been written and committed to stable storage can still issue a fsync(), otherwise writes to the disks still happen at around the same rate as UFS, it just gives control back to the application immediately and lets the ZFS IO scheduler worry about actually emptying the caches. Assuming I lost power half-way through these writes, the filesystem would still be guaranteed to be safe – but I may still lose data (data loss as opposed to data corruption, which ZFS is designed to protect against end-to-end). It can also give you a staggering performance advantage and conserve IO bandwidth, just so long as you’re careful you don’t get misled into believing that your storage is faster than it actually is and have enough memory to handle the caching!

For other ZFS benchmarks see also http://milek.blogspot.com/2007/04/hw-raid-vs-zfs-software-raid-part-iii.html

According to my blog statistics, I have on average 15 visitors a day who found my blog using the following Google search terms:

ZFS Oracle RAC

I don’t pretend to know how that search stuff works (in classic Unfrozen Caveman Lawyer style) because I …

Interesting comment to ZFS performance at ZFS and caching for performance: Watching iostat again also showed that during the read tests, I was using far less of the SAN bandwidth. While performing “read” iozone tests on UFS, I was nearly maxing ou…

I like to say every ten to fifteen years there is a fundamental architectural change in filesystems. The last big shift, in the mid-1990s, was the emergence of extent-based filesystems, such as IBM JFS, Veritas VxFS, SGI XFS, and Microsoft NTFS, along with many others which followed.