Read-after-write validation is occasionally offered as an integrity option but is seldom used – again, for performance reasons. And it can’t correct data trashed by a wild write: it can only ensure that the correct location is also written.

The virtues you claim for ZFS are surpassed in WAFL, which collects updates in its NVRAM and then coalesces them to write back to disk as a single request – and it doesn’t have to do so as often, since the NVRAM stabilizes the data without requiring as frequent ’syncs’ to disk. Your closing comments above are so vague (and apparently confused) that I’ve really got to wonder whether you know what you’re talking about in this area – but I’m always willing to be educated if I’m missing something, so if you’d like to be more specific feel free to continue.

- bill

ZFS is my current favorite not due to write anywhere, which I feel is a very risky behavior for a file system, but because of the low transaction overhead. Its one of the things they have gotten right. I personally file systems should behave very predicatibly, the less variance, better chance for reliability. One major advantage is write allocation coalescingand meta data written to anywhere on the disk, but I think meta data writes should be under a little closer control for workload, capacity, performance planning, consistent workaround for undocumented features and repeatable operation. I tend to treat application servers more like big dedicated embedded systems. Especially for synchronous clustering and remote data synchronization. Treating a server and its systems like a general purpose computer, well, then you may as well put your apps on a desktop. I’m not saying that WAFL is junk, it just doesn’t scale linearly as I like. I have a preference for very linear scaling servers. Its a personal perference – no I don’t have a real favorite for linear scaling either ( I get asked that often after being on a soapbox).

These are indeed data integrity deficiencies in the block interface. A lost write cannot be detected by mechanisms such as DIF, nor can the fact that a wild write did not update the block it was supposed to – but at least when the block that a wild write updated in error is subsequently read the DIF mechanism should catch that.

I’m also curious about precisely what worries you about WAFL – especially since ZFS (which you seem to prefer) also uses a very similar ‘write-anywhere’ approach.

- bill

Back onto T10 DIF…. The DIF is a block data integrity scheme targeting the transfered between the CPU/system memory and the storage medium. DIF wasn’t intended to enforce or detect data integrity outside of the block interface. It also works well for volume to volume block copies. Its not intended to detect missing blocks, ie the file system forgot to request or write data and the app reads stale data from the buffer. File based data integrity checks need to occur at another layer in the system. This case one size does not fit all.

ZFS is my new favorite technology of the year. More so than WAFL, no insult to the WAFL advocates. Working with different remote data synchonization technologies since the mid ’80’s any “anywhere” file system sends up redflags and flashbacks of long torch(turous) nights. When the sun is shining, everyone is your friend, but once the storm hits… with marginal interconnects between data stores, live systems that cannot go down, hard SLAs and long resync periods, life quickly becomes interesting. Its better than nothing at all.

As I’ve observed before, calling ZFS “the only game in town” in this respect really isn’t fair to WAFL. WAFL provides the same level of end-to-end validation that ZFS does: the main difference is that by virtue of running only on a file server rather than as a local file system WAFL can’t provide end-to-end protection all the way to client RAM – but it does provide end-to-end protection from server RAM to disk and back again, and the normal network checksums provide protection from client RAM to server RAM and back again, so the validation is scarcely weaker than local ZFS protection.

- bill

The 10^7 number underscores the urgency of end-to-end data protection. So far, ZFS seems to be the only game in town.

The specific causes of the data corruption at CERN don’t seem that interesting given that they seem to be spread out over the entire I/O system. The data chain is so long and subject to so many kinds of error that errors are pretty much baked into the system. If you weren’t CERN would you even figure out the WD/3ware problem?

This is on top of the fact that RAID 5 no longer protects our large drives from URE’s. The whole data protection model is up for a serious re-think.

xfer, thanks for the T10-DIF reference. Hadn’t heard of that, even though I agree with Bill’s comment. Half a loaf is better than none.

Rob, good catch on the CERN using the PDF. Honestly though I save a lot of interesting web pages as PDF’s myself and use an iPhoto like PDF viewer to cruise through them.

Robin

Both ZFS and its older progenitor WAFL do provide this form of separate checksuming (truly end-to-end in ZFS’s case, only end-to-end-within-server in WAFL’s case). It’s interesting that the CERN study reportedly stated that this form of protection doubles the disk write overhead, since reasonable implementations can have far lower average impact than that.

- bill

Your right, the errors experienced at CERN are not unusual and many times are not repeatable. I have 20TB and another 10Tb array, I see bad data juju once a month.

The 3 * 10^7 BER, is not unexpected. I’m glad to see someone has finally blogged and brought the issue out of the closet. Its also good to see an end user has finally come out and disclosed the data errors they’re actually experiencing.

There is a lot of work done in the telco and satcom to model and correct different types of errors now experienced in the data storage arena.

Sometimes in the open systems world, unaddressed critical issues like data integrity is treated like the “crazy aunt locked in the basement” — no one wants to talk about. In the past, the open systems computer industry typically responds to this particular issue by passing the buck… Drive mfgs say “its the raid controller’s problem”, the raid controller mfgs push it off to the host computer systems, who in turn, push the problem off to the OS, where it either gets pushed back down the food chain or passed up to the applications to deal with. No one in the industry really wants to take ownership of this issue, its a complex problem, it requires participation from several diffent compnent vendors and requires significant changes to products lines. In other words…. its expensive to fix.

Are there any heroes in this story of corruption and buck passing intrigue ? That is besides Robin and his blogs.

In 2003, the T10 group took on an effort to standardize an “end to end” error detection scheme. The called it the T10 Data Integrity Field standard, commonly referred to as T10-DIF. It doesn’t correct the data corruptions, but at least they can be detected, until the data capacities get larger. T10-DIF uses an additional data field (DIF) generated by the host and accompanies the data from the host application to the disk and is returned to the host during data read operations. If any discrepancies occur between the DIF and the data during read operations, the host should able to detect it. The spec is still a work in progress in some areas. Although not complete, T10-DIF still provides significant value to storage data integrity issues.

The good new about T10-DIF ? – almost all 2GB fibre channel and Infini-band, all 4GB fibre channel HBAs and some midrange disk arrays support it. Also, there is a newly formed Data Integrity Initiative (Oracle/Emulex/LSI/Seagate) that will try to iron out the rest of the technical issues and try to insure all their products will interoperate at some level.

The bad news of T10-DIF… not yet supported by SAS, iSCSI and not supported by any low cost disk arrays. Also, their’s no independent authority validating that T10-DIF operates properly across products and platforms and there is no specification classifying different levels of DIF support ( a marketing jackbox). Most of these issues are temporary and should be worked out in the next 24 months.

We should see T10-DIF rolling out in most midrange arrays in the next 12 to 24 months. When is it going to reach the lower end of the markets, when the three 800 pound gorillas demand it. Is it going to help your 1 TB drive in your PC ? Sorry, you’ll have to wait another 10 years.

The way ZFS uses checksums on all blocks really helps with the integrity. In generally reading over all the ZFS documentation, they’ve thought hard about integrity issue and seem to have covered a lot of bases others are missing.

One bit of irony with the link you have to CERN above. Where was the world wide web created? At CERN as an “information management system” exactly for distributing information like this (http://en.wikipedia.org/wiki/World_Wide_Web#History). So what format is the paper at CERN in? PDF of course. *sigh*