Comments on: ZFS: Threat or Menace? Pt. I http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/ Data storage info & analysis Fri, 21 Nov 2008 20:09:56 +0000 http://wordpress.org/?v=2.6.3 By: Macwintel » ZFS in new Leopard build http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-12101 Macwintel » ZFS in new Leopard build Tue, 19 Dec 2006 10:29:10 +0000 http://storagemojo.com/?p=108#comment-12101 [...] Also for you ubergeeks (is there any other kind?), who like me, might have noticed a similarity between google’s famed GFS and ZFS there is a discussion of it here [...] [...] Also for you ubergeeks (is there any other kind?), who like me, might have noticed a similarity between google’s famed GFS and ZFS there is a discussion of it here [...]

]]> By: StorageMojo » ZFS On Leopard: How Cool Is That? http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-2381 StorageMojo » ZFS On Leopard: How Cool Is That? Thu, 29 Jun 2006 12:15:24 +0000 http://storagemojo.com/?p=108#comment-2381 [...] But That’s Not All! For in-depth treatment of ZFS see here and here. Includes links to more technical info and benchmarks. [...] [...] But That’s Not All! For in-depth treatment of ZFS see here and here. Includes links to more technical info and benchmarks. [...]

]]> By: Robin Harris http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-528 Robin Harris Tue, 13 Jun 2006 12:53:45 +0000 http://storagemojo.com/?p=108#comment-528 No cluster support today. I'm told it is high on the list though, with implementation of a global namespace. No cluster support today. I’m told it is high on the list though, with implementation of a global namespace.

]]> By: Dmitry Kulagin http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-522 Dmitry Kulagin Tue, 13 Jun 2006 05:41:40 +0000 http://storagemojo.com/?p=108#comment-522 Interesting article. But there is a question, as soon as you compared ZFS with Google's GFS: Does ZFS support distribution over a cluster? Reading the docs, I have no indications of that. Effective and fault tolerant clustering is a major feature of GFS. Interesting article. But there is a question, as soon as you compared ZFS with Google’s GFS: Does ZFS support distribution over a cluster? Reading the docs, I have no indications of that. Effective and fault tolerant clustering is a major feature of GFS.

]]> By: Jeff Bonwick http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-108 Jeff Bonwick Sun, 28 May 2006 08:15:17 +0000 http://storagemojo.com/?p=108#comment-108 Sure. I've explained this to the best of my ability <a href="http://blogs.sun.com/roller/page/bonwick?entry=zfs_end_to_end_data" rel="nofollow">here</a>. If you find this unsatisfactory or unclear in any way, please let me know so I can improve it. Sure. I’ve explained this to the best of my ability here. If you find this unsatisfactory or unclear in any way, please let me know so I can improve it.

]]> By: Robert Pearson http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-107 Robert Pearson Sun, 28 May 2006 07:51:18 +0000 http://storagemojo.com/?p=108#comment-107 Are you referencing Jeff Bonwick's post of--- "Friday December 09, 2005 ZFS End-to-End Data Integrity " in your reply? If so, this issue is at the core of developing the definitive Search, Find, and Obtain (SFO) "killer app" (function) that is the cornerstone of Peter Morville's Findability. [My words, not Peter's.] Findability is the key element in determining the User Experience (UX) defined by Peter in "Ambient Findability". True, everything above the SFO is way above the file system level. However, a big stumbling block for developing a working SFO has been Information Integrity (II [my acronym]). In my concept of End-to-end Information on Demand (E2EIoD) SFO is mission critical and depends totally on Information Integrity (II). Particularly with regard to returning useful Information in a manner and time frame that generates a pleasing User Experience (UX). The "Operational Definition" of "pleasing User Experience (UX)" is, "Did you get what you asked for"? Was it fast enough to be of use? Other requirements to this definition are: 1) Did you get "value add" Information that was helpful? 2) If the Information was not available did the reply contain meaningful information as to when it would be available? 3) If the "exact" Information was not available was any meaningful "fuzzy search' or "sounds like" Information returned? I have been working for years on ways to overcome the very problems addressed by the ZFS developers. EMC has been particularly obtuse when I raised these issues with their equipment. I raised these issues with other Storage vendors as well but I made the mistake of thinking EMC understood the problem. They talked like they did. Turns out it was EMC"Show-biz" talk. Are you referencing Jeff Bonwick’s post of—

“Friday December 09, 2005

ZFS End-to-End Data Integrity ”

in your reply?

If so, this issue is at the core of developing the definitive Search, Find, and Obtain (SFO) “killer app” (function) that is the cornerstone of Peter Morville’s Findability. [My words, not Peter's.]
Findability is the key element in determining the User Experience (UX) defined by Peter in “Ambient Findability”. True, everything above the SFO is way above the file system level. However, a big stumbling block for developing a working SFO has been Information Integrity (II [my acronym]). In my concept of End-to-end Information on Demand (E2EIoD) SFO is mission critical and depends totally on Information Integrity (II). Particularly with regard to returning useful Information in a manner and time frame that generates a pleasing User Experience (UX).
The “Operational Definition” of “pleasing User Experience (UX)” is, “Did you get what you asked for”? Was it fast enough to be of use? Other requirements to this definition are:
1) Did you get “value add” Information that was helpful?
2) If the Information was not available did the reply contain meaningful information as to when it would be available?
3) If the “exact” Information was not available was any meaningful “fuzzy search’ or “sounds like” Information returned?
I have been working for years on ways to overcome the very problems addressed by the ZFS developers. EMC has been particularly obtuse when I raised these issues with their equipment. I raised these issues with other Storage vendors as well but I made the mistake of thinking EMC understood the problem. They talked like they did. Turns out it was EMC”Show-biz” talk.

]]> By: Robin Harris http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-102 Robin Harris Sun, 28 May 2006 02:16:05 +0000 http://storagemojo.com/?p=108#comment-102 Good question, Jonathan. The basic issue is if the checksum is in the packet with the data, all it tells you is that the data is correct. But it may not be the data you want. <a href=http://blogs.sun.com/roller/page/bonwick?catname=%2FZFS rel="nofollow">Jeff Bonwick</a>, the ZFS architect cites phantom writes and misdirected reads (corruption in a pointer file?) as reasons that the wrong block might be returned. Since ZFS keeps the checksum seperate from the data, it knows if the wrong data has been returned. Since the checksum is calculated while the data is in the host's RAM, I suppose it could be corrupt, but if that were a regular occurence the host would crash pretty quickly. How often does this happen? I have no idea. Since this is a statistical universe, I'd assume that as data stores continue to grow the incidence will cross a cognitive threshold and boom! all the CIOs will get excited. From a marketing perspective, no storage vendor has gone broke scaring the hell out of customers. Someone is going to pick this up and run with it. Sun? Not likely -- they've got a lot of Storagetek to sell. More likely some hungry mid-tier vendor will start the ball rolling. Hey, ZFS guys, any comment? Good question, Jonathan. The basic issue is if the checksum is in the packet with the data, all it tells you is that the data is correct. But it may not be the data you want. Jeff Bonwick, the ZFS architect cites phantom writes and misdirected reads (corruption in a pointer file?) as reasons that the wrong block might be returned. Since ZFS keeps the checksum seperate from the data, it knows if the wrong data has been returned.

Since the checksum is calculated while the data is in the host’s RAM, I suppose it could be corrupt, but if that were a regular occurence the host would crash pretty quickly.

How often does this happen? I have no idea. Since this is a statistical universe, I’d assume that as data stores continue to grow the incidence will cross a cognitive threshold and boom! all the CIOs will get excited.

From a marketing perspective, no storage vendor has gone broke scaring the hell out of customers. Someone is going to pick this up and run with it. Sun? Not likely — they’ve got a lot of Storagetek to sell. More likely some hungry mid-tier vendor will start the ball rolling.

Hey, ZFS guys, any comment?

]]> By: Jonathan http://storagemojo.com/2006/05/26/zfs-threat-or-menace-pt-i/#comment-99 Jonathan Sat, 27 May 2006 22:42:04 +0000 http://storagemojo.com/?p=108#comment-99 Great article - maybe im missing something fundamental, but when you say, "Any checksum stored with the data it is supporting can only tell you that this data is uncorrupted. It could be the wrong data" How is this different than ZFS? Is the checksum calculated at a different time? If the checksum is calculated on corrupt data, then you'd have the same problem of a valid checksme with invalid data. Garbage In, Garbage out seems to continue to apply either way. Can you give an example of a failture that ZFS would recover from, that a traditional system would not? Great article - maybe im missing something fundamental, but when you say,

“Any checksum stored with the data it is supporting can only tell you that this data is uncorrupted. It could be the wrong data”

How is this different than ZFS? Is the checksum calculated at a different time? If the checksum is calculated on corrupt data, then you’d have the same problem of a valid checksme with invalid data.

Garbage In, Garbage out seems to continue to apply either way. Can you give an example of a failture that ZFS would recover from, that a traditional system would not?

]]>