I’m confused… ZFS (SUN’s) doesn’t have a built in file replication engine. Only file system to disk integrity.

If your underlying execution platform is having data integrity errors, like the one you have described (it must be linux), the integrity of the application must be called into question. File systems, like Ext3 in lnux, has some very famous bugs including throwing data away without informing the application. While there are programmatic techniques to detect these errors, mos developers do not apply them The linux swap file system does NO end to end integrity checking. The fact that the mail system’s program modules are executing on platform with a poor level of data integrity, brings the integrity of the executing application environment into question.

Its great to have data integrity checking, and most people agree not to rely on a single point for any system feature. However, most developers ignore the application execution space when it comes to integrity validation. It wouldn’t be the first time I’ve seen multiple copies of files corrupted by the application that wrote them. Most data recover actions are not due to hardware failures, but due to operator error and application defects - yes, both copies.

Before being lulled into a false sense of security in your application, realize there are other issues much more significant, in terms of “true” integrity, other than some hash value with the data.

Data integrity is the responsibility of every layer. The underlying storage (like ZFS indeed) can only guarantee the integrity of what it’s given. If your replication engine makes a mistake, or your email software contains a bug somewhere which causes it to go on an index hosing spree, then your filesystem can present a perfectly good copy of the hosed files. It doesn’t understand the format enough to know that they’re wrong.

This is the same reason that we don’t use block level replication for our filesystem, because we have seen filesystem bugs before, and we don’t want a filesystem bug to lose both copies. Better to have the replication engine bail out saying the sha1 doesn’t match when it tries to copy the message.

So yeah, it’s great to have a filesystem that does integrity checking for you, but concepts like “defence in depth” from the security world map very nicely to the data integrity world as well. Don’t trust everything to one layer.

Sendmail (and Postfix as well) is strictly an MTA (Mail Transfer Agent), that is, it really only moves mail from one host to another. Once at the right host, it will hand the mail to a backend to deliver the mail to the user’s inbox. So, the question then becomes, how does the backend handle it?