FastMail fights data corruption
Email is the largest personal database for most people. Easy to search, my gigabyte of email contains contacts, documents, notes and the record of many relationships. I back it up both locally and remotely.
But how do I know it isn’t corrupted?
FastMail’s email data protection
FastMail is, I think [guys, how about a "what is FastMail" paragraph?], an open source email system an email hosting provider. In a recent blog post someone there talks about how FastMail protects user data from data corruption:
. . . we ensure that as soon as an email is delivered to a mailbox, a SHA-1 checksum of that email is generated and stored in the email index.
When the email is replicated, the email content and the checksum are sent separately. We then generate the checksum on the replicated email content and ensure that it matches the original checksum to see that the email was replicated correctly.
We also repeat this procedure when the email is backed up, ensuring that the backup of the email is correct.
We also run a regular check process that takes blocks of emails and recomputes their checksum to see it matches what is in the index. If there’s any issues, we’re alerted and can find which of the master, replica or backup email are correct and can correct the problem.
What do other email systems do?
To my untutored eye, this seems like comprehensive protection against data corruption.
Two questions:
- Is it?
- What do other email systems do?
Gmail presumably relies upon the triple redundancy of GFS to ensure data integrity. What do Exchange and Sendmail do? Are any of them demonstrably better?
The StorageMojo take
Email is looming ever-larger as a personal information management system. As volume and attachment size continue to grow, multi-gigabyte mailboxes will become the norm, if they aren’t already.
Are the data protection measures in email systems up to the task?
Update: Alert reader - more alert than me, anyway - Nathan, found the FastMail “About” page. They are another hosted email provider. I updated the post to reflect that. Thanks, Nathan!
Comments invited, of course.
on December 19th, 2007 at 7:55 am
Seems like there is an “About” page (http://www.fastmail.fm/pages/fastmail/docs/about.html). They just look like another hosted mail provider. They give you 10M of storage space for free and IMAP access. Whoo.
on December 19th, 2007 at 1:02 pm
Hi Robin,
Sendmail (and Postfix as well) is strictly an MTA (Mail Transfer Agent), that is, it really only moves mail from one host to another. Once at the right host, it will hand the mail to a backend to deliver the mail to the user’s inbox. So, the question then becomes, how does the backend handle it?
on December 19th, 2007 at 2:50 pm
Shouldn’t data integrity be the responsibility of the underlying storage (like ZFS) and not the application (messaging in this case) that runs on top?
on December 19th, 2007 at 5:46 pm
Disclaimer: I’m the FastMail developer who wrote most of our integrity checking systems.
Data integrity is the responsibility of every layer. The underlying storage (like ZFS indeed) can only guarantee the integrity of what it’s given. If your replication engine makes a mistake, or your email software contains a bug somewhere which causes it to go on an index hosing spree, then your filesystem can present a perfectly good copy of the hosed files. It doesn’t understand the format enough to know that they’re wrong.
This is the same reason that we don’t use block level replication for our filesystem, because we have seen filesystem bugs before, and we don’t want a filesystem bug to lose both copies. Better to have the replication engine bail out saying the sha1 doesn’t match when it tries to copy the message.
So yeah, it’s great to have a filesystem that does integrity checking for you, but concepts like “defence in depth” from the security world map very nicely to the data integrity world as well. Don’t trust everything to one layer.
on December 20th, 2007 at 3:30 pm
Hi Bron,
I’m confused… ZFS (SUN’s) doesn’t have a built in file replication engine. Only file system to disk integrity.
If your underlying execution platform is having data integrity errors, like the one you have described (it must be linux), the integrity of the application must be called into question. File systems, like Ext3 in lnux, has some very famous bugs including throwing data away without informing the application. While there are programmatic techniques to detect these errors, mos developers do not apply them The linux swap file system does NO end to end integrity checking. The fact that the mail system’s program modules are executing on platform with a poor level of data integrity, brings the integrity of the executing application environment into question.
Its great to have data integrity checking, and most people agree not to rely on a single point for any system feature. However, most developers ignore the application execution space when it comes to integrity validation. It wouldn’t be the first time I’ve seen multiple copies of files corrupted by the application that wrote them. Most data recover actions are not due to hardware failures, but due to operator error and application defects - yes, both copies.
Before being lulled into a false sense of security in your application, realize there are other issues much more significant, in terms of “true” integrity, other than some hash value with the data.
on December 29th, 2007 at 4:51 am
I know first hand of a rather large email system (3GB free account, 4+ million active users) which is running on ZFS.