Email is the largest personal database for most people. Easy to search, my gigabyte of email contains contacts, documents, notes and the record of many relationships. I back it up both locally and remotely.

But how do I know it isn’t corrupted?

FastMail’s email data protection
FastMail is, I think [guys, how about a “what is FastMail” paragraph?], an open source email system an email hosting provider. In a recent blog post someone there talks about how FastMail protects user data from data corruption:

. . . we ensure that as soon as an email is delivered to a mailbox, a SHA-1 checksum of that email is generated and stored in the email index.

When the email is replicated, the email content and the checksum are sent separately. We then generate the checksum on the replicated email content and ensure that it matches the original checksum to see that the email was replicated correctly.

We also repeat this procedure when the email is backed up, ensuring that the backup of the email is correct.

We also run a regular check process that takes blocks of emails and recomputes their checksum to see it matches what is in the index. If there’s any issues, we’re alerted and can find which of the master, replica or backup email are correct and can correct the problem.

What do other email systems do?
To my untutored eye, this seems like comprehensive protection against data corruption.

Two questions:

  • Is it?
  • What do other email systems do?

Gmail presumably relies upon the triple redundancy of GFS to ensure data integrity. What do Exchange and Sendmail do? Are any of them demonstrably better?

The StorageMojo take
Email is looming ever-larger as a personal information management system. As volume and attachment size continue to grow, multi-gigabyte mailboxes will become the norm, if they aren’t already.

Are the data protection measures in email systems up to the task?

Update: Alert reader – more alert than me, anyway – Nathan, found the FastMail “About” page. They are another hosted email provider. I updated the post to reflect that. Thanks, Nathan!

Comments invited, of course.