Comments on: Our changing file workloads

By: Build a 135TB array for $7,384 | ZDNet

Build a 135TB array for $7,384 | ZDNet — Wed, 20 Jul 2011 14:29:58 +0000

[...] peoples of the world gift by open-sourcing their storage pod design. Most business files are opened only a few times, so why put them on the most costly storage you can [...]

By: Dell TechCenter

Dell TechCenter — Fri, 29 May 2009 19:39:51 +0000

How Well Do You Know Your Data?…

If you are sensing a pattern to some of my blogs, you are probably correct. One of the reasons I write…

By: HPCC

HPCC — Wed, 15 Apr 2009 13:31:16 +0000

How Well Do You Know Your Data?…

If you are sensing a pattern to some of my blogs, you are probably correct. One of the reasons I write…

By: Bobby Moulton

Bobby Moulton — Thu, 16 Oct 2008 16:23:24 +0000

Good article Robin –

This data is important. This is the kind of information our company – Seven10 – has used to promote a data management paradigm change. And while I think you are almost there in your view of how data should be managed, I still think you are putting too much emphasis on migration.

Data placement across tiers with retention policies is the key.

Take a look at our EAS product – tiered storage offering support for disk, cas, and tape (as an archive) and I think you will see that there is a better approach to managing the growth of fixed content.

We see an end of days for back-up as an archive – it simply is too costly to manage.

I look forward to hearing from you.

Bobby Moulton
President
Seven10
bmoulton@seventenstorage.com

By: Brian

Brian — Wed, 01 Oct 2008 22:43:55 +0000

Why not virtualize the NAS? Utilize the workflow, data to a degree that can use the all the storage capacity, resources available. Product like Acopia for taking advantage of the NetApp, EMC, and Network Attached Storage.

That would be a better way.

By: Cinetica Blog » Troppo BELLO per essere VERO !!!

Cinetica Blog » Troppo BELLO per essere VERO !!! — Mon, 22 Sep 2008 14:17:50 +0000

[...] riduce i costi dello storage: uno studio pubblicato da poco e ripreso anche su StorageMojo dimostra che l’80% dei dati che avete nei vostri storage è inattivo!!!! con Compellent [...]

By: Martin G

Martin G — Tue, 16 Sep 2008 15:11:19 +0000

The 76% of files only accessed by a single user, makes you wonder why we provide shared file-systems for most users. There’s got to be a better way of doing this.

By: Steve Jones

Steve Jones — Fri, 12 Sep 2008 12:37:14 +0000

There’s no major surprises here. As the available storage capacity for a given amount of money increases on an exponential curve, then the tendency will be to store more and more low-access, low-value data. The costs of managing this storage (by the individual creating it) do not so decline – or at least without automation.
Measured another way, it’s just as well that the access frequency of storage is on the decline. The rate of increase of I/O throughput bandwidth and IOPs are on much lower curves (especially the latter). Quite simply, if the access density per GB of today’s data looked anything like it did a decade ago then the storage systems would be unusable.

By: Jered Floyd

Jered Floyd — Thu, 11 Sep 2008 22:26:51 +0000

Robin,

I saw this paper, and while I appreciate the data, I find it unfruitful for drawing conclusions. The particular customer involved appears to make extraordinarily little use of their storage. Their needs could likely be met with a 386 running Linux running with drives from the same era.

I’d really like to see aggregate data from a wide variety of customers, and I hope the authors will go on to do so. I’ll say for a fact that zero of the conclusions you can draw from that report apply to our development file server, and I bet that’s likely true for many others!

Regards,
Jered Floyd
CTO, Permabit Technology Corp.

By: Ryan Malayter

Ryan Malayter — Thu, 11 Sep 2008 13:03:00 +0000

Funny, we just did a similar study last year internally on all of our storage, including DB and mail servers. The results were startingly similar. 2:1 read:write ratio, 12 KB average IO size, and very few files are ever touched beyond 30 days of creation.

That said, I do not think a study of two NAS installations is statistacilly significant. WHat is really needed is a long-term study of hundreds of disparate organizations. Something only a vendor could do, I suppose, with remote monitoring and opt-in from customers.

Also, the fact that such studies are “blind” to access patterns inside datbases and virtual machine files is also quite a flaw.

By: Ernst Lopes Cardozo

Ernst Lopes Cardozo — Wed, 10 Sep 2008 21:21:08 +0000

To me these observation suggest a different policy: write new files to backup storage first, then copy to primary storage when requested. Primary storage becomes a cache that does not need backup. Effectively, the backup process is spread over the day. Motivation: all new files need to be backuped, but only a fraction will be accessed. Backup is to disk and needs to be replicated to disk or tape at a DR site. The capacity of primary storage becomes flexible, there will be no ‘disk full’ conditions, only sub-optimal performance when the primary storage (the cache) is too small.

By: Matthew Emery

Matthew Emery — Wed, 10 Sep 2008 13:40:44 +0000

Thanks for this, I have found the study interesting.

One recommendation though has been left out from the end of section 5

“While access to file metadata should
be fast, this indicates much file data can be compressed,
de-duplicated, or placed on low power storage, improving
utilization and power consumption, without significantly
impacting performance. In addition, our observation
that file re-accesses are temporally correlated (see
Section 4.5) means there are opportunities for intelligent
migration scheduling decisions.”

… or data can be deleted.

By: Steve Todd

Steve Todd — Wed, 10 Sep 2008 13:38:47 +0000

Robin,
Thanks for sharing the pointer to the paper. I find it interesting from an industry and product perspective. The SNIA XAM initiative (www.snia.org/forums/xam) is an industry effort that deals with exactly this issue (managing fixed or unchanging content). At EMC we’re also seeing strong demand for products that silently migrate untouched data, whether it’s performed at the server, in the network, or in the filer.
Steve

By: Open Systems Storage Guy

Open Systems Storage Guy — Wed, 10 Sep 2008 12:35:04 +0000

Interesting data- I wish someone would do this for Oracle, SQL, Exchange, and some other heavy hitting DB applications…