Comments on: Storage for version control

By: cliff

cliff — Wed, 27 Jan 2010 11:58:23 +0000

robin-
First time I have made a comment. This is a great blog and love what you write. I work as a senior sales engineer for Rackspace (NYSE:RAX). These comments are my own. It sounds like Chris might be a customer. I would be happy to work with him directly if he wants and you can share my email with him.

comment for the post, feel free to edit for length-
there are really two ways to this answer this. You see that kind of through the other comments. there is the hardware based approach (netapp) or the web scale approach (the github example is good). Notice the difference in level of expertise required to admin the solution. there are tradeoffs along a couple of different axis. cost vs complexity. complexity and ease of use.

The iops here are really small, so the cheapest option will be to use commodity server running NFS with a bunch of TB sized drives, then shard the users so that X% go to one server after another. I assume the service has enough users that this can work.

cliff

By: Chris

Chris — Mon, 25 Jan 2010 16:49:21 +0000

Thanks everyone for such great advice and feedback. There are a lot of options and it helps a lot. Now we have to choose a good path

Once we figure it out I will post with an update.

Chris

By: Jeff Darcy

Jeff Darcy — Sat, 23 Jan 2010 22:01:25 +0000

There’s more than “a little bit” of complexity involved in clustering your NAS heads if you want them to present a single namespace. Fortunately, at 500 or even 800 IOPS you can serve that very easily with a single pair of servers in an active/standby arrangement. For metadata-heavy workloads *any* cluster/parallel filesystem (I’ve worked on several) will struggle to keep up with a single-active-server setup for workloads where the latter suffices. If it were me, I’d just go buy a couple of Joe L’s boxes and be done with it.

By: Tim

Tim — Sat, 23 Jan 2010 18:48:24 +0000

how about getting a drobo?

http://drobo.com/products/droboelite.php

By: Jeffrey W. Baker

Jeffrey W. Baker — Fri, 22 Jan 2010 22:21:14 +0000

It sounds to me like you need to forget your storage subsystem and switch to a better VCS. Instead of trying to use modern storage to host the crusty old Subversion, why not switch to a modern, distributed, and therefore self-parallelizing, self-backing-up VCS like Mercurial, Git, or Bzr?

By: Wes Felter

Wes Felter — Thu, 21 Jan 2010 16:38:52 +0000

Chris, instead of DRBD I would suggest two servers with a dual-ported SAS JBOD for HA.

By: Olivier

Olivier — Thu, 21 Jan 2010 14:21:50 +0000

What is the ratio of the ‘dead’ files (files that are not frequently used) versus the ‘live’ files (files that are often requested)? The SUN 7000 series have SSD’s that are used as read cache, and write SSD’s to take all IOPS as buffer in front of the SATA disks. If only 500G of your data is used frequently, you will be very happy with a SUN 7000 system with 500G SSD read cache, and some SSD’s as write cache to take the IOPS. The SUN 7000 has built in dedup as well. We use some 7410 clusters (which are overkill for your requirements), and the NFS performance is quite impressive for the price. We’ve tested with Netapp but their best price was still far more expensive.

By: Martin Scholl

Martin Scholl — Thu, 21 Jan 2010 08:09:54 +0000

Hello Chris,

for failover and replication, DRBD is used at many high-profile sites with great success. I can definitely recommend using DRBD for replication. Used together with heartbeat is DRBD a great foundation for having highly-available shared-nothing block device.
Regarding the hardware used: we are about to launch a storage product especially for *aaS/cloud-providers.
If you would like to learn more about it, please contact me for more: ms [at] mystoragepod.com

By: Karl Katzke

Karl Katzke — Thu, 21 Jan 2010 05:23:04 +0000

Hmn, that’s REALLY not that many IOPS. I would look more closely at the Sun offerings — the Unified Storage Systems mentioned above. Google around a bit for ‘fishworks’ — you’ll get an ear/eye/mind-full. The NetApp system you have is massive overkill for what you’re doing with it.

By: Chris

Chris — Thu, 21 Jan 2010 05:01:00 +0000

Thanks so much for all of the detailed responses. I sent the question in to Robin late at night after researching some options. Let me clarify some of the items:

* The VCS is SVN
* The NetApp was a 2040
* 500 iops is the usual, but we experience peaks during a given period of 800 for writes and 500 for reads
* 3T is our current usage, but the NetApp would give us around 5T usable.
* We did some tests with dedupe on netapp. Interesting thing is that vmdk was about 50% savings, but directly on disk it was almost no savings.
* We used to use GFS, which caused all sorts of problems.

One advantage we have is sharding. Our app is already setup for this, so we are considering a completely “shared nothing” environment with local disks. Getting away from a shared file system would be ideal. While NFS works well, it will eventually be a bottleneck (I think).

If we go with local disks, we just need to figure out the failover/replication scenario. Any suggestions there would be great. So far ZFS/Nextenta looks great for its flexible management and snapshots.

Btw, I am not an architect or engineer, just trying to personally explore high level options for the biz moving forward.

Thanks again.

By: Anonymous Sysadmin

Anonymous Sysadmin — Thu, 21 Jan 2010 02:22:59 +0000

Ah, prices from an unnamed major North American managed hosting company who is based in Texas. I know them well.

At the type of scale that your reader describes, there are a variety of viable options, particularly with the fairly low 500 IOPs numbers stated. That could very easily be handled by a Dell MD3000 with 15 x 600GB 15k RPM drives connected to a pair of front-end hosts to act as redundant NFS heads connected via SAS to redundant controllers. At RAID10, you’d wind up in the neighborhood of 4TB of usable space and at least a couple of thousand IOPs. Admittedly, that’s a roll your own solution that does have a little bit of complexity involved in the clustering of the NFS heads.

The next step from there would be something like a smaller Isilon cluster. You get NFS out of the box and scale-out capability for both IOPs and storage. Isilon has gotten much more aggressive on pricing in recent months as well, so the price point can be very competitive.

That’s my $0.02 anyway.

By: Jonathan Ragan-Kelley

Jonathan Ragan-Kelley — Thu, 21 Jan 2010 01:36:08 +0000

Hosted version control is an embarrassingly horizontal problem. Assuming that no one repository is represents a major portion of their load, simply partitioning the repositories horizontally across machines should allow them to use stupid-simple commodity storage in each node.

For a great perspective on the practical issues in scaling an entire large web + version-control service, cf. http://github.com/blog/530-how-we-made-github-fast

By: Uday Mohan

Uday Mohan — Wed, 20 Jan 2010 23:34:04 +0000

ParaScale is really simple to manage. They offer a free 4 TB download that can be installed on your own selected commodity hardware running Linux. It can be installed and configured in a morning and does not need much maintenance.
Also once you have a cloud running you can scale it online without any service disruption. They have Thin Provisioning so you can provision a multi TB file system, and throw in additional storage only when needed.

$8,000/mo over the lifetime of the data can really add up. However, if you repurpose some supermicro or dell boxes, or purchase some new ones, you can build a storage cloud for much less.
The download is available at http://www.parascale.com

Uday

By: Wes Felter

Wes Felter — Wed, 20 Jan 2010 20:09:54 +0000

With such a small amount of data, $/GB may not be the appropriate metric; perhaps the reader should look at $/IOPS. I also question using a cluster when a single controller could easily satisfy the requirements. Heck, if there is locality you might be able to fit this in four hard disks and three SSDs…