Friday hike blogging: Airport Mesa

by Robin Harris on Friday, 29 August, 2014

Often named as one of the most scenic airports in America, the airport sits on a mesa high above the town. There’s a trail that circles the mesa and – more important for me – I can walk to it.

According to my iPhone app, the hike is 7.22 miles with 1788 feet of vertical climbing. The highest point of the trail is only 500 feet higher than my house, but there’s lots of up and down on the trail.

While it isn’t my favorite hike, it is convenient and the 360° views are fabulous. Here’s looking south and seeing, from left to right, one of the twin buttes, Courthouse Rock, Bell Rock, Castle Rock and Cathedral Rock.

Click for large size. Click for large size.


The network crunch: DAS and SAN

by Robin Harris on Thursday, 28 August, 2014

Several years ago an Intel briefer promised me $50 10Gb Ethernet ports. The shocker: prices have dropped little in the last 8 years – well more than a decade in Internet time.

I don’t look back as often as I should. But a note from a ZDNet reader prompted some retrospection and research into network prices.

Consumer adoption is behind the most dramatic price declines in high tech. CPUs, disks, monitors, printers and GigE got cheap when hundreds of millions of people were buying them.

Network effects
Who has a 10Gig home network? Some StorageMojo readers, no doubt, but the rest of the world is sitting on its hands.

This is seen in some remarkably frozen specs. Apple introduced its first GigE system in 2000. 15 years later GigE is still standard on Mac systems – along with, now, 20Gb/s Thunderbolt.

Obviously, Apple’s – and everyone else’s – networking investments have been going into Wi-Fi, not Ethernet. Consumers are willing to pay for the convenience of faster Wi-Fi; not so much for faster Ethernet.

That shows in the pricing. The lowest priced 10Gig PCIe adapter online is about $100. Same with the lowest cost Infiniband adapter.

But here’s a point off the curve: the lowest cost 20Gb Thunderbolt adapter: $72. The adapter does require motherboard support.

But the StorageMojo question is: what impact has the price/performance differential had on datacenter architectures?

Thumb rules
The StorageMojo rule of thumb is that storage and networks are about 80% fungible. If we had infinite bandwidth, zero latency networks, 80% of today’s storage would be in the cloud; if we had no networks 80% of today’s storage would be local.

Another rule of thumb: 80% of the cost of an array is bandwidth, not capacity.

The net effect favors direct-attach storage. PCIe, introduced in 2003, started out at 250MB/s per lane and is inching its way to 2GB/s: an 8x improvement in ≈10 years.

Fiber Channel, over the same interval, has managed a 4x increase in bandwidth to 3.2GB/s, with higher rates projected in the next couple of years. Assuming, of course, that sales support the engineering costs.

10Gig Ethernet has been spec’d for 10 years, but 40 and 100GigE uptake has been slow due to the cost of processing high data rates. 40GigE is seeing more uptake, but is hardly common. Essentially, Ethernet bandwidth has been static for much of the decade.

Network upgrades have always been lumpy. The order-of-magnitude uplifts are more difficult than a CPU speed bump. But we’re seeing something more than uptake indigestion.

The StorageMojo take
Backplane bandwidths have increased much faster than network interconnect speeds. Coupled with PCIe flash storage and the bandwidth’s low cost, this explains much of the rapid growth in direct-attached architectures and shared-nothing clusters – and the growing problems in the SAN market.

The larger problem seems to be that semiconductors aren’t getting faster. We needed GaAs to make gigabit FC work 18 years ago but now it seems even that isn’t fast enough for 100GigE.

The performance crunch is affecting CPUs most visibly. But if CPUs can’t go faster and crunch more data, maybe we don’t need faster networks either.

In any case DAS seems to have long-term legs. The advantages SANS had 16 years ago are evaporating as DAS improves and networks stagnate.

Courteous comments welcome, of course.


Why are array architectures so old?

by Robin Harris on Monday, 25 August, 2014

25 years ago I was working on DEC’s earliest RAID array. When I look at today’s “high-end” arrays, it’s shocking how little architectural change the big iron arrays have embraced.

The industry is ripe for disruption, only part of which is coming from cloud vendors. 21st century problems demand 21st century architectures.

Here’s a list of the obsolete technologies still embraced by most large arrays.

RAID. After it turned out that the governing assumption behind RAID was not accurate – non-correlated failures – and as drive sizes increased, RAID data losses and rebuild times are both too high. Rebuild times have rendered traditional RAID 5 and 6 enterprise arrays functionally obsolete.

Active/active controllers. CPU cycles used to be costly. Now they’re cheap. If performance and availability are paramount, triple controllers – active/active/active – or better are the way to go.

Low density drive enclosures. Why do we need to install drives at a moment’s notice? RAID. Drive failures in a stripe – even with RAID 6 – threaten data integrity due to failure correlation and silent data corruption. Get rid of RAID in favor of modern redundancy techniques and drive replacements can be handled by calm, awake people.

Fiber channel drives. With the rise of much faster SSDs with dual SAS connectors there is little reason for fiber channel drives. Dump ‘em.

Hot spares. Parking an expensive and wearing resource in an expensive slot – and not using it – was a good idea only compared to the alternatives. Getting rid of RAID means that all of your unused capacity can be used for fast replication, not just dedicated drives.

Back up. Enterprise storage should be engineered to make back up unnecessary. We know how to do it, it’s been done, data sizes are exploding and back up windows are imploding.

Custom hardware. Hardware, formerly a differentiator, is now a boat anchor. Low-volume, costly, and little benefit. FPGAs and ASICs make sense for the bleeding edge, but if you aren’t using high-volume hardware for 98% of your kit, you’re last century.

The StorageMojo take
Storage has always been conservative – almost as conservative as the backup and archive. Your data is, after all, why you invest in infrastructure.

But the “new” technologies that have rendered old architectures obsolete are now 10 years old. Times have changed.

Object storage is powering the world’s largest storage systems – and they aren’t using RAID either. That makes high-density drive enclosures – 45-60 drives in 4U – feasible.

Chunk data across enough drives or use SSDs and you don’t need fibre-channel drives. With advanced erasure codes and fast snapshots you can lose backup too – and the expense of the systems it requires. Archiving remains a different problem.

All this and more without custom hardware. Server-hosted storage works as Google, Amazon and Azure have proven.

In five years these old architectures should be on the ash heap of history. But they won’t, because too many buyers buy what makes them comfortable, rather than what maximizes utility at the lowest cost.

But as those folks retire or get eased out, the market change will accelerate.

Courteous comments welcome, of course. What’s your favorite obsolete array technology?


Friday hike blogging: clouds!

by Robin Harris on Saturday, 23 August, 2014

There’s a reason Arizona doesn’t have Daylight Saving Time: we have as much daylight as we can stand. With over 300 sunny days a year – and the sunniest are often the hottest – it’s a relief to see the sun set as the perceived temps drop 10-15°F.

Despite that, summer is my favorite season because of the clouds. We get much of our clouds and rain in summer – low pressure in the warm mountain west sucks up moisture from the gulf and coast – and the clouds dapple the mountains. If we’re lucky we get low clouds drifting among the rocks while the setting sun illuminates them.

This view looks north from the old Jordan Trail across Mormon Canyon to Wilson Mountain.

Click for high-res. Click for high-res.

Yes, it’s Saturday. Took PTO Friday. Picture on Tuesday.


DataGravity launches

by Robin Harris on Wednesday, 20 August, 2014

The co-founder of Equallogic, Paula Long, is heading up the new startup DataGravity. Their system takes advantage of active/active controllers to bring deep storage inspection to small and medium businesses.

What they do
DataGravity brings a new level of information awareness to network storage arrays, enabling important new capabilities within the array at no extra cost:

  • Full-text index of all documents.
  • Automatic scanning for personally identifiable information such as credit card and Social Security numbers.
  • Complete data access logging: who looked at what, when – handy for legal discovery.
  • Multiple views of stored data by size, time, content and more.
  • Even cooler: you can look at who is creating content and who is looking at it. Great for seeing the hidden info flows in a company.

DataGravity’s v.1 supports SMB, NFS, iSCSI and VMware.

How they do it
The key insight is that in any active/active system, all the metadata is replicated across both controllers. Commonly the 2nd data set is trashed once writes are fully committed.

But DataGravity saves that data from the 2nd controller. It’s a low-overhead way to capture the critical data.

Over that they have an easy-to-use GUI for array management and searching, filtering and presenting the data. I’d like to show you the coolest one – info flows between employees – but no graphics online. Update: Suzanne promptly got me what I wanted! Added below. End update.



The StorageMojo take
A storage array that tells you how your data is being used. Pretty cool.

The architecture is innovative and meets a real business need for smaller companies.

Assuming the product works as advertised – and I do – the only real question is how effectively DG can communicate the value proposition. EqualLogic had a simple value prop: storage automation and a complete feature set in one box for a fixed price. Easy to grasp.

DG’s “. . . storage, protection, data governance, search and discovery powered by an enterprise-grade hardware platform and patent-pending software architecture . . . .” seems like a harder sell. Maybe you have to see it to get it.

Similar functionality can be had with layered software at extra cost. How quickly those vendors will cede market share is a question.

Finally, in these social media drenched days, a significant share of communications takes place outside of the traditional storage infrastructure. Tracking tweets and IMs is a hole in the DG model.

But it’s way better than what’s out there and that will be enough for many businesses who’ve experienced the pain of legal discovery or worry about trade secret theft. Ultimately though this seems more like a feature than a market.

How will other network storage vendors react? Similar functionality, should DG gain traction, would seem to be within their capabilities as well.

Courteous comments welcome, of course.


Friday hike blogging: Mormon Canyon

by Robin Harris on Friday, 15 August, 2014

After several days where I couldn’t get out I left at 6am for my favorite hike, the Cibola-Jordan-Soldiers Pass-Brins Mesa loop. Big decision: clockwise or counter-clockwise?

Clockwise is gentler with an uphill bias until reaching the highest point – 5138 feet – after 4.5 miles. Counter-clockwise gets the bulk of the vertical done in the first 1.7 miles.

Which would you prefer?

This view from the Brins Mesa trail on the Mormon Canyon rim is looking southeast down the canyon. The haze is due some small fires on the Mogollon rim and the morning angle of the sun.



Scale Computing: infrastructure made simple

by Robin Harris on Friday, 15 August, 2014

Google and Amazon have armies of PhDs to design, manage and diagnose their scale-out systems. Few small to medium sized businesses do – nor should they – but they should still have the advantages of scale-out infrastructure.

Imagine infrastructure that comes in a box with no costly VMware licenses, great support and good scalability. That is the idea behind Scale Computing. As one customer put it: “It’s not VMware – it’s better.”

Scale-out for the rest of us
Scale Computing is aiming at the midsize company market. The folks with a tiny, overworked IT group or person.

The key to making this work is simplification. Instead of the typical infrastructure of servers, switches and arrays, layered over with hypervisors and VMs, Scale integrates servers, storage and virtualization in one box.

Actually, several clustered boxes for redundancy. But managed as a single server, through a simple GUI.

Making clustering and virtualization easy isn’t easy. Let’s start with the hardware.

Three models – the HC1000, HC2000, HC3000 – that support anywhere from 3 quad core to 6 hex core CPUs, from 96GB to 384GB DRAM and from 6 to 28.8TB raw storage per box.

Software makes it go.

  • SCRIBE is the layer that aggregates all block storage into a single pool that mirrors data chunks for performance, redundancy and fast rebuilds.
  • HyperCore bare metal hypervisor – based on KVM – that accesses the SCRIBE virtual storage devices directly as local storage. VM-level snapshots, thin VMs and VM replication and recovery standard.
  • The HC3 Manager GUI is the single control point for all system resources, OSs, VMs and applications.

The system currently scales to 16 nodes. Supports up to 400 VMs per node.

3 node systems – the minimum – start at an MSRP of $25.5k and range up to $67.5k for the top of the line servers. Additional nodes are 1/3 those prices.

Customers seem to like it too. They advertise a Net Promoter score of 75, giving them higher customer loyalty than the iPhone.

The StorageMojo take
What a difference 5 years makes. Back then Scale was based on GPFS, cool in theory but less workable in practice.

Since then they’ve written SCRIBE and made good use of open source software to build a system that delivers good scale and industry-leading ease of use. At a very reasonable price, too.

Any small to medium sized business should give Scale a close look. It’s 21st century infrastructure from the ground up.

Courteous comments welcome, of course.


Got interoperability testing?

by Robin Harris on Saturday, 9 August, 2014

At Flash Memory Summit StorageMojo spoke to David Woolf and Kerry Munson of the University of New Hampshire InterOperability Lab. It’s been around for decades and still is.

It is primarily staffed by college students – cheap labor – managed by senior engineers.

Protocol testing is it’s primary function. The IOL tests many protocols such as Ethernet, Fibre Channel, iSCSI, SAS, SATA, PCIe and more. As most protocol “standards” are festooned with options this isn’t as simple as it seems.

For example, NVMe (non-volatile memory express) testing. One of the options in the NVMe spec is reservations for storage resources – pinning a resource to a server or app.

As an option it may or may not be implemented. But if the vendor says it is implemented, then the IOL will test it for correctness.

IOL maintains integrators lists that show who has been tested, and when. But IOL doesn’t make their testing report – where they list features and results – public.

As the property of the vendor they aren’t available unless the vendor makes them public. Or released to you under NDA. To use them you’d still have to match up the tested features with whatever you want to interoperate with.

The StorageMojo take
The IOL is funded by industry – and UNH student labor – so the limited distribution of its results isn’t surprising, though it is sad. This reflects the fact that interoperability – or lack thereof – is a competitive weapon almost as much as it is a useful feature.

This was evident in Fibre Channel’s heyday, when frequent plug fests convinced buyers that FC was open and interoperable. But niggling details made multi-vendor FC networks rare in production.

Despite the limitations, the IOL is a useful tool for ensuring that protocol implementations meet the letter, if not the spirit, of industry standards.

Courteous comments welcome, of course.

{ 1 comment }