they definitely run FreeBSD 4.x (haven’t remembered x)
(while Ontap-GX for example runs AFAIK FreeBSD 6.2 !)

The FS SEEMS like a clustered-FS, which all nodes mount under /onefs.
Internally the box calls the fs ‘efs’ – whatever that stands for.
Unlike a traditional array – they stripe/distribute FILES, not blocks, and they can RE-stripe files – which seems to make it possible to change RAID-levels on individual files.
(and to reconstruct of course – which is a regen of parity/data)

Due to the fact, that they do not want to expose any details of their ‘efs’ to the world, each node does act as an NFS-server.
(or CIFS server – but that’s just samba)
This has now several effects IMHO:

a.) clients don’t need any fs-driver to access it
(which sounds good)
OTOH – clustered NFS isn’t really trivial, and causes issues.

b.) each of the node acts as an own NFS-server for /onefs.
So all nodes ‘see’ and ‘export’ the same FS via multiple nodes.
But this causes all kind of questions and issues, which I couldn’t check:
1.) is the NFS-reply-cache replicated ?

2.) is there an optional NFS-failover ? (I couldn’t see any vif)
(no need for reply-cache replication if there’s no NFS-failover)

3.) you have to take care which clients mount which brick – as it could otherwise create ‘hot bricks’ …
NFS itself has ZERO mechanisms to ‘move/migrate’ a mount from one IP to another IP without unmount/mount on the client.
(this is something NTAP tries to avoid talking about too)

c.) the generic arch isn’t capable to scale single-client or single-stream performance to more than one gigE.
With a client, you could mount /onefs several times, but whenever a client writes to a file – this stream goes only over one mountpoint – and hence over one isln-interface/brick.
So the max sustainable read/write is 1 gigE PER FILE – while reality says it’s even less (rumours say ~75mb/sec write speed)
Maybe they’ll go to 10gigE frontends – then it’ll change – maybe.

Also – if several clients (mounted on different bricks) read/write to the same file – what’s the realistic aggregate speed ?
(depends on the style of caching/locking)

d.) the IB-backend:
implies, that the remote-disks are connected via (TCP) IP !
(otherwise you couldn’t use gigE instead of IB)

Latency: like every distributed FS, which PROXIES requests to the back-end on other nodes – it normally causes latency – esp. for small files…
As this system is from my understanding for large-files – no problem – but for home-dirs/small files – hmm …
(esp. if many clients mounted on different bricks do many creates/renames/removes.)

e.) metadata coordination:
Like every distributed FS – some metadata operations MUST GET serialized (like mkdir/rmdir/creat()/rename()/unlink()/link() ).
You can’t parallelize ‘mkdir’ or ‘rm/mv’ )
Not sure how they handle this – they’re very quiet on this.
But as the system is limited in size – you could do local caching – depends on the number of files etc… (memory consumption)

f.) failure behavior:
Like with every distributed FS – the failure-behavior is definitely tricky – and is definitely different from non-clustered FSs.

1.) If a node goes down (OS crash or power) – the other nodes have to decide what to do:
reconstruct all files from the failed node – and reinitialize the node (clean) when it rejoins. (very inefficient)
or hope and wait that the node comes back. (need to wait)
(how long do they wait untill they give up the hope ?)

Can you write during this time NEW files ? (I think – yes)
Can you write/append/modify during this time to EXISTING FILES ?
(likely not – but maybe yes – don’t know)

Reconstruction of failed nodes:
Seems like the system is FILE based – so reconstruction of failed nodes only recons REALLY-USED-space, not like block-arrays everything …
But still – with 6+ disks per brick – and assuming they’re close to full – this can take time … – maybe 1 day ??

So remains the question of what’s this system REALLY good for:

a.) it’s mainly useless for HPC – at least for the workloads I know.
too slow single-stream perf. – NFS has issues with N-to-1 writes.
(multiple clients write through different bricks to the same file)

b.) it’s mainly useless for ‘commercial’ NAS
homedirectory-files (small) won’t go too well, and some good space/quota management doesn’t seem to exist.

c.) not sure whether it’s usefull for databases (like ORCL) – esp. would need more details how the crash/recovery of individual nodes is handled.

d.) To me it looks more like a READ-optimized archive system.
(esp. the degraded write/append/modify has questions)
So somewhere at web-farms or other high-read-load scenarios.
But for such areas – most cheaper systems should do well too.

hirni

They don’t charge more for Infiniband, which is a plus, given its much lower latency and higher bandwidth. As I recall though, Iband switches are actually much simpler than ethernet switches. So not terribly surprising. I’m sure Isilon gets a major performance boost from it.

I haven’t looked that closely at the actual hardware, so thanks for the overview. I’ve asked Isilon to comment, so let’s see if they do.

Wes,

What a great site! Thanks for the link.

At this point in time … it seems to be specialized for ‘read mostly ’ sequential multi-stream video delivery… if the reported 96 KB chunk size is correct.

Each node contains 12 disks which are locally managed as a RAID group, protected by Reed Solomon algorithms … which is RAID 6.

So far… not too much mojo.

As usual, the issue of performance and scalability is more complex.

It would help if someone at Isilon could come up with a system diagram & comment on their locking mechanizm & dataflow, much as Netapp did on …

http://drunkendata.com/?p=622

and … http://gridguy.net/?p=16#comments