Max throughput of 140.8MB/sec (with 65k blocks)
Max IO rate of 4743.6IOPS (with 8k blocks)

The T5240 does have a Hardware RAID controller on-board but as yet I have not been able to test this

http://communities.vmware.com/blogs/WichitaDataCenters/2009/08/10/san-on-the-cheap

So, can someone give me some numbers?

I have been looking for other filebench results to make a comparison but I am having a hard time finding sample benchmarks of other vendors using filebench and varimail, online. Does anyone have numbers available for hardware in the same league? I am not asking for numbers on a sym 6 but maybe a cx500, ps100e, a LH setup with 3 NSMs and maybe an x4500 with 12-14 drives so we have some idea of how zraid compares with hardware solutions and scales in comparison to the 6 drive benchmarks.

If i had a 2 raid controllers I would put Raid0 on and mirror the stripes giving best performance and a Raid10 setup although this does have the problem of one disk failure breaking the raid just fix it and bring it back online. Different disks require different settings to get best performance/reliability ZFS normaly for me has around 128k stripe in RAIDz.

To paraphrase, there are lies, damn lies and storage performance numbers – I hesitate to call them statistics for fear of giving statistics an even worse reputation than they’ve already got.

I didn’t dig into the benchmarks Robert used for his test to see what the mix of I/O sizes are. The typical strategy for describing array performance is to run a test that will give the absolute best possible number for the attribute one is measuring.

For bandwidth that means reading and writing really big files – which is fine if you are doing video production or 3D seismic analysis – and totally irrelevant for almost all common workloads. For IOPS numbers that means the smallest possible I/Os as fast as possible – which usually means everything is sitting in cache. While that is nice when it happens, that is also an unlikely event in the real world.

So other than storage marketing people being lying scum, what is the point of benchmarks that only reflect un-real-world performance? Consider all storage benchmarks as simply telling you what the absolute maximum you could ever see in that metric – the vendor’s guaranteed absolutely “will never exceed” number. If you have good reason to believe you’ll need more than that then be afraid – be very afraid.

In the real world, with a mix of I/O sizes and rates, you’d be shocked at what “performance” looks like. Running 2k I/Os on the biggest Sym or Tagma you can imagine – ’cause you certainly can’t afford it! – on dozens of servers across multi-dozen FC’s and I suspect you’d see, maybe, with luck, 100MB/sec of bandwidth. That 3ware controller would probably do single digits. No bad guys here, this is just the nature of the storage I/O problem.

To me, the point of Robert’s benchmarks is not the absolute numbers, which are respectable for either case, but that the money spent for the RAID controllers bought nothing. I’d argue that even if the software were 20% slower, you’d still want to lose the hardware RAID and its associated bugs, power consumption, cost and maintenance.

You also need to add specifics about the storage configurations before the data can be judged. How many disk per RAID volume, RAID level, stripe size, etc.

Can ZFS replicate such performance using just software? The 3ware hardware controller is making parallel read/writes to multiple RAID 5 discs simultaneously. Something you can not do with most disc controllers when accessing them via the standard OS using ZFS.

I’m wondering if I’m missing something? Does this assume multiple disc arrays?

Thanks.

Roughly:
-Latency is usually lower with a smaller array, since you don’t have millions of lines of code and multiple switch operations to traverse
-IOPS scale for large arrays mostly as a function of parallelism – more I/O ports, more I/O processors, more cache, more interconnect bandwidth, more spindles – not because each individual I/O unit is blindingly fast
-there are only a few vendors of most of these components, so the big arrays are built out of commodity parts. Architecture and firmware are the major differentiators. So, for example, cache access times are fairly constant unless using expensive static RAM. FC chips come from what, two vendors? Microcontrollers from four? Disks from three? What do you expect?

Of course the price-point engineered stuff will be slow. But I bet there is little difference in per-port performance between an enterprise modular array and the big iron Sym’s and Tagmas.

I’ve never seen a direct comparison of single FC port performance across big iron and modular arrays, which also suggests that it isn’t all that different. If you have data that suggests otherwise I encourage you to post it. I’d love to be proven wrong.

Really? Show your work.