Slow iSCSI Performance - Nimble Storage CS220

Hi!

I have a new environment setup.

• 2x HP DL560 G8's with 2 8-core Xeons and 128GB of RAM, I have 4 NIC ports dedicated to iSCSI traffic, they are connected to Brocade enterprise class 1GBe switches.

• 1x Nimble CS220 12TB iSCSI unit with 4 Active NICS and 4 failover NICs

I have configured ESXi 5.1 with MPIO and Round Robin enabled, I have tried settings the IOPS to 0, 1, 4 and default of 1000, neither option makes any difference to the throughput. As far as I can tell, the traffic is being distributed evenly across the NICs at about 22.8MBps or roughly 90MBps combined.

I am told by the vendor that we have configured the unit to best practice standards and that nothing is wrong, but the throughput number we are saying are WAY off from what they would normally expect.

During a speed test using CrystalDiskMark, we are seeing the following:

Sequential Read :   108.458 MB/s
          Sequential Write :   100.017 MB/s
         Random Read 512KB :    59.763 MB/s
        Random Write 512KB :    88.862 MB/s
    Random Read 4KB (QD=1) :     8.711 MB/s [  2126.6 IOPS]
   Random Write 4KB (QD=1) :     6.916 MB/s [  1688.4 IOPS]
   Random Read 4KB (QD=32) :   164.283 MB/s [ 40108.0 IOPS]
  Random Write 4KB (QD=32) :    72.232 MB/s [ 17634.8 IOPS]

  Test : 1000 MB [C: 58.5% (23.2/39.7 GB)] (x5)
  Date : 2012/12/05 14:53:10
    OS : Windows Server 2012 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

The Sequential Read and Write are, by their standards, about 1/4th of what they would expect to see.

Here is the kicker! When I use the iSCSI Initiator with MPIO enabled on my Windows box, on the SAME network, I am seeing increased throughput with even fewer paths connected, except with slightly slower Random Reads/Writes.

Sequential Read :   188.052 MB/s
          Sequential Write :   128.094 MB/s
         Random Read 512KB :    64.522 MB/s
        Random Write 512KB :    91.783 MB/s
    Random Read 4KB (QD=1) :     9.009 MB/s [  2199.4 IOPS]
   Random Write 4KB (QD=1) :     5.352 MB/s [  1306.7 IOPS]
   Random Read 4KB (QD=32) :   110.950 MB/s [ 27087.4 IOPS]
  Random Write 4KB (QD=32) :    65.868 MB/s [ 16081.0 IOPS]

  Test : 1000 MB [E: 0.3% (0.1/30.0 GB)] (x5)
  Date : 2012/12/05 14:38:20
    OS : Windows Server 2012 Server Standard Edition (full installation) [6.2 Build 9200] (x64)

I am at a loss and Nimble Storage have spoken with me and troubleshot this thing and can't figure out what could be wrong.

Any input or suggestions would be awesome!

Thanks!