HP, a great company, I like the hardware design of HP ProLiant server, it’s pretty easy for datacenter maintenance and operation, do you like it? Today, I’ll introduce a storage issue on HP ProLiant BL460, BL480 blades. This issue happened on Qlogic HBA with VC-FC module. I have two dual port Qlogic HBAs on each ESXi5.x host, one port of each HBA was zoned together on SAN switch.
For example, vmhba1 and vmhba3 are zoned for LUN allocation, each LUN have two paths on each HBA port.
I observed all LUNs disappeared on random HBA port sometimes, it’s not happening very frequently, but it can lead to ALL VM DEAD if you get storage outage when LUNs disappeared!!! This problem becomes more frequently more your virtual infrastructure grows bigger.
This is the symptoms when the issue happening:
And if you login SSH console and check HBA card status by:
less /proc/scsi/qla2xxx/[Device ID]
You will find following differences of two HBA ports:
See? All targets show Offline status on problem HBA.
scsi-qla3-target-0=500a09859d812da0:030098:1000:<Offline>
You have two options to fix it:
- Reseat blade. Downtime and local resource is required.
- Reset HBA by following step:
Record the Device ID, and force HBA do rescan:
echo "scsi-qlascan" > /proc/scsi/qla2xxx/adapter_id
Wait few seconds, force LIP login:
echo "scsi-qlalip" > /proc/scsi/qla2xxx/adapter_id
- Wait few minutes, LUNs come back online… JYou could refer to KB 1031199 for more detail.
This is a temporary remediation, the problem will repeat. I’ll show you some permanent solution in next blog.