Hi guys,
We just upgraded our ESXi installation from ESXi 5.0 to 5.1. We have 4 Dell T-410 computational servers (named vmhost1, vmhost2, vmhost3,vmhost4) each with at least 32 GBs of RAM. Each server is also connected to a couple of iSCSI LUNs exported from 2 QNAP boxes. (The QNAP boxes export 9 LUNs and each host is connected to 2 - 3 of them for storing the VMs).
Everything was running fine until last night. Last night at 2:30 AM, all of a sudden ONE of the hosts, vmhost3, dropped 1 of its iSCSI connections (LUN id 1 on one of the QNAP's). LUN 1 was mapped as 'qboxraid1' in vmhost3.
I restarted vmhost3 and after the restart, I had to/could re-add qboxraid1 from the "Add new storage" option and ESXi recognized that the LUN had come back and all the "Unknown (inaccessible)" VMs refreshed and I could see my VMs again. I even started a few and everything seemed to coming back up fine.
In the mean time on my QNAP box, I started a "bad blocks" scan on all my disks in case there was a corrupt disk or something that caused the RAID to fail. Not sure if this actually happened, but just double checking the disks to be sure...
However, 2 hours later, vmhost3 dropped the connection to just that one LUN again! The second LUN is operating just fine...
Now if I go on vmhost4 and "add new storage", I can see LUN 1 showing up as 'qboxraid1'. I can add it to vmhost4 and bring up all the VMs stored on qboxraid1 on vmhost4. The VMs seem to run fine and all is well.
Anybody have any ideas on what could be going on and how to fix it? I am a little new to debugging iSCSI and storage issues on ESXi so please be patient with me if I ask clarifying questions.
Thank you in advance!