I am having a problem and need some advice tracking it down. I'll try to give an overview of my setup and what the problem is.
We have some software that talks on 3 different networks. Op-Primary, Op-Secondary, Controller-Primary. The vendor has some proprietary software that handles routing between the clients (which only exist on the Op-Primary and Op-Secondary networks) and the controllers (which only exist on the Controller-Primary network and are non-virtual, physical devices).
I have multiple hosts in a cluster running ESXi 5.0.0 623860. My clients and servers are spread across the hosts. My clients on Host1 are working fine. On every other Host my clients can't connect to my controllers. If I migrate the VM to Host1 it immediately works and if I migrate it back off it immediately stops working. The entire time this is happening I can ping the controller directly. I never lose network communication. If I add a NIC to my client directly on the Contoller-Primary network everything works fine.
These are HP Blade servers running in a c7000 chassis. Both chassis are physically hooked up to the same switches with the same port settings. The host configurations are very similar, but they are currently both out of Profile Compliance. It appears to be a problem with routing or security and whatever proprietary protocol this vendor is using for communication between clients and controllers.