Hello Everybody,
We have hit an interesting issue which, I believe, it will be interesting for all of us to discuss.
At first let me explain the configuration we have.
Two ESXi hosts (version 4.1.0, 874690) used in a cluster and two pfSense boxes running inside as guests virtual machines.
The Issue
As you may probably guessed already, we have CARP interfaces.
Everything was up and running until we decided to move one of the machines. When migrated to the other host - both machines reported itself as backup nodes and the CARP IP address was not served anymore.
We figured out that most probably this was caused by the two pNICs used as uplinks on the vSwitch on the target host.
It seems that somehow this was caused by both of the nature of working of CARP and the promiscuous mode on the vSwitch.
We found out that the originated multicast traffic from the pfSense machine was bounced back through the pSwitch (used between the ESXi hosts) to the secondary physical uplink on the vSwitch.
And the result is that pfSense sees the traffic originating from itself bouncing back, and as result stays as backup node and not serving the virtual IP address.
So the questions are:
1. How exactly the promiscuous mode, set on the portgroup level, affects physical NIC's? Is it actually such relation exists?
2. How to interpret the information listed above:
When promiscuous mode is enabled at the portgroup level, objects defined within that portgroup have the option of receiving all incoming traffic on the vSwitch. Interfaces and virtual machines within the portgroup will be able to see all traffic passing on the vSwitch, but all other portgroups within the same virtual switch do not.
When promiscuous mode is enabled at the virtual switch level, all portgroups within the vSwitch will default to allowing promiscuous mode. However, promiscuous mode can be explicitly disabled at one or more portgroups within the vSwitch, which override the vSwitch-defined default.
Source:
VMware KB: How promiscuous mode works at the virtual switch and portgroup levels
Duplicate multicast packets are generated when the vSwitch has at least two vmnics and promiscuous mode enabled
Consider a vSwitch that has more than one uplink and has the promiscuous mode enabled. Some of the packets that come in from the uplinks that are not currently used by the promiscuous port, are not discarded. This behavior might mislead some applications, such as the CARP protocol instance.
This issue is resolved in this release. Starting with this release the Net.ReversePathFwdCheckPromisc configuration option is provided to explicitly discard all the packets coming in from the currently unused uplinks, for the promiscuous port.
Note: If the value of the Net.ReversePathFwdCheckPromisc configuration option is changed when the ESX instance is running, you need to enable or re-enable the promiscuous mode for the change in the configuration to take effect.
Source:
VMware vSphere ESX 4.0 Update 2 Release Notes
For me the information I have read so far is misleading. At least it is not clear enough if the promiscuous mode on the vSwitch level or the port groups level affects somehow the physical adapters?
Thank you in advance for eventual participation!