I have a HP Proliant DL380 G5 running ESXi 5.1.0 connected to a Cisco 3750 Switch Stack.
While performing a tcpdump on an unrelated host I discovered that it was being bombarded with ethernet frames destined for the MAC address of a VM located on a host on the other side of the data centre. After some brief troubleshooting I discovered that I was able to stop these erroneous frames by disabling NIC teaming on my VMWare host.
I believe that in my situation when ESXi is configured to use two network adapters that it takes all frames received on vmnic0 and re-transmits them on vmnic1 and vice-vesa. I have experienced these symptoms when ESXi is configured for Fail-over with one adapter active and one adapter in standby and I have also experienced the same symptoms in the load-balancing configuration described below.
Symptoms: A short time after enabling load-balancing, all Ethernet Frames destined for the MAC address of a VM on the affected host are broadcast out every switch port in the entire data centre.
Steps to Reproduce:
1) Implement configuration below.
2) Disconnect ethernet cable running between vmnic1 and switch1 port gi2/0/4
3) Run tcpdump -q -n -e host 1.1.1.1 on any physical linux machine in the data centre (Doesn't even need to be connected to switch 1 directly).
Confirm there are no packets seen with the destination IP of 1.1.1.1
4) Reconnect ethernet cable between vmnic1 and switch1 port gi2/0/4
5) Wait 60-120 seconds
6) Observe a flurry of ethernet frames with the destination MAC address of the VM (that owns 1.1.1.1) in the tcpdump output
Cisco Configuration:
hostname switch1
!
port-channel load-balance src-dst-ip
!
interface GigabitEthernet1/0/4
description vmnic0.host0 (NIC 1)
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 4 mode on
spanning-tree portfast trunk
end
!
interface GigabitEthernet2/0/4
description vmnic1.host0 (NIC 2)
switchport trunk encapsulation dot1q
switchport mode trunk
channel-group 4 mode on
spanning-tree portfast trunk
end
!
interface Port-channel4
description host0
switchport trunk encapsulation dot1q
switchport mode trunk
spanning-tree portfast trunk
end
ESXi Configuration
vSwitch0
NIC Teaming / Load Balancing: Route based on IP Hash
NIC Teaming / Network Failover Detection: Link status only
NIC Teaming / Notify Switches: Yes
NIC Teaming / Fallback: Yes
NIC Teaming / Active Adapters: vmnic0, vmnic1
NIC Teaming / Standby Adapters: Nil
NIC Teaming / Unused Adapters: Nil
Security / Promiscious Mode: Reject
Security / MAC Address Changes: Accept
Security / Forged Transmits: Accept
Virtual Machine Port Group #1
Network Label: "Public"
VLAN: 27
NIC Teaming: All Unchecked (Inherited)
Virtual Machine Port Group #2
Network Label: "Trunk"
VLAN: 4095
NIC Teaming: All Unchecked (Inherited)
VM Kernel Port #1
Network Label: "Management"
VLAN: 2
NIC Teaming: All Unchecked (Inherited)
VM #1
OS: Windows Server 2003
NIC 1 / Adapter: Flexible
NIC 1 / Network Label: "Public"
IP Address: 1.1.1.1/24