Hi Everyone,
Having a bit of a strange issue with vMotion today. It's been working in my lab for months, and was still working fine after I upgraded to 5.1 last week.
Now, today, for whatever reason, it just refuses to progress beyond 14%, failing with timeout. Here's the log snippet:
2012-09-18T23:52:14.117Z| vmx| I120: VMXVmdbCbVmVmxMigrate: Got SET callback for /vm/#_VMX/vmx/migrateState/cmd/##1_6438/op/=to
2012-09-18T23:52:14.117Z| vmx| I120: Could not identify IP address family of in/srcLogIp:
2012-09-18T23:52:14.117Z| vmx| I120: Could not identify IP address family of in/dstLogIp:
2012-09-18T23:52:14.117Z| vmx| I120: VMXVmdbVmVmxMigrateGetParam: type: 1 srcIp=<10.5.132.60> dstIp=<10.5.132.61> mid=4ca02aa12688e uuid=4c4c4544-004c-4a10-8031-c8c04f4c4b31 priority=high checksumMemory=no maxDowntime=0 encrypted=0 resumeDuringPageIn=no latencyAware=yes diskOpFile=
2012-09-18T23:52:14.117Z| vmx| I120: VMXVmdbVmVmxMigrateGetParam: type 1 unsharedSwap 0 memMinToTransfer 0 cpuMinToTransfer 0 numDisks 0 numStreamIps 1
2012-09-18T23:52:14.117Z| vmx| I120: Received migrate 'to' request for mid id 1348012698921102, src ip <10.5.132.60>, dst ip <10.5.132.61>(invalidate source config).
2012-09-18T23:52:14.117Z| vmx| I120: SVGA: Maximum display topology 2560x1600.
2012-09-18T23:52:14.120Z| vmx| I120: MigrateSetInfo: state=1 srcIp=<10.5.132.60> dstIp=<10.5.132.61> mid=1348012698921102 uuid=4c4c4544-004c-4a10-8031-c8c04f4c4b31 priority=high
2012-09-18T23:52:14.120Z| vmx| I120: MigrateSetState: Transitioning from state 0 to 1.
2012-09-18T23:52:14.120Z| vmx| I120: VMXVmdb_SetMigrationHostLogState: hostlog state transits to emigrating for migrate 'to' mid 1348012698921102
2012-09-18T23:53:44.117Z| vmx| I120: VMXVmdb_SetMigrationHostLogState: hostlog state transits to failure for migrate 'to' mid 1348012698921102
2012-09-18T23:53:44.121Z| vmx| I120: MigrateSetStateFinished: type=1 new state=5
2012-09-18T23:53:44.121Z| vmx| I120: MigrateSetState: Transitioning from state 1 to 5.
2012-09-18T23:53:44.121Z| vmx| I120: Migrate_SetFailureMsgList: switching to new log file.
2012-09-18T23:53:44.122Z| vmx| I120: Migrate_SetFailureMsgList: Now in new log file.
2012-09-18T23:53:44.139Z| vmx| I120: [msg.migrate.expired] Timed out waiting for migration start request.
2012-09-18T23:53:44.139Z| vmx| I120: Migrate: cleaning up migration state.
2012-09-18T23:53:44.139Z| vmx| I120: MigrateSetState: Transitioning from state 5 to 0.
I can vmkping between the relevant interfaces just fine:
# vmkping 10.5.132.61
PING 10.5.132.61 (10.5.132.61): 56 data bytes
64 bytes from 10.5.132.61: icmp_seq=0 ttl=64 time=0.131 ms
64 bytes from 10.5.132.61: icmp_seq=1 ttl=64 time=0.162 ms
64 bytes from 10.5.132.61: icmp_seq=2 ttl=64 time=0.122 ms
--- 10.5.132.61 ping statistics ---
3 packets transmitted, 3 packets received, 0% packet loss
round-trip min/avg/max = 0.122/0.138/0.162 ms
I've tried restarting vCenter itself, no help. Tried multiple different VMs. Tried various combinations of my 4 hosts, and none of them work.
I just dont know where else to go - Ideas?
Edit: Other things I've confirmed: Time sync is good. Disk free space is good. FWD and REV name resolution is good.