Hi,
I am facing issue with two of my hosts in the cluster of 6Hosts. Hosts are getting disconnected and connected back on vCenter Server.I found out many portals wherein it defines resolution to this problem , but the logs states different than what is actually mentioned in other blogs.
I need help in fining out the resolution for this as it is impacting my production work,
Below is the error I had captured from the time when esx get disconnected form VCS.
Logs are getting updated properly till it starts generating coredump:
May 7 07:09:13 [2013-05-07 07:06:40.100 10AE5B90 info 'App' opID=HB-host-13@93-a7] [VpxLRO] -- FINISH task-internal-25 -- -- vmodl.query.PropertyCollector.Filter.destroy -- 522acf41-44d6-2f56-da36-6adcd544df9b [2013-05-07 07:06:40.102 1091EB90 verbose 'SoapAdapter.HTTP
May 7 07:09:13 Service'] User agent is 'VMware-
May 7 07:09:13 Vpxa: ------ In-memory logs end --------
May 7 07:09:13 vmkernel: 598:19:58:09.633 cpu1:221044419)User: 2428: wantCoreDump : vpxa -enabled : 1
May 7 07:09:14 vmkernel: 598:19:58:09.826 cpu8:221044419)UserDump: 1485: Dumping cartel 226750129 (from world 221044419) to file /var/core/vpxa-zdump.003 ...
Here after 42seconds logs starts getting generated and on vcs I could see host in non responding state and all VMs in disconnected.
May 7 07:09:56 vmkernel: 598:19:58:52.801 cpu10:221044419)UserDump: 1587: Userworld coredump complete.
May 7 07:09:56 Hostd: [2013-05-07 07:09:56.999 39BEDB90 warning 'Proxysvc Req79023'] Server localhost:8089 disconnected with a response pending.
May 7 07:09:56 Hostd: [2013-05-07 07:09:56.999 39D40B90 warning 'Proxysvc Req79026'] Server localhost:8089 disconnected with a response pending.
May 7 07:09:56 Hostd: [2013-05-07 07:09:56.999 3B240B90 warning 'Proxysvc Req79022'] Server localhost:8089 disconnected with a response pending.
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.066 3B840B90 verbose 'Proxysvc Req79028'] New proxy client SSL(TCP(local=207.129.230.60:49197, peer=10.108.2.51:443))
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.069 3B440B90 verbose 'Proxysvc Req79029'] New proxy client SSL(TCP(local=207.129.230.60:49198, peer=10.108.2.51:443))
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.070 FFB77E80 verbose 'Proxysvc Req79030'] New proxy client SSL(TCP(local=207.129.230.60:49199, peer=10.108.2.51:443))
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.187 39FD2B90 warning 'Proxysvc Req79029'] Connection to localhost:8089 failed with error N7Vmacore15SystemExceptionE(Connection refused).
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.187 3B440B90 warning 'Proxysvc Req79028'] Connection to localhost:8089 failed with error N7Vmacore15SystemExceptionE(Connection refused).
May 7 07:09:57 Hostd: [2013-05-07 07:09:57.187 FFE6EB90 warning 'Proxysvc Req79030'] Connection to localhost:8089 failed with error N7Vmacore15SystemExceptionE(Connection refused).
May 7 07:09:58 watchdog-vpxa: '/opt/vmware/vpxa/sbin/vpxa ++min=0,swap,group=vpxa' exited after 267 seconds
May 7 07:09:58 watchdog-vpxa: Executing '/opt/vmware/vpxa/sbin/vpxa ++min=0,swap,group=vpxa'
May 7 07:09:58 vpxa: VmkSyscall_init failed: 2 No such file or directory
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.136 FFDA8B10 info 'App'] Current working directory: /var/log/vmware/vpx
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.136 FFDA8B10 verbose 'ThreadPool'] Thread info: Min Io, Max Io, Min Task, Max Task, Max Thread, Keepalive, thread kill, max fds: 2, 9, 2, 4, 13, 4, 10, -1
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.138 FFDA8B10 info 'App'] Log path: /var/log/vmware/vpx
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.138 FFDA8B10 info 'App'] Initializing SSL
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.142 FFDA8B10 info 'App'] Vmacore::InitSSL: doVersionCheck = false, handshakeTimeoutUs = 120000000
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 verbose 'VpxMutex'] Profiling mutex: InvtLock
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 info 'App'] [Vpxd_EarlyInit] Setting malloc mmap threshold to 32KB
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 info 'App'] Creating SSL Contexts
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.145 FFDA8B10 info 'SysCommandPosix'] ForkExec(/usr/bin/vmware) 229504479
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.149 FFDA8B10 verbose 'App'] Output: VMware ESXi 4.1.0 build-348481
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.152 FFDA8B10 warning 'Libs'] ObtainHardwareID unexpected failure: 22.
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.152 FFDA8B10 warning 'Libs'] Hostinfo_MachineID ObtainHardwareID failure (Invalid argument); providing default.
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] vmware-vpxa: Removing stale symlink /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] Removing stale cnx files in /var/run/vmware/root_0/1367910331587505_226750129
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] vmware-vpxa: Setup symlink /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 -> /var/run/vmware/root_0/1367910598150197_229803473
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.136 FFDA8B10 verbose 'ThreadPool'] Thread info: Min Io, Max Io, Min Task, Max Task, Max Thread, Keepalive, thread kill, max fds: 2, 9, 2, 4, 13, 4, 10, -1
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.138 FFDA8B10 info 'App'] Log path: /var/log/vmware/vpx
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.138 FFDA8B10 info 'App'] Initializing SSL
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.142 FFDA8B10 info 'App'] Vmacore::InitSSL: doVersionCheck = false, handshakeTimeoutUs = 120000000
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 verbose 'VpxMutex'] Profiling mutex: InvtLock
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 info 'App'] [Vpxd_EarlyInit] Setting malloc mmap threshold to 32KB
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.143 FFDA8B10 info 'App'] Creating SSL Contexts
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.145 FFDA8B10 info 'SysCommandPosix'] ForkExec(/usr/bin/vmware) 229504479
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.149 FFDA8B10 verbose 'App'] Output: VMware ESXi 4.1.0 build-348481
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.152 FFDA8B10 warning 'Libs'] ObtainHardwareID unexpected failure: 22.
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.152 FFDA8B10 warning 'Libs'] Hostinfo_MachineID ObtainHardwareID failure (Invalid argument); providing default.
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] vmware-vpxa: Removing stale symlink /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] Removing stale cnx files in /var/run/vmware/root_0/1367910331587505_226750129
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.153 FFDA8B10 info 'Libs'] vmware-vpxa: Setup symlink /var/run/vmware/f4a0dbedb2e0fd30b80f90123fbe40f8 -> /var/run/vmware/root_0/1367910598150197_229803473
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.159 FFDA8B10 info 'App'] Starting VMware VirtualCenter Agent Agent Daemon 4.1.0 build-345043
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.160 FFDA8B10 verbose 'App'] Init: Succeeded with directory = /var/log/vmware/journal
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.161 FFDA8B10 info 'SysCommandPosix'] ForkExec(/usr/bin/vmware) 225506784
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.165 FFDA8B10 verbose 'App'] Output: VMware ESXi 4.1.0 build-348481
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.166 FFDA8B10 verbose 'App'] [VpxSecPolicy] Security policy allocated
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 28 max LROs
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 0 reserved internal LROs
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 0 reserved blocker LROs
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 6 reserved short LROs
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 2 reserved long LROs
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'App'] [VpxLRO] 600-second task lifetime
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.168 FFDA8B10 info 'ThreadPool'] [SetConfig] Thread info: Min Io, Max Io, Min Task, Max Task, Max Thread, Keepalive, thread kill, max fds: 2, 9, 14, 38, 47, 16, 10, -1
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.169 FFDA8B10 info 'App'] [VpxaInvtHost] Manager IP: 207.129.230.60:902 Host IP: 10.108.2.51 Preserve IPs: false
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.169 FFDA8B10 verbose 'App'] [VpxaInvtHost] Increment master gen. no to (1): HostConfig:VpxaInvtHost::Init
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.169 FFDA8B10 verbose 'App'] [VpxaInvtHost] Increment master gen. no to (2): ResourcePool:VpxaInvtHost::Init
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.169 FFDA8B10 verbose 'App'] [VpxaVMAPSyncInfo] _countThreshold=5, _timeThreshold=300000000
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.169 FFDA8B10 verbose 'App'] [VpxaInvtHost] Increment master gen. no (3): Init
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.170 FFDA8B10 info 'App'] [VpxaMoDvs::Init] dvs dvport sync rate set to [300], portgroup sync batch limit set to [500], port sync batch limit set to [1000000]
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.170 FFDA8B10 verbose 'App'] [VpxaAAM::Init]
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.173 FFDA8B10 verbose 'App'] [VpxaHalCnxHostagent] Creating temporary connect spec: localhost:443
May 7 07:09:58 Hostd: [2013-05-07 07:09:58.193 FFE6EB90 verbose 'Proxysvc Req79031'] New proxy client SSL(TCP(local=127.0.0.1:57551, peer=127.0.0.1:443))
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.198 FFDA8B10 verbose 'App'] [VpxVim] Local and Remote Version are the same. Talking with version vim.version.version6
May 7 07:09:58 Vpxa: [2013-05-07 07:09:58.199 FFDA8B10 verbose 'App'] [VpxaHalCnxHostagent] Connecting to hostd with version vim.version.version6
May 7 07:09:58 Hostd: [2013-05-07 07:09:58.362 3B881B90 verbose 'DvsManager'] PersistAllDvsInfo called
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.174 106DCB90 verbose 'App'] [VpxaAAM::Start]
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.175 106DCB90 verbose 'App'] [VpxaAAM::Start] setting environment var FT_PROC_NAME to VMap_ukelhpbc001esx02
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.175 106DCB90 warning 'App'] [VpxaAAM::Start] Not logged into to hostd!
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.175 106DCB90 verbose 'App'] [VpxaAAM::Start] Defining sensor: PolicyChange
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.176 106DCB90 verbose 'App'] [VpxaAAM::Start] Defining sensor: StateChange
May 7 07:10:02 Vpxa: [2013-05-07 07:10:02.176 106DCB90 verbose 'App'] [VpxaAAM::Start] Defining sensor: VMDeleted
May 7 07:10:03 Hostd: [2013-05-07 07:10:03.314 39F40B90 verbose 'Locale'] Default resource used for 'TooManyWrites.summary' expected in module 'fault'.
May 7 07:10:03 Hostd: [2013-05-07 07:10:03.451 39F40B90 verbose 'App'] Looking up object with name = "firewallSystem" failed.
May 7 07:10:03 Hostd: [2013-05-07 07:10:03.536 39F40B90 verbose 'NetConfigProvider'] FetchFn: List of pnics opted out
May 7 07:10:04 Hostd: [2013-05-07 07:10:04.471 39F40B90 verbose 'NetConfigProvider'] FetchFn: List of pnics opted out
May 7 07:10:05 Hostd: [2013-05-07 07:10:05.342 39F40B90 verbose 'DvsTracker'] FetchDVPortgroups: added 0 items
May 7 07:10:05 Hostd: [2013-05-07 07:10:05.399 39F40B90 verbose 'Locale'] Default resource used for 'host.SystemIdentificationInfo.IdentifierType.AssetTag.label' expected in module 'enum'.
May 7 07:10:05 Hostd: [2013-05-07 07:10:05.399 39F40B90 verbose 'Locale'] Default resource used for 'host.SystemIdentificationInfo.IdentifierType.AssetTag.summary' expected in module 'enum'.
May 7 07:10:05 Hostd: [2013-05-07 07:10:05.400 39F40B90 verbose 'Locale'] Default resource used for 'host.SystemIdentificationInfo.IdentifierType.AssetTag.label' expected in module 'enum'.
May 7 07:10:05 Hostd: [2013-05-07 07:10:05.400 39F40B90 verbose 'Locale'] Default resource used for 'host.SystemIdentificationInfo.IdentifierType.AssetTag.summary' expected in module 'enum'.
Quick help on this would be much appreciated.
My VCS Version is : 4.1
ESX host version is :ESXI 4.1 U1
Regards