Hey!
I configured a fresh installed VMware vSphere cluster with 3 single node and when I should try to testing NFS share performance the VM hasn't started. I watched vmkernel logs in /var/log what happened with datastore and I saw this:
019-09-18T15:44:05.459Z cpu30:2098703)WARNING: NFS41: NFS41FSAPDNotify:6027: Lost connection to the server 10.1.2.57 mount point ssd, mounted as 56016377-f996d3c5-0000-0000000 00000 ("/volume1/test")
2019-09-18T15:44:05.459Z cpu30:2098703)StorageApdHandler: 1203: APD start for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:44:05.459Z cpu5:2097602)StorageApdHandler: 419: APD start event for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:44:05.459Z cpu5:2097602)StorageApdHandlerEv: 110: Device or filesystem with identifier [56016377-f996d3c5-0000-000000000000] has entered the All Paths Down state.
2019-09-18T15:44:15.989Z cpu35:2101468)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:44:18.475Z cpu4:2099935)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:44:25.037Z cpu30:2099328)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:44:28.879Z cpu38:2100473 opID=69e0f6e3)World: 11943: VC opID k0gphp04-74422-auto-1lfc-h5:70012138-75-77 maps to vmkernel opID 69e0f6e3
2019-09-18T15:44:28.879Z cpu38:2100473 opID=69e0f6e3)Unmap6: 7133: [Unmap] 'Datastore':device(0x430717722f70)does not support unmap
2019-09-18T15:44:30.991Z cpu35:2101468)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:44:33.478Z cpu4:2099935)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:44:34.901Z cpu10:2101481)J6: 2651: 'Datastore': Exiting async journal replay manager world
2019-09-18T15:44:36.706Z cpu33:2101417)WARNING: NFS41: NFS41FileIOSync:731: Synchronous IO on obj 0x430744eb39e0 (fh 0x430bd7e3a098) failed on #2 retry: IO was aborted
2019-09-18T15:44:36.706Z cpu14:2097550)WARNING: NFS41: NFS41FileIOSync:731: Synchronous IO on obj 0x430744e90040 (fh 0x430bd7e3fcc8) failed on #2 retry: IO was aborted
2019-09-18T15:44:36.706Z cpu7:2098714)WARNING: SunRPC: 3913: fail all pending calls for client 0x43072e3d7240 IP 10.1.2.57.8.1 (socket half closed)
2019-09-18T15:44:36.707Z cpu3:2098701)NFS41: NFS41ProcessExidResult:2301: clientid 533f825d10000000 roles 0x10000
2019-09-18T15:44:36.708Z cpu30:2098703)NFS41: NFS41ProcessClusterProbeResult:3875: Reclaiming state, cluster 0x430bd7e36e70 [0]
2019-09-18T15:44:36.708Z cpu20:2098707)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744eb39e0 open status: Failure; rqst 0x430bd7e40560 cmpd stat us: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu12:2098705)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744e92510 open status: Failure; rqst 0x430bd7e42340 cmpd stat us: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu20:2098707)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744e90040 open status: Failure; rqst 0x430bd7e461f0 cmpd stat us: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu6:2098708)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744ccd210 open status: Failure; rqst 0x430bd7e45090 cmpd statu s: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu3:2098706)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744e87c80 open status: Failure; rqst 0x430bd7e45940 cmpd statu s: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu20:2098707)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744e815d0 open status: Failure; rqst 0x430bd7e46aa0 cmpd stat us: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.708Z cpu12:2098705)WARNING: NFS41: NFS41OpenReclaimDoneWork:5345: Failed to reclaim objDesc 0x430744e91590 open status: Failure; rqst 0x430bd7e40330 cmpd stat us: NFS4ERR_NO_GRACE
2019-09-18T15:44:36.710Z cpu30:2098703)StorageApdHandler: 1315: APD exit for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:44:36.710Z cpu30:2098703)NFS41: NFS41FSAPDNotify:6093: Restored connection to the server 10.1.2.57 mount point ssd, mounted as 56016377-f996d3c5-0000-000000000000 ("/volume1/test")
2019-09-18T15:44:36.710Z cpu5:2097602)StorageApdHandler: 507: APD exit event for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:44:36.710Z cpu5:2097602)StorageApdHandlerEv: 117: Device or filesystem with identifier [56016377-f996d3c5-0000-000000000000] has exited the All Paths Down state.
2019-09-18T15:44:51.718Z cpu6:2099935)WARNING: NFS41: NFS41FileOpOpenDir:4331: Failed to process the open directory result: IO was aborted
2019-09-18T15:45:03.780Z cpu27:2101457 opID=4896837a)World: 11943: VC opID sps-Main-293449-809-4d-9d-e6c4 maps to vmkernel opID 4896837a
2019-09-18T15:45:03.780Z cpu27:2101457 opID=4896837a)Unmap6: 7133: [Unmap] 'Datastore':device(0x430717722f70)does not support unmap
2019-09-18T15:45:09.787Z cpu33:2101503)J6: 2651: 'Datastore': Exiting async journal replay manager world
2019-09-18T15:45:21.725Z cpu15:2098690)WARNING: NFS41: NFS41SessionSlotUnassign:4133: sess 0x430bd7e43e60 slot table usable slots (15) below 50%: abandoning
2019-09-18T15:45:21.725Z cpu30:2098703)WARNING: NFS41: NFS41FSAPDNotify:6027: Lost connection to the server 10.1.2.57 mount point ssd, mounted as 56016377-f996d3c5-0000-0000000 00000 ("/volume1/test")
2019-09-18T15:45:21.725Z cpu30:2098703)StorageApdHandler: 1203: APD start for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:45:21.725Z cpu5:2097602)StorageApdHandler: 419: APD start event for 0x430bd7e35d70 [56016377-f996d3c5-0000-000000000000]
2019-09-18T15:45:21.725Z cpu5:2097602)StorageApdHandlerEv: 110: Device or filesystem with identifier [56016377-f996d3c5-0000-000000000000] has entered the All Paths Down state.
2019-09-18T15:45:42.176Z cpu35:2101468)WARNING: NFS41: NFS41FileOpOpenDir:4331: Failed to process the open directory result: IO was aborted
2019-09-18T15:45:51.728Z cpu0:2099408)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:51.728Z cpu10:2099935)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:51.731Z cpu22:2099841 opID=d4a00b45)World: 11943: VC opID k0gphp04-74521-auto-1li6-h5:70012161-94-6d maps to vmkernel opID d4a00b45
2019-09-18T15:45:51.731Z cpu22:2099841 opID=d4a00b45)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:51.732Z cpu5:2099853 opID=18d4e77)World: 11943: VC opID k0gphp04-74400-auto-1les-h5:70012126-44-22 maps to vmkernel opID 18d4e77
2019-09-18T15:45:51.732Z cpu5:2099853 opID=18d4e77)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:51.732Z cpu24:2099328)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:53.421Z cpu2:2099410 opID=190efaad)World: 11943: VC opID k0gphp04-75176-auto-1m0b-h5:70012209-45-56-e6d8 maps to vmkernel opID 190efaad
2019-09-18T15:45:53.421Z cpu2:2099410 opID=190efaad)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
2019-09-18T15:45:57.178Z cpu35:2101468)WARNING: NFS41: NFS41FileOpGetFileAttributes:4239: Failed to get file attributes for object 0x430744eb2870: IO was aborted
Details:
NAS: Synology FS 2017
VMware: 6.7 update2 in hypervisors (NFS VAAI Plug-in has installed too) & fresh and latest vsphere
Network: 1 uplink based DSwitch
In NAS I set all permission what vmware need to see and communicate to the NFS server.
Have you got any idea for solving this problem ?
Thanks a lot!