Hi,
an attempt to recover a VM using vSphere Replication and WebClient fails with the following message:
"Processing recovered virtual machine 'testvm' configuration file failed."
The setup:
- one vCenter 5.1U1 on Windows 2008
- 2 ESXi Hosts 5.1U1
- Source VM on ESXi1 on local datastore
- Replica on ESXi2 on FC datastore
- one VRS running on ESXi2 (VRS 5.1.1.0)
- RPO 15min
- replication was setup more than 8 hours ago, replica can be seen on datastore
- user "Administrator" has full priviledge (default)
Trigger the failure:
- Poweroff ESXi1 (VM is lost, because it's on local datastore)
- recovery attempt through WebClient fails at 40 percent
I use a different folder in "VMs & Templates"-View to avoid VMs having the same name in the same folder.
I generated a log bundle and had a look at "_var_log_vmware_hbrsrv-1.log". There I can see, that VRS has access to the datastore, where the replica is. VRS successfully copy/rename files like .vmx, .vmxf and .nvram.
Then VRS tries the disk:
from log file:
---
2013-06-20T08:06:27.948Z [7FE40B723700 verbose 'HostPicker' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] AffinityHostPicker choosing host host-12 for context '[] /vmfs/volumes/4fc738a3-11a8f647-b2e
2-0022196a6aad/testvm'
2013-06-20T08:06:27.954Z [7FE40B723700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] Waiting for result for task haTask--vim.VirtualDiskManager.queryVirtualDiskInfo-421473350
2013-06-20T08:06:27.957Z [7FE4080D5700 info 'RemoteTask'] Task haTask--vim.VirtualDiskManager.queryVirtualDiskInfo-421473350 got result
2013-06-20T08:06:27.959Z [7FE40B723700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] Completing task haTask--vim.VirtualDiskManager.queryVirtualDiskInfo-421473350 after 0 sec (success)
2013-06-20T08:06:27.959Z [7FE40B723700 verbose 'HostPicker' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] AffinityHostPicker choosing host host-12 for context '[] /vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm'
2013-06-20T08:06:27.961Z [7FE40B723700 info 'Host' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] NFC creating session to target: 10.10.50.3 port: 902
2013-06-20T08:06:28.008Z [7FE403F3C700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:28.008Z [7FE403F3C700 verbose 'SessionManager'] hbr.replica.Group.GetCurrentRpoViolation: authorized
2013-06-20T08:06:28.096Z [7FE40B723700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: SELECT * FROM ReplicatedDisk WHERE (diskID == 'RDID-2346a4fe-c786-4ec8-a902-c7a5bd6192cf');
2013-06-20T08:06:28.096Z [7FE40B723700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: SELECT * FROM DiskInstance WHERE (diskInstanceKey == 186);
2013-06-20T08:06:28.097Z [7FE40B723700 verbose 'HostPicker' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] AffinityHostPicker choosing host host-12 for context '[] /vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm'
2013-06-20T08:06:28.100Z [7FE40B723700 info 'Host' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] NFC creating session to target: 10.10.50.3 port: 902
2013-06-20T08:06:28.375Z [7FE40B723700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: UPDATE OR ROLLBACK GroupImage SET isReapable = 0 WHERE (groupInstanceKey == 186);
2013-06-20T08:06:28.380Z [7FE40B723700 info 'Image' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05:6aad99db-136f-4685-a1ec-b89cde8e8c05] Made fail-over image hbr.replica.GroupImage:Hbr.Replica.GroupImage.00000000000000ba/GID-b0fd0850-80e9-48ce-9763-c3a975b48115:1FO
2013-06-20T08:06:28.388Z [7FE408116700 verbose 'SessionManager'] hbr.replica.GroupImage.GetVirtualMachines: authorized
2013-06-20T08:06:29.011Z [7FE408053700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:30.014Z [7FE40B723700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:31.016Z [7FE408116700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:32.019Z [7FE40B723700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:33.021Z [7FE408094700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:34.024Z [7FE408157700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:35.027Z [7FE40B8B2700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:36.029Z [7FE403F7D700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:37.032Z [7FE40B7E6700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:38.035Z [7FE403F7D700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:38.863Z [7FE40B7E6700 verbose 'SessionManager'] hbr.replica.ReplicationManager.GetServerStats: authorized
2013-06-20T08:06:39.037Z [7FE40B7A5700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:40.039Z [7FE408094700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:41.042Z [7FE40B723700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:42.045Z [7FE40B7E6700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:43.047Z [7FE40B7A5700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:44.050Z [7FE408116700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:45.052Z [7FE408094700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:46.055Z [7FE408116700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:47.057Z [7FE40B723700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:48.060Z [7FE408094700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
2013-06-20T08:06:49.063Z [7FE408053700 verbose 'SessionManager'] vmodl.query.PropertyCollector.waitForUpdatesEx: authorized
---
and then VRS deletes the files:
---
2013-06-20T08:06:49.845Z [7FE40B7A5700 verbose 'SessionManager'] hbr.replica.GroupImage.revert: authorized
2013-06-20T08:06:49.845Z [7FE40B7A5700 info 'Image' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] Reverting image Hbr.Replica.GroupImage.00000000000000ba/GID-b0fd0850-80e9-48ce-9763-c3a975b48115:1FO; type FO; state created
2013-06-20T08:06:49.845Z [7FE40B7A5700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: UPDATE OR ROLLBACK GroupImage SET isReapable = 1 WHERE (groupInstanceKey == 186);
2013-06-20T08:06:49.853Z [7FE40B7A5700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: INSERT OR ROLLBACK INTO PersistentCleanups (relPath, cleanupType, dsPath, diskType) VALUES ('testvm/testvm.vmx', 0, '4fc738a3-11a8f647-b2e2-0022196a6aad', 0);
--> SELECT LAST_INSERT_ROWID();
2013-06-20T08:06:49.857Z [7FE40B7A5700 verbose 'HostPicker' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] AffinityHostPicker choosing host host-12 for context '[] /vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm'
2013-06-20T08:06:49.864Z [7FE40B7A5700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] Waiting for result for task haTask--vim.FileManager.delete-421473364
2013-06-20T08:06:49.870Z [7FE40B7A5700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] Completing task haTask--vim.FileManager.delete-421473364 after 0 sec (success)
2013-06-20T08:06:49.870Z [7FE40B7A5700 info 'PersistentCleanup' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] The file '/vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm/testvm.vmx' (key=565) was cleaned up successfully.
2013-06-20T08:06:49.870Z [7FE40B7A5700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: DELETE FROM PersistentCleanups WHERE (key == 565);
2013-06-20T08:06:49.875Z [7FE40B7A5700 verbose 'Db' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] SQL code: INSERT OR ROLLBACK INTO PersistentCleanups (relPath, cleanupType, dsPath, diskType) VALUES ('testvm/testvm.vmxf', 0, '4fc738a3-11a8f647-b2e2-0022196a6aad', 0);
--> SELECT LAST_INSERT_ROWID();
2013-06-20T08:06:49.879Z [7FE40B7A5700 verbose 'HostPicker' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] AffinityHostPicker choosing host host-12 for context '[] /vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm'
2013-06-20T08:06:49.885Z [7FE40B7A5700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] Waiting for result for task haTask--vim.FileManager.delete-421473366
2013-06-20T08:06:49.890Z [7FE40B7A5700 verbose 'RemoteTask' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] Completing task haTask--vim.FileManager.delete-421473366 after 0 sec (success)
2013-06-20T08:06:49.890Z [7FE40B7A5700 info 'PersistentCleanup' opID=6aad99db-136f-4685-a1ec-b89cde8e8c05] The file '/vmfs/volumes/4fc738a3-11a8f647-b2e2-0022196a6aad/testvm/testvm.vmxf' (key=566) was cleaned up successfully.
[...]
---
Does anyone know the reason for the failure?
Thank you,
vmmerlin