Spoiler alert: there is PLENTY disk-space available on the vsanDatastore...
Hello,
The VSAN System in our Lab is running into more and more problems when it comes to creating snapshots. At the moment, there are several VM's which whom it has become completely impossible to create snapshots at all.
It started out a few weeks ago when i notices strange retries in our Backup-software (Veeam v8 latest version). It needed 1 or sometimes 2 retries because it could not create a snapshot due to the mention error.
Then, over time, it took 2 or 3 retries. So it would fail to create a snapshot once, twice but the third time it would work just fine. Job retries are 5 minutes aparts. So all of this happens in a timeframe of say 15 to 20 minutes or so.
At the moment, several VM's cannot be backed up anymore because it has become impossible to create snapshots. The reason is always the same: "Insufficient disk space on datastore" which is totally bolloks as we have over 5 TB free.
At first, it thought it was due to a problem in Veeam but as it turns out, it has nothing to do with Veeam because I can't create snapshots in the VI /Web Client either. The problem lies in vSphere and Veeam is mere a victim.
Looking at logfiles, I always see this error when trying to consolidate snapshots: "An error occurred while consolidating disks: msg.disklib.NOSPACE"
The error "msg.disklib.NOSPACE" is a generic error which i also see when i simply want to make a snaphot.
This lead me to KB Article VMware KB: Consolidating disks in VMware Virtual SAN fails with the error: An error occurred while consolidating dis…
None of the conditions like "When the disk being consolidated has more than 255GB data, consolidation fails." are valid in my environment.
Each and every blog-post or KB article I found has brought me nothing.
When i look inside such a VM's directory, there is carnage. A VM with two disks started out with names like the usual "vm01.vmdk" and "vm01_1.vmdk" (for the second disk). After a couple of days, it can look like ""vm01.vmdk" and "vm01_17.vmdk" and there are a lot of snapshots lying around. If also leads to the usual "VM need consolidation" messages which always fail due to locking problems ("Consolidation failed for disk node 'scsi0:1': msg.fileio.lock").
Consolidating a VM requires some handy work but I can always get them back on track. The wierd _17.vmdk names (or similar) I can fix by renaming the .vmdk and -flat.vmdk files to _1 again and editing the .vmdk file and change the _17 to _1 etc. etc. etc.
So I can always "repair" a VM and we never have dataloss. But it's pain in the bum and it requires downtime for the VM in question.
I can repair a VM and it will be able to make snapshots just fine but it will get into trouble within 3 or 4 days with the exact same problem. Sometimes i go as far as deleting the VM from the inventory, repair the VMDK names (when it went cookoo again and calls a disk _15.vmdk or whatever instead of the original _1.vmdk etc.) and creating a brand new VM and attach the old vmdk's. That "new" VM will then be quiet and fine and dandy for a few days before the same shit starts happening all over again, getting worse over time.
Each and every blog-post or KB article I found has brought me nothing.
I desperately need your help. The VSAN runs fine. But snapshots, and with that the backups, are getting more and more problematic as ever more VM's start suffering from this issue. I could move some low I/O VM's to a cheap NFS NAS and presto, all problems are gone. Move them back, fine for a few days and then the misery starts all over again, getting worse over time. it IS a VSAN problem.
VMware is not going to help me. Reason: this is not a production environment and there is no dataloss so they don't care. Period. I would end up with a level-1 support employee which starts asking all the basic questions which I am already way beyond and they will never escalate and allocate level-2 personell to my case, making the entire exercise frustrating, lengthy (weeks) and as useless as tits on a fish (pardon my french).
Help me Obi-Wan Kanobi.