Every year, before Christmas, the update window is coming up for some of my customers. One of these customers was due for minor VMware vSphere updates today. Actually not a big deal even if the customer has not activated vSphere Distributed Resource Scheduler (DRS) on his cluster due to missing licenses. As in previous years, the task was to manually evacuate the individual ESXi hosts one by one and then standardize them via the vSphere Update Manager (VUM). At the beginning everything was running without any issues until I wanted to evacuate the vCenter Server Appliance (VCSA) as the last VM of the host. For whatever reason, the migrate function was grayed out in the context menu of the VM.
So far, vMotion has actually never caused any problems in this cluster. So it was once again time for a little round of troubleshooting.
As a first step, I checked all other VMs to see if it was a general problem in the cluster. But I was able to move all other VMs between hosts without any issues. After that I moved an already evacuated VM back to the ESXi host with the affected VM to check if it could be a host related problem. But both the migration on and the migration off the host went without problems. After my tests, the affected VM was again running alone on the host designated for the update. I briefly considered restarting the VM, but then abandoned it because we had not planned for any downtime for the applications.
So I did a short internet research and found what I was looking for pretty quickly in the VMware Knowledge Base. KB1029926 describes this behaviour as the cause of an entry in the vCenter Server database table vpx_disabled_methods that was not deleted after a backup. Since I had not found any errors in the nightly backup in preparation for the upgrade, I briefly checked with the customer’s administrators to see if there had been any errors or warnings in the backup of the corresponding VM recently. And in fact, about two weeks ago, there was a snapshot deletion error for this very VM. The snapshot was deleted manually the following day. Since then, there have been no further abnormalities with the backup of the VM. After the behavior was confirmed, I immediately worked on the solution described in the KB article, which I would like to outline briefly below.
Step 1: Identify the MOB ID of the affected VM
In the vSphere Client, select the affected VM and look at the URL in your browser’s address bar. Be on the lookout for this section in the URL:https://ExamplevCenterFQDNorIP/ui/#?extensionId=vsphere.core.vm.summary&objectId=urn:vmomi:VirtualMachine:vm-26:fa9a744e-07bf-472d-b4a1-462dd6a4badf&navigator=vsphere.core.viTree.hostsAndClustersView
In my case the MOB ID of the VM is vm-26.
Step 2: Access the Virtual Machine Operations
Go to the following URL in your browser and then log in with your administrative SSO credentials (administrator@yourssodomain.local):https://ExamplevCenterFQDNorIP/mob/?moid=AuthorizationManager&method=enableMethods
You will then land on the following page:
Step 3: Edit/fill in the required parameters and invoke the method
Replace within the first parameter (entity) MOID with the MOB ID of the affected VM determined in the first step. Insert exactly the following value for the second parameter (method):<method>RelocateVM_Task</method>
After you have edited or filled in the parameters, your mask should look something like this.
Now you are ready to invoke the method by clicking on Invoke Method.
Step 4: Refresh vCenter and check VM for migration options
After you have successfully executed the method you have to refresh thevCenter Server view in the browser and then check if you can now migrate the VM as usual with vMotion. For me it worked right away.
Wrap-Up
So far, I have never had to deal with this issue. But I am always happy to learn something new and to see that there are vendors (in this case VMware) with well maintained and publicly available support databases.
If a migration is not possible even after refreshing the vCenter Server view in the browser, I recommend to restart the VCSA. I have not had to test it myself, but at least the last step of the Knowledge Base article refers to restarting all vCenter Server services. So it is certainly worth to wait for the restart of the vCenter Server before opening a support call with VMware.
