Lately one of my customers had some issues with his VMware vSphere environment where the semi-outage of one ESXi took down quit a huge amount of his VMs. After fixing the issue in his productive infrastructure and bringing back his IT everything seamed working again. Some days later we noticed Errors in some of his Veeam Backup Jobs. Strangely not all VMs where affected and those affected had no direct correlation (e.g. same ESXi host, same datastore, some network, etc.).
The error message shown inside the Veeam Backup & Replication console was an error I have not seen before:
Error: DiskLib error: .A file error was encountered -- Failed to read the file
Error Failed to retrieve next FILE_PUT message. File path: [[<DATASTORE>] <VMFOLDER>/<VM>.vmx]. File pointer: . File size: .
Today I demonstrated some of the new features of VMware vSAN 7.0 Update 3 (7.0 U3) related to 2-node deployments to one of our customers who uses this topology extensively. We have focused particularly on improvements to the resilience of 2-node clusters and the witness Host. You can find a short video about the features here. An extensive list of the new features and the release notes can be found here and here.
Another new feature I noticed but forgot zu demonstrate to the customer ist the new vSAN cluster shutdown and restart operation. I already tested the shutdown and restart feature with my 4-node cluster but as the demo was finished I decided to give the feature a try with the 2-node cluster and the external witness appliance.
Every year, before Christmas, the update window is coming up for some of my customers. One of these customers was due for minor VMware vSphere updates today. Actually not a big deal even if the customer has not activated vSphere Distributed Resource Scheduler (DRS) on his cluster due to missing licenses. As in previous years, the task was to manually evacuate the individual ESXi hosts one by one and then standardize them via the vSphere Update Manager (VUM). At the beginning everything was running without any issues until I wanted to evacuate the vCenter Server Appliance (VCSA) as the last VM of the host. For whatever reason, the migrate function was grayed out in the context menu of the VM.
So far, vMotion has actually never caused any problems in this cluster. So it was once again time for a little round of troubleshooting.
To get some more flexibility in my Homelab I added another domain controller (Active Directory, DNS and DHCP). Unlike my first domain controller, which runs directly on the physical ESXi host (details can be found here), I installed the second domain controller inside the nested vSAN cluster. After configuring all services I wanted to use the new domain controller as an additional DNS server in my VMware vSphere environment. So I quickly adjusted the network and NTP settings of the vCenter Server appliance and the ESXi hosts and then everything should be fine. So far so good. No problem until then. Shortly after I added the additional domain controller in all locations a warning message appeared in my vSphere cluster.
After my „little“ homelab outage last year and the delivery of a new SSD I found some time to redeploy the nested cluster quite some time ago. During the preparation to my VCAP-DCV Deploy exam I deployed a second VCSA (vCenter Server Appliance) on my old Intel NUC and joined them to a single SSO domain to learn and try different things in the linked mode setup. That’s the reason why I received the „Could not connect to one or more vCenter Server Systems: https://<vcsaFQDN>:443/sdk“ every time I logged in to the second VCSA. Because I planned to redeploy the nested environment using the same IPs/FQDNs I wanted to make sure the orphaned VCSA is cleanly removed from the SSO configuration. This week one of my customers asked me for help with the same problem.A quick search and I found the following VMware KB article (again): Using the cmsso command to unregister vCenter Server from Single Sign-On (2106736). This time I decided to write a short blog post on the topic.
Most of my customers now use tag-based backup with Veeam Backup & Replication to protect their business-critical applications and services. This ensures that they no longer need to perform any configuration within the backup software to protect their workloads. Only the individual adjustments of the Guest Credentials for Application-Aware Processing have to be done in the Veeam Console. The added value you have by doing this I have already covered in another blog post. In the ever accelerating IT world and the changes that come along with it, it is extremely important for my customers to be able to make fast and reliable statements about the data backup status of certain systems. Since we use vSphere tags to perform nearly all of our backup administration from within vCenter, it would be consistent to have the appropriate status information available at this location as well. This is where the Notification Settings within the Advanced Backup Settings come into play. How you can configure simple status updates in vCenter without installing additional plug-ins or tools, and what you should consider when doing so, I‘ ll show you below.
In one of their latest updates for macOS Catalina Apple has introduced some new requirements for the acceptance of SSL certificates. The changes are documented here: https://support.apple.com/en-us/HT210176. This means that pages without a corresponding certificate are no longer accessible in Google Chrome. Unfortunately, the default vCenter Server certificate is one of the affected certificates. Unlike other certificate warnings, this error cannot be easily bypassed using Advanced options.
In the following I would like to show you how you can temporarily work around this issue.
As every year, some of my customers use the weeks after Christmas to update their environments. Nearly all of them run their ESXi hosts with vendor-specific Custom Images that provide additional drivers or agents over the standard VMware image. Unfortunately there are almost always problems with conflicting VIBs when they are updated. Of course it was the same this time. In my case, this time it was about a custom image from Fujitsu.
To perform the update successfully the problematic VIB must be removed. The necessary steps for this I would like to point out below.
Today I learned something very useful that I want to share with you. We have some smaller customers from the SMB segment. Here we often find small vSphere clusters with two or three ESXi hosts. These are usually licensed with vSphere Essentials Plus to use the benefits of vSphere High Availability (vSphere HA). The functions contained in this bundle are quite sufficient for regular operation. One function that is missing in vSphere Essentials Plus is the ability to move running VMs from one Datastore to another using Storage vMotion. In this post I want to share with you a way on how you can still move your running VMs between datastores.
As a solution provider, my company often installs VMware vSphere environments for customers who do not administer the solution themselves. In addition to restricting user permissions, we have been working with individual login messages in the vCenter Server for some time now. We usually use the login message to remind the customer that the environment is managed by our company and that any changes must be approved in advance.
This post gives you a short overview on how to configure the login message using the vCenter Server administration interface.