Every year, before Christmas, the update window is coming up for some of my customers. One of these customers was due for minor VMware vSphere updates today. Actually not a big deal even if the customer has not activated vSphere Distributed Resource Scheduler (DRS) on his cluster due to missing licenses. As in previous years, the task was to manually evacuate the individual ESXi hosts one by one and then standardize them via the vSphere Update Manager (VUM). At the beginning everything was running without any issues until I wanted to evacuate the vCenter Server Appliance (VCSA) as the last VM of the host. For whatever reason, the migrate function was grayed out in the context menu of the VM.
So far, vMotion has actually never caused any problems in this cluster. So it was once again time for a little round of troubleshooting.
Today’s Homelab session dealt with the creation of a short customer demo of the Veeam Backup & Replication functionality SureBackup. As I have already implemented several SureBackup jobs for other customers, I was confident that I could quickly finish configuring the environment. For those who have not worked with SureBackup before, Veeam provides an excellent guide in their Help Center. You can find this guide here. Unfortunately the whole thing did not work out as expected. Already at the beginning I made a crucial mistake which made the creation of the demo a nerve-wracking adventure. More on this in a moment. First of all, for those of you who have no idea how the creation of a SureBackup job works, I would like to give a short outline.
To get some more flexibility in my Homelab I added another domain controller (Active Directory, DNS and DHCP). Unlike my first domain controller, which runs directly on the physical ESXi host (details can be found here), I installed the second domain controller inside the nested vSAN cluster. After configuring all services I wanted to use the new domain controller as an additional DNS server in my VMware vSphere environment. So I quickly adjusted the network and NTP settings of the vCenter Server appliance and the ESXi hosts and then everything should be fine. So far so good. No problem until then. Shortly after I added the additional domain controller in all locations a warning message appeared in my vSphere cluster.
Recently I got the chance to redesign the existing backup environment of a customer. The customer has been successfully using Veeam Backup & Replication for years, but the environment has been growing over the years and therefore the backup environment has been adjusted from time to time. After an unpleasant data loss mid-year, the customer decided to give us the task of revising his backup environment.
As part of the design, we decided to leave all components of the backup environment outside the customer’s productive Active Directory domain and only connect where absolutely necessary. In general this works without problems, only the installation and configuration of additional, Windows-based Managed Servers to use them for example as remote backup repository requires some additional attention. If we try to add the Managed Server in the Veeam Backup & Replication Console and do not use the default Account Administrator (UID 500), the installation of the necessary Veeam Services fails due to Remote UAC being enabled by default.
As we do not want to use a standard account in our design (a general recommendation), we use dedicated service accounts and therefore rely on the second solution. For those of you who don’t want to read the Microsoft article here are the steps you need to perform on the managed server.
Launch regedit (Click Start, type regedit and press ENTER).
Navigate to the following register entry: HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System
Create a new DWORD value (32-Bit) with the following name: LocalAccountTokenFilterPolicy
Assign the new DWORD the value 1.
After creating the registry entry, adding the Managed Server and installing the Veeam Services works without any problems.
Right before my vacation I reached a goal, which was actually already on my list for 2019. I tried and managed to get the Veeam Certified Engineer (VMCE) certification in the 2020 edition. This was also my first online proctored exam. So in the following I would like to tell you why it took so long since my Veeam Configuration and Management training in 2019 and what experiences I made with the 2020 exam in the „Remote Edition“.
After my „little“ homelab outage last year and the delivery of a new SSD I found some time to redeploy the nested cluster quite some time ago. During the preparation to my VCAP-DCV Deploy exam I deployed a second VCSA (vCenter Server Appliance) on my old Intel NUC and joined them to a single SSO domain to learn and try different things in the linked mode setup. That’s the reason why I received the „Could not connect to one or more vCenter Server Systems: https://<vcsaFQDN>:443/sdk“ every time I logged in to the second VCSA. Because I planned to redeploy the nested environment using the same IPs/FQDNs I wanted to make sure the orphaned VCSA is cleanly removed from the SSO configuration. This week one of my customers asked me for help with the same problem.A quick search and I found the following VMware KB article (again): Using the cmsso command to unregister vCenter Server from Single Sign-On (2106736). This time I decided to write a short blog post on the topic.
Most of my customers now use tag-based backup with Veeam Backup & Replication to protect their business-critical applications and services. This ensures that they no longer need to perform any configuration within the backup software to protect their workloads. Only the individual adjustments of the Guest Credentials for Application-Aware Processing have to be done in the Veeam Console. The added value you have by doing this I have already covered in another blog post. In the ever accelerating IT world and the changes that come along with it, it is extremely important for my customers to be able to make fast and reliable statements about the data backup status of certain systems. Since we use vSphere tags to perform nearly all of our backup administration from within vCenter, it would be consistent to have the appropriate status information available at this location as well. This is where the Notification Settings within the Advanced Backup Settings come into play. How you can configure simple status updates in vCenter without installing additional plug-ins or tools, and what you should consider when doing so, I‘ ll show you below.
Yesterday I had a scheduled update of his Veeam Backup & Replication installation with one of my customers. We planned to go from version 9.5 Update 4b (220.127.116.1166) to version 10 GA (10.0.0.4461).
As usual, I created an encrypted configuration backup before the update for safety reasons. How this works and why you should encrypt the configuration backup you can read here and here. I prefer to be a little more cautious at this point, before I have the trouble in hindsight. However, I did not need the configuration backup. The update went smoothly and without problems.
Since I carried out the update during the day, it was not possible, in agreement with the customer, to perform a complete backup run directly after the update. Therefore, I did a short functional test using the Quick Backup capabilities of Veeam Backup & Replication. There were no problems here either.
Today the customer called and reported about failed backup jobs. So I looked into it:
In one of their latest updates for macOS Catalina Apple has introduced some new requirements for the acceptance of SSL certificates. The changes are documented here: https://support.apple.com/en-us/HT210176. This means that pages without a corresponding certificate are no longer accessible in Google Chrome. Unfortunately, the default vCenter Server certificate is one of the affected certificates. Unlike other certificate warnings, this error cannot be easily bypassed using Advanced options.
In the following I would like to show you how you can temporarily work around this issue.
As every year, some of my customers use the weeks after Christmas to update their environments. Nearly all of them run their ESXi hosts with vendor-specific Custom Images that provide additional drivers or agents over the standard VMware image. Unfortunately there are almost always problems with conflicting VIBs when they are updated. Of course it was the same this time. In my case, this time it was about a custom image from Fujitsu.
To perform the update successfully the problematic VIB must be removed. The necessary steps for this I would like to point out below.