Conflicting VIBs when updating ESXi using Custom Images/Offline Bundles

As every year, some of my customers use the weeks after Christmas to update their environments. Nearly all of them run their ESXi hosts with vendor-specific Custom Images that provide additional drivers or agents over the standard VMware image. Unfortunately there are almost always problems with conflicting VIBs when they are updated. Of course it was the same this time. In my case, this time it was about a custom image from Fujitsu.

Dependency error when updating ESXi with the custom offline bundle

To perform the update successfully the problematic VIB must be removed. The necessary steps for this I would like to point out below.

Continue reading „Conflicting VIBs when updating ESXi using Custom Images/Offline Bundles“

How to migrate running VMs to different datastores without Storage vMotion

Today I learned something very useful that I want to share with you. We have some smaller customers from the SMB segment. Here we often find small vSphere clusters with two or three ESXi hosts. These are usually licensed with vSphere Essentials Plus to use the benefits of vSphere High Availability (vSphere HA). The functions contained in this bundle are quite sufficient for regular operation. One function that is missing in vSphere Essentials Plus is the ability to move running VMs from one Datastore to another using Storage vMotion. In this post I want to share with you a way on how you can still move your running VMs between datastores.

Continue reading „How to migrate running VMs to different datastores without Storage vMotion“

Datadomain replication fails with „send queue: sending aborted by caller“

Last friday I struggled with some of our Dell EMC Datadomain (DD) Systems. I tried to establish a MTree replication between two identical systems.
The first few steps worked without an issue, so I was confident to get into the weekend early. But then things changed.

Ambiguous error message after replication setup

But first things first. Let’s start with the step by step process on how I enabled the replication.

Continue reading „Datadomain replication fails with „send queue: sending aborted by caller““

Configure the vCenter Server Login message

As a solution provider, my company often installs VMware vSphere environments for customers who do not administer the solution themselves. In addition to restricting user permissions, we have been working with individual login messages in the vCenter Server for some time now. We usually use the login message to remind the customer that the environment is managed by our company and that any changes must be approved in advance.

This post gives you a short overview on how to configure the login message using the vCenter Server administration interface.

Continue reading „Configure the vCenter Server Login message“

Update to ESXi 6.7 U3 using esxcli fails with error „Failed to setup upgrade using esx-update VIB“

After the gerneral availability of ESXi 6.7 U3, I decided it is time to update my homelab. I used the integrated vSphere Update Manager (VUM) for my nested vSAN Cluster without any issues. During my VCAP-DVC Deploy preparation earlier this year I used an old Intel NUC system to deploy an additional vCenter Server Appliance (VCSA) in order to play around with Enhanced Linked Mode (ELM). I wanted to have the VCSAs in different Layer 3 networks, so I copied the design from the nested cluster and installed a nested router alongside with a nested ESXi host. The nested ESXi is used to host the second VCSA. Due to this design I can not use the integrated VUM of the second VCSA to update this standalone ESXi host.

The virtual machines (VMs) on the standalone ESXi host are not very important, so I decided to shut them down together with the VCSA and update the nested ESXi host via CLI. For details on how to update ESXi 6.x using the CLI, take a look at one of my previous blog posts here. I used this method quite some time so I was certain to have no issues with the update. But this time it was different, the update failed.

esxcli software profile update failes
Continue reading „Update to ESXi 6.7 U3 using esxcli fails with error „Failed to setup upgrade using esx-update VIB““

Using PowerCLI in preparation of storage migration

In the last weeks I accompanied a customer during the preparation of his storage migration. Among other things, we came up with the usual questions on this subject:

  • On which datastores are my virtual machines located?
  • Are there virtual machines with snapshots?
  • Are CD/DVD drives still connected to certain VMs?

That’s why I played around with PowerCLI and wrote some scripts to get answers to these questions. I leave the setup of the vCenter connection and the parameterization of individual commands to each of you and concentrate here on the central command(s).

List datastores of certain VMs

This small command returns a list of VMs sorted by the name of the VM in a specified cluster, including their datastore and the VM folder.

Get-Cluster -Name <ClusterName> | Get-VM | Select-Object Name,@{N=”Datastore”;E={[string]::Join(‘,’,(Get-Datastore -Id $_.DatastoreIdList | Select-Object -ExpandProperty Name))}},@{N=”Folder”;E={$_.Folder.Name}} | Sort-Object Name

The following example shows the output of the command.

List the datastores of certain VMs

List VMs with Snapshots

The second command creates a list of VMs with snapshots, sorted by name, with information about the name of the snapshot, the date the snapshot was created, and the size of the snapshot.

Get-Cluster -Name <ClusterName> | Get-VM | Get-Snapshot | Sort-Object VM | Format-Table VM,Name,Description,Created,@{ n="SizeGB";e={[math]::round($_.SizeGB,0)}} -AutoSize

Find below a screenshot of the output.

List of VMs with Snapshots

List VMs with CD/DVD drive attached

The last command lists all VMs with a connected CD/DVD drive and the path to the inserted ISO image.

Get-Cluster -Name <ClusterName> | Get-VM | Get-CDDrive | Select-Object @{N="VM";E="Parent"},IsoPath | Where-Object {$_.IsoPath -ne $null} | Sort-Object VM

The following is an example of the output of the command.

List of VMs with attached ISO images


PowerCLI makes it really easy for an administrator to find out what dependencies exist prior to storage migration. With the commands shown above, you can quickly and easily get an overview of the current state. If you wish, you can also extend or rewrite the commands accordingly and, for example, completely automatically remove all attached ISO images from the VMs.

My VCAP-DCV Deploy experience

After attending VMware Empower 2019 Europe and receiving an exam voucher for a VCAP Deploy exam of my choice I decided to sit the VMware Certified Advanced Professional 2018 – Data Center Virtualization Deployment (3V0-21.18) exam two weeks ago. Due to the expiry date of the voucher I was forced to take the exam prior to the 15th june which meant that there wasn’t much preparation time after returning from Empower.

Because I went down the VCAP-DCV Design track last year, I was already aware that the Deloy exam is a practical exam based on VMware Hands-on Labs (HOL) and and not comparable with the usual multiple-choice exams. The following steps should give you a brief overview of my preparation and my experience with the exam.

Step 1: Read the blueprint

The blueprint or Exam Preparation Guide as it is called now is the single source of truth regarding the scope of the exam. After you read it, read it again and make sure you are well aware of the exam sections outlined in the guide. As described in the paragraph about the Minimally Qualified Candidate (MQC) the exam requirers practical knowledge about complex virtualized environments across multiple locations and technologies. So be prepared to work with such an environment and the corresponding dependencies and constraints.
Another very important fact is the section Product Affiliation. The exam is associated with VMware vSphere 6.5. Some features or options you might know from vSphere 6.7 might not be available or behave differently in the exam.

Step 2: Familiarize yourself with the HOL GUI and operation

Because the exam is based on the HOLs it is a very good idea to have a look at the GUI and operation of the HOLs prior to the exam. You do not want it to be the first time you see the GUI and limitations when sitting in the exam room. The screen size is limited, some keyboard shortcuts might not work and quite a bit more. Due to all these restrictions, VMware has published an own Exam UI Guide. You’ll also find a good video on this topic on YouTube.

Step 3: Know your stuff and the documentation

Sounds like a phrase, but it’s true. You don’t have to know everything 100%, but knowing which functions and options are hidden in which menu is essential for this exam. There are plenty of good blog entries with exam preparations and test exams that you can find with your favorite search engine. Regardless of which one you use, I recommend that you first work out your own solution before you look at the approach or solution in the guide.

During the exam you will not have access to the internet, not even to the VMware Knowledge Base. Nevertheless, you will find a very good selection of VMware documentation as PDFs on the desktop of your test system. Get to know the different documentations and their contents in advance and use them like you would normally use a search engine. Please don’t waste any time and read the complete documentation during the exam.

Step 4: Keep calm and manage your time

You are given 205 minutes (+30 minutes for non-native English speakers) for 17 exam questions. That’s about 12 minutes (14 minutes for non-native speakers) per question. That sounds very generous at first, but you’ll need the time. Some of the questions consist of multiple parts, others may require vSphere actions that take longer than others. Don’t waste time on questions that you already know when you read them that you don’t have a starting point or solution right away. Use the washable paper you get to write down such questions and save them for later. A good tip I read was to write the numbers from 1 to 17 on the left side of the paper below each other. Behind it a short keyword so that you still know after 3 hours what the question was about without having to browse through the manual. I also used this technique to check off questions I had already worked on and keep track of them during the exam.

Step 5: Patience until the evaluation of the exam

Because these are not the usual multiple-choice questions but a practice-oriented exam, you will not get your result directly after the exam. It took a week to get my badges from Acclaim. I haven’t received an e-mail or similar with my certification results yet.


It felt like it was the most difficult exam I’ve ever had in my career. Regardless of the challenge, it was also by far the exam I enjoyed the most. I really liked solving problems and performing tasks in a „real“ environment and not just answering theoretical questions about a particular product. Someone on Twitter found a great description for the exam: „Like a day with a new customer. Here is our environment and now you have four hours to solve the problems.“ At the end of the VCAP-DCV Deploy adventure, however, the really nice VCIX badge awaits – assuming the successful VCAP-DCV design certification.


Using Veeam with vSphere Tags and individual credentials

This week I reworked the backup of a customer. The customer is using Veeam Backup & Replication to back up their VMware vSphere infrastructure. For whatever reason (the customer couldn’t give me the exact reasons), a single backup job was created for each VM.

Single VM per Veeam backup job

Apart from the poor deduplication rate, these over 100 backup jobs were anything but easy to manage and monitor. We looked at different ways to group the backups and after a short time on the whiteboard we came to a first solution: VMware vSphere Tags.
These tags exist since vSphere 5.1 and are the successors of the Custom Attributes.
Within Veeam Backup & Replication, they can be used as selection criteria for the VMs to be backed up in a backup job.

First we had to create the appropriate category (Backup) and tags within vSphere. Then we were able to use the newly created tags as selection criteria within the backup job.

Using vSphere Tags as selection criteria in a Veeam Backup & Replication backup job

The customer operates some hosted VMs for external customers. The VMs to be backed up are in different domains with different credentials. For this reason, we faced the challenge of having to use different credentials for different VMs within a single job. This is generally no problem for Veeam Backup & Replication. But when working with container objects like tags (but also clusters, folders, datastores, etc.), the functionality is a bit tricky to find. If we look at the settings for Credentials under Guest Processing, we first see only the container object for which we can assign certain credentials.

After the job has been created, only the container object is available for configuration

But if we click the Add… button and switch to VMs and Tags again, we have the possibility to expand our selected tag and see all VMs to which our tag is currently assigned.

Selection of VMs for which individual access data are to be assigned

If we now select all VMs for which we would like to assign individual credentials and press Add, we will be able to store the correct credentials in the next step.

Assignment of individual credentials to individual VMs


Even if it is not immediately recognizable at first glance, the assignment of individual credentials works not only with the dedicated selection of individual VMs, but also with the use of container objects such as tags. This enables a highly automated implementation of backup jobs even in complex environments with different credentials. New VMs only need to be given the appropriate tag and the backup administrator must receive the appropriate credentials if more than just backing up the entire VM is required.