Latest posts

VMware vCenter Reduced Downtime Upgrade with Automatic Switchover

VMware vCenter RDU (Reduced Downtime Upgrade (or Update)) is a relatively new feature that allows you to update your vCenter to the next version with limited downtime, just like the name indicates. It works similarly to the process for upgrading from vCenter 7 to vCenter 8, which is also very similar to doing a fresh install of VMware vCenter.

In October 2021, VMware introduced the vCenter Reduced Downtime Upgrade feature. However, the feature was not available for on-premises vCenters. In September 2023, vSphere 8 Update 2 introduced the feature for on-premises vCenters. With the release of vSphere 8 Update 3 in June 2024, more features have been added, including one called automatic switchover, allowing the whole process to be even more seamless.

The magic that makes vCenter RDU work is the vCenter installer ISO. When you mount the ISO to your existing vCenter, the RDU process will create a new upgraded vCenter VM. Once that part is completed, it will transfer the settings from your current vCenter to the newly upgraded vCenter VM and cut you over. This process reduces the time that VMware vCenter is down and can also reduce some risks of in-place upgrades.

My blog post, Install VCSA Updates, covers the traditional method of upgrading VMware vCenter in-place.

In this post, I will show you step-by-step how to upgrade VMware vCenter using the Reduced Downtime Upgrade with Automatic Switchover.

Prerequisites

  • Backup of vCenter.
  • VMware vCenter ISO.
  • Temporary IP for the new upgraded vCenter VM.
  • Temporary root password for the new upgraded vCenter VM.

The Process

  • Upload the VMware vCenter Server Appliance ISO to a datastore in vCenter.
  • Attach the VCSA ISO to your current vCenter VM.
  • Click on your vCenter and select the Updates tab.
  • Under the vCenter Server section, click on Upgrade.
  • The process will check and confirm that your upgrade path is supported. If all is good, click Next.

In my example, I am upgrading from vCenter 8 Update 2b to vCenter 8 Update 3.

  • Confirm that you have a backup and click Next.
  • Click on Upgrade Plug-in to upgrade the vCenter Server Life-Cycle Manager plug-in.

I’m Going to VMware Explore

I am very excited about VMware Explore this year, as it will be my first time at VMware Explore and the first large tech conference I’ve attended in person.

VMware Explore is a conference about VMware products that VMware (technically Broadcom now) organizes. It used to be known as VMworld, but in 2022, it was renamed VMware Explore. VMware Explore usually takes place in Las Vegas and Barcelona.

The VMware Explore event that I am attending takes place from August 26th to August 29th in Las Vegas. What’s really cool about VMware Explore is that it covers a wide range of topics, from entry-level to very technical. One of the items at VMware Explore that I’m excited about is the hands-on labs, which will allow me to play with some of the VMware products I’ve only read about.

I’m eager to learn more about VMware Cloud Foundation, Tanzu Kubernetes, NSX, and so much more. I’m also very excited to network with other like-minded people and nerd out about all the technical things. I will also be able to meet other vExperts with whom I’ve only spoken with online.

There are over 400 sessions at VMware Explore, and here are a few of the sessions I plan on attending.

vCenter ESXi Config Backup Script

When using VMware vCenter, you may only occasionally need a configuration backup of each VMware ESXi host. However, there are some situations where having a config backup of each ESXi host is nice to have.

I didn’t want to back up each ESXi host manually, as it doesn’t scale well. Instead, I created a PowerShell script called vCenter ESXi Config Backup to do everything for me.

You can find the script on my GitHub. https://github.com/thedxt/VMware#vcenter-esxi-config-backup

Prerequisites

How It Works

The vCenter ESXi config backup script connects to VMware vCenter, uses vCenter to connect to each ESXi host, and takes a configuration backup for each host. Because the script uses vCenter, you don’t need to enable SSH on any of the ESXi hosts for the backup to work.

By default, the vCenter ESXi config backup script assumes you are not connected to vCenter and will prompt you to connect to vCenter. You can suppress this behavior if you are already connected to vCenter by setting the optional parameter named connected to the value of Yes.

The script checks to see if the backup folder you defined exists. If the folder does not exist, the script will create it.

Next, the vCenter ESXi config backup script enumerates all of the ESXi hosts in vCenter, it connects to each one and takes a configuration backup. The script outputs the backup into the folder you defined.

UniFi Network Application MongoDB Upgrade

If you have been using the UniFi controller for a very long time, there’s a chance you are running an older version of MongoDB. When Ubiquiti released version 8.1 of the UniFi network application server, they finally bumped up the supported MongoDB version from 4.4 to 7.0.

The MongoDB upgrade path only supports jumping one major version at a time, and some manual steps are needed. Yes, you could take a settings-only backup and reroll the whole setup, but where’s the fun in that?

In this post, I will show you step-by-step how to upgrade the Mongo database version from 4.4 to 7.0 for the Unifi network application server running on docker.

Prerequisites

  • Backup your UniFi config. My blog post, UniFi Network Server Settings Backup and Export, covers how to do this.
  • Running UniFi network application version 8.1 or newer.
  • The service name of the MongoDB for UniFi from your docker compose file.
  • The container name of your MongoDB for UniFi.

The Process

The entire process revolves around docker exec as we will execute commands directly into the Mongo database container.

You can follow this process on other UniFi controller setups that aren’t done with docker. However, I’ve only validated the process with on a docker setup, but the command structure would be similar.

To be extra safe during this process, I will bring down my UniFi docker stack and only bring up my UniFi database container.

I will run the command docker compose down to bring the UniFi controller offline. This helps ensure the UniFi network application doesn’t try to write to the database while the database is upgrading.

  • Bring up the MonogoDB container for UniFi with the following command. Replace Mongo_Service_Name with the service name for MongoDB from your docker compose file. sudo docker compose up Mongo_Service_Name -d

If you are using my docker compose setup from my blog post, UniFi Network Server with Docker, the MongoDB service name will be unifi-net-app-db

For me, the command will be sudo docker compose up unifi-net-app-db -d

Before starting any upgrades, let’s double-check which version of MongoDB is installed and what the compatibility level is currently set to. We will do that by using the MongoDB shell with the command mongo. We will tell Mongo that we will provide some commands from which we want the output by using the argument eval followed by our commands.

The command to check which version of MongoDB we are running and what our compatibility level is set to is the following command db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )

To run this against our Mongo docker container, we will use docker exec with a shell command followed by the mongo shell commands we want to run.

  • Run the following command to check what MongoDB version is running. Replace Mongo_Container_Name with the name of the MongoDB container that you are using with UniFi. sudo docker exec Mongo_Container_Name sh -c 'mongo --eval "db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )"'

For me, the command looks like

sudo docker exec unifi-net_DB sh -c 'mongo --eval "db.adminCommand( { getParameter: 1, featureCompatibilityVersion: 1 } )"'

Your output should be something like this

MongoDB server version: 4.4.29
{ "featureCompatibilityVersion" : { "version" : "4.4" }, "ok" : 1 }

The output above confirms that we are running MongoDB 4.4, and our feature compatibility is set to the same version.

If you get an error that mongo is not found, this is due to running a newer version of MongoDB that no longer supports the mongo command as that command has been deprecated in newer versions.

The replacement command for mongo is mongosh. However, the output differs from the old mongo command when gathering the compatibility levels.

UniFi Network Server Settings Backup and Export

When you self-host the UniFi Controller or the UniFi Network Application server, it’s a good idea to take a backup before making major changes. Backups are always a good thing to have.

In this post, I will show you step-by-step how to take a settings only backup of the UniFi Network Application server.

The Process

  • Login to the UniFi Network Server.
  • Click on Settings.
  • Click on System.

Omnissa Horizon Unmanaged VDI Black Screen Issue

I recently encountered a strange issue with Omnissa Horizon (formerly VMware Horizon) that resulted in a black screen presenting to the VDI users when they logged in. When you search for this issue, you will find a lot of information about various solutions to the blank screen issue. However, none of them solved the problem I was running into.

VMware Horizon black screen on login

In a traditional VMware Horizon setup, there can be several causes for the black screen issue. If your setup is VDI VMs managed by VMware Horizon, then the black screen issue you may have is likely covered in my friend Stephen’s blog post, VMware Horizon Blank Screen.

His post did not cover my black screen issue. To replicate the problem I encountered, you need to have VDI VMs in VMware vCenter, and VMware Horizon configured not to use the VMware vCenter integration.

While having your VMware VDI VMs unmanaged by VMware vCenter in VMware Horizon is still a supported configuration, it is rare. However, there can be some edge cases where, even though your VDI VM exists in vCenter, you don’t want to let Omnissa Horizon access vCenter. My blog post, Omnissa Horizon Desktop Pool without vCenter, covers how to configure this.

The Why

The black screen issue presents itself when Omnissa Horizon does not have the integration to VMware vCenter to manage the VMware VDI VM. Due to this, a required setting in vCenter for the VDI VM is not configured when adding the VM to a desktop pool, as Horizon has no access to vCenter to make the change. That setting is called screen DMA (Direct Memory Access).

In version 6 of VMware vCenter, the default setting for screen DMA was changed to disabled by default. Screen DMA is required to enable proper streaming of the VDI desktop to the end users.

In this post, I will show you step-by-step how to enable the screen DMA setting to fix the black screen issue on a VMware Horizon VDI VM in VMware vCenter when the Omnissa Horizon integration to VMware vCenter is not used.

The Fix

Confirm The Issue

  • Login to VMware vCenter and select the VDI VM experiencing the black screen issue.
  • Click on Edit Settings.
  • Click on Advanced Parameters.

If you are on vCenter version 7, click on VM Options, then expand Advanced, and click on Edit Configuration.

Windows Autopilot Hardware ID

Windows Autopilot uses several mechanisms to function. A common one is the device’s hardware hash, also called the hardware ID. There are several ways to gather the hardware ID from a device to use in Windows Autopilot.

In this post, I will show step-by-step how to get the hardware ID from a system with PowerShell and how to import it into Windows Autopilot. The process is the same regardless of whether the system is a virtual machine or a physical machine, and it doesn’t matter if the system is running Windows 10 or Windows 11.

Get Hardware ID

  • Open PowerShell as Admin.
  • Run the following command to enable TLS 1.2 [Net.ServicePointManager]::SecurityProtocol = [Net.SecurityProtocolType]::Tls12
  • The following command will create a folder named HWID on the root of C for the hardware ID file New-Item -Type Directory -Path "C:\HWID"
  • Switch into that directory with the following command Set-Location -Path "C:\HWID"
  • Add C:\Program Files\WindowsPowerShell\Scripts to the path environment variable by running the following command $env:Path += ";C:\Program Files\WindowsPowerShell\Scripts"

SSL Inspection

Inspecting TLS/SSL traffic on corporate networks is very common, as over 80% of all web traffic is encrypted. If you aren’t performing TLS/SSL traffic inspection, you are potentially leaving your network exposed.

The simple act of inspecting SSL connections helps reduce your attack surface. SSL inspection can make it harder to establish malicious outbound connections, such as malware trying to communicate with the C2 (Command and Control) server.

To reduce the attack surface further, I like to deploy a block on common file types that could be malicious, such as EXEs. If SSL inspection isn’t performed, any EXE downloaded via SSL won’t be blocked.

In this post, I will go over the basics of SSL inspection. I won’t go into details about certificate chains. If you want to understand more about certificate chains, my post Certificate Chain goes into more detail.

An SSL inspector sits in the middle of all SSL traffic, inspecting everything by decrypting and re-encrypting encrypted traffic on the fly. At its core, SSL inspection is like a MitM (man-in-the-middle) attack you control.

To inspect SSL traffic, the SSL inspector needs to decrypt it first. It can do this by tricking clients into communicating with it instead of the actual website, and the SSL inspector will communicate with the website on the client’s behalf. An SSL inspector is also a proxy.

To keep everything secure even when decrypting SSL traffic, the SSL inspector needs to be its own intermediate CA to resign and re-encrypt everything to send back to the client. All public-facing websites or web applications have certificates issued by a public root CA.

The SSL inspector can’t just issue certificates on behalf of a public root CA, as that would break the chain of trust. Even if you reach out to an intermediate CA, they will not sell you a certificate that could allow you to issue certificates on their behalf, even if it’s just for your internal network, as that also breaks the chain of trust.

To deploy an SSL inspector, you have two options. The first option is to deploy the certificate from your SSL inspector to all devices, making it and the certificates it issues trusted. The second option is to deploy your own internal root CA and allow your SSL inspector to be a subordinate CA (also known as an intermediate CA) and issue certificates on behalf of your internal root CA.

Broken Certificate Chain

Certificate chain issues are extremely widespread and more common than you think. You start to see how many websites have broken certificate chains when you start performing SSL inspection on a network.

If you want to read more about certificate chains, my post Certificate Chain goes into more detail.

My website, thedxt.ca, has a server certificate for thedxt.ca, issued by the intermediate CA GTS CA 1P5. The intermediate CA GTS CA 1P5 certificate is issued by the public root CA GTS Root R1.

Correct certificate chain for thedxt.ca

The image above is an example of a correctly formed certificate chain. Each item is issued by the previous one, and because the root CA is trusted, the entire chain is trusted.

In this post, I will go over what can cause a broken certificate chain and how to verify your certificate chain properly.

There are two main types of broken certificate chains. First is the malformed certificate chain or the out of order certificate chain. The second is an incomplete certificate chain.

Malformed Certificate Chain

If my certificate chain for my website, thedxt.ca, was configured in the following order:

  • A server certificate for thedxt.ca.
  • The public root CA GTS Root R1 certificate.
  • The intermediate CA GTS CA 1P5 certificate.

This order would cause a broken certificate chain as the certificate chain would be out of order and considered malformed, which can cause issues.

Out of order certificate chain for thedxt.ca

The image above is an example of a malformed certificate chain, as each item is not issued by the previous one.

Incomplete Certificate Chain

Another common issue that causes broken certificate chains is an incomplete certificate chain. An incomplete certificate chain means a certificate is missing in the certificate chain. Usually, this is when one or more intermediate CA certificates are missing. It’s an issue because the path to the root CA can’t be completed.

Incomplete Certificate Chain for thedxt.ca

The image above is an example of an incomplete certificate chain, as there are no links to follow for the path to the root CA.

The Why

Many people don’t realize they have a broken certificate chain, as most things will just work even when the certificate chain is broken. Web browsers try to fix broken certificate chains for websites so users don’t run into issues.

A very common practice for checking if your SSL/TLS certificate is working on a public-facing item is to use a web browser to browse to the item and see if you get any certificate issues. Because web browsers auto-fix broken certificate chains, you end up with a false sense that everything is working correctly.

If you can’t trust a web browser to check if your certificate chain is correct, how do you check it?