We use cookies on this page to better serve you. You can find more information about cookies here.

Latest posts

ESXi Config Restore Bug
by

While I was looking into various ways to restore an ESXi config backup I came across a bug.

If you read VMware’s documentation about how to restore an ESXi config backup (you can find that here) you will see that it is full of references saying the build numbers must match.

The bug is that you can restore an ESXi config backup even if the build numbers don’t match. Which according to the VMware documentation should not be possible.

Even though it is possible to restore an ESXi config backup when the build numbers don’t match, I do not recommend doing this as there has to be a reason why VMware says that the build numbers must match.

In my testing I was able to replicate the bug in ESXi 7 and ESXi 8. I even went all the way back to ESXi 6.7 which had some interesting findings. I didn’t fully test everything in ESXi 6.7 as general support has ended on that version.

Here are my findings and how to replicate the bug.

Generating Backup Files

In order to test everything as much as possible I wanted to have a few ESXi config backup files. This is how I generated the ESXi config backup files that I used in my testing.

  • Fresh install of ESXi 7 Update 2e Build Number 19290878.
  • Took an ESXi config backup from ESXi 7 Update 2e Build Number 19290878.
  • Fresh install of ESXi 7 Update 3d Build Number 19482537.
  • Took an ESXi config backup from ESXi 7 Update 3d Build Number 19482537.
  • Fresh install ESXi 7 Update 3k Build Number 21313628.
  • Took an ESXi config backup from ESXi 7 Update 3k Build Number 21313628.

Restoring from ESXi 7 Update 3d to ESXi 7 Update 3k

Once ESXi 7 Update 3k Build Number 21313628 was installed I tried to restore an ESXi config backup that was taken from ESXi 7 Update 3d Build Number 19482537.

When I do the restore using the vim-cmd hostsvc/firmware/restore_config method it works.

No errors when restoring ESXi 7 U3d to an install of ESXi 7 U3k using the restore_config method

Due to the fact that the restore worked I then reinstalled ESXi 7 Update 3k Build Number 21313628 again.

…
Read more
IT
BugESXiRabbit HoleVMware

ESXi Build Number without vCenter
by

Knowing your ESXi Build Number can be very useful. It’s really easy to do with vCenter. Without vCenter it’s not as straight forward. Here are a few ways to get your build number when you don’t have vCenter.

Console

If you have access to the console of the ESXi host via IPMI or iLO or iDRAC or physical access, you can get your ESXi build number right from there, you don’t even need to login.

Help Menu

You can also get your ESXi build number right from the Help menu in the Web UI.

  • Login to the Web UI of your ESXi host
  • Click on Help > About

You will now get a screen that show you your ESXi build number.

It should look something like this

In this example we know that my ESXi build number is 19482537.

SSH

  • Enable SSH on your ESXi Host by right clicking on the host and selecting Services > Enable Secure Shell
  • Login to your ESXi host with SSH
  • Enter the following command vmware -v

You will get an output that looks something like this

VMware ESXi 7.0.3 build-19482537

In this example we know that my ESXi build number is 19482537.

ESXi Config Backup File

You can also get your ESXi build number from an ESXi config backup file, which can be helpful if you want to know which ESXi build number was installed when a backup was taken.

To do this we will need something that can open a tgz archive. I like to use 7-Zip.

…
Read more
IT
Build NumberESXiHow ToVMware

Cisco Aironet Won’t Connect to Wireless LAN Controller
by

I ran into an issue where some older Cisco Aironet APs (Access Points) stopped connecting to a Cisco WLC (Wireless LAN Controller). No config changes had been made and some of the Cisco Aironet APs would connect and some wouldn’t. All of them were the same model, the Cisco Aironet APs were able to ping the Cisco WLC and vice versa.

What happened is the Cisco MIC (Manufacture Installed Certificate) expired and the default setup of a Cisco WLC is to reject any Cisco Aironet AP with an expired MIC.

It looks like this could impacts every Cisco WLC when used with older Cisco Aironet APs that have an expired Cisco MIC. Cisco has a Field Notice about this issue, you can read it here FN63942.

Any Cisco Aironet AP that was manufactured from July 18, 2005 until 2017 will have a Cisco MIC that expires 10 years after the manufacture date. There seems to be no way to replace or renew that Cisco MIC, this will keep being an issue that could randomly show up until 2027 when all of them should be broken.

The reason some of my Cisco Aironet APs worked and some didn’t is because they were manufactured at different times even though they have the same model number.

The fix is super quick we just need to tell the Cisco WLC to ignore expired Cisco MICs.

…
Read more
FixesIT
AironetCertificatesCiscoFixHow ToLogsNetworkingWiFiWireless LAN Controller

Cisco UCS Upgrade with Firmware Auto Install
by

Recently I needed to upgrade a Cisco Unified Computing System (UCS) system and while Cisco does have documentation about it. The process can be scary and could use more screenshots. Here is my step by step guide on how to upgrade a Cisco UCS via the Firmware Auto Install with the Cisco UCS Manager.

Fully planning your Cisco UCS upgrade is very important. There are a lot of things that need to be checked to make sure the upgrade is a success and that everything is compatible. In the past I’ve needed to upgrade my ESXi to as high as they can go based on the VMware and Cisco Compatibility Matrix and then upgrade the Cisco UCS. Then go back and upgrade ESXi again and then upgrade Cisco UCS again.

Quick Cheat Sheet

…
Read more
IT
BladesCiscoFirmwareHow ToNetworkingServersUpgrade

Update Teams Machine Wide Installer
by

Recently I’ve ran into an issue with Microsoft Teams where the user sees a blocking message that says “Teams needs an update”. The user needs to action it before they can use Microsoft Teams again. Typically the user can just click on Update Teams which will take them to a download page for Microsoft Teams, they also need to pick the correct version of Microsoft Teams because now there are two of them.

Let’s have some faith in the user and say they do download the correct version, now they need to run the downloaded file and then it will update Microsoft Teams for them and finally they can use Microsoft Teams again.

That whole process isn’t a good user experience and that’s way too many steps. It’s worse if the user is in a rush because let’s say the system with the outdated version of Microsoft Teams is a meeting room computer and they are trying to load Microsoft Teams for their meeting.

There has to be a way to prevent this and a way to reliably fix the Teams needs an update problem. Here’s what I found in this rabbit hole along with the solutions I found.

…
Read more
FixesITScripts
EUCFixHow ToMicrosoftMicrosoft TeamsPowerShellRabbit HoleScriptUpdate

Cloudflare Tunnel with Docker
by

There’s been a few times where I needed to setup access to an internal web application but I couldn’t put it on 443 or 80 because something else was using those ports and a reverse proxy would break one of the applications. A solution to this is Cloudflare Tunnel.

Cloudflare Tunnel used to be called Warp when it was in beta and was eventually renamed to Argo Tunnel. When Cloudflare made Argo Tunnel free they renamed it to Cloudflare Tunnel. The magic of Cloudflare Tunnel is handled by a small but powerful client that is known as cloudflared.

What makes a Cloudflare Tunnel awesome is the fact that you can use it to host an application externally without opening any ports on your firewall. It does this by creating an outbound only tunnel directly to Cloudflare.

For my situation I needed to setup access to an internal web application but I didn’t want to do another port forward to make it work. My solution was Cloudflare Tunnel with Docker.

The way I set it up is slight different than what Cloudflare’s documentation says as I wanted to use the Zero Trust dashboard and Docker but also have it in a Docker Compose file, as cloudflared seems to get updated at least once a month and I wanted it to be easy enough to recreate. Here’s how I did it and how everything works.

…
Read more
IT
CloudflareCloudflare TunnelCloudflare Zero TrustDockerHow ToProxySelf-hostedWeb App

Upgrade Palo Alto Firewall HA Pair (Active/Passive)
by

Palo Alto has some great documentation about how to do basically everything. Sometimes it’s a bit buried. These are my short and long cheat sheets for upgrading a Palo Alto Networks firewall in an Active/Passive High Availability Pair.

If you want to preform the upgrade using CLI only please see my post Upgrade Palo Alto HA Pair (Active/Passive) with CLI for more details.

Quick Cheat Sheet

Long Cheat Sheet

Upgrade path and sanity checks

For my example, FW01 is the Primary firewall and currently Active firewall and FW02 is the Secondary firewall and currently Passive firewall and they are both running PAN-OS 10.1 version 10.1.6-h6 which is the current preferred release for that version. We will be upgrading them to PAN-OS 10.2 version 10.2.2-h2 which is the current preferred release for that version.

…
Read more
IT
FirewallHigh AvailabilityHow ToNetworkingPalo Alto NetworksPAN-OSUpgrade

Azure AD Connect 2.0 Won’t Start
by

I recently ran into an issue where an install of Azure AD Connect failed to start. It seems like the root cause was due to the SQLLocalDB Model database becoming corrupt, which caused it to fail at upgrading itself. This is a known issue in versions older than 2.1.1.0 of Azure AD Connect.

While looking at the event logs it looks like the chain of events was that it tried to do the auto upgrade as auto upgrade is enabled but then failed to restart the SQLLocalDB due to the corruption which then caused Azure AD Connect to break.

…
Read more
FixesIT
Azure ADAzure AD ConnectFixHow ToMicrosoft

Setting Up Cloudflare Access
by

I’ve been a fan of Cloudflare for a while now. I love how fast they can propagate DNS changes and I typically like to use them as a DNS resolver. An issue that I’ve ran into many times, is how to protect something with MFA (Multi-Factor Authentication) that doesn’t have any support for MFA.

This problem is common with legacy web applications and very common with SCADA (Supervisory Control And Data Acquisition) web applications. The issue I was trying to solve was how to put MFA in front of a SCADA web application.

I decided to use Cloudflare Access. Cloudflare Access goes by a few names some of them are Cloudflare Zero Trust Network Access (ZTNA), Cloudflare Access, and Cloudflare Zero Trust Access. For simplicity I’m going to refer to it as Cloudflare Access.

Cloudflare Access is really nice because you can put it in front of any web application and it will require the user to authenticate before they can even reach the website.

If you have a lot of applications like this you can even set it up so the users can login to a portal to see all of the applications that are available to them. You can do all of this without the user having to install anything. What’s even better is that it’s free for 50 users.

Here’s how I setup a SCADA web app with Cloudflare Access.

…
Read more
IT
CloudflareCloudflare AccessCloudflare Zero TrustHow ToMFASelf-hostedSSOWeb App

Windows Recovery Partition
by

When you install Windows it always creates a Recovery Partition, which runs a very lite version of Windows with a few tools. There’s been a few times where I’ve used those tools to fix a system.

I wanted to know what would happen if I deleted the Recovery Partition. This is what I found down that rabbit hole.

Your Windows partitioning will look something like this.

Disk Partitioning with Recovery Partition
…
Read more
IT
EUCMicrosoftPartitionRabbit HoleRecoveryWindows
theDXT on GitHub
theDXT on X (Twitter)
theDXT on YouTube