ESXi Config Restore Bug

While I was looking into various ways to restore an ESXi config backup I came across a bug.

If you read VMware’s documentation about how to restore an ESXi config backup (you can find that here) you will see that it is full of references saying the build numbers must match.

The bug is that you can restore an ESXi config backup even if the build numbers don’t match. Which according to the VMware documentation should not be possible.

Even though it is possible to restore an ESXi config backup when the build numbers don’t match, I do not recommend doing this as there has to be a reason why VMware says that the build numbers must match.

In my testing I was able to replicate the bug in ESXi 7 and ESXi 8. I even went all the way back to ESXi 6.7 which had some interesting findings. I didn’t fully test everything in ESXi 6.7 as general support has ended on that version.

Here are my findings and how to replicate the bug.

Generating Backup Files

In order to test everything as much as possible I wanted to have a few ESXi config backup files. This is how I generated the ESXi config backup files that I used in my testing.

• Fresh install of ESXi 7 Update 2e Build Number 19290878.
• Took an ESXi config backup from ESXi 7 Update 2e Build Number 19290878.
• Fresh install of ESXi 7 Update 3d Build Number 19482537.
• Took an ESXi config backup from ESXi 7 Update 3d Build Number 19482537.
• Fresh install ESXi 7 Update 3k Build Number 21313628.
• Took an ESXi config backup from ESXi 7 Update 3k Build Number 21313628.

Restoring from ESXi 7 Update 3d to ESXi 7 Update 3k

Once ESXi 7 Update 3k Build Number 21313628 was installed I tried to restore an ESXi config backup that was taken from ESXi 7 Update 3d Build Number 19482537.

When I do the restore using the vim-cmd hostsvc/firmware/restore_config method it works.

Due to the fact that the restore worked I then reinstalled ESXi 7 Update 3k Build Number 21313628 again.

When I do the restore using the firmwareConfig.py script method it also works.

PuTTY unexpectedly closing the sessions is due to the ESXi host rebooting as that is part of the restore process and is expected.

Both methods work and do not throw any errors, the same is true with ESXi 8.

Restoring from ESXi 7 Update 2e to ESXi 7 Update 3k

Due to the fact the restore worked when the build numbers didn’t match I wanted to see what would happen if the release levels didn’t match. I was not able to test this on ESXi 8 as there is no update 1 yet.

To start off fresh I reinstalled ESXi 7 Update 3k Build Number 21313628.

Once ESXi 7 Update 3k Build Number 21313628 was installed I tried to restore the ESXi config backup that was taken from ESXi 7 Update 2e Build Number 19290878.

If I try to restore the ESXi config backup using the vim-cmd hostsvc/firmware/restore_config method I get the following error.

(vim.fault.MismatchedBundle) {
faultCause = (vmodl.MethodFault) null,
faultMessage = ,
bundleUuid = "",
hostUuid = "",
bundleBuildNumber = 19290878,
hostBuildNumber = 21313628
msg = "Received SOAP response fault from [<, >, /sdk>]: restoreConfiguration
fault.MismatchedBundle.summary"
}

If I try to restore the ESXi config backup using the firmwareConfig.py script method I get the following error.

Mismatched Bundle: Host release VMware ESXi 7.0 Update 3 Bundle release VMware ESXi 7.0 Update 2
Mismatched Bundle: Host build 21313628 Bundle build 19290878

This is good I expected both methods to throw an error when trying to restore from Update 2 to Update 3.

It looks like the bug only exists when the release levels match.

Extra Findings

I was curious to see if I could figure out when this bug might have started, so I installed ESXi 6.7.

With ESXi 6.7 I noticed that the ESXi config backup files have no mention of the build number in the Manifest.txt file.

RELEASELEVEL=VMware ESXi 6.7.0 Update 3
UUID=00000000-0000-0000-0000-AC1F6B950B36
KERNELOPTS=installerDiskDumpSlotSize=2560 no-auto-partition
USEROPTS=

In ESXi 6.7 there’s a script named firmwareConfig.sh which can be used to take an ESXi config backup or restore an ESXi config backup, however there is zero mention of build numbers in the script.

Here’s the restore part of the firmwareConfig.sh script.

#
# Restore configuration from the specified configuration bundle.
#
# Usage: RestoreConfiguration(<bundleFile>, <force>)
#
RestoreConfiguration()
{
   local bundle=${1} force=${2}
   local tmpdir="/tmp/firmware-restore.$$"
   local host_uuid=$(esxcfg-info -u) bundle_uuid=
   local host_release=$(vmware -l) bundle_release=
   local manifest_file="${tmpdir}/Manifest.txt"

   mkdir -p "${tmpdir}"

   tar xzf "${bundle}" -C "${tmpdir}"
   if [ ! -e "${manifest_file}" ] ; then
      echo "Invalid Bundle: Missing Manifest File"
      rm -rf "${tmpdir}"
      exit 1
   fi

   bundle_uuid=$(GetValue "${manifest_file}" UUID)
   bundle_release=$(GetValue "${manifest_file}" RELEASELEVEL)

   # validate version and UUID
   if [ "${bundle_release}" != "${host_release}" ] ; then
      echo "Mismatched Bundle: Host release level: ${host_release} Bundle release level: ${bundle_release}"
      rm -rf "${tmpdir}"
      exit 1
   fi

   if [ "${host_uuid}" != "${bundle_uuid}" ] && [ ${force} -ne 1 ] ; then
      echo "Mismatched Bundle: Host UUID: ${host_uuid} Bundle UUID: ${bundle_uuid}"
      rm -rf "${tmpdir}"
      exit 1
   fi
Code language: Bash (bash)

Due to this I did not do any further testing in ESXi 6.7. I believe it is safe to say that ESXi 6.7 has no validation on the build numbers only the release numbers.

It looks with the release of ESXi 7 VMware added build numbers to the ESXi config backups as the ESXi config backup files now list the build number in the Manifest.txt file.

RELEASELEVEL=VMware ESXi 7.0 Update 3
BUILDNUMBER=19482537
UUID=00000000-0000-0000-0000-AC1F6B950B36
KERNELOPTS=autoPartition=FALSE
USEROPTS=

Another change with ESXi 7 is that the firmwareConfig.sh script is now a python script called firmwareConfig.py and there are mentions of build numbers in it.

Here’s the restore part of the firmwareConfig.py script.

      # Validate RELEASELEVEL and UUID
      # Error message format will be parsed by FirmwareSystemImpl:
      # 'Mismatched Bundle: Host KEY VALUE Bundle KEY VALUE'
      hreleaseLevel = _hostReleaseLevel()
      if mDict['RELEASELEVEL'] != hreleaseLevel:
         logger.error('Mismatched Bundle: Host release {} Bundle release {}'
                      .format(hreleaseLevel, mDict['RELEASELEVEL']))

         # FirmwareSystemImpl want's to throw a MismatchedBundle exception
         # which reports the build number if they don't match.
         build = getBuildNum()
         if mDict['BUILDNUMBER'] != build:
            logger.error('Mismatched Bundle: Host build {} Bundle build {}'
                         .format(build, mDict['BUILDNUMBER']))
         return False

      huuid = getHardwareUuid()
      if not mDict['UUID'] == huuid and not force:
         logger.error('Mismatched Bundle: Host UUID {} Bundle UUID {}'
                       .format(huuid, mDict['UUID']))
         return False
Code language: Python (python)

My Thoughts

I think what might be happening is the build number check only happens if the release level check fails as that seems to be the only time it throws an error that includes the build number.

Even though you can restore an ESXi config when the build numbers don’t match I would not recommend doing so, as it could cause unexpected behavior and based on VMware’s documentation would likely not be supported if something goes wrong later on.

I’ve place a support request with VMware to report this and to find out for sure if this is actually a bug or maybe the documentation is wrong.

Related posts:

ESXi Config Restore vCenter ESXi Config Backup Script ESXi Build Number without vCenter ESXi Config Backup