Introduction
A software interoperability issue exists between HXDP [3.5(x), 4.0(x)] and ESXi 6.7P04 (build 17167734) and later. Customers should avoid this software combination.
NOTE: This Issue is extended to any 6.7 ESXi version above 6.7P04
The compatibility issue is resolved in HXDP 4.0(2e). This issue does not impact HXDP 4.5(1a) and later.
Requirements
ESXi 6.7P04 (build 17167734) and later
HXDP Version - 3.5(x), 4.0(x)
More Information
Defect
The related bug ID is CSCvv88204 - ESXi OpenSSH Interoperability Issue with HXDP
The issue occurs in ESXi 6.7P04, due to VMware upgrading the openSSH library to: OpenSSH_8.3p1. This new version of OpenSSH removes support for the key exchange method used internally by HXDP when communicating to ESXi directly via SSH. Below is a snippet from the OpenSSH changelog describing the breaking change made in that version:
ssh(1), sshd(8): this release removes diffie-hellman-group14-sha1 from the default key exchange proposal for both the client and server.
Software Advisory
Refer Software Advisory for more details - Cisco Software Advisory for ESXi 6.7 P04
Impacted Areas
Some functional areas of HX will be impacted including:
- Fresh cluster creation (may fail with Algorithm negotiation fail)
- Cluster expansion (may fail with Algorithm negotiation fail)
- Cluster reregistaration (stcli cluster reregister may fail with "Algorithm negotiation fail")
- System information page in HX Connect
- Upgrades may fail with "Failed to Establish SSH Connection to host" or "Errors found during upgrade"
ESXi upgradae fails with ssh exception-
2020-12-16-10:31:04.675 [] [] [vmware-upgrade-pool-9] ERROR c.s.sysmgmt.stMgr.SshScpUtilImpl - Failed to establish SSH connection to host: Host is not reachable, or in lockdown mode
com.jcraft.jsch.JSchException: Algorithm negotiation fail
Workaround
The HXDP release notes have been updated to specifically call out this version of 6.7 not being supported on 3.5(x) and 4.0(x) releases. This issue is fixed in the HXDP 4.0 patch - 4.0(2e) and in all releases 4.5(1a) and later.
- Use the rollback mechanism built into ESXi to roll back to a compatible ESXi version.
- Another possible workaround is to re-enable the removed key exchange method by updating sshd_config on each ESXi host and restarting the SSH service.It is recommended that this workaround only be implemented temporarily only.
NOTE: The goal should be to move the cluster to a fixed HXDP release and remove this workaround as soon as possible. Clusters should not remain in this state long term with this extra key algorithm setting added to sshd_config.
Steps for Workarounds
If you are unable to upgrade HXDP to a fixed release, use the following workarounds -
Workaround 1
Workaround 2
Re-enable the removed key exchange method by updating sshd_config on each ESXi host and restarting the SSH service.
- Add +diffie-hellman-group14-sha1 to the KexAlgorithms under /etc/ssh/sshd_config on each ESXi host
# echo "KexAlgorithms +diffie-hellman-group14-sha1" >> /etc/ssh/sshd_config
- Confirm that KexAlgorithms +diffie-hellman-group14-sha1 shows in the /etc/ssh/sshd_config
# /etc/init.d/SSH restart
- Re-start or resume the previously failed workflow.