Turns out this is a known issue and solved in vCenter 6.5B.
Upgrade Issues
VMware vCenter Server Appliance configured with VMware vCenter Server High Availability might fail after an upgrade to 6.5 Update 1
A vCenter Server Appliance configured with vCenter HA might fail after an upgrade to 6.5 Update 1, because the hostname of the appliance might change to the default localhost.localdomain.
This issue is resolved in this release.
Recently had the pleasure of doing an upgrade on a vCenter 6.5 HA solution.
This is a new feature in 6.5 which gives you the option to run vCenter in a HA config. Active, Passive and Witness Node. The setup of this is fully automated in vCenter to set it up.
To upgrade a vCenter HA there is a specific path you need to follow. Upgrade Witness, Then the passive node and then do a failover from the Active to the upgraded passive node and upgrade the last node.
Here’s a good guide to doing that upgrade.
Before you begin, check if the passwords are not expired of the VCHA user on all three nodes. If so then fix it before putting the VCHA cluster in maintenance mode
The upgrade of the nodes went perfectly but afterward, I was unable to get the cluster out of maintenance mode. All options were greyed out and left with the option remove vCenter HA.
vCenter
Further on checking i saw that the passive node was not receiving the appliance configuration and was out of sync.
When failing over to the passive node via cli with “vcha-reset-primary” you will be greeted with an error. Which is explainable because the config is not correct.
So VCHA.. why are you so mean..
Eventually ended up destroying the config via the CLI and rebuilding the HA config from the working vCenter node. If you try to delete the config via the GUI it will fail..
To destroy the VCHA via CLI you need to log in as root to the Active node. Stop VPXD and VCHA services. Then you run “vcha –destroy -f” and the active vCenter will be the only one left with the HA config reset to none.
Then power off the passive and witness node so you can delete them.
Now you can follow the vCenter GUI and rebuild the vCenter HA config from the working vCenter and all will be good again.
