Update:
After an upgrade to vCenter Server 7.0.0c, you see high CPU usage.
Patch for VMware vCenter Server Appliance 7.0.0d
After you upgrade your vCenter Server system to vCenter Server 7.0.0c, CPU usage continuously stays high. On a single-core, CPU usage might spike up to 100% for hours. The Workload Control Plane service causes the issue, even if you do not have Workload Management enabled in your environment.
This issue is resolved in this release.
https://docs.vmware.com/en/VMware-vSphere/7.0/rn/vsphere-vcenter-server-700d-release-notes.html
The issue
After upgrading my vCenter server appliance to the latest patch, 7.0.0.10600 Build 16620007, I found out that the CPU usage was abnormally high than before. Specifically on “observability”. Constant 4 to 5Ghz usage with just 4 hosts connected. Top shows java being the pain and vimtop shows observability hogging the CPU.
After some troubleshooting and consulting in the VMware Slack channel for vCenter we saw that the WCP service is consuming a lot of CPU. Ended up stopping that service and everything was back to normal. Thanks, Martin Gustafsson for pointing it out 🙂
Be aware!
When disabling the WCP Service you will be unable to put a host in maintenance mode. For a lab, this is not a problem but disabling it in a production environment is not recommended at all.
You will be warned with “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node.”
So now what?
To disable the Workload Control Plane service login to vCenter on the management page with https://vCenterFQDN:5480 and go to services. Then on the bottom, you will find the Workload Control Plane where you can disable it. If you are running Kubernetes then you might need to contact GSS for another fix because the Workload Control Plane is kinda an important part of that 🙂
After disabling the service you can see that my CPU usage went down to normal.
The bug has been reported and waiting on the fix. Bugzilla 2615223 for who has access to it as a VMware Employee or PSO consultant.
Disabling it via 5480 is just a one-timer. After reboot, it’s back on hogging CPU cycles. Want to disable it permanently? Follow this guide:
vCenter Appliance 7.0 – Disable Services
or
How to change the startup setting of services in vCenter Server Appliance 6.x using command line
Provided by Sidney Laan
So 3 ways to go with this issue.
- Restore to a previous backup.
- Disable the Workload Control Planes service and lose maintenance mode.
- Wait for the fix and ignore the CPU usage.
Thank you for this posting workaround. I’m on vCenter 7.0c and ESXi 7.0 GA and I just stopped the WCP service in the GUI (did not do anything more permanent). Now I’m seeing an error blocking maintenance mode – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node.”
Just wanted to bring awareness in case anyone else encounters this problem. Hoping VMware is able to patch this soon. In the meantime, I’ll just stop and start the service as needed to get things done.
I confirm, stopping the wcp service prevents entering maintenance mode.
Tested again and i can confirm it too.. changed blog with remarks thanks. Weird that it worked for me before:(
As Im going to run vSphere 7 with Kubernetes VMware vCenter Server 7.0.0b is solution for me :/
Not able to disable services from the GUI anymore (at least not in my three VCSA’s), easiest way for me is using: “vmon-cli -U wcp -S MANUAL” in the bash shell.. That way I am also able to start it (and stop it again) to put hosts in or out of maintenance mode.
@Ronald de Jong, look in the VAMI interface. Services are now listed there. https://vcenter:5480/#/ui/services
Yeah, I can stop and start them there, but I can’t disable them there or set them to manual (which I have done with this service, to be able to put hosts in maintenance mode, when necessary).
Confirm this fix is working perfectly 🙂
Pingback: vCenter Server 7.0.0c で CPU 使用率が高騰する | kurokobo.com
Hi there,
I can confirm it works for sure, but from my end, I reverted my home lab vCenter to 7.0b as 100% CPU just consumes too much ressources (and power at the end). Having to restart the service to put hosts in maintenance was too painful! 🙂
On the other end, that it takes more than 3 weeks to VMware to release a fix… not so good… 😉
The issue has been solved! Patch for VMware vCenter Server Appliance 7.0.0d is available.
As a user of mostly a HomeLab for learning purposes, VERY happy to have a fix released (and also hugely grateful for your workaround tips that lessened the impact until the fix was released)!
Thankfully all resolved with the patch (thank you for the heads up that there was a fix, had been watching this post closely!)
HI there,
Any fix after i have stopped WCP service in VCSA 7.0.0.c. The host can’t go to maintenance mode – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node”
How can i fix that issue?
Thanks.
Chris
Upgrade to 7d:)
Thanks for reply.
I have already update to 7.0.0.d to fix high CPU usage problem. But the host still can’t go to maintenance mode and show – “Host cannot enter maintenance mode because it failed to enter Namespaces maintenance mode on the node”.
Any solution to fix that problem?
Many thanks.
We’re having the same issue in 7.0c and after upgrading to 7.0d issue still persists across both vCenters in ELM. We upgraded from 6.5 a few weeks ago (lots of issues…)
Logged call with VMware Support, lets see what they say.
Don’t know exactly if I had the same issue, but I’ve seen many vMotion problems that I didn’t had in the past since I’ve upgraded to vSphere 7.X. Sometimes, to put a host in maintenance mode, I’ve to vmotion manually some VMs. With 6.7, I had zero issues (and nothing changed in configuration)
Pingback: Technology Short Take 131 - s0x
Problem prevails in 7.0.1.00200
Pingback: 【vSphere7.0u2】マシンSSL証明書だけ更新していけば問題ないのか？