vcenter rarely causes me problems (except during the 5.1 upgrade), it usually just runs and reboots are a non-issue. last week i rebooted the vcenter server to install windows updates and to fix something messed up with the VUM service.

after the reboot, i was experiencing two issues. one was that i couldn’t connect to vcenter at all from the vsphere client and two was that when i did connect i would get a blank inventory screen (with messages like retrieving… and loading…). this was all pretty unusual, but i figured hmm weird crap happens and i restarted the vcenter service. the service took a little while to fully restart and then i experienced the same exact issues again. the windows application log listed an event id 1000 for the vmware virtualcenter server. the closest KB i could find was this one: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025664. there was no way that was my issue. i went ahead and rebooted the server completely once more in hopes that the oddity would go away. also there was nothing terribly useful in the vcenter logs (c:\users\all users\vmware\vmware virtualcenter\logs\vpxd-*.log)

upon reboot, i experienced the same problem. even the locally installed vsphere client on the vcenter server encountered the same issues. i checked windows task manager and noticed that 80-99% of the memory was in use. this was fairly odd for a server with 12GB of RAM and running nothing else besides vcenter and sql server. i noticed that the vcenter process “vpxd.exe” was using a little more than half of the available RAM. the rest was being used up by sqlservr.exe and various instances of java.exe. i tried restarting the vcenter service once more and noticed that vpxd.exe started out with very little memory usage and then immediately shot up to 8-9GB of usage. that didn’t seem right at all.

after much digging online i found a thread that said vcenter can go nuts if you reboot it during VDR backups. i’m pretty sure i had indeed done that. i SSH’d into the VDR appliance and shut it down. i then restarted the vcenter service once more and everything was back to normal.

lesson learned: halt VDR activity before you reboot vcenter.

update 11/22/2013:
this problem occurred around 3 more times. i noticed in general that various processes were consuming a lot of memory on the vcenter server…so i first upgraded to vcenter 5.1 update 1c, which did drop the memory usage some, but did not prevent the issue mentioned above from happening. the 5.1u1c update was fairly straight forward. simply mount the iso and upgrade the vcenter modules in order. SSO first, inventory service, vcenter, and so on. the process took about an hour and a half  to upgrade everything.

i’m not 100% certain the issue is VDR related, but most things still point to it being the culprit. there’s a bug fixed in the latest release of VDR regarding causing memory leaks with vpxd.exe (http://www.vmware.com/support/vdr/doc/vdr_202_releasenotes.html). so i had an option to move to the latest VDR release, but that would entail deploying a new appliance. i decided i would rather redirect that effort into migrating to VDP since VDR technically isn’t even supported in a 5.1 environment.

finally, i also doubled the RAM in the vcenter server from 12GB to 24GB. that along with the upgrade to 5.1u1c, the decommissioning of 4.1 hosts, and the VDR appliance being shut down…i doubt this problem will reoccur.

update 1/8/2014:
one more final point of interest. i was looking into the vcenter 5.5 info and noticed that the new recommended min. requirements for RAM is 12GB for colocating all the vcenter services. and it says it’s even *higher* if the database server is also colocated. when we initially spec’d out the vcenter server for 4.x, 8GB was the recommended amount of RAM for our environment. with 12GB we thought we were golden at the time.
source: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052334

update 5/11/2014:
at a VMUG conference in february it was said that 24GB is the recommended amount of RAM for a windows-based vcenter server running vcenter 5.5.

Comments are closed.