recently, we needed unity connection to attempt a transfer to an extension in the UCCX partition.

first in unity we created a mailbox for the extension and set the greetings to play no recording and simply attempt transfer to this mailbox user’s extension.

then in UCM under Advanced Features->Voice Mail->Cisco Voice Mail Port, we examined the voice mail ports to see which CSS they were set for. and finally…clicked on the CSS and added the UCCX partition to the selected partitions listing.

i was able to deploy the VDP appliance after upgrading the majority of the ESX(i) hosts to 5.1. ESX 4.1 hosts are supported, but to deploy the 2TB version of VDP you need a datastore that’s actually a little over 3TB in size. the ability to create datastores >2TB debuts with ESXi 5.1.

after deploying it and powering it on, i used the https://vdp:8543/vdp-configure URL to initially set up the appliance. it seems like they really want you to set up a dedicated service account for VDP so i did so in AD. the configuration takes a good while to apply once you complete the wizard. also booting up this appliance in general seems to take a considerable amount of time.

after the deployment finished, i was looking around the vsphere client wondering where VDP was. i couldn’t find it so i skimmed through the documentation and found that it’s only accessible form the vsphere web client.

the good news is that the layout of the VDP UI is very similar to VDR, so there’s not much to relearn. one glaring shortcoming of VDP is that you can’t choose which vmdks to include/exclude, you can only choose whole VMs to back up. this feature will show up in VDP 5.5 apparently.

i stumbled upon a method to migrate VDR data to VDP. i was really excited about this and began the process. unfortunately, this process only migrates your backups and not your backup jobs. also this data migration takes an extremely long time. the migration process has been running for nearly a week and it’s still sitting at 10% done (about 21 VM backups migrated so far). initially, i ran into a problem attaching the VDR appliance to the VDP migration app. this is a known issue that can be resolved by renaming the VDR appliance to a name without spaces such as VDR01 (source: kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052476). also the VDP migration URL for future reference is https://vdp:8543/vdp-migration.

update/final thoughts 12/6/2013:
the VDR-to-VDP migration pretty much wrapped up around 8 days. the migration task hung at 90% for about 2 days even though it looked like all migration activity was complete. perhaps it was waiting for the VDR appliance to be detached. i’m fairly uncertain on what held up the task because i lost network connectivity to the web session of “vdp-migration”. it looks like once you’ve closed out that session you can’t get back into it. checking the VDP appliance’s logs (/usr/local/avamar/var/vdp-migration/server_logs) revealed no activity. i ended up simply canceling the migration task and then disabling the VDR plugin and deleting the VDR appliance. the migration process successfully migrated 94 backups and failed on 45 backups. that’s around a 68% success rate. the majority of the backups that failed seemed to be VMs with multiple disk drives.

quick update 12/9/2013:
to initially get the maintenance services running you’ll need to access the “vdp-configure” URL and go to Status and start the maintenance services.

vdp info:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2016565
http://blogs.vmware.com/vsphere/2013/04/vdp-5-1-10-ga.html

(or password)

this was a new one for me. after some problems involving the vcenter server a particular host was issuing this error. according to the vmware kb (proof: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2014275) and various blogs the easiest solution is to disconnect and reconnect the host to vcenter. unfortunately, after disconnecting the host i could not reconnect the host to vcenter.

i tried the usual bag of tricks: restarting management agents, checking top for oddities (this was full ESX), checking the vpxa.log, even a iscsi vmhba rescan. i couldn’t reboot the host because production vms were still running on it.

eventually i found some blogs that mentioned uninstalling the vcenter agent altogether and letting a new agent reinstall. this did the trick.

rpm -qa | grep vpx
rpm -e <package name from above>
service mgmt-vmware restart

i could then reconnect the host back to vcenter. also none of this is disruptive to the host connectivity or the vms running on the host.

sources:
http://h10025.www1.hp.com/ewfrf/wc/document?cc=us&lc=en&docname=c02055373
http://vmwise.com/2011/03/21/when-a-reconnect-is-not-enough/
http://geekysolution.blogspot.com/2013/02/cannot-contact-specified-host-host-may.html

vcenter rarely causes me problems (except during the 5.1 upgrade), it usually just runs and reboots are a non-issue. last week i rebooted the vcenter server to install windows updates and to fix something messed up with the VUM service.

after the reboot, i was experiencing two issues. one was that i couldn’t connect to vcenter at all from the vsphere client and two was that when i did connect i would get a blank inventory screen (with messages like retrieving… and loading…). this was all pretty unusual, but i figured hmm weird crap happens and i restarted the vcenter service. the service took a little while to fully restart and then i experienced the same exact issues again. the windows application log listed an event id 1000 for the vmware virtualcenter server. the closest KB i could find was this one: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=1025664. there was no way that was my issue. i went ahead and rebooted the server completely once more in hopes that the oddity would go away. also there was nothing terribly useful in the vcenter logs (c:\users\all users\vmware\vmware virtualcenter\logs\vpxd-*.log)

upon reboot, i experienced the same problem. even the locally installed vsphere client on the vcenter server encountered the same issues. i checked windows task manager and noticed that 80-99% of the memory was in use. this was fairly odd for a server with 12GB of RAM and running nothing else besides vcenter and sql server. i noticed that the vcenter process “vpxd.exe” was using a little more than half of the available RAM. the rest was being used up by sqlservr.exe and various instances of java.exe. i tried restarting the vcenter service once more and noticed that vpxd.exe started out with very little memory usage and then immediately shot up to 8-9GB of usage. that didn’t seem right at all.

after much digging online i found a thread that said vcenter can go nuts if you reboot it during VDR backups. i’m pretty sure i had indeed done that. i SSH’d into the VDR appliance and shut it down. i then restarted the vcenter service once more and everything was back to normal.

lesson learned: halt VDR activity before you reboot vcenter.

update 11/22/2013:
this problem occurred around 3 more times. i noticed in general that various processes were consuming a lot of memory on the vcenter server…so i first upgraded to vcenter 5.1 update 1c, which did drop the memory usage some, but did not prevent the issue mentioned above from happening. the 5.1u1c update was fairly straight forward. simply mount the iso and upgrade the vcenter modules in order. SSO first, inventory service, vcenter, and so on. the process took about an hour and a half  to upgrade everything.

i’m not 100% certain the issue is VDR related, but most things still point to it being the culprit. there’s a bug fixed in the latest release of VDR regarding causing memory leaks with vpxd.exe (http://www.vmware.com/support/vdr/doc/vdr_202_releasenotes.html). so i had an option to move to the latest VDR release, but that would entail deploying a new appliance. i decided i would rather redirect that effort into migrating to VDP since VDR technically isn’t even supported in a 5.1 environment.

finally, i also doubled the RAM in the vcenter server from 12GB to 24GB. that along with the upgrade to 5.1u1c, the decommissioning of 4.1 hosts, and the VDR appliance being shut down…i doubt this problem will reoccur.

update 1/8/2014:
one more final point of interest. i was looking into the vcenter 5.5 info and noticed that the new recommended min. requirements for RAM is 12GB for colocating all the vcenter services. and it says it’s even *higher* if the database server is also colocated. when we initially spec’d out the vcenter server for 4.x, 8GB was the recommended amount of RAM for our environment. with 12GB we thought we were golden at the time.
source: http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2052334

update 5/11/2014:
at a VMUG conference in february it was said that 24GB is the recommended amount of RAM for a windows-based vcenter server running vcenter 5.5.

files related to snapshots…
.vmsn file = snapshot state/”the snapshot file” (will also contain VM memory data if included in snapshot)
00000#-delta.vmdk = delta disk
00000#.vmdk = descriptor file for delta disk (plain text)
.vmsd = snapshot list (plain text)

additional info:
http://www.vmware.com/support/ws55/doc/ws_learning_files_in_a_vm.html
and the video “vsphere-anatomy-of-snapshots”

in case there was ever a legal reason why an admin would need to access the contents of another user’s mailbox, the following specifies how to…
http://help.outlook.com/140/gg709759.aspx (i’ve never tested this)

in exchange 2007 from the management console – “Server Configuration”. in this case it was version 8.3 enterprise and 8.3 means 2007 sp3.

more info here:
http://support.microsoft.com/kb/152439/en-us

with 5.x there are some kind of annoying host warnings regarding the ESXi Shell and SSH access being enabled on a host. while i don’t think people enable the ESXi Shell unless they really need to; i do imagine lots of people leave SSH access on 24/7 for remote management purposes.

i don’t like to see warnings when i feel like something is operating within normal parameters. luckily, this warning can be suppressed by going to the host Configuration tab->Advanced Settings->UserVars.SuppressShellWarning and enabling this setting with a “1”.

source:
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2003637

i’m still learning the ropes of the onboard sd cards in the r720. about a week or so ago, i was installing a pcie expansion card into an r720. the pcie expansion slots are located directly above the sd card slots. after i installed the expansion card i went about installing esxi. from the drac interface, i noticed that the sd card redundancy status was “Lost” and under removable flash media the sd card IDSDM SD2 was set to “Offline”.

i figured i might have dislodged one of the cards so i opened the server up again and reseated both sd cards. came back and checked, same thing. i went back once more and opened the server and this time instead of simply reseating the cards i swapped their places. i noticed that when i moved one particular sd card between the slots that an amber light lit up each time i seated the card. so even though the possibility of an sd card going bad is fairly slim…i started considering it.

i was just about to put a replacement request into dell when i noticed on the drac virtual console a message regarding sd card redundancy. all i had to do was confirm the request to restore redundancy and everything was back to normal.