VMWare Virtual Center and SQL 2005 Native Driver, Part 2

A quick follow up on a post I did last December, VMWare Virtual Center and SQL 2005 Native Driver. When upgrading your VMWare Virtual Center (vCenter, whatever), it will require you to use the SQL Server Native Client 10.0 in your ODBC when using an external 2005 SQL server.

If you remember, we installed the SQL Server Native Client 10.0 during installation as asked, but couldn't get the service to run after install. To fix the issue, we had to delete the DSN and create a new one using the standard SQL Server driver with the same settings.

Before doing the upgrade, delete your current DSNs and create new ones using the Native Client 10.0 with the same exact settings. Run the upgrade, let it do its thing, then it'll fail when attempting to upgrade the Converter and Update Manager (because at this point the VirtualCenter Server service can't start with the Native 10.0 driver). That's ok, just delete the OBDC connections and recreate them using the "SQL Server" driver. Then go into the Services MMC (Start - Run - services.msc) and start the VMWare VirtualCenter Server service. Give it a few minutes to figure itself out (Virtual Center will reconfigure HA, etc). Then run the upgrade again, this time only selecting Converter and Update Manager to upgrade.

vSwitches and vmnic Linking - When the NIC Name Changes

I had one of my ESX servers die over the weekend. Upon inspection (and some calls to support to verify) it seems the system board needed replacing. I pulled the machine and had it fixed. This machine has two on-board NICs and a quad-port NIC PCI-X card. When I booted up the server (after the repair) I noticed that the server tech reinstalled the quad port card into the wrong PCI-X slot. I shut down the machine and corrected the mistake.

Then I powered on the machine and plugged in the network cables. Unfortunately I could no longer access this server via network (nor could it access the network). ifconfig showed that all the NIC ports on the quad port card had different labels, going from vmnic0 and vmnic5 to vmnic10 and vmnic14. Nice, but not too much of an issue.

When you set up the service console earlier (during install), it puts the configuration in a virtual switch (such as vSwitch0) and then "links" that switch to your physical NIC (such as vmnic0). So this is what we need to do to get network connectivity. Run the command esxcfg-vswitch -l. This lists all of your virtual switches and their configuration. Look for the vSwitch that contains your Service Console port group. This is the switch you need. In this example, it is vSwitch0.

Next we need to link the vSwitch to the correct NIC. If all of the NIC ports are on the same vLAN or physical network, then this part isn't hard. But each NIC port on this card is plugged into a different vLan (set by our network switches), so it's important that I find the correct vmnic.

If you have physical access to the machine, and you need to determine which NIC port is assigned to which vmnic# label, do this:

• Unplug all network cables from the machine.
• As root, type esxcfg-nics -l in the console. This lists all the vmnic ports, the manufacturer, and link status.
• Next, take a network attached Cat5 cable and plug it into a port.
• Run the command again, noting wich vmnic show UP for the link status.
• Repeat the process for each NIC port.

(note: You can also just link each vmnic to the vSwitch, one by one, until your network access is achieved, if you don't have physical access to the machine. But, if you're not careful, that method may not be accurate.)

When you've determined the vmnic to use, linking it to the vSwitch is just one command: esxcfg-vswitch -L vmnic# vSwitch#. In my case, it was esxcfg-vswitch -L vmnic10 vSwtich0. Once I entered that, I could get on the network. Now, you can fix the rest of the nics/switches with this command, or you can connect to the server via Virtual Center and do it through the GUI.

VMWare Virtual Center and SQL 2005 Native Driver

So, I'm setting up a new VMWare Virtual Center server, using a separate server for my database (running SQL 2005). This is on a Windows Server 2003 R2 SP2 machine. Installation goes as planned until it asks for the ODBC connection. I set it up, choosing SQL server and everything tests out ok. But when VMWare decides to use it, it complains that the driver I'm using can't be used with SQL 2005. So I found and installed the "SQL Native Client 10.0" and the install continued.

After the server and client components installed, the add-on components Update Manager and Converter attempted to install. After asking for credentials, the installation failed because the Virtual Center service could not be reached. A quick check with the VMWare Infrastructure Client failed as well.

Going into the Services MMC, the VMWare Virtual Center Server service was stopped. Starting it took a few seconds and it showed started. Hitting refresh showed that it stopped pretty much immediately after starting. The Event Log showed the same thing, although was useless to find the cause.

I did a repair, no luck. Uninstall/reinstall, no go. Uninstall, manually remove registry entries, reinstall, nope.

According to the troubleshooting guide (Troubleshooting the VMware VirtualCenter Server service when it does not start or fails), there are a few things to check.

One is the SQL connection/setup. The ODBC connections tested perfectly and the SQL server was up. I also verified that there was plenty of open disk space. (See: Troubleshooting the database data source used by VirtualCenter Server (1003928))

Next is permissions. The service was set to run as local System. I configured it to run as a domain administrator account, same results. I also tried a bad password, to see what error appeared. It actually complained about an authentication error, which is not what it's doing otherwise. I set it back to local System.

Port conflicts is another thing to check. In a command prompt, enter netstat -ban and see if anything is taking up port 80, 902, or 443 on the machine. In my case, it was not. (See: Verifying if a port is in use (1003971))

After some other troubleshooting, I decided to check out the file the service wants to start, "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe". So I opened an command prompt and navigated to that folder. Entering vpxd.exe /? showed a list of options. I tried vpxd.exe -u to unregister the service. Then I tried vpxd.exe -r to register the service back. Still was getting the same issue. I tried vpxd.exe -s to run Virtual Center Server in non-service mode. After a bunch of text flashed on the screen, it failed but this time with an error.

Part of the error it left was "Fractional second precision exceeds the scale specified", and mentioned a bunch of tables it couldn't update. It also mentioned the SQL Native Client 10.0 driver. Google searching this error didn't help me much since I'm not an SQL programmer. However, this was leading me in the right direction. After thinking for a minute, I wondered what would happen if I removed the SQL Native Client 10.0 ODBC connection and set up a new connection with the same name and settings, but using the standard SQL Server ODBC driver.

It worked. Now I'm running my new Virtual Center server without issue. It does seem odd that the standard SQL Server driver won't work during setup, but is what works after setup.

A Couple of VMWare Issues

Sorry for the lack of posting this month; building a 200 person call center from scratch has been dominating my time. It is almost done though, but I still need to find time to post.

Today we discuss a couple VMware issues.

"Operation Failed Since Another Task Is In Progress"

I've seen this error a couple of times. Basically what happens is that a VM will show running but is actually frozen. Any attempts to force a VM shutdown or restart results in the error "Operation Failed Since Another Task Is In Progress". Same if attempted to Vmotion the machine.

Basically this turns out to be a snapshot issue. To fix this without rebooting the ESX server, we can just kill the VM process via command line. I took this tip from a VMware Communities post and cleaned it up a little. The post can be found here.


1. SSH into the ESX server that is currently running the affected VM (or you can use the console).
2. At the cmd prompt enter: cat /proc/vmware/vm/*/names

This lists the running VM's on the host server you are logged on to. Look for the vmid=##

vmid=1069 pid=-1 cfgFile="/vmfs/volumes/45.../server1/server1.vmx"
uuid="50..." displayName="server1"
vmid=1107 pid=-1 cfgFile="/vmfs/volumes/45.../server2/server2.vmx"
uuid="50..." displayName="server2"
vmid=1149 pid=-1 cfgFile="/vmfs/volumes/45.../server3/server3.vmx"
uuid="50..." displayName="server3"
vmid=1156 pid=-1 cfgFile="/vmfs/volumes/45.../server4/server4.vmx"
uuid="50..." displayName="server4"

3. At the cmd prompt enter: less -S /proc/vmware/vm/1149/cpu/status

It will now clear the console screen and show a bunch of numbers and stats. Hit the right arrow key until you see the section about group. Example:

group
vm.1058

With this ID number you can safely kill the VM without corrupting it.

4. At the cmd prompt enter: /usr/lib/vmware/bin/vmkload_app -k 9 1058

(Then number 1058 in the command is an example; your VM's group number goes here.)

5. If you see "Warning: Apr 20 16:22:22.710: Sending signal '9' to world 1058." this means your VM has been closed successfully. You can now start your VM back up and run it.

Unable to migrate due to "Remote Backing" issues with CD/DVD

When trying to VMotion two VM machines, I received the error: "Unable to migrate from VMESX2 to VMESX1: Virtual machine is configured to use a device that prevents migration: Device 'CD/DVD Drive 1' is a connected device with a remote backing.". Going into Edit Settings didn't help; all CD/DVD options where greyed out. I had recently set both systems to use the client device and to disconnect.

I was able to fix one of them; the VMware Tools was still waiting to be installed. I right-clicked on the VM and choose "End VMware tools install". That did the trick and it VMotioned fine.

The other system didn't have that option. I ssh'd into the esx host and from the command prompt ran service mgmt-vmware restart

After about two minutes (including a scary "disconnected" state in Virtual Center that lasted about a minute), it allowed me once again mess with my CD/DVD settings and I was once again able to Vmotion.

More on this issue in this VMWare Communities thread.