Zone Assignments and GPO settings

Let's say you have a handful of websites that you want to assign to particular zones. You have an internal site you want everyone's browser to handle in the Intranet zone. You have a few external sites/vendors that your users need to interact with and those must be in the Trusted zone. Or even if you set your drive mappings in your login scripts to use the FQDN of the file server, and Windows/Office automatically treats every file on those drive mappings as "Internet" files and won't trust them (you need to set your domain in the Intranet zone).

 There are a couple of ways to handle this type of situation. First, you could just teach all the users how to do their own zone assignments, which is never a fun task. You could script the changes, adding the sites directly to the ZoneMap in the HKCU in the registry.  Or you could push it all out via Group Policy.

 There are two ways to push these settings via Group Policy; the strict way and the flexible way, depending on what you're trying to accomplish.

 If you want to set the Zone Assignments and not allow the user to modify them in any way, create a new policy and navigate to User Configuration\Administrative Templates\Windows Components\Internet Explorer\Internet Control Panel\Security Page.  Locate the "Site to Zone Assignment List" setting. If you disable this setting, no user will be able to set any zone assignments. If you Enable the setting, you can set the zone assignments for the user.

 HOWEVER, THEY WILL NOT BE ABLE TO MODIFY (ADD) ANY ZONE ASSIGNMENTS. 

Also, their existing zone assignments will be lost.  This is important. The user will not be able to add that one-off site that they need and will have to wait on you to add it to the GPO. However, sometimes it is necessary to do this. If you are ok with this, enable the setting and click the "Show" button next to "Enter the zone assignments here". I tend to enter my domain assignments using a wildcard, so any child/sub domains are covered. Of course, you can enter specifics here as well.

For the "Value", you must enter a number from 1-4 that designates what zone to put the domain in. They are:
1 = Intranet Zone
2 = Trusted Sites Zone
3 = Internet Zone
4 = Restricted Sites Zone

After you are finished, assign the GPO to the OU's you want to apply it to.

Let's say you want to be flexible.  You know there are a few users out there that might need to use another vendor's site for whatever reason and they don't want to wait for you to add it to the GPO.  Or an existing vendor made a change to their website and requires it to be in the Trusted Zone suddenly.  Or your helpdesk wants to troubleshoot an issue by moving site assignments around.  We want to assign sites and still allow the user to add their own.

TO ALLOW USERS TO ADD THEIR OWN SITES, DO NOT SET THE "SITE TO ZONE ASSIGNMENT LIST" SETTING.

Leave that setting to Not Configured.  I learned this the hard way.  Instead, navigate to User Configuration\Preferences\Windows Settings\Registry.  Right-click and choose New - Registry Item.

1. For Action, choose Update.
2. For Hive, choose HKEY_CURRENT_USER
3. For Key Path, enter Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap\Domains\blogger.com  
1. Replace blogger.com with the domain you want to add.
2. If you want to cover the entire domain, just put the domain name.
3. If you want to cover only a sub domain, put it instead (example: client.blogger.com)
4. If you want to cover only www, put that as well (example: www.blogger.com)
4. For Value Name, you have a few options.
1. You can use a wildcard to cover anything .blogger.com (*.blogger.com)
2. You can specify a protocol (http, https).  This will only cover that one protocol (example: www.blogger.com, with Value http = http://www.blogger.com)
5. Value type: REG_DWORD
6. Value Data: Enter the value of the zone you want to assign.
1. 1 = Intranet Zone
2. 2 = Trusted Sites Zone
3. 3 = Internet Zone
4. 4 = Restricted Sites Zone
7. Base: Decimal.

Let's say you want to add an IP address or an IP range.  This is a bit trickier but it is possible.  First, your Key Path will be different.  Instead of "Domains" under ZoneMap, you will be placing the registry setting in Ranges.  Also, for each "Range" you will have to create a sub-key and it will require two settings instead of one.

If you run into an issue or need to know how to add a specific setting, you can always add the Zone Assignment on your computer and look in the registry to see how it works.   You can also do this to verify that the GPO is applying correctly.

Open Regedit and go to:

HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Internet Settings\ZoneMap

Your domain will each be a subkey under "Domains".  Your IP addresses will be under "Ranges".

Yahoo Messenger and Internet Proxies

I hate proxies but they seem necessary in the Corporate world. Getting all the websites, programs, widgets, etc working as people want can cause much pain and suffering.

This is a simple post, but hopefully it'll save someone a few hours of frustration.

We have a group that has to use Yahoo! Messenger for chat. So I put them on our web filter/proxy, allowing the Yahoo IM access. (Note, our proxy requires NTLM/Active Directory authentication to allow traffic). Those not using Yahoo's client (like those using Pidgin) connected fine. But the official Yahoo client seemed like it wasn't passing the Windows Authentication to the proxy and not connecting. Setting the client's proxy settings to Direct Connection did not work; Yahoo was still pulling Internet Explorer's proxy settings.

After a few hours of searching, I came across this Microsoft KB article: How to Set Up Yahoo Messenger to Use Only Integrated Authentication on Your ISA Server Computer. In the article, it advises that you set your Internet Explorer settings to enable "Use HTTP 1.1 through proxy connections".

Apparently Yahoo Messenger client doesn't send a keep-alive header in it's NTLM request, thus not keeping itself authenticated. Telling it to use HTTP 1.1 forces this, since the keep-alive header is default in 1.1.

Check out the Microsoft article (linked above) for more information, but this simple checkbox fixed the issues for my users. It's not JUST for ISA; I'm not using ISA and it worked for me.

iSCSI, Windows 2003, SANSurfer, and VDS

So I have a site with two servers having identical problems, a iSCSI volume will not initialize in Window's Disk Management utility (or create a partition in DiskPart).

In the GUI Disk Management, I get the error "Logical Disk Manager: The operation did not complete. Check the System Event Log for more information on the error." In DiskPart, I get the error "The disk management services could not complete the operation." The event log doesn't have an error directly explaining these errors, but sometimes I'll get an Application Error 1004, mentioning vds.exe (Virtual Disk Service) faulted. Google and multiple forum searches don't yield much result.

A little about the setup. I have an IBM DS3300 iSCSI SAN attaching LUNs to two Dell PowerEdge 1950s via a Qlogic QLE4062c cards, using SANSurfer software. (They're not sharing a LUN, each server gets a different one).

So I won't bore you too much, I'll leave out the majority of my troubleshooting and homicidal hatred towards these two systems and just give you the answer (or at least the answer that worked for me).

Turns out that the issue lies in the Virtual Disk Service, which Microsoft has a hotfix for here: Article ID: 948699. Once I installed the fix and rebooted I was able to initialize the disk, format, label, and use.

iSCSI File Shares Disappear

A quick tip for those new to iSCSI, the Microsoft iSCSI Initiator, and using the two for file sharing.

Setting up the LUN, mapping it as a disk, and putting your file shares on it is great, but there are a few configuration steps you need to take if you experience your share settings disappear every time you reboot (even thought the files remain).

First, ensure that the Server service is dependent on the Microsoft iSCSI Initiator Service. To do this, go into the Services MMC, open the Server service properties, and check under the Dependencies tab. No Microsoft iSCSI Initiator? Open Regedit (Run - regedit.exe) and navigate to HKEY_LOCAL_MACHINE\System\CurrentControlSet\Services\lanmanserver. Find the key "DependOnService" and set it's value to MSiSCSI. The Server service is responsible for creating the shares; if this service starts up before the iSCSI LUNs are ready on the server, then the shares will not appear.

Second, make sure you set up the Microsoft iSCSI Initiator to automatically restore the connection and drive letters. Under the Targets tab, when you highlighted the target and clicked "Log On", did you check "Automatically restore this connection when the system boots"? If not, remove the connection and log it back on, this time selecting the correct option.

Third, after the drive has been set up and you assigned it a drive letter, go back into the Microsoft iSCSI Initiator software and click on the Bound Volumes/Devices tab. You'll see some funky volume path that looks like gibberish. Highlight it and click Add. Put in the drive letter you assigned and click Ok. This will cause the iSCSI Initiator service to not completely start until the volume has been mapped to the drive letter. Since the Server service now depends on the iSCSI service starting up, it'll have to wait until the volume is available before it can assign share settings.

VMWare Virtual Center and SQL 2005 Native Driver, Part 2

A quick follow up on a post I did last December, VMWare Virtual Center and SQL 2005 Native Driver. When upgrading your VMWare Virtual Center (vCenter, whatever), it will require you to use the SQL Server Native Client 10.0 in your ODBC when using an external 2005 SQL server.

If you remember, we installed the SQL Server Native Client 10.0 during installation as asked, but couldn't get the service to run after install. To fix the issue, we had to delete the DSN and create a new one using the standard SQL Server driver with the same settings.

Before doing the upgrade, delete your current DSNs and create new ones using the Native Client 10.0 with the same exact settings. Run the upgrade, let it do its thing, then it'll fail when attempting to upgrade the Converter and Update Manager (because at this point the VirtualCenter Server service can't start with the Native 10.0 driver). That's ok, just delete the OBDC connections and recreate them using the "SQL Server" driver. Then go into the Services MMC (Start - Run - services.msc) and start the VMWare VirtualCenter Server service. Give it a few minutes to figure itself out (Virtual Center will reconfigure HA, etc). Then run the upgrade again, this time only selecting Converter and Update Manager to upgrade.

Proxy PAC Files, How to Use With Laptops and Local Bypass

First, I *HATE* proxy .pac files. With that said, here's some tips on setting up your file.

Here's the challenge. You have a web filter/proxy in your network and you need everyone to go through it. You also need to allow unfettered access to local resources, which can sometimes cause fits with your proxy. Some examples of such fits: not being able to route back to certain local resources, having issues with with special applications that install a local http server (Google Desktop comes to mind here), access to local webservers with certain programs (Microsoft Frontpage), or local webservers that run code that the proxy doesn't like (we have a site that does a lot of perl, designed for internal use only that the proxy just has issues with).

If your client machines are local desktops or Terminal Server sessions that don't leave the network, setting up bypasses for all of this with Group Policy is easy. Just put the IP address or subnet in the Exceptions part of the Proxy Settings configuration window. But what if your clients have laptops that leave the network? Forcing them to use the VPN for web traffic is one way to work around this; it forces all Internet usage on the company machine to be accounted for in the web filter and is easy to configure in their GPO. But what if management doesn't like this option; the clients have to be filtered while in the office but can hit whatever they want when they leave? The answer to this problem is to set up a proxy .PAC file.

The PAC file allows you to configure a user's proxy in many different ways. It's a text file that is referenced by the browser for proxy configuration, and uses a JavaScript function (FindProxyForURL(url, host)) to pull this off. In this post, I'm only going to focus on my requirements to build my PAC file: determining if the user is on the office network and bypassing local resources.

There's quite a few examples of PAC files on the Internet, and I'll provide links to these sites on the bottom of this post. They're all worth a look; considering the custom nature and the many configuration options to PAC files it's a good idea to see what everyone else is doing and develop your own configuration from there.

First, let's determine a way to figure out if you're on the local network or not. Some examples I've seen attempt to determine if the machine is part of the local subnet, then bases the configuration on that. If you're like me and have a lot of different subnets (either internal vlans or different sites), that may not work. So, what I do is configure the PAC file to determine if it can communicate with the proxy. If yes, it uses it. If not, then it goes "DIRECT" to the Internet. For me, it's a simple one-liner:

function FindProxyForURL(url,host) { return "PROXY 192.168.10.100:3128; DIRECT"; }

This will cause the web browser to look for the PROXY IP address (in the example 192.168.10.100:3128). If it finds it, then it uses it and you should see the traffic hit your logs. If it can't find it, it'll take a minute to time out then drop the user directly on the Internet with no proxy.

Ok, the first requirement is done; mobile users will be on the web filter while at the office and will be off of it away from the office. Now, we need to bypass local resources; a task that causes a little more fuss. Let me note that before I bash on web filters, or at least come off as bashing them, most of my local resources tested just fine with no further configuration. A lot of my monitoring tools, local wikis, etc had no issue. But some, such as our phone system configuration site, didn't work at all in testing. Your mileage may vary; for a lot of people stopping at this point would work for them just fine. But for those who do have issues, we'll continue.

Next determine the subnets you want to bypass. In this case, we'll use 10.10.1.x, 10.100.x.x, 192.168.10.x, and all local addresses 127.0.0.x).

function FindProxyForURL(url,host) {

if (
(isInNet(host,"10.10.1.0","255.255.255.0")) ||
(isInNet(host,"10.100.0.0","255.255.0.0")) ||
(isInNet(host,"192.168.10.0","255.255.255.0")) ||
(isInNet(host,"127.0.0.0","255.255.255.0"))
)
return "DIRECT";
else
return "PROXY 192.168.10.100:3128; DIRECT"; }

Now, when testing, any host that uses an IP address or resolves to an IP address in the subnets you're bypassing should not show up in your logs. All other traffic should show up just fine. Make sure you set the appropriate subnet mask. The || pipes mean "OR", so you can add as much as needed.

Now, lets say you have a specific site you want to bypass, on your network or not. For example, you want to bypass traffic to your external web site, or you use a hosted email solution that has issues in your web filter. Just add the following line within the "if ( )" statement:

(dnsDomainIs(host, ".bypassed_url.com")) ||

Note that there is no need to put anything more than the domain name; no need for http://, *://, or the full URL. Also, don't forget the OR (||) if you intend to put any more statements below it. Your last line in the if( ) should not have ||.

Now that all of our requirements are met it's time to save the file and test it. There are several methods of the PAC file placement involving web servers or network shares, but since the client is on a laptop I prefer to place the file locally. Save your PAC file on the local drive. Next, open Control Panel and Internet Options. Click the Connections tab and the LAN Settings button. Uncheck everything except "Use automatic configuration script". For the address, put in file://c:/PAC_file_location/name_ of_PAC_ file, example: file://c:/pxy/pxy.pac

In a future posting I'll go over how I deploy the PAC file via login script and how I use Group Policy to force my users to use the file. I may even do a write-up on blocking other browsers, such as Firefox, Chrome, Safari, and Opera.

A quick note: PAC files are read when the browser is opened. If your laptop user has an open browser then pulls their laptop off the network (hibernate) then attempts to use the same browser window off the network, it most likely will not work. They'll have to restart the browser. Same concept when they enter the network; if they were working in a browser window off the network unfiltered, the same browser window will remain in DIRECT configuration until it is restarted.

As promised, some useful links:

Wikipedia:Proxy_auto-config
Craig Johnson Consulting: Autoconfigure Scripts for Proxy Settings - good overview, but I couldn't get the if (shExpMatch(url, method to work..
Novell: Cool Solutions: Proxy Failover (without Clustering) - great article, the author dissects the code and explains it.
Jason Curnow: Writing Effective Proxy PAC Files - wow, I JUST found this site.. Read this.
Microsoft Technet: Using Automatic Configuration, Automatic Proxy, and Automatic Detection - lots of examples here.

vSwitches and vmnic Linking - When the NIC Name Changes

I had one of my ESX servers die over the weekend. Upon inspection (and some calls to support to verify) it seems the system board needed replacing. I pulled the machine and had it fixed. This machine has two on-board NICs and a quad-port NIC PCI-X card. When I booted up the server (after the repair) I noticed that the server tech reinstalled the quad port card into the wrong PCI-X slot. I shut down the machine and corrected the mistake.

Then I powered on the machine and plugged in the network cables. Unfortunately I could no longer access this server via network (nor could it access the network). ifconfig showed that all the NIC ports on the quad port card had different labels, going from vmnic0 and vmnic5 to vmnic10 and vmnic14. Nice, but not too much of an issue.

When you set up the service console earlier (during install), it puts the configuration in a virtual switch (such as vSwitch0) and then "links" that switch to your physical NIC (such as vmnic0). So this is what we need to do to get network connectivity. Run the command esxcfg-vswitch -l. This lists all of your virtual switches and their configuration. Look for the vSwitch that contains your Service Console port group. This is the switch you need. In this example, it is vSwitch0.

Next we need to link the vSwitch to the correct NIC. If all of the NIC ports are on the same vLAN or physical network, then this part isn't hard. But each NIC port on this card is plugged into a different vLan (set by our network switches), so it's important that I find the correct vmnic.

If you have physical access to the machine, and you need to determine which NIC port is assigned to which vmnic# label, do this:

• Unplug all network cables from the machine.
• As root, type esxcfg-nics -l in the console. This lists all the vmnic ports, the manufacturer, and link status.
• Next, take a network attached Cat5 cable and plug it into a port.
• Run the command again, noting wich vmnic show UP for the link status.
• Repeat the process for each NIC port.

(note: You can also just link each vmnic to the vSwitch, one by one, until your network access is achieved, if you don't have physical access to the machine. But, if you're not careful, that method may not be accurate.)

When you've determined the vmnic to use, linking it to the vSwitch is just one command: esxcfg-vswitch -L vmnic# vSwitch#. In my case, it was esxcfg-vswitch -L vmnic10 vSwtich0. Once I entered that, I could get on the network. Now, you can fix the rest of the nics/switches with this command, or you can connect to the server via Virtual Center and do it through the GUI.

VMWare Virtual Center and SQL 2005 Native Driver

So, I'm setting up a new VMWare Virtual Center server, using a separate server for my database (running SQL 2005). This is on a Windows Server 2003 R2 SP2 machine. Installation goes as planned until it asks for the ODBC connection. I set it up, choosing SQL server and everything tests out ok. But when VMWare decides to use it, it complains that the driver I'm using can't be used with SQL 2005. So I found and installed the "SQL Native Client 10.0" and the install continued.

After the server and client components installed, the add-on components Update Manager and Converter attempted to install. After asking for credentials, the installation failed because the Virtual Center service could not be reached. A quick check with the VMWare Infrastructure Client failed as well.

Going into the Services MMC, the VMWare Virtual Center Server service was stopped. Starting it took a few seconds and it showed started. Hitting refresh showed that it stopped pretty much immediately after starting. The Event Log showed the same thing, although was useless to find the cause.

I did a repair, no luck. Uninstall/reinstall, no go. Uninstall, manually remove registry entries, reinstall, nope.

According to the troubleshooting guide (Troubleshooting the VMware VirtualCenter Server service when it does not start or fails), there are a few things to check.

One is the SQL connection/setup. The ODBC connections tested perfectly and the SQL server was up. I also verified that there was plenty of open disk space. (See: Troubleshooting the database data source used by VirtualCenter Server (1003928))

Next is permissions. The service was set to run as local System. I configured it to run as a domain administrator account, same results. I also tried a bad password, to see what error appeared. It actually complained about an authentication error, which is not what it's doing otherwise. I set it back to local System.

Port conflicts is another thing to check. In a command prompt, enter netstat -ban and see if anything is taking up port 80, 902, or 443 on the machine. In my case, it was not. (See: Verifying if a port is in use (1003971))

After some other troubleshooting, I decided to check out the file the service wants to start, "C:\Program Files\VMware\Infrastructure\VirtualCenter Server\vpxd.exe". So I opened an command prompt and navigated to that folder. Entering vpxd.exe /? showed a list of options. I tried vpxd.exe -u to unregister the service. Then I tried vpxd.exe -r to register the service back. Still was getting the same issue. I tried vpxd.exe -s to run Virtual Center Server in non-service mode. After a bunch of text flashed on the screen, it failed but this time with an error.

Part of the error it left was "Fractional second precision exceeds the scale specified", and mentioned a bunch of tables it couldn't update. It also mentioned the SQL Native Client 10.0 driver. Google searching this error didn't help me much since I'm not an SQL programmer. However, this was leading me in the right direction. After thinking for a minute, I wondered what would happen if I removed the SQL Native Client 10.0 ODBC connection and set up a new connection with the same name and settings, but using the standard SQL Server ODBC driver.

It worked. Now I'm running my new Virtual Center server without issue. It does seem odd that the standard SQL Server driver won't work during setup, but is what works after setup.

VMWare: Orphaned Templates

Stolen from this Message Board Thread.

• Right click on your orphaned template
• Select "Remove from Inventory"
• From the host you would like the template to reside on...
• Select you host
• Select the "Configuration" tab
• Select "Storage" under the Hardware pane
• Double click on the storage device in the right pane..this will bring up the "Datastore Browser"
• Browse to your orphaned template's location
• Right click on the template file..it will have a ".vmtx" extention with the Type displayed as "Template VM"
• Choose "Add to Inventory"
• Go through and answer the wizard information to complete the "fix"

Local to Mapped Printer Migration

Sorry I haven't been posting regularly, I've been really really busy at work
lately and the little time I've had off I didn't want to do ANYTHING
sysadmin related. Hopefully things will be a bit slower now and I can post
some of the stuff I've been saving up. On to the topic...


I love terminal servers. Unfortunately they can be a bit fragile at times,
especially under higher loads. One load-inducing problem that can occur is the existence of local printer queues on the server itself. Locally mapping a user's printer to the terminal server may seem like a good idea, but typically it isn't the best way to do things. Local printers cause a lot of I/O traffic on your local disks, the drivers take up memory (some drivers will load a nice hunk of memory for each logged in user), and the spooler service will also take up resources that will affect other users. Throw in memory leaks that some printer drivers may have and you'll end up with a pretty good issue as your users pile into the server.

On my network I ran into this exact issue with one of my oldest stand-alone terminal servers. I'm running Windows Server 2003, older but still decent hardware, and about 50-60 heavy users all getting a full desktop. Since I didn't know the issues with local printers when I first deployed the server, I ended up with over 30 local printers, most of which were mapped to printers over slower WAN links. Users started complaining about the server being slow. Investigating, I found a memory leak in the spooler service (restarting it helps a little), and two printer drivers taking up 5 megs each in every user session. (That's about 500 megs of ram wasted when I've got 50 users on the server). At this point I decided to do something about it.

There are two options at this point. I could either manually move each user to a mapped printer on another server (which involved contacting each user, taking over their session, and migrating them), or I could script the move (and the users won't know the difference). I chose the scripting method, especially since I don't like to track down and interrupt the users when it can be avoided.

First things first, set up each printer queue on a print server. This is necessary in either option. Hopefully you have a standard naming convention, because scripting is a lot easier if you don't have to change the printer names. (I'll show you a way to handle printer renames later in this post, but it gets messy if there's more than just a few exceptions).

Next, write up your script. Here's what I've done:


'Find the default locally mapped printer, move it to a print server mapping.
Dim ws, dflt, nCount, to_server
Dim WshNet, WshShell
Dim objNetwork
Set objNetwork = CreateObject("WScript.Network")

' The server where the new print queues are hosted.
to_server = "\\printserver01"

' Find the default printer from the registry, strip out unneeded text.
Set ws = WScript.CreateObject("WScript.Shell")
dflt = ws.RegRead("HKEY_CURRENT_USER\Software\Microsoft\Windows NT\CurrentVersion\Windows\Device")
nCount = instr(dflt,",") -1
dflt = left(dflt,nCount)

' Create the printer mapping for the client.
On error Resume Next
objNetwork.AddWindowsPrinterConnection to_server+"\"+dflt

' Set the created printer mapping as the default.
On error Resume Next
objNetwork.SetDefaultPrinter to_server+"\"+dflt


To map to a renamed printer queue, add this to the script, right before the part of the script where the printer mapping occurs. If the printer name matches what you specify, it will rename it to the new, renamed print queue.


' Specific mapping for Joe Blow's printer. Change dflt to the printer you want to match.
if dflt="JoeBlow_HP4000" then
' Change dflt to the renamed print queue.
dflt = "Site2_HP4100-JoeBlow"
to_server = "\\printserver02"
End if


Also, if you have a client that uses more than one printer, use the below script to determine this, map them to all their printers, and then set their default printer. The script will try to compare their mapped printer with any one of the printers they need to activate this section of script. This script does an Exit Function, which assumes that this migration piece is part of a function.


' Specific mappings for store 3's printers. This will see if dflt is the name of either of this store's printers
if dflt="str3_HP4250-1" or dflt="str3_HP4250-2" then
' Map both printer that the store employee's use.
objNetwork.AddWindowsPrinterConnection "\\printserver01\str3_HP4250-1"
objNetwork.AddWindowsPrinterConnection "\\printerserver01\str3_HP4250-2"
' Set the default printer to the default the user previously had.
objNetwork.SetDefaultPrinter to_server+"\"+dflt
Exit Function
End if


Personally, I incorporated this code as part of my login script, which runs each time a user logs in. After a few days, all of my users should have their printers mapped to the print server and I'll delete the local print queues and remove the printer drivers.

Using Wireshark to determine bandwidth needs and top bandwidth users

Today I'll show a couple of tricks to determine bandwidth of a conversation between two hosts and a trick to help quickly determine top bandwidth users on a network. To do this, download and install the massively powerful free Network Protocol Analyzer, Wireshark.

First, we'll look at how to determine the average bandwidth between two hosts. I first came across the need for this information when I was tasked to plan a move of a department from one site to another. The users of this department need to use a specialized application hosted at the original site, but bandwidth may be an issue. To test, we bought one user's computer to the second site and had him log in and use the software. Even if the test is successful, we need to know the bandwidth impact of not just one user, but of the entire department (in this case, ten users). We needed to find this one user's usage, then we could multiply by the number of users and get an approximate average bandwidth need. To find this:

1. Open Wireshark.
2. Click on Capture, then Interfaces.
3. Your available network interfaces will appear. Find the interface you wish to monitor, then click Options.
4. The Capture Options window will appear. Since we're monitoring the communication between two hosts, we only want to see the traffic between the two. Type host {IP of one host} && host {IP of other host} next to the Capture Filter button. This feature is actually really powerful, allowing you to monitor just per port, per network, exclude hosts or port or networks, etc. For a good list of capture filter options, look at The Wireshark Wiki or this site: http://home.insight.rr.com/procana/.
5. It's a good idea to save your capture to a file. To set this up, enter a location and file name next to the File field.
6. Click the Start button to start your capture.
7. At this point you'll start seeing the packets being captured. Start running your tests.
8. After you're done testing, click Capture, then click Stop. Depending on the size of the capture, it may take a minute or two for the capture to fully stop.
9. Now, click Statistics and click Summary. The Wireshark:Summary window will appear.



Here is the summary of the communications between the two hosts. In my example, the Avg bytes/sec is 4555.791, or approx 4.6Kbps. This can help you determine your bandwidth needs for an application or will help you recognize if one client is taking up more bandwidth than it should. When determining bandwidth needs, you have to realize that this is the AVERAGE bytes per second, not the maximum. There may be certain times that the host could take exceed that, such as on an application open or save. With proper testing, such as taking measurements during each section of the test, you can verify if this is the case and can help you plan accordingly.

You can also use this method to determine if a client is taking up much more bandwidth than it should. If you can capture all communications on a network, such as mirroring your WAN (or Internet) port, you can find out what host is transmitting the most packets or the most bytes. To do this:

1. Start a capture like listed above, but skip the Capture Filter.
2. Once you feel that you have an adequately sized sample, stop the capture.
3. Click Statistics and choose Conversations.
4. I typically choose the IPv4 tab.
5. You can sort by Packets or by Bytes.



In my example, you can see that the top conversation between Address A and Address B has sent many more packets and many more Bytes than the next host. This can help you find out your biggest bandwidth users and will show you whom they are talking to. I've actually used this method to find out some of my bigger users of my Internet bandwidth and was able to determine that the biggest bandwidth hog was those users listening to streaming Internet radio, which gave justification for web filtering.

Wireshark is powerful tool as shown above, but it is much more powerful if utilized well. Also to note, Wireshark was previously known as Ethereal. When searching for more information, most Ethereal information should be applicable to Wireshark.

Software Review: The Hobbit Monitor

When looking for server and network device monitoring, there are quite a few options, ranging from very expensive to free. Today I focus on a free solution that I use, The Hobbit Monitor.

Hobbit is a system based on a plug-in for the Big Brother Monitoring software. Big Brother has been around for quite some time and was bought by Quest Software a few years ago. Hobbit is very comparable the Big Brother system, keeping most of the same general interface and functionality but adding many new features and overall speed improvements.

Hobbit is a monitoring solution for servers and network devices and allows you to write or use extensions to monitor just about anything that responds over a network connection. A central server controls and collects the monitoring and displays the results via a fairly easy to use web interface. It will track history and trends (via rrd) and provides a built-in reporting tool. If there is an issue (that you've defined), such as a down host, the interface will turn red and will performs any alerting actions that you've defined, such as sending an email or sms message.

A member of the Hobbit team provides a live demo here: http://www.hswn.dk/hobbit/

The alerting function has some great features and is very customizable, but is less than straight forward in its setup. An example the configuration file:

$PHONE=MAIL mycell@acme.com SERVICE=conn REPEAT=2h FORMAT=sms DURATION>10m
$SYSADMIN=MAIL admin@acme.com REPEAT=2h DURATION>10m TIME=*:0600:2300

PAGE=servers/siteone TIME=*:0700:2100
MAIL a_sysadmin@acme.com SERVICE=conn REPEAT=50h FORMAT=plain

HOST=%^win.*
$PHONE
$SYSADMIN

Breaking it down a bit, the configuration is in two parts, the definition and rules of the targets and definitions of the monitoring rules.

The $PHONE definition will email mycell@acme.com only if the conn test (ping test) fails, will repeat the alert every two hours, send the alert in sms format. It will also only trigger if the system has been in alert status for more than ten minutes. The $SYSADMIN definition will email admin@acme.com every two hours on ANY failed test lasting over ten minutes and only between the hours of 6am and 11pm. Those two lines define the targets (whom to email) and the conditions at which to email those addresses.

The PAGE=servers/siteone definition will monitor all the hosts on a page on the server. If your server is http://hobbit.acme.com, then it will monitor all the hosts on http://hobbit.acme.com/servers/siteone. Continuing on that line, the section TIME=*:0700:2100 will only trigger the alert if a host is in alert status between the hours of 7am and 9pm. The next line specifies a target, in this case an email address with some extra rules. The HOST=%^win.* will monitor any hosts that has win. in the name. For example, if you name several servers inside Hobbit win.server, like win.mailserver, win.fileserver, win.appserver. No matter where in Hobbit you have these servers, they will be monitored under this rule. The next two lines, $PHONE and $SYSADMIN just call the predefined targets and use the rules defined there.

As you can see, the alerting functionality is very customizable and, even if the setup isn't point and click, not that hard to set up once you have a little understanding about it. Hobbit also features an easy way to pause or stop alerts via it's web interface. You can stop alerts by test type (like ping test, telnet test, etc), set a duration for the stopage (like no alerts for this host for the next two hours, or until the test turns ok), or even schedule a stopage when you are scheduling some downtime for a host.

As I mentioned before, Hobbit can be extended and customized for greater functionality. You can add more tests either by enabling those built in (refernce the help file for details), by writing your own port tests (also in the help), or by adding extensions. See deadcat.net for a lot of extensions and additional tests. Although the majority of these are geared towards Big Brother, with a little bit of code tweaking they can be easily adapted to Hobbit.

Although this system seems to be Linux/Unix oriented at a quick glance, it provides a lot of functionality for Windows systems utiliziing an agent called BBWin. With BBWin, you can monitor resources such as CPU usage, disk usage, memory usage, running processes and services, uptime, and netstat results. You can also add additional extensions (called externals in BBWin) to test for other things. Configuration is done in an XML file on each server and is very customizable.

You can customize the default warn and panic levels for the CPU usage:



Disk monitoring can be configured with a default warning and panic levels:



Or you can specify specific levels per drive based on a percentage or just an amout of space left:



Remote drives and optical drives can be monitored as well:



Services can be monitored wheter they are running or not. You can also automatically restart the service if you so configure it. You can specify any process running on the server just by adding another line with it's service name. Processes are configured similarily:



Example of the Hobbit overview of some Windows servers utilizing BBWin:



Example of the CPU usage monitoring:



Example of the OpenManage extension I use for my Dell servers:



As you can see, Hobbit is a very powerful and customizable alternative to the other server and network monitoring products out there. With a little bit of reading and some work, you can get this system up and monitoring your systems without too much hassle. Hobbit is definately worth a try if you need a solution and don't have the funds to drop for a commercial solution.

Awesome Utility: TestDisk

So, a family member brought me a laptop from a small business owner who he helps with computer issues. Well, the laptop is broke. It looks like it'll boot into Windows (XP Home) and then blue screens. Safe mode does the same thing. Although the BIOS will see the disk, the Windows install media doesn't. And of course the laptop's owner really needs the company data off of it, can't afford data recovery, and, of course, has no backups.

We pulled the drive out of the laptop and used a IDE to USB converter to hook it up to my laptop. Windows recognizes the disk and assigns it a drive letter, but took forever (like 10 minutes) before it showed up in My Computer. Attempting to access the drive via My Computer, command prompt, or even by Run (e:\) would error out. So I figured the disk is in some way corrupt and a third party recovery software was needed.

I tried several recovery softwares, but the one that eventually worked was TestDisk. TestDisk is OpenSource freeware designed specifically for drives with lost partitions or recovering data from non-bootable drives.

From thier website, TestDisk can:

* Fix partition table, recover deleted partition
* Recover FAT32 boot sector from its backup
* Rebuild FAT12/FAT16/FAT32 boot sector
* Fix FAT tables
* Rebuild NTFS boot sector
* Recover NTFS boot sector from its backup
* Fix MFT using MFT mirror
* Locate ext2/ext3 Backup SuperBlock

It can also run under DOS, Windows, Linux, BSD, MacOS, and SunOS and can handle MANY different file systems.

For my issue in particular, I did the following:

1. Hook the drive up your computer. I used an IDE to USB adapter, but I'm sure setting the drive into slave mode and installing it into a PC will work as well.
2. Allow Windows to find the drive (I'm not sure if this is necessary since Windows XP found the drive for me. It may work without Windows recognition).
3. Open TestDisk (did I mention that no install is required?).
4. It asks to create a log file, I chose Create.
5. Select the drive and choose Proceed.
6. Choose the partition table type. Since this drive was running Windows, I chose Intel.
7. Here's the meat of the software. I chose Advanced.
8. Choose your partition you want to analyze. Some drives have more than one partition; even if there's only one presented to Windows, some manufacturers have a Diagnostic or Restore partition.
9. The next option I chose is List.
10. This should list the files on the drive. Select the drive by using the Up or Down arrows. Enter will bring you into a folder. The Left arrow will bring you up a level in the folder tree.
11. Select the file or folder you want to recover and hit the C key to copy. It will present you with an option to choose the directory on the local machine (the machine you're running TestDisk from) where you want to copy the file to. Hit Enter with your choice.
12. After the copy is complete, the text "Copy done!" will appear in green text. You can now choose another file or directory to copy or hit the Q key to quit.

Also be aware that if you copy a large amount of data it will be fairly SLOW. Or at least slower than most people's standards. But you will have your data, so a little time should be no big deal.

Seriously, add TestDisk to your Admin toolbox immediately.

A Couple of VMWare Issues

Sorry for the lack of posting this month; building a 200 person call center from scratch has been dominating my time. It is almost done though, but I still need to find time to post.

Today we discuss a couple VMware issues.

"Operation Failed Since Another Task Is In Progress"

I've seen this error a couple of times. Basically what happens is that a VM will show running but is actually frozen. Any attempts to force a VM shutdown or restart results in the error "Operation Failed Since Another Task Is In Progress". Same if attempted to Vmotion the machine.

Basically this turns out to be a snapshot issue. To fix this without rebooting the ESX server, we can just kill the VM process via command line. I took this tip from a VMware Communities post and cleaned it up a little. The post can be found here.


1. SSH into the ESX server that is currently running the affected VM (or you can use the console).
2. At the cmd prompt enter: cat /proc/vmware/vm/*/names

This lists the running VM's on the host server you are logged on to. Look for the vmid=##

vmid=1069 pid=-1 cfgFile="/vmfs/volumes/45.../server1/server1.vmx"
uuid="50..." displayName="server1"
vmid=1107 pid=-1 cfgFile="/vmfs/volumes/45.../server2/server2.vmx"
uuid="50..." displayName="server2"
vmid=1149 pid=-1 cfgFile="/vmfs/volumes/45.../server3/server3.vmx"
uuid="50..." displayName="server3"
vmid=1156 pid=-1 cfgFile="/vmfs/volumes/45.../server4/server4.vmx"
uuid="50..." displayName="server4"

3. At the cmd prompt enter: less -S /proc/vmware/vm/1149/cpu/status

It will now clear the console screen and show a bunch of numbers and stats. Hit the right arrow key until you see the section about group. Example:

group
vm.1058

With this ID number you can safely kill the VM without corrupting it.

4. At the cmd prompt enter: /usr/lib/vmware/bin/vmkload_app -k 9 1058

(Then number 1058 in the command is an example; your VM's group number goes here.)

5. If you see "Warning: Apr 20 16:22:22.710: Sending signal '9' to world 1058." this means your VM has been closed successfully. You can now start your VM back up and run it.

Unable to migrate due to "Remote Backing" issues with CD/DVD

When trying to VMotion two VM machines, I received the error: "Unable to migrate from VMESX2 to VMESX1: Virtual machine is configured to use a device that prevents migration: Device 'CD/DVD Drive 1' is a connected device with a remote backing.". Going into Edit Settings didn't help; all CD/DVD options where greyed out. I had recently set both systems to use the client device and to disconnect.

I was able to fix one of them; the VMware Tools was still waiting to be installed. I right-clicked on the VM and choose "End VMware tools install". That did the trick and it VMotioned fine.

The other system didn't have that option. I ssh'd into the esx host and from the command prompt ran service mgmt-vmware restart

After about two minutes (including a scary "disconnected" state in Virtual Center that lasted about a minute), it allowed me once again mess with my CD/DVD settings and I was once again able to Vmotion.

More on this issue in this VMWare Communities thread.

Using DSADD.exe to Bulk Create Users in Active Directory

So I had to add about 70 user accounts to Active Directory in preparation for a new call center. Sounds like boring, tedious work if you ask me. Well, it would be without the magic of dsadd.exe, a command in Windows 2003 command line that allows you to create Active Directory objects, such as users, computers, groups, contacts, and OUs. My focus here is on adding multiple user accounts.

Focusing on my needs, I wanted to add the user, set the display name, set a password, set a description, set the office, their title and department, and their logon script while forcing a password change and the ability to change the password. I also wanted these accounts to start disabled since it might be a week or two before the users are ready for them. Have the accounts created in the proper OU would also be nice. Also, my users would be logging with accounts based on their phone extension numbers, since high turnover is a concern.

So, I set up a user, called cc70215. Since I want him in his proper OU, I set him up as cn=cc70215,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com. This was no big deal, I already had the list of users, just copy/paste and some text replacement set up the list of users. With all I wanted to do, I set up the command as such:

DSADD user cn=cc70215,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com -display cc70215 -pwd mypassword -office "Call Center" -title "Customer Service Associate" -dept Collections -loscr cc_li.vbs -mustchpwd yes -canchpwd yes -disabled yes

A success message will return if successful and navigating to the CallCenter, Users OU will reveal my new account. But this is a pain to set up 70 times. And it was 30 minutes before time to go home. So, I got dirty a bit and cheated with the batch script FOR command. First, I got all my users in a comma-separated list. I also had to put quotes around each user. A quick text replacement in my favorite text editor (Notepad2) did the trick. Then I created a batch file, and put in the following:

FOR %%D in ("cn=cc70216,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com", "cn=cc70217,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com", "cn=cc70218,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com", "cn=cc70219,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com", "cn=cc70220,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com") DO DSADD user %%D -display %%D -pwd mypassword -office "Call Center" -title "Customer Service Associate" -dept Collections -loscr cc_li.vbs -mustchpwd yes -canchpwd yes -disabled yes

For this example I only used 5 users, but you get the point.

Put a pause and exit in there and run it as a domain admin. With all luck, your accounts will show up in no time. Now, I did find one issue with this method. Here I'm telling it to set the -display (Display Name) with the variable %%D. What this does is set the Display Name for the account as "cn=cc70216,ou=Users,ou=CallCenter,dc=sysadminhell,dc=com", which is not ideal. Since I was under some time constraints, I just changed the Display Name for the new accounts manually (took me about 10 minutes to prep the script, 2 minutes to run it, then another 10 to fix the Display Name issue). Researching other ways to do this now that I have some free time, I could have done this via wscript (using arrays), used the built-in Windows command CSVDE.exe (see this Technet article for more info), or bought one of several different commercial applications. Even with the one flaw, it did all I wanted it to do for free and under 30 minutes.