Tim's trash of thoughts

A note on Western Digital 2001FASS drives

2010-12-22T09:07:00.003+01:00

About a year ago, I decided to open my wallet and cough up some serious money for a good NAS solution for my home usage. With an ESX whitebox, a growing number of pictures and other digital parafernalia that I like to (permanently) store, I decided that a standalone NAS solution would be more reliable than relying on a single (now aging) RAID controller in my ESX whitebox. After all, a NAS is "system independent" so it can be accessed from any device, as long as there is a network. A few weeks later, I ordered the Thecus N7700 NAS from eBay, together with three Western Digital Caviar Black 2 TB disks (type: WD2001FASS). In the meantime, I upgraded to 5 disks.

Neglecting some configurational complexities between ESX and the Thecus (see my previous blogpost on ESX's iSCSI implementation changes in 4.1), everything has been running very fine... until yesterday.

At 17:04 yesterday evening, I received a gmail notification from the Thecus NAS (yes, it sends mails through gmail) indicating one of the Caviar Black disks had failed and that my RAID5 array was now degraded. I was a bit surprised and already fearing another "Sea-gate" incident with another series of continuously failing disks (the "gate" prefix being so popular with "cablegate", I decided to introduce another one :) ).

I decided to remove the affected disk and run the Western Digital drive diagnostic tools on it (which took a dreadfully long 4 and a half hours). Sure enough, a Full Drive test revealed that there were some bad sectors on the drive but that they were succesfully remapped to the spare capacity that drives get exactly to compensate for a few bad blocks. Still, the RAID array was degraded and the drive was reported as being failed (even though it seems to be very easily fixable), so I decided to dive a little deeper into what happened in an attempt to discover why this is not automatically fixed by the drive when such a bad block is discovered.

What I found out, seriously pissed me off. Western Digital does support a mechanism to automatically remap bad blocks to the spare capacity on the drive. However, this can take a few moments so the question rises how the drive should communicate with the RAID controller to report that it is currently busy to do some block remapping. Western Digital has a technology which they refer to as TLER - Time Limited Error Recovery to delay the RAID array of marking a drive as failed.

Fantastic! The only problem is that this software feature is disabled in the 2001FASS drives, simply because it is considered a "consumer" drive. The even more expensive (and trust me, I had to use all my tactics to convince my wife to cough up the money for what I consider a really expensive drive) RE or "RAID edition" drives are in fact almost identical to the 2001FASS drives, with the exception that they have the TLER feature enabled.

Basically, this means that the 2001FASS drive is not suitable for RAID arrays. When a drive encounters a bad block, it will immediately marked as failed even though this is not the case. Talking about a serious bummer! Some report that TLER is not needed for Linux (which is basically what the Thecus NAS is, a Linux box) but my experience seems to contradict this slightly.

For me, this is an important reason not to buy Western Digital anymore -- you need to cough up an additional bucket of money for a feature that should be enabled in any drive -- after all, all motherboards today support a basic RAID functionality! Or, if you want to upgrade at a given time from one drive to multiple drives...

A note on ESX 4.x and my iSCSI devices

2010-11-24T22:32:00.006+01:00

A few weeks ago, I decided to extend my iSCSI NAS (Thecus N7700) from 3x 2TB Western Digital Caviar Black disks to 5x 2TB Western Digital Caviar Black disks.

Trouble has been my companion ever since. I have been experiencing some serious performance issues since the RAID extension, and was fearing that the different firmware versions of the new Caviar Blacks was confusing my NAS system; mixing firmwares in RAID systems does not seem to be a best practice. The symptoms were very simple: from the moment a lot of I/O was generated (think: 160 MB/s write speeds to the NAS), ESX would loose the iSCSI link to the NAS, which was choking on all that traffic with a 100% CPU usage. As you very well know, storage is ESX's Achilles heel, and very shortly after that, the vmkernel logs would be flooding with messages indicating a path failure to the NAS:

0:00:41:06.581 cpu1:4261)NMP: nmp_PathDetermineFailure: SCSI cmd RESERVE failed on path vmhba36:C0:T0:L3, reservation state on device t10.E4143500000000000000000040000000AE70000000000100 is unknown.
0:00:41:06.581 cpu1:4261)ScsiDeviceIO: 1672: Command 0x16 to device "t10.E4143500000000000000000040000000AE70000000000100" failed H:0x2 D:0x0 P:0x0 Possible sense data: 0x0 0x0 0x0.

After a multitude of firmware up- and downgrades on the Thecus N7700 and a lot of conversation with Thecus Support (which by the way I want to thank for their patience with a guy like me working in an unsupported scenario!), I stumbled across some a strange error message that I had not seen before on an ESX host:

0:00:41:06.733 cpu0:4113)FS3: 8496: Long VMFS3 rsv time on 'NASStorage04' (held for 3604 msecs). # R: 1, # W: 1 bytesXfer: 2 sectors

Some googling quickly pointed me to a few interesting threads, which talked about a VMware KB 1002598 discussing performance issues on EMC Clariion systems with iSCSI. It seems that the iSCSI initiator in ESX allows for for delayed ACK's which apparently is important in situations of network congestion. Knowing that the N7700's CPU usage can sometimes peak to 100% and that this can very briefly can lock up the network link on the N7700, I decided to disable the Delayed ACK's, following the procedure in the VMware KB...

Great success! Performance was rock solid again, and I have no longer experienced ESX hangs ever since!

This made me think a bit, and I remember that I first noticed the performance issues a few weeks after upgrading to ESX 4.0 Update 2 -- I suppose some default setting has changed from a vanilla ESX 4.0 (which I was running earlier) to ESX 4.0 Update 2 that seems to disturb the good karma that I had going between my ESX host and N7700 NAS earlier. Let it be known to the world that also the N7700 with firmwares 2.01.09, 3.00.06 and 3.05.02.2 (the ones I tried) also is subject to the iSCSI symptoms described in VMware KB 1002598.

The joy of WSUS

2010-11-05T17:40:00.006+01:00

After a rather unpleasant electrical powerspike earlier this week had made some of my harddisks go wierd (crashing my ESX server with an equally unpleasant PSOD), a quick inspection revealed that no real harm was done -- except for one of the dozen RAID arrays that I have decided to do an automatic rebuild (no real issue). That finished after a few hours so I was able to go back to my comfortable sofa and enjoy some more quality prime time TV (lol). At least, so I thought...

A few hours later I discovered that my domain controller had not survived the ESX crash and was very unpleasantly complaining about a corrupted registry. Deciding that a bare metal (or virtual metal) Active Directory disaster recovery was not really necessary on my home network (recreating the three user accounts was less effort ;) ), I decided to reinstall my entire domain controller. About 30 minutes after that decision, I was again running a new AD domain with the users recreated and the most important servers already rejoined to the domain.

So what did I forget to configure in my enthousiasm to just reinstall the entire bunch? Certificate services, DFS namespace, DHCP server, re-ACL of file server, recreation of user profiles and also my own WSUS server (which were all happily running on my domain controller as well -- beat that SBS!).

My own WSUS server I hear you say? Well yes, with the very unpleasant (which you will have noticed already is the word of today) bandwidth limitations we have in Belgium, my ISP decides to punish me with some low-bandwidth connection after transferring more than 80 GB of data. That is quite sufficient but I prefer not spending it on downloading all my Windows updates 14 times (which is about the total number of virtual machines, physical laptops and desktops I have running on a frequent basis).

Given that my WSUS partition was about 120 GB and 98% filled, the doom scenario of seeing my entire data transfer that my ISP allows me for this month being entirely consumed by frikkin' Windows updates after reinstalling WSUS & synchronizing for the first time, slowly started to set in. An entire month of "small band" in this digital age? The horror... the horror...

So I decided to spend a few megabytes of datatransfer of very actively googling whether it is possible to prevent WSUS from downloading all the updates from the internet. After all, the registry corruption of the domain controller had completely borked its functionality, yet the separate partition (and separate VMDK) which was holding the WSUSContent directory was undamaged.

Most fora and blogs I found on recycling WSUSContent when performing a new installation, refer to a TechNet page called "Set Up a Disconnected Network (Import and Export Updates)" , which explains how the WSUSContent can be copied from one server to other -- however, they are always exporting & importing the WSUS database as well; unfortunately this database got lost when I -- again -- enthousiastically wiped the entire corrupted OS VMDK.

So I just decided to have a go and installed WSUS from scratch, and I pointed the WSUSContent directory to the partition which already contained the updates from the old server. Then I did the following:

Configured the WSUS server exactly has before (with the same products to update)
Performed the first initial synchronization (this took a long time but using the network bandwidth monitoring in the vSphere client I could clearly see that only minimal amounts of data were transferred during this synchronization -- no actual content was downloaded!)
Approved all the updates that were previously also approved.

This turns out to work quite nicely; apparently when WSUS detects that the updates are already downloaded to disk, it will recycle the existing content! Hurray for WSUS and for not torturing me with small band for an entire month!!

ESX Whitebox & RAID controller failures - an epic struggle

2010-04-01T18:00:00.004+02:00

The past few days have been a bit tense. Not only was there a deadline at work (an interesting study at one of our customers that had to be finished before end of March 2010), but also yesterday, my ESX whitebox decided to die on me. Of course, I took my screwdriver and box of recovery CD's and went to work.. A reconstruction of the epic struggle to get everything back to work (yes, ):

March 31, 8:00 AM. The (old & faithful 100 Mbps) 3Com switch that my PC's are currently connected to -- after having moved and being too lazy to install CAT6 cabling in my new house so I don't live between UTP cables, the wife loves it-- has crashed and had a blinking "Alert!" light; after disconnecting the power, the switch got back up again.
March 31, 8:05 AM. No internet connectivity; road works again, like the day before? Nope, turns out my ESX box, which runs a virtual m0no0wall router, has completely frozen and can only be brought back by a hard reset.
March 31; 8:10 AM. Thirdly, I discovered my Dell Perc 5i controller now freezes the computer after the power has been cycled. Interesting. Trying to enter the Perc 5i BIOS for configuration also freezes the computer. Fear kicks in.

About a year ago, I already burned a Perc 5i controller (including the sizzling, smoke and fireworks) and I decided to buy a second hand controller from eBay again. That replacement never fully worked as I liked it (for example, after resetting the computer, the controller is no longer recognized -- in fact it is only recognized after a power cycle; strange!). A bit pissed off, I blame myself for accepting a half-and-half working controller for hosting all my data (family pictures, personal documents, ...). I'm already fearing that I will have to buy a replacement controller & restore all my data from Amazon S3 & JungleDisk (which I subscribed to after the previous controller went up on smoke)... weeks of downtime.

March 31, 8:30 AM. I remember that shortly after I got the Perc 5i controller, I got a few warnings about ECC errors being discovered in the DIMM that provides the read/write cache. I decide to replace the DIMM as BIOS's crashing all of the sudden seems a bit unreal. Unfortunately, to no avail.
March 31, 8:45 AM. After some fiddling around with the controller, I notice the Perc 5i BIOS is accessible without any drives connected. Puzzling, but after performing a factory reset of the card (erasing the FlashROM) and performing a "foreign array import" of my two RAID arrays, the disks are discovered again & the computer tries to boot up. All this is followed by a little dance of happiness around the computer, thanking the computer gods for resurrecting the RAID array.
March 31, 8:55 AM. Immediately after the import, all volumes seem to report suspicious RAID consistency and an automated consistency check & back initilization is automatically started. The just recovered peace of mind is disturbed and fear for data corruption kicks in. Anyway, the only thing to do is wait several hours for the data consistency checks to complete, so I just boot into ESX.

March 31, 8:57 AM. ESX now freezes somewhere halfway in the boot. Turns out I am running an unpatched vSphere 4.0 which still has an older megaraid_sas. I remember issues were reported with this driver and this is confirmed when inspecting the vmkernel logs. They reveal that the megasas driver is receiving tons of AEN events (Automated Event Notifications):

esx01 vmkernel: 0:03:28:31.377 cpu3:4193)<6>megasas_hotplug_work[6]: event code 0x006e
esx01 vmkernel: 0:03:28:31.387 cpu3:4193)<6>megasas_hotplug_work[6]: aen registered
esx01 vmkernel: 0:03:28:31.518 cpu1:4485)<6>megasas_service_aen[6]: aen received
esx01 vmkernel: 0:03:28:31.518 cpu0:4196)<6>megasas_hotplug_work[6]: event code 0x006e
esx01 vmkernel: 0:03:28:31.528 cpu0:4196)<6>megasas_hotplug_work[6]: aen registered
esx01 vmkernel: 0:03:29:51.334 cpu3:4251)<6>megasas_service_aen[6]: aen received
esx01 vmkernel: 0:03:29:51.334 cpu2:4205)<6>megasas_hotplug_work[6]: event code 0x0071
esx01 vmkernel: 0:03:29:51.349 cpu2:4205)<6>megasas_hotplug_work[6]: aen registered
esx01 vmkernel: 0:03:29:54.318 cpu3:4246)<6>megasas_service_aen[6]: aen received
esx01 vmkernel: 0:03:29:54.318 cpu0:4207)<6>megasas_hotplug_work[6]: event code 0x0071
esx01 vmkernel: 0:03:29:54.334 cpu0:4207)<6>megasas_hotplug_work[6]: aen registered
esx01 vmkernel: 0:03:29:57.405 cpu3:4246)<6>megasas_service_aen[6]: aen received
esx01 vmkernel: 0:03:29:57.405 cpu2:4193)<6>megasas_hotplug_work[6]: event code 0x0071
esx01 vmkernel: 0:03:29:57.421 cpu2:4193)<6>megasas_hotplug_work[6]: aen registered

For an unknown reason, the ESX server is unable to cope with the massive amount of events received and slows down dreadfully (In retrospect I noticed it did not actually crash).

I decide to boot back into the Perc 5i BIOS and let the consistency check finish. Turns out again everything freezes before I can enter the BIOS so I need to disconnect all drives again, perform a factory reset & re-import my RAID arrays. I let the consistency checks start & hurry to work.

March 31, 21:00 PM. Consistency checks have finished but now ESX refuses to boot up, no longer finding the service console VMDK & reports:
```
VSD mount/Bin/SH:cant access TTY job control turned off.
```
Interesting. I discover a VMware KB that describes this behavior, which explains that sometimes LUN's can be discovered as snapshots when changes are made at the storage array. I conclude that my consistency checks & foreign array importing might have messed up the identifiers such that now ESX can no longer find the Service Console VMDK and goes berserk. After following the steps in the KB (basically resignaturing all VMFS volumes), everything works again. Afterwards, I discover that I had switched the two cables connecting both of my RAID arrays (cable 1 got attached to port 2 and vice versa). Doh!!!
March 31, 21:30 PM. Time to install ESX 4.0 update 1a; yet again, another issue: not enough diskspace to install the patches! After cleaning up the /var/cache/esxupdate, sufficient diskspace is available.
March 31, 22:00 PM. After having booted up everything, I again notice a very bad performance of ESX, and my suspicion is confirmed when I notice again the same megaraid_sas AEN events in the vmkernel logs. Strangely enough the error only occurs when I access my fileserver virtual machine, which is the only virtual machine that runs on the second of two RAID arrays... hmmm.
April 1, 13:00 PM. Some time for further analysis. I start a virtual machine running on my first RAID array and see that no AEN events are logged in the vmkernel log. Then I decide to add the VMDK's of my fileserver, all hosted on my second RAID array, one by one. The first VMDK is hotadded to a Windows 2008 virtual machine fine and I can see the data is still intact. Big relief! But indeed, when adding the second and third VMDK, the AEN events are flooding the vmkernel logs again.

At this time, I am becoming more and more convinced that not the Perc 5i controller is involved for the issues, but one or more disks in the second RAID array.
April 1, 14:00 PM. I decide I want to have a look at the Perc 5i controller logs to see if errors are logged at the HBA level. Since the Perc 5i uses a LSI logic chip, I use the procedure I blogged about a while back to install the MegaCLI tool again.

At this point, I discover that it is no longer possible to use the LSI MegaCLI tools under vSphere. I guess VMware finally decided that the Service Console has to run as a virtual machine and the Perc 5i card is no longer exposed inside the Service Console. LSI MegaCLI therefor reports that no compatible controllers are present. Bummer! Apparantly some people report in the VMware Community forums that LSI MSM (remote management server?) seems to work with limited functionality but I decide not to try to install this.
April 1, 17:00 PM. Time to think of an alternative way of discovering what is wrong in the second RAID array. It is a RAID5 array of 4 Seagate 1 TB disks (yes, the ST31000340AS series that had the firmware issues), and my suspicion is now that a single disk has failed, but the failure is not picked up by the Perc 5i controller, or not reported by the disk firmware. That is particularly bad because I don't want to pull the wrong disk out of a RAID5 array with a failed disk -- obviously causing a total data loss, which would be very, very, very, VERY depressing after all the happiness that I still had my data ;).

Time to pull out the Seagate selftests and indeed, testing each drive individually revealed that one of the drives had failed.

So the conclusion is: time for another RMA! I now have had each of my four Seagate 1 TB disks fail on me. In fact, out of the 8 Seagate drives I own, I have already requested 7 RMA's. At times like these I remember why I coughed up a massive amount of money to get my hands on the Western Digital Caviar Black edition (which AFAIK is the last consumer disk to provide a 5 year warranty).

Hosting your DNS on vSphere 4 - caveat

2009-08-31T13:19:00.011+02:00

For a while now, I was having an issue with my whitebox ESX4.0 server: after rebooting this machine, I was unable to connect to it using the vSphere client. The error I was receiving was a simple "503: Service unavailable". The hostd.log on the host was filled with errors like:

--F637FB90 warning 'Proxysvc Req00002'-- Connection to localhost:8309 failed with error N7Vmacore15SystemExceptionE(Connection refused).

and I noticed that the /var/log/messages contains a lot of vmware-authd start & stop messages. I struggled and managed to find a workaround which consisted of:

Logging onto the service console as root

Edit the /etc/vmware/hostd/config.xml file and disabling the "proxysvc" component of hostd.

Restart the hostd process (service mgmt-vmware restart)

Wait for all my autostart VM's to come online

Re-enable the "proxysvc" and restart hostd once again

Today, I discovered this thread on the VMware communities which contained the answer I was looking for: the DNS servers I had configured on my ESX box were virtual machines running on the box itself (in my case: a m0n0wall virtual appliance and a Windows 2008 domain controller with DNS). Apparently this disrupts the proxysvc component of hostd (since the virtual DNS servers are not reachable at the time hostd is first started - autostart is yet to kick in), causing it to fail to start properly and preventing vSphere client connections. Furthermore, this prevented the autostart of VM's all together, thus never getting DNS to get up and running at all.

The solution was to clear my /etc/resolv.conf file and now everything works fine immediately after a reboot (no more attempts to connect to a virtual machine that is not yet running)! This completely slashes DNS support (in particular if you are using HA, you'll need to do good /etc/hosts maintenance). Since your typical production environment probably is not running the entire DNS infrastructure as a or several virtual machine(s), you probably are never exposed to this issue anyway.

Active Directory over SSL in VMware Lifecycle Manager

2009-04-09T18:45:00.009+02:00

I recently have been playing around with VMware's Lifecycle Manager appliance, and one of the small "gotcha's" I ran into was how to configure secure communications between the LCM appliance and the Active Directory backend I was authenticating against.

After configuring LCM to use Active Directory and SSL, I was getting the following error message:

Error: Unable to connect to LDAP Server / simple bind failed: dc.pretnet.local:636

In order to get the SSL authentication working for Active Directory (or LDAP in general), you need to be sure that the Certificate Authority that issues your domain controller certificates is trusted by the appliance (you don't need to actually import the domain controller certificate itself, just the issuing CA is sufficient). This is done by going through the following steps:

First, obtain a copy of the issuing certification authority's certificate (without private key obviously). Ensure that it is in the X.509 format, Base64 encrypted or DER encrypted. The appliance doesn't seem to support certificate containers (P7B format), so when you export the certificate using the Certificates MMC, ensure you select one of the first two options as the export format!!
To add the X.509 certificate to the appliance, go to the "Network" tab and select the "SSL Certificate" configuration pane. Here, import the certificate file.

Next, restart the "VMO Configuration Server", which you can find at the bottom of the "Server" tab in the GUI.

Note: if you get an error message that first you need to fix your LDAP configuration (and "Plugins" section) before you can restart the VMO Configuration Service, go back to the LDAP configuration and disable SSL for a moment.

That's it! Secure Active Directory authentication (which is what we all want) is now working properly! It's a good idea to import the certificate right away, because your other configuration tasks are severily limited when the authentication (either using the built-in OpenLDAP server on the appliance, or using Active Directory) is not working properly.

As a sidenote, I would like to add that, despite VMware recommending to run Lifecycle Manager on a dedicated Windows box (LCM Administration Guide v1.01, p21), the appliance is a really convenient way of running and upgrading this product without too much hassle. Of course, don't forget to offload the configuration database from the appliance (use a dedicated SQL or Oracle server)!

Counting ESX Server storage paths

2008-12-22T11:09:00.005+01:00

At a customer, we have been hitting with one of the built-in storage limits of ESX Server: you can only present up to 1024 storage paths to a single ESX host. Depending on your SAN topology, each LUN that you present over a fiber fabric uses 4, 8 or even 16 storage paths. You can check this using the esxcfg-mpath command:

Disk vmhba1:9:2 /dev/sdf (102400MB) has 8 paths and policy of Fixed
FC 13:0.0 10000000c96e8972<->50001fe15009264e vmhba1:9:2 On active preferred
FC 13:0.0 10000000c96e8972<->50001fe15009264a vmhba1:10:2 On
FC 13:0.0 10000000c96e8972<->50001fe15009264c vmhba1:11:2 On
FC 13:0.0 10000000c96e8972<->50001fe150092648 vmhba1:12:2 On
FC 16:0.0 10000000c96e8ccc<->50001fe15009264f vmhba2:12:2 On
FC 16:0.0 10000000c96e8ccc<->50001fe15009264b vmhba2:13:2 On
FC 16:0.0 10000000c96e8ccc<->50001fe15009264d vmhba2:14:2 On
FC 16:0.0 10000000c96e8ccc<->50001fe150092649 vmhba2:15:2 On

To count the total number of paths presented to a single ESX host, you can use the following service console command:

esxcfg-mpath -l | grep paths | awk '{ split($0, array, "has "); split(array[2], array2, " paths"); SUM +=array2[1] } END { print SUM}'

Probably the awk syntax can be greatly shortened but I am no awk/grep/sed expert :). Nevertheless, you can script this command into a cron job such that you can receive reports on whether or not you are hitting this limit.

App-V 4.5 Certificate Galore

2008-11-30T21:42:00.012+01:00

1) Setting
This weekend I finally found some time to delve a bit deeper into properly configuring an App-V 4.5 infrastructure for large scale deployments. One of the first things that I investigated was the usage of RTSPS for smoother firewall tunneling: as you know, when using RTSP a series of ports is dynamically chosen, which means that you need to open up entire portranges in your firewall. This is not something your firewall guys will like if you work in a larger environment.

Going for RTSPS means you need to use a server public certificate and a corresponding private key in order to let the App-V server sign and encrypt its communications. I have blogged before about how to configure this in SoftGrid 4.1/4.2 -- luckily the procedure for configuring an SSL certificate got a lot simpler. At least, that is what I thought. Some issues I ran into that might save you some valuable troubleshooting time:

As always, when requesting a certificate from your Enterprise PKI, use the Virtual Application Server's FQDN as the subject. It is probably also a good idea to use the hostname as a subject alternate name for those people that still refer to servers by their shortnames.
After the App-V 4.5 Web Management Service has been installed, don't forget to configure the certificate for the IIS Default Website. In IIS7, that requires adding a binding & selecting the proper certificate. It is not clear to me why the App-V installer cannot handle this automatically!?
App-V 4.5 runs under the NETWORK SERVICE account by default and no longer under the SYSTEM account as SoftGrid 4.1/4.2 used to. This has some consequences when it comes to Windows PKI: you need to grant the NETWORK SERVICE account read permissions on the private key.

This later action is a lot harder than you think when reading them ;). Read on for more information.

2) Configuring permissions on private keys
You have three options to get this working:

If you are using a Windows 2008 Enterprise CA and are using your own certificate templates, then you can modify the template to automatically grant the NETWORK SERVICE account read permissions on all certificates issued using that template.

Since you will typically be creating a new certificate template for server deployment (to enable longer than 2 years validity & exporting of private keys), this is probably the easiest solution if you have a Windows Server 2008 Enterprise CA.
In a pre-Windows 2008 CA world, you will have to use the WinHTTPcertcfg.exe tool, the Windows HTTP Services Certificate Configuration tool. In our situation, we need to modify the ACL of the certificate to grant read access to the service account of the Management Service (which is the NETWORK SERVICE by default).

winhttpcertcfg -g -c LOCAL_MACHINE\My -s (subjectname) -a NetworkService

Verify that everything went ok by listing the permissions:

winhttpcertcfg –l –c LOCAL_MACHINE\My –s (subjectname)
It is also possible to explicitly set the permissions on the private key file. This information is based on information obtained from the App-V blog, with some corrections below.
- First, obtain the certificate thumbprint. You can find this in the details tab of the certificate:
  
  Copy/paste the thumbprint for the next commandline.
- Next, use the FindPrivateKey.exe utility to locate the private key file on disk (compiled version available here -- download & use untrusted executables from the internet at your own risk). Use the following syntax:
  
  FindPrivateKey.exe My LocalMachine -t "your thumbprint"
  
  This will give you the full path. Read the caveat message below if this path looks awkward.
- Grant the NETWORK SERVICE account read & execute permissions on the private key file.

CAVEAT:

the location of the private key should be in a publicly accessible location. For WinXP/Win2K3 the default is:

C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys

For W2K8/Vista, this changed to:

C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys

If you have a different location, then take actions to deplace the private key. I requested my certificate through the Web Enrollment pages of Active Directory Certificate Services on Windows 2008. This stores the public & private key in your user account's profile by default. I knew this and drog & dropped the public certificate from the "Certificates (My User)" to the "Certificates (My Computer)" MMC and when your private key was marked as exportable, this is indeed possible. However, this does not actually move the private key and leaves it in your user profile location (for example:

C:\Users\Administrator\AppData\Roaming\Microsoft\Crypto\RSA

). I fixed this by explicitly exporting the certificate & private key from my user account and then explicitly importing everything again. So huge warning for all you regular crypto-users: no more drag 'n dropping of public/private keypairs!

4) Conclusion
A bit messy... yet secure! The move towards the NETWORK SERVICE account for the App-V Management service (... and other Microsoft products as well) is obviously a good choice, yet it brings along some difficulties that probably can be streamlined from within the App-V Management Server's installer.

PS: You didn't forget to grant the NETWORK SERVICE account also read permissions on your content directory, since otherwise your streaming won't work?

VMware Tools without a reboot?

2008-11-21T10:31:00.005+01:00

Every now and then, you see blogposts appearing on the "issue" that you need to reboot a guest operating system after you install or update the VMware Tools. Many people have pondered about whether a reboot is in fact really necessary and if it can be avoided all together. Recent posts about this can be read here and here, refering to this VMware community thread -- the question is still alive in multiple-year spanning threads like this one right here. I usually frown my eyebrowses when reading on these "no reboot" topics, yet I am interested in keeping up with the advancements in that subject for some of the large customers that I come in contact with professionaly.

The scripts and methods outlined in these blogposts sound a bit tricky at first if you ask me, and I feared they might not have the outcome you expected. I would think the VMware tools really require a reboot on some operating systems because you update parts of the virtual device drivers and those need to be reloaded by a reboot of the operating system (Note: strictly speaking you don't need a reboot for all types of device drivers, only under a specific set of circumstances documented by Microsoft. The VMware disk drivers host a boot device so that would fit under the "requires a reboot" category from that document). This means that just running the installer with a "Suppress Reboot" parameter on all your machines will place the new VMware Tools files on your harddisk, but will not actively load all of them... I am not sure if that is a state I would want my production virtual machines in!? And to be very clear: what these scripts do is request an automatic postpone of the reboot, not trigger some hidden functionality in VMware Tools not to really reboot after all!

To remove all suspicion, I did a little test on a Windows 2003 virtual machine and upgraded the tools from ESX 3.0.2 to ESX 3.5U2 without rebooting (using the commandline setup.exe /S /v"REBOOT=R /qb" on the VMware Tools ISO). This effectively updates the following services and drivers without rebooting:

VMware services (bumped from build 63195 to build 110268)
VMware SVGA II driver, VMware Pointing Device driver

It left the following drivers untouched:

VMware Virtual disk SCSI Disk Device ("dummy" harddisk driver - Microsoft driver)
NECVMWar VMware IDE CDR10 (virtual CD-ROM driver)
Intel Pro/1000 MT Network Connection (vmnet driver - Microsoft driver)
LSI Logic PCI-X Ultra320 SCSI Host Adapter (storage adapter - Microsoft driver)

It turned out that these drivers didn't require updating for my specific virtual machine (even after a reboot). In fact, I wasn't immediatelly able to find one machine in the test environment at work that required updating any bootdisk device drivers (and some still had 3.0.2 VMware Tools running!).

To conclude, I would say that in some circumstances it is safe to postpone the reboot of your virtual machine, if at minimum the boot disk device drivers are not touched. Postponing the reboot is very convenient if you use it in the context of a patch weekend where you want to postpone the restart to one big, single reboot at the end of all your patches.

Update: as Duncan Epping points out in a recent blogpost, be also advises that updating the network driver effectively drops all network connections. This is for all practical purposes maybe just as bad as actually rebooting your server, so beware with the "fake level of safety and comfort" that you might have by postponing a VMware Tools reboot!

Matching LUN's between ESX hosts and a VCB proxy

2008-08-14T09:43:00.015+02:00

One of the problems that I encountered at a customer was to discover what VMFS partitions were presented to a VCB proxy. It turned out to be a bit more complex than I had first expected.

Introduction
VMware released the VCB framework (VMware Consolidated Backup) to make a backups of a virtual machine. The VCB framework is typically installed on a Windows host (the VCB proxy), and in order to make SAN backups, you need to present both the source LUN, which contains the virtual machines to backup, and the destination LUN, where the backup files are stored, to that VCB proxy.

This setup is relatively simple to maintain in smaller environments. However, once you get in a big environment were a dozen teams are involved (separate networking teams, separate SAN teams, separate Windows teams and separate VMware teams), it can become quite challenging to find out which of the 12 LUN's that are presented to a Windows host in fact belong to a specific ESX host.

Finding unique identifiers for a LUN
The mission is to find a unique identifier (UID) that can be used both on the ESX host and the Windows box. The first two obvious candidates to uniquely identify a ESX managed LUN on a SAN network are:

The VMFS ID for the partition
Upon the initialization of a VMFS partition, it is assigned a unique identifier that can be found by looking in the /vmfs/volumes directory on an ESX host, or by using the esxcfg-vmhbadevs -m command on the ESX host. The output looks like this:

vmhba1:0:2:1 /dev/sdb1 48858dc4-f4e218d1-d3a8-001cc497e630
vmhba1:4:1:1 /dev/sdc1 483cf914-29b60dc5-dbfd-001cc497e630
vmhba1:4:2:1 /dev/sdd1 479da7c1-4494cd90-d327-001cc497e630

The first disk is the (remainder) of the locally attached storage, and the two other disks are presented from the SAN. The first column indicates that HBA 1, SCSI target 4 and LUN's 1 and 2 are used (and partition 1 on each LUN); the second column lists the Linux device name under the Service Console and the third column lists the VMFS ID.
The WWPN (World Wide Port Name) of the disk on the SAN
On a fiber-channel SAN network, each device is assigned a unique identifier called the WWPN. You can compare the WWPN as performing the same function as a MAC address on an Ethernet network. The WWPN's of the disks that are presented to an ESX host can be obtained from the Service Console using the esxcfg-mpath -l command:

Disk vmhba1:4:1 /dev/sdc (256000MB) has 16 paths and policy of Fixed
FC 13:0.0 10000000c96e8972<->500507630308060b vmhba1:4:1 On
FC 13:0.0 10000000c96e8972<->500507630313060b vmhba1:5:1 On
FC 13:0.0 10000000c96e8972<->500507630303060b vmhba1:6:1 On active preferred
FC 13:0.0 10000000c96e8972<->500507630303860b vmhba1:7:1 On
FC 13:0.0 10000000c96e8972<->500507630308860b vmhba1:8:1 On
FC 13:0.0 10000000c96e8972<->500507630313860b vmhba1:9:1 On
FC 13:0.0 10000000c96e8972<->500507630318060b vmhba1:10:1 On
FC 13:0.0 10000000c96e8972<->500507630318860b vmhba1:11:1 On
FC 16:0.0 10000000c96e8ccc<->500507630303460b vmhba2:4:1 On
FC 16:0.0 10000000c96e8ccc<->500507630308460b vmhba2:5:1 On
FC 16:0.0 10000000c96e8ccc<->500507630313460b vmhba2:6:1 On
FC 16:0.0 10000000c96e8ccc<->500507630303c60b vmhba2:7:1 On
FC 16:0.0 10000000c96e8ccc<->500507630308c60b vmhba2:8:1 On
FC 16:0.0 10000000c96e8ccc<->500507630313c60b vmhba2:9:1 On
FC 16:0.0 10000000c96e8ccc<->500507630318460b vmhba2:10:1 On
FC 16:0.0 10000000c96e8ccc<->500507630318c60b vmhba2:11:1 On

Disk vmhba1:4:2 /dev/sdd (256000MB) has 16 paths and policy of Fixed
FC 13:0.0 10000000c96e8972<->500507630308060b vmhba1:4:2 On
FC 13:0.0 10000000c96e8972<->500507630313060b vmhba1:5:2 On
FC 13:0.0 10000000c96e8972<->500507630303060b vmhba1:6:2 On
FC 13:0.0 10000000c96e8972<->500507630303860b vmhba1:7:2 On
FC 13:0.0 10000000c96e8972<->500507630308860b vmhba1:8:2 On
FC 13:0.0 10000000c96e8972<->500507630313860b vmhba1:9:2 On
FC 13:0.0 10000000c96e8972<->500507630318060b vmhba1:10:2 On
FC 13:0.0 10000000c96e8972<->500507630318860b vmhba1:11:2 On
FC 16:0.0 10000000c96e8ccc<->500507630303460b vmhba2:4:2 On
FC 16:0.0 10000000c96e8ccc<->500507630308460b vmhba2:5:2 On active preferred
FC 16:0.0 10000000c96e8ccc<->500507630313460b vmhba2:6:2 On
FC 16:0.0 10000000c96e8ccc<->500507630303c60b vmhba2:7:2 On
FC 16:0.0 10000000c96e8ccc<->500507630308c60b vmhba2:8:2 On
FC 16:0.0 10000000c96e8ccc<->500507630313c60b vmhba2:9:2 On
FC 16:0.0 10000000c96e8ccc<->500507630318460b vmhba2:10:2 On
FC 16:0.0 10000000c96e8ccc<->500507630318c60b vmhba2:11:2 On

In this output, you can see two HBA's (that have WWPN's 10000000c96e8972 and 10000000c96e8ccc) that see two LUN's vmhba1:4:1 and vmhba1:4:2 that are presented over 16 paths.

On the VCB proxy / Windows box, I used the Emulex HBAnywhere utility to retrieve the WWPN's of the LUN's that were presented. The output is shown in the following screenshot:

It is also possible to use the HbaCmd.exe AllNodeInfo command to retrieve a list of all WWPN's that a certain HBA sees.

Looks nice, what's the problem?
Using the WWPN seemed to be the obvious answer to identifying the LUN's on both the ESX host and the VCB proxy. Until I discovered that two different LUN's where presented using the same WWPN (obviously they were on two different SAN's and presented to two different hosts). On one of our ESX hosts, a 256 GB LUN was presented using WWPN 50:05:07:63:03:08:06:0b, and on the VCB proxy, a 500 GB LUN was presented using that same WWPN -- apparently our SAN team recycles the WWPN's on the different fibre channel fabrics.

To make matters even worse, I noticed that the same LUN was presented using one WWPN to an ESX host, and with another WWPN to the VCB proxy (I am no SAN expert myself but I assume it is possible to present the same LUN in different SAN zones using different WWPN's). I was able to verify this since VCB was able to do a SAN backup of a virtual machine that resides on a LUN with a WWPN on the ESX side that is not presented to the VCB proxy.

The next step: VMFS ID's as a unique identifier
So, if you cannot rely on the WWPN's to uniquely identify a LUN on a host that is connected to multiple SAN's, then surely VCB must use the VMFS ID to know what LUN to read the virtual machine data from? Right?

On the VCB proxy & Windows machine, I tried to discover the VMFS ID's using the vcbSanDbg.exe tool (included in the VCB framework and available as a separate download from the VMware website -- careful, the separate download is an older version than the one included in the VCB 1.5 framework). An excerpt from its lengthy output:

C:\Program Files\VCB>vcbSanDbg | findstr "ID: NAA: volume"
[info] Found logical volume 48761b97-a4f562bd-6875-0017085d.
[info] Found logical volume 48761bc5-3f508baa-2f5d-0017085d.
[info] Found logical volume 483cf913-05b4f526-45b5-001cc497.
[info] Found logical volume 479da7ac-55fe7dfe-378c-001cc497.
[info] Found logical volume 477c2b4a-7db36616-30ea-001cc495.
[info] Found logical volume 48843bec-154cf784-871a-001cc495.
[info] Found SCSI Device: NAA:600508b10010443953555534314200044c4f47494341
[info] Found SCSI Device: NAA:60060e801525180000012518000000374f50454e2d56
[info] Found SCSI Device: NAA:600508b4000901eb0001100003230000485356323130
[info] ID: LVID:48761b97-dacedf9f-ebb9-0017085d0f91/48761b97-a4f562bd-6875-0017085d0f91/1
Name: 48761b97-a4f562bd-6875-0017085d
[info] Found SCSI Device: NAA:600508b4000901eb0001100003260000485356323130
[info] ID: LVID:48761bc6-7b4afa63-97d9-0017085d0f91/48761bc5-3f508baa-2f5d-0017085d0f91/1
Name: 48761bc5-3f508baa-2f5d-0017085d
[info] Found SCSI Device: NAA:6005076303ffc60b0000000000001049323130373930
[info] ID: LVID:483cf913-458f9fa5-a749-001cc497e630/483cf913-05b4f526-45b5-001cc497e630/1
Name: 483cf913-05b4f526-45b5-001cc497
[info] Found SCSI Device: NAA:6005076303ffc60b000000000000104a323130373930
[info] ID: LVID:479da7b6-877867e9-dd06-001cc497e630/479da7ac-55fe7dfe-378c-001cc497e630/1
Name: 479da7ac-55fe7dfe-378c-001cc497
[info] Found SCSI Device: NAA:6005076303ffc403000000000000128d323130373930
[info] ID: LVID:477c2b4a-969e01e0-8d49-001cc495fb46/477c2b4a-7db36616-30ea-001cc495fb46/1
Name: 477c2b4a-7db36616-30ea-001cc495
[info] Found SCSI Device: NAA:6005076303ffc403000000000000128e323130373930
[info] Found SCSI Device: NAA:600508b40006e8890000b000010a0000485356323130
[info] Found SCSI Device: NAA:600508b40006e8890000b00003770000485356323130
[info] ID: LVID:48843bec-28cc17a4-ca9e-001cc495fb46/48843bec-154cf784-871a-001cc495fb46/1
Name: 48843bec-154cf784-871a-001cc495

Unfortunately, I was not able to discover the VMFS ID's I saw on the ESX host in this output, even though there are some resemblances:

ESX host VMFS ID 483cf914-29b60dc5-dbfd-001cc497e630 looks a lot like vcbSanDbg.exe output's logical volume 483cf913-05b4f526-45b5-001cc497.
ESX host VMFS ID 479da7c1-4494cd90-d327-001cc497e630 looks a lot like vcbSanDbg.exe output's logical volume 479da7ac-55fe7dfe-378c-001cc497.

Furthermore, I found out that current versions of VCB do not rely on the VMFS ID to discover virtual machines on a LUN. In Andy Tucker's talk "VMware Consolidated Backup: today and tomorrow" at VMworld 2007, it is clearly stated (slide 19) that there...

No “VMFS Driver for Windows” on proxy

And furthermore that the usage of VMFS signatures is on the "todo" list for identifying LUNs on the SAN network (slide 34).

Other ideas?
So where does one turn when all possible solutions seem to lead to a dead end? Right: the VMware community forums. The answer came in this thread by snapper.

What I learned today is that besides the WWPN on a fiber channel network, there is another unique identifier called the NAA (Network Address Authority) to identify devices on the FC fabric. You can obtain the NAA for the LUN's on an ESX host using the esxcfg-mpath command in verbose mode using:

esxcfg-mpath -lv | grep ^Disk | grep -v vmhba0 | awk '{print $3,$5,$2}' | cut -b15-

The output on our ESX host looks much like this:

6005076303ffc60b0000000000001049323130373930 (256000MB) vmhba1:4:1
6005076303ffc60b000000000000104a323130373930 (256000MB) vmhba1:4:2

The NAA can be seen in the vcbSanDbg.exe output shown above, and can be filtered as follows:

vcbSanDbg.exe | findstr "NAA:"

The output should look like this:

C:\Program Files\VCB>vcbSanDbg | findstr "NAA:"

[info] Found SCSI Device: NAA:600508b10010443953555534314200044c4f47494341
[info] Found SCSI Device: NAA:60060e801525180000012518000000374f50454e2d56
[info] Found SCSI Device: NAA:600508b4000901eb0001100003230000485356323130
[info] Found SCSI Device: NAA:600508b4000901eb0001100003260000485356323130
[info] Found SCSI Device: NAA:6005076303ffc60b0000000000001049323130373930
[info] Found SCSI Device: NAA:6005076303ffc60b000000000000104a323130373930
[info] Found SCSI Device: NAA:6005076303ffc403000000000000128d323130373930
[info] Found SCSI Device: NAA:6005076303ffc403000000000000128e323130373930
[info] Found SCSI Device: NAA:600508b40006e8890000b000010a0000485356323130
[info] Found SCSI Device: NAA:600508b40006e8890000b00003770000485356323130

Et voila, now I can start running the esxcfg-mpath command on all our ESX hosts and start matching these NAA's with those in the output of vcbSanDbg to discover what our Windows VCB proxy has access to.

VMWare D-Day: 12/08/2008

2008-08-12T15:35:00.016+02:00

I recon "12 August 2008" will be long remembered by all VMWare enthousiasts out there.

That is the day that a major bug caused ESX 3.5 Update 2 no longer to recognise any license, even if the license file at your license server was perfectly valid. There is no need to sketch the horror that follows when your ESX clusters no longer detect a valid license: Vmotion fails, DRS fails, HA fails, powering on virtual machines is no longer possible... Ironically, today is also Microsoft's Patch Tuesday of August, which probably means that quite some system admininistrators where caught with their pants down (and their VM's powered off during a scheduled maintenance window) when this bug struck.

The symptoms and errors that we have been experiencing are the following:

Unable to VMotion a host from ESX 3.0.2 to ESX 3.5. The VMotion progresses until 10% and then aborts with error messages such as "operation timed out" or "internal system error".
HA agent getting completely confused (unable to install, reconfigure for HA does not work).
Unable to power on new machines:

[2008-08-12 14:11:16.022 'Vmsvc' 121330608 info] Failed to do Power Op: Error: Internal error
[2008-08-12 14:11:16.065 'vm:/vmfs/volumes/48858dc4-f4e218d1-d3a8-001cc497e630/HOSTNAME/HOSTNAME.vmx' 121330608 warning] Failed operation
[2008-08-12 14:11:16.066 'ha-eventmgr' 121330608 info] Event 15 : Failed to power on HOSTNAME on esx.test.local in ha-datacenter: A general system error occurred

VMWare is promising a patch tomorrow, but several forum posts (here and here) are wondering how this patch will be distributed and -- given the deep integration of the licensing components within ESX -- whether this will require a reboot of the ESX host or not (which can be quite problematic if you cannot VMotion machines away). A possible workaround for this issue is to introduce a 3.0.2 host in the cluster as I have seen in our environment that VMotioning from 3.5 to 3.0.2 still works.

Edit (21:20 PM): hopes are up that VMware should be able to release a patch that doesn't require the ESX host to reboot. See what Toni Verbeiren has to say about it on his blog.

Edit (9:00 AM 13 AUG): a patch has been released by VMware. Regarding whether hosts need to be rebooted or not... there is good news and there is bad news: "to apply the patches, no reboot of ESX/ESXi hosts is required. One can VMotion off running VMs, apply the patches and VMotion the VMs back. If VMotion capability is not available, VMs need to be powered off before the patches are applied and powered back on afterwards."

You can follow the developing crisis at the following sources:

Even our dear friends at Microsoft write about the problem, see the blogpost "It's rude to laugh at other people's misfortunes - even VMware's" here.

WM6 and self-signed certificates

2008-08-08T20:47:00.004+02:00

When playing around with a new (unofficial) WM6.1 rom for my Mio A701, I bumped into a well known problem with installing self-signed certificates on (homebrew?) WM6 ROMs: it is not possible to install a new CA certificate with the error message "The certificate was not successfully added; please restart your device and try again". Obviously, restarting the device did not fix the problem.

A few months ago, I already encountered the problem and I knew you could bypass it by importing the certificate directly into the mobile device's registry. However, the procedures that I read all involved:

flashing Windows Mobile 5 (or a WM6 version that was patched to accept any certificate),
importing the certificate in that temporary ROM,
exporting the relevant registry data,
reflashing back to the rom that has the certificate problem,
importing the certificate through the registry file you obtained earlier in step 3.

As you can imagine, this is quite some work and since I am a lazy person by nature, I did not want to go back to WM5 after just having flashed my Mio to a brandnew and shiny WM6. Therefore, I decided to develop a shorter workaround that doesn't involve reflashing.

The tricky part is that you need to create the proper registry file to import. This file looks like:

Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\Comm\Security\SystemCertificates\Root\Certificates\824AF72AB87E17AC777098A4164D7A90C90C0D69]
"Blob"=hex:19,00,00,00,01,00,00,00,10,00,00,00,4f,e5,c4,01,4e,7d,89,4a,da,42,\
3f,f7,24,0f,7f,a2,19,00,00,00,01,00,00,00,10,00,00,00,cb,bc,40,37,8a,45,2c,\
...

(please disregard the unintentional wrapping of the registry location; everything between the square brackets should be on one line).

The difficult part is converting your self-signed certificate to the proper registry format. Here's how I did that:

On a regular PC, use Internet Explorer to go to a website with the certificate that you want to install on your mobile device (typically this will be Outlook Web Access or something). Open the certificate and install it on your local PC (let the certificate import wizard automatically place the certificate in whatever store it finds necessary).
View the certificate (in Internet Explorer or by using the Certificate MMC) and go to the "Details" tab. There you will find the "Thumbprint" of the algorithm. You will need to look up this number in a few moments, so be sure to remember the first few digits. In the case for the company I work for, the thumbprint is "824af72ab8somethingsomething".
Open your registry editor and go to the following location:

HKEY_CURRENT_USER\Software\Microsoft\SystemCertificates\Root\Certificates\

There should be a registry key that has the thumbprint of your certificate as its name:

Rightclick that registry key and click "Export...". Choose a location for the exported registry data.
Next, open the registry export in Notepad. Replace the registry key location (between the square brackets) to HKEY_LOCAL_MACHINE\Comm\Security\SystemCertificates\Root\Certificates\ followed by the thumbprint. Next, replace the first 12 bytes in the "Blob" registry value by: hex:19,00,00,00,01,00,00,00,10,00,00,00.
Your result should look like this:
Windows Registry Editor Version 5.00

[HKEY_LOCAL_MACHINE\Comm\Security\SystemCertificates\Root\Certificates\824AF72AB87E17AC777098A4164D7A90C90C0D69]
"Blob"=hex:19,00,00,00,01,00,00,00,10,00,00,00,4f,e5,c4,01,4e,7d,89,4a,da,42,\
3f,f7,24,0f,7f,a2,19,00,00,00,01,00,00,00,10,00,00,00,cb,bc,40,37,8a,45,2c,\
...
Compare this with the original registry export that I have shown above, the differences are shown in bold.
Save the registry file, copy it to your mobile device and import it there. Voila! Finished!

You can use the "Certificates" control panel to verify that your certificate is properly recognized!

Note: you must either restart the ActiveSync process on your device because it will not immediately recognize the new certificate; you can kill the ActiveSync process or restart your device (but first wait at least a few minutes such that Windows Mobile can commit your registry changes to memory!).

Obviously, this is completely not supported or endorsed by anybody on this planet. Perform these actions at your own risk and be sure you know what to do in case you brick your device!

Full backups of virtual machines and Windows VSS

2008-07-29T09:57:00.026+02:00

Introduction
One of the new features that is appearing in backup products that take backups of an entire virtual machine, as opposed to using an agent inside the guest operating system, is the ability to cooperate with Windows VSS (Volume Snapshot Service) inside the guest. For example, the recently released version of VMWare's Consolidated Backup 1.5, now supports VSS quiescing for Windows 2003, Windows Vista, Windows 2008; vizioncore's vRanger Pro backup utility has been supporting VSS for Windows 2003 for some versions already.

Several opinions exist on whether this is in fact a useful feature or not; for example, not so long ago the developers of esXpress talked about not including VSS quiescing into their product at that time because it adds additional complexity and does not offer any significant benefits in their opinion (see here). This discussion is still alive as you can see for example here, and the big question is indeed: can you rely on live backups of database virtual machines?

The early days of VSS
The root of the discussion is at the intended use of VSS: on a physical machine that is running a database application such as SQL Server, Exchange or even Active Directory or a DHCP server for that matter, you cannot directly read the database files since they are exclusively locked by the database application. This used to be particularly troublesome because the only way to get a backup of the data inside such a database is to use some sort of export function that had to be programmed into the database application (think of the BACKUP TSQL command or a brick-level backup of an Exchange server).

Microsoft tackled this problem by introducing VSS, which presents a fully readable point-in-time snapshot of a filesystem to the (backup) application that initiates the snapshot. That way, a backup application can read the database file contents and put it away safely in case it is ever needed.

However, there are two problems when reading files from a filesystem that is "frozen" in time:

a file can be in progress of being written (i.e. only 400 bytes of a 512-byte block are filled with actual data).
data still in a filesystem cache or buffer in memory and not yet written to the disk (in the filesystem journal).

On top of the filesystem issues, there are two problems when reading a database that is still in use but "frozen" purely at a filesystem level:

at the time of the snapshot, a transaction could still be in progress. This can be an issue when the transaction is not supposed to be committed to the database at the end: as you know, a database query can initiate thousands of changes and perform a ROLLBACK at the end to reset any changes made since the start of the transaction.

A good (ficteous) example here is when you try to draw 1000 euros in cash from an ATM: if you change your mind right before clicking the "confirm transaction" button on the ATM screen, then you don't want your 1000 euros to be really gone if at the same time a database snapshot is taken and your final "ROLLBACK" command is not included in the database!
some data could still be in memory and not written to a logfile or a database file (so-called "dirty pages").

Crash consistency versus transactional consistency
If you don't take these four problems into account, then restoring a snapshot of such a filesystem would be in fact the same as bringing back up the server after you suddenly pulled the power plug. Such a snapshot is said to be in a crash-consistent state, i.e. the same state as a sudden power-loss.

Modern filesystems have built-in mechanisms (so-called "journalling") to tackle these problems and to ensure that when such a "frozen" filesystem is restored from a backup, the open files are put back in a consistent state as possible. Obviously, any data that only existed in memory and never was written to a filesystem journal/disk is lost. Databases rely on transaction logging to recover from a crash-consistent state back to a consistent database; this is typically done by simply rolling back all unfinished transactions, effectively ignoring all transactions that were not committed or rolled back.

Windows VSS wants to go beyond a crash-consistent snapshot and solves both the filesystem and database problem by not only freezing all I/O to the filesystem but also asking both the filesystem and all applications to flush its dirty data to disk. This allows the creation of both a filesystem consistent and an application-consistent backup. VSS has built-in support for several Windows-native technologies such as NTFS filesystems, Active Directory databases, DNS databases, ... to flush their data to disk before the snapshot is presented to the backup application requesting the snapshot. Other programs, such as SQL/Oracle databases or Exchange mailservers, use "VSS Writer" plugins to get notified when a VSS snapshot is taken and when they have to flush their dirty database pages to disk to bring the database in a transactionally consistent state.

From Technet:

[...] If an application has no writer, the shadow copy will still occur and all of the data, in whatever form it is in at the time of the copy, will be included in the shadow copy. This means that there might be inconsistent data that is now contained in the shadow copy. This data inconsistency is caused by incomplete writes, data buffered in the application that is not written, or open files that are in the middle of a write operation. Even though the file system flushes all buffers prior to creating a shadow copy, the data on the disk can only be guaranteed to be crash-consistent if the application has completed all transactions and has written all of the data to the disk. (Data on disk is “crash-consistent” if it is the same as it would be after a system failure or power outage.). [...] All files that were open will still exist, but are not guaranteed to be free of incomplete I/O operations or data corruption.

Under this design, the responsibility for data consistency has been shifted from the requestor application to the production application. The advantage of this approach is that application developers — those most knowledgeable about their applications — can ensure, through development of their own writers, the maximum effectiveness of the shadow copy creation process.

Conclusions for the physical world: the above makes clear that there is a huge benefit in using VSS when working on physical machines: VSS is a requirement to be able to backup the entire database files and to ensure that the database is not in an inconstent state when you want to do the restore the database- and logfiles and attempt to mount them. The main advantage here is that a restored database does not have to go through a series of consistency checks that typically take up many, many hours.

Going to the virtual world
In the virtual world, there are several different types of backups that can be performed:

Performing the backup inside the guest OS.
Performing a backup of the harddisk files (VHD/VMDK) when using a virtualization product that is hosted on another operating system, such as Microsoft Virtual Server or VMWare Workstation/Server.
Performing a backup of the harddisk files (VHD/VMDK) when using a bare-metal hypervisor based product such as Microsoft Hyper-V or VMWare's ESX/ESXi Server.

Obviously, when you perform the backup inside the guest OS, you still encounter the same problems as when attempting to back up a physical host: open files and database files are locked and thus cannot be backed up directly, so you have to revert to using VSS for the reasons discussed above.

But what about the other two ways of performing a virtual machine backup, when attempting to back up the entire harddisk file? For starters, it is important to realize that "file locking" now occurs at two levels:

The VHD/VMDK harddisk files themselves are opened and locked by the virtualization software (be it the hypervisor for bare-metal virtualization or the executable when using hosted virtualization);
Files can be opened and locked inside in the guest operating system.

The first issue of the open VHD/VMDK harddisk files is solved depending on the virtualization product: if you are using host-based virtualization, you can obtain a readable VHD/VMDK file by using VSS on the host operating system and asking to present an application-consistent variant of the VHD/VMDK files. If you are using a bare-metal hypervisor, a typical mechanism is by taking a snapshot of a virtual machine (which, for example in VMWare ESX, shifts the file lock from the VMDK file to the snapshot delta file, thus releasing the VMDK file for reading).

Open files inside the guest OS
Ironically, the solution of the first problem of open VHD/VMDK host files introduces the second problem of open files inside the guest os: once you have your snapshot of the VHD/VMDK files (be it through VSS for host-based virtualization or a VM snapshot for bare-metal hypervisors)... that snapshot is only in a crash-consistent state! After all, it is a point-in-time "freeze" of the entire harddisk and restoring such an image file would be equivalent to restarting the server after a total powerloss occured.

VMWare attempted to tackle this problem by introducing a "filesystem sync driver" in their VMTools (which you are supposed to install in every virtual machine running on a VMWare product). This filesystem sync driver mimics VSS in the sense that it requests that the filesystem flushes its buffer to disk, guaranteeing that the snapshot -- and thus corresponding full virtual machine backup -- is in a filesystem consistent state. Obviously, this does not solve the problem for databases which tend to react quite violently to these kind of non-VSS "freezes" of the filesystem. Prototype horror stories can be read here (AD) and here (Exchange).

So what are the real solutions for this problem? I can think of two at this moment:

After taking a snapshot, do not only backup the disks but also the memory. Then, when restoring the backup, do not "power on" the virtual machine but instead "resume" it. At first, the machine will probably be "shocked" to see that the time has lept forward and that many TCP/IP connections are suddenly being dropped, but the database server you are running should be able to handle this and properly commit any unsaved data from memory to disk.
Trigger a VSS operation inside the guest OS to commit all changes to disk and ensure filesystem- and applicationlevel consistency, and only then take the full virtual machine snapshot.

The VSS interaction with the guest operating system was first introduced by vizionCore in their vRanger Pro 3.2.0 -- which required the installation of an additional service inside the guest VM, .NET 2.0 and was only officially supported for Windows 2003 SP1+ in 32bit. With the release of VMWare Consolidated Backup 1.5, VMWare announced the default queiscing of disks on ESX 3.5 Update 2 would now be done using the new VSS driver -- supported on Windows 2003/2008/Vista in both 32 & 64-bit variants. Hurray! Problem solved, right?

So VSS seems nice, but is it necessary?
Obviously, your gut feeling will tell you that it is "nicer" and "more gentle" to the guest virtual machine when using VSS when taking a snapshot and a backup. The arguments on the difference between crash-consistency, filesystem consistency and application-level consistency (which translates to transactional consistency for databases) give solid grounds to this gut feeling.

Personally, I cannot find an argument that states that VSS is also really necessary to create a full virtual machine backup. In the physical world, filesystems and databases have been hardened to recover from the crash-consistent state that you obtain when taking a snapshot of a running virtual machine to back up and restore. Hands-on experience about this robustness can be read on several informal channels such as forum posts here.

However, if you want to be sure that your database is in a consistent state (for a faster recovery) and certainty that those few seconds of data that were not yet committed from memory to disk are in fact included in your snapshot, then VSS is what you need. The next question to answer is: what is the risk of VSS messing up and is this probability larger than not being able to restore a non-VSS-based snapshot?

Conclusion
Performing live backups of virtual machines seems like an interesting and simple feature of virtualisation at first. However, at a second glance, there are some important decisions to be made regarding the use of VSS/snapshotting technology that can impact your restore strategy and success. Even without any quiescing mechanism, the operating system should be able to handle the crash-consistent backups that are taken by performing live machine backups and should therefore be sufficiently reliable. With the ready availability of VSS in the new VMWare Tools that come with ESX 3.5 Update 2, much more than crash-consistent backups can be guaranteed without the need to install additional agents. The increased reliability and faster restore time (no filesystem/database consistency checks) that come with VSS quiesced snapshots make full virtual machine backups now a fully mature solution without the need to worry for possibly inconsistent backups.

Side remarks
Some additional remarks regarding full virtual machine backup:

Full VM backups can be an addition to guest-based file level backups, but they can never be a complete replacement:
- you might take a full VM based snapshot of your Exchange or SQL database every day, but a filebased/bricklevel backup (which is far more convenient to use for your typical single file/single mailbox restore operations) might be taken several times a day, depending on the SLA that your IT department has with the rest of the company.
- a full vm backup is a good place to start a full server recovery. It is a bad place to start a single-file or a single mailbox restore.
- a full VM backups using VSS do not allow the backup of SQL transaction logs (see "what is not supported" in the SQL VSS Writer overview), nor do they commit transaction logs to the database in order to clear up the transaction logs (an absolute necessity for Exchange databases or for several types of SQL databases).
Microsoft does not support any form of snapshotting technology on domain controllers. For more information, see MSKB 888794 on "Considerations when hosting Active Directory domain controller in virtual hosting environments".

Edit (12 Aug 2008): VeeAm has released a very interesting whitepaper that discusses not only the necessity for VSS awareness during the backup process, but also during the restore process. They give the example of a domain controller that performs USN rollbacks when being backed up using VSS but not restored using a VSS aware software. Another nice example is Exchange 2003 that requires VSS aware restore software in order to be supported by Microsoft.

Postscriptum: I started writing this article a few days before VCB 1.5 was released, and the original point I was trying to make at that time was that there were too many disadvantages to the available VSS implementations (yet another service to install, .NET 2.0, very limited OS support) to really profit from the benefits that VSS could offer. Of course, in the meantime, VMWare has taken away most of those objections by including VSS support in their VMTools for a wide range of server operating systems. This forced me to reconsider my view on whether VSS would be a good idea or not.

SQL Server 2005 Express Edition on Windows 2008 x64

2008-06-19T23:15:00.004+02:00

While experimenting with the Microsoft App-V 4.5 Release Candidate (more on that soon), I decided to go for a full-blown installation on Windows 2008 x64. Since this is only on my home network, I don't run a dedicated SQL server so I went for the natural choice of installing SQL Server 2005 Express Edition SP2 on my freshly installed Windows 2008 x64 App-V server.

This turned out to be less trivial than I thought. The short answer is: if you want to have a painless install of SQL Server 2005 Express Edition, take the download that includes the “Advanced Services” and simply don’t install them. The “smaller” download package does not include some necessary files for a successful x64 installation.

If you want to go the hard way and patch the setup for easier automated deployment (or just to be ‘1337 and be able to say that you fixed Microsoft’s SQL Server installer for 64-bit systems…), then follow these steps:

First of all, you should know that SP2 is the first Vista/Windows 2008 certified edition (think UAC, think session zero hardening, think enhanced security). Secondly, SQL Server 2005 Express Edition SP2 is supported to run under WOW64. That is very comforting to know, and I hadn't expected a true 64-bit edition for free. So why does it complain about installing a 32-bit version on a 64-bit machine then?

"The installation package has a missing file, or you are running a 32-bit only Setup program on a 64-bit computer"

Of course, what you don't see is that SQL is first installing the SQL Native Client in the background (as a prerequisite) and the error message conveniently forgets to mention that this is in fact the installation that is not succeeding. The error message was indeed accurate, but the error was not that I was trying to run a 32-bit installer on a 64-bit machine, but that the 64-bit installer for the SQL Native Client is not included in the package! What’s even worse, some other essential x64 packages are also not included in the smallest SQL Express 2005 SP2 download.
So you have to include the missing files manually:
1. Download the “SQL Server 2005 Express Edition SP2 with Advanced Services” package.
2. Run both the SQL Express installers with the /X switch to extract the setup files (to different directories):
  
  sqlexpr.exe /x
  sqlexpr_adv.exe /x
3. Next, locate the 64-bit SQL Native Client (sqlncli_x64.msi) and 64-bit SQL VSS Writer (SqlWriter_x64.msi) from the Advanced Services setup and copy them to the "Setup" directory of the regular SQL Express installation.

Et voila! The installer works now. One day, we will live in a perfect world of unambiguous error messages...

Now off to do some more SoftGri... ehr.. I mean Microsoft Application Vir... ehr... I mean App-V testing!

Installing LSI Logic RAID monitoring tools under the ESX service console

2008-05-25T14:29:00.009+02:00

As I discussed in a recent post, I used a Dell Perc 5i SAS controller in my ESX whitebox server. One of the nice features of this controller is that it is a rebranded LSI Logic controller (with a different board layout!), supported by LSI Logic firmwares and the excellent monitoring tools that LSI offers.

Of course, it is important to keep track of your RAID array status, so I decided to install the MegaCLI monitoring software under the ESX Server 3.5 Service Console. Here's how I did it and configured the monitoring on my system:

The MegaCLI software can be downloaded from the LSI Logic website. I used version 1.01.39 for Linux, which comes in a RPM file.
After uploading the RPM file to the service console, it was a matter of installing it using the "rpm" command:

rpm -i -v MegaCli-1.01.39-0.i386.rpm

This installs the "MegaCli" and "MegaCli64" commands in the /opt/MegaRAID/MegaCli/ directory of the service console.

That's it, MegaCLI is ready to be used now. Some useful commands are the following:

/opt/MegaRAID/MegaCli/MegaCli -AdpAllInfo -aALL
This lists the adapter information for all LSI Logic adapters found in your system.
/opt/MegaRAID/MegaCli/MegaCli -LDInfo -LALL -aALL
This lists the logical drives for all LSI Logic adapters found in your system. The "State" should be set to "optimal" in order to have a fully operational array.
/opt/MegaRAID/MegaCli/MegaCli -PDList -aALL
This lists all the physical drives for the adapters in your system; the "Firmware state" indicates whether the drive is online or not.

The next step is to automate the analysis of the drive status and to alert when things go bad. To do this, I added an hourly cron job that lists the physical drives and then analyzes the output of the MegaCLI command.

I created a file called "analysis.awk" in the /opt/MegaRAID/MegaCLI directory with the following contents:

# This is a little AWK program that interprets MegaCLI output

/Device Id/ { counter += 1; device[counter] = $3 }
/Firmware state/ { state_drive[counter] = $3 }
/Inquiry/ { name_drive[counter] = $3 " " $4 " " $5 " " $6 }
END {
for (i=1; i<=counter; i+=1) printf ( "Device %02d (%s) status is: %s <br/>\n", device[i], name_drive[i], state_drive[i]); }
This awk program processes the output of MegaCli, as you can test by running the following command:

./MegaCli -PDList -aALL | awk -f analysis.awk

when being in the /opt/MegaRAID/MegaCLI directory.
Then I created the cron job by placing a file called raidstatus in /etc/cron.hourly, with the following contents:

#!/bin/sh

/opt/MegaRAID/MegaCli/MegaCli -PdList -aALL| awk -f /opt/MegaRAID/MegaCli/analysis.awk >/tmp/megarc.raidstatus

if grep -qEv "*: Online" /tmp/megarc.raidstatus
then
/usr/local/bin/smtp_send.pl -t tim@pretnet.local -s "Warning: RAID status no longer optimal" -f esx@pretnet.local -m "`cat /tmp/megarc.raidstatus`" -r exchange.pretnet.local
fi

rm -f /tmp/megarc.raidstatus
exit 0

Don't forget to run a chmod a+x /etc/cron.hourly/raidstatus in order to make the file executable by all users.

In order to send an e-mail when things go wrong, I used the SMTP_Send Perl script smtp_send.pl that was discussed by Duncan Epping on his blog.

Renaming a VirtualCenter 2.5 server

2008-05-22T23:03:00.002+02:00

After running my VirtualCenter server on a standalone host for quite some time, I decided to join it into the domain that I am running on my ESX box (in order to let it participate in the automated WSUS patching mechanism). This also seemed like a perfect opportunity to rename the server's hostname from W2K3-VC.pretnet.local to virtualcenter.pretnet.local. However, after the hostname change, the VMWare VirtualCenter service would no longer start with an Event ID 1000 in the eventlog.

Somehow, this didn't come as a surprise ;). This has been discussed before on the VMWare forums (here and here), but I post it here because I did not immediatelly find a step-by-step walkthrough.

The problem was in fact twofold, the solution rather simple:

Renaming SQL servers is a bad idea in general (so it appears). For my small, nonproduction environment, I use SQL Server 2005 Express edition that comes with the VirtualCenter installation. If you rename a SQL server, you need to internally update the system tables using a set of stored procedures in order to make everything consist again. This is done by installing the "SQL Server Management Studio Express" and then executing the following TSQL statements:

sp_dropserver 'W2K3-VC\SQLEXP_VIM'
GO
sp_addserver 'VIRTUALCENTER\SQLEXP_VIM', local
GO
sp_helpserver
SELECT @@SERVERNAME, SERVERPROPERTY('ServerName')

The first statement removes the old server instance (replace W2K3-VC with your old server name), the second statement adds the new server instance (replace VIRTUALCENTER with your new server name). The sp_helper and SELECT statement query the internal database and variables for the actually recognized SQL server instances. You need to perform a reboot in order to get the proper instances with the last two statements.
Secondly, the System ODBC connection that is used by VMWare required an update to point to the new SQL Server instance. This was of course done using the familiar "Data Sources (ODBC)" management console.

Afterwards, the VMWare Virtual Center Server service started just fine again.

Enabling Subject Alternate Name certificates

2008-05-02T17:39:00.008+02:00

When requesting certificates from your freshly installed Certification Authority, it can come in handy to specify multiple DNS names that this certificate should be valid for. This principle is known as specifying a list of "subject alternate names" that the server is also reachable under.

Unfortunately, this mechanism doesn't work out of the box with Windows CA's. On your CA, you first need to enable a setting that allows the usage of SAN attributes. Open a command box and type (on one line):

certutil -setreg policy\EditFlags +EDITF_ATTRIBUTESUBJECTALTNAME2

net stop CertSvc & net start CertSvc

Afterwards, use the SAN:dns=&dns= attribute when requesting certificates to enable multiple DNS names.

Windows 2008 Certificate Authority and Windows 2000/XP/2003 clients

2008-04-30T16:39:00.011+02:00

I was experimenting with Windows 2008 Certificate Services the other day in order to create certificates for WSUS 3.0 and for doing SSL tunneling of RDP towards the internet. I noticed that several of my clients were unable to automatically install the WSUS client, with vague errors in the event log (Win32HResult=0x00000000):

I had quickly discovered that the problem was related with the certificate that I had issued for the WSUS IIS server. It turned out that Windows 2008 WSUS clients could connect without any problem to the WSUS webpage, but Windows 2003 and Windows XP clients could not. What made it even more puzzling is that on a Windows XP system, connecting to the IIS homepage didn't succeed using Internet Explorer, but worked perfectly fine using Firefox.

Opening the certificate of my WSUS server gave the following result:

with a "This certificate has an nonvalid digital signature" error in the "Certification Path" details for both the issued certificate and my CA certificate.

Root cause:
The answer is the bleeding obvious: Windows 2008 has several new additions to the cryptography API, called Cryptography Next Generation (CNG), that are used in the V3 certificate templates for CA's and Webservers in Windows 2008. Amongst those new features is support for new certificate signing algorithms (in my case SHA512, a SHA-2 variant) which is not recognized by older clients. Windows XP SP3 adds support for XP, I suppose a future hotfix will add compatibility for Windows 2003.

Solution:
In absense of a worldwide XP SP3 deployment and a working hotfix for W2K3, the only option here is to ensure that the Windows 2008 CA certificate is created with a non-CNG cryptographic provider. If you already created a CA certificate using the new CNG features, the only option is to reinstall your CA and regenerate your CA certificate --- remember how mum always told you to think things over twice before just plainly installing a W2K8 CA... I bet you regret that now (just like I did :D) ? Reinstalling your CA could be messy, and make your PKI infrastructure go berserk, so this time do think twice before going down that road!

Step by Step plan of attack (POA)
So you have decided you want to proceed? First verify that you are indeed using a CNG CSP. To do this, open your registry editor and navigate to the following key:

[HKLM\SYSTEM\CurrentControlSet\Services\CertSvc\
Configuration\{CAname}\CSP]

If you find a CNGHashAlgorithm REG_SZ value, and the HashAlgorithm DWORD is set to 0xFFFFFFFF, then you are using a CNG CSP. If the HashAlgorithm is set to a value such as 0x00008003, then you are already using a "classic" CSP. You can also use the following command on the CA to retrieve the CSP:

certutil -getreg ca\csp\HashAlgorithm
certutil -getreg ca\csp\Provider

which will return the HashAlgorithm and the name of the CSP. For more information, I refer to the Microsoft whitepaper "Active Directory Certificate Server Enhancements in Windows Server Code Name Longhorn", you crypto-boys out there will love it.

Keep in mind that when you are adding the Certificate Services Role to your Windows 2008 server, that you need to specify the proper cryptographic service provider. The image below displays some of the options, what is important to remember here is that all the service providers that contain a hash sign ("#") are CNG providers and thus incompatible with Windows XP SP2/Windows 2003 and earlier clients.

The default cryptographic service provider for Windows 2003 is the "Microsoft Strong Cryptographic Provider", so that is what you want to use. Notice how selecting this provider reduces the number of certificate signing options... SHA-2 algorithms are no longer included! Proceed as usual to end up with a CA that produces certificates that can be handled by legacy clients.

ESX 3.5 on a whitebox

2008-03-09T15:03:00.006+01:00

It has been very quiet from my end for the past weeks because I was very busy at a client & at the same time spending all my free time working on my ESX-on-whitebox hardware project. After being inspired by some colleagues, I decided to order the following hardware:

Asus P5BP-E/4L motherboard
This motherboard supports an Intel S775 processor, has VGA and audio onboard and most importantly, the LAN controllers on this motherboard are ESX certified (Broadcom 57xx chipset).
Intel Q6600 Quad Core processor (2.4 GHz) and 8 GB ECC RAM (4x 2GB)
Just to be sure I have enough CPU power and memory resource pools :)
Dell Perc 5i Integrated SAS Controller
My colleagues advised me that storage was the biggest bottleneck in their ESX whiteboxes (based around the very nice Asus P5M2/SAS board). I decided to go for a dedicated hardware controller. I picked up the Dell Perc 5i controller, which is more or less a rebranded LSI Logic 8408 SAS controller on EBay with 256MB of RAM and a battery backup unit for about 175 EUR.

The main advantage of SAS controllers is that they also support the (cheaper) SATA consumer drives. A quick test confirmed this; I had absolutely no problems at all with this controller & even flashed the latest LSI Logic firmware to it :).

Maybe of interest for some: the later Dell firmwares and also the later LSI logic firmwares for this controller provide support for Write Back without a BBU present.
SATA to SAS cables
The Dell Perc 5i has SFF-8484 SAS connectors on board, so I purchased two Adaptec SFF-8484 to 4xSATA cables from a nearby store to attach all the drives.
8 Seagate SATA harddisks (4x 1TB and 4x 200GB)
Space... loads of space.

The hardest thing was getting all these disks in my Silentmaxx ST11 casing; it required some case modding and loads of patience to get everything well fitted. The 500W PSU that is necessary to provide enough juice, was recycled from an Antec Sonata case. I also added a small 3Com 3C905 100Mbps card for my ISP modem connection.

The installation of ESX 3.5 was a piece of a cake & and I can confirm that the above hardware works like a charm. For those interested, I also noticed that ESX 3.5 supports the ICH7 SATA controllers (found on many consumer motherboards as well). I think -- but this has to be confirmed by someone else -- that you need to configure your ICH7 disks in a RAID before the ESX kernel will accept them as a storage pool.

MAV 4.5: How to perform a Dynamic Suite Composition

2008-02-03T13:59:00.000+01:00

The blog of Justin Zarb details a step-by-step guide of how the Dynamic Suite Composition (DSC) in Microsoft Application Virtualization 4.5 functions. He describes how to include a Snag-It bubble into an existing Office 2007 bubble. There are not many technical details about how DSC works, but some interesting facts are mentioned there:

Appearantly, it is possible to compose multiple bubbles but only one level deep. If you attempt to include an OSD file that in itself has another DSC, this third bubble is not included.
Sequencing tip: make sure your sequencer workstation has all the software installed that you want your second bubble to hook onto. For example: if you are sequencing an application that integrated with Office 2007, do a fat installation of Office 2007 first, and only then start the monitoring and sequencing of the add-ins.

Personal note: also for applications that depend on Java or Oracle clients, you obviously first need to prepare your sequencer workstation by installing those core components.
The user changes that are made in the dynamically composed bubbles are all redirected to the primary bubble's UsrVo_sftfs.pkg files.

I am already planning for a few weeks to delve deeper into the DSC of MAV 4.5 beta and to check in more details what Justin describes, but a project at a customer currently prioritizes my spare time into non-MAV related things. Be sure to check regularly here again for more information on DSC.

Microsoft SoftGrid 4.1 SP1 and 4.2 Hotfixes

2008-01-05T10:20:00.000+01:00

The Microsoft SoftGrid blog contains the announcement for the first Hotfix Rollup Packages for SoftGrid 4.1 SP1 and SoftGrid 4.2. The main new feature for these two packages is support for the MSI Utility that was released at Christmas.

Further improvements include:

Better ActiveUpgrade and better downgrade of a package version.
Improvements to nonpaged pool usage when sequencing large applications.
Improvements when you sequence applications that use both the Microsoft .NET Framework 1.1 and the .NET Framework 2.0.
Improvements to command-line parameter handling of virtualized child processes.

The new versions are downloadable from the Microsoft Support site:

MSKB 938497 for SoftGrid 4.1 SP1
MSKB 941408 for SoftGrid 4.2

Enjoy!

MSI Utility 1.0.0.16 released

2007-12-26T21:49:00.000+01:00

The long awaited replacement for the SMS Connector has been released a few days by Microsoft. The MSI Utility 1.0.0.16 can be used to convert a SoftGrid package into a MSI file that can be deployed using third party electronic software distribution systems. When executed, the MSI wrapper will import the virtualized application on a 4.1 SP1/4.2 SoftGrid client, thus allowing to run virtualized applications in situations where no "MCVAS" server is available. The MSI utility is no longer needed for the 4.5 beta release since the 4.5 sequencer allows direct saving of MSI files (compatible with those produced by the MSI utility).

Mixed-mode scenarios, such as using MSI delivery for initial deployment and streaming delivery for updates, is not supported.

The MSI Utility is freely downloadable from the Microsoft site. In order to use the MSI Utility, you need to upgrade your SoftGrid clients to versions 4.1.1.21 or 4.2.1.21 respectively. These versions, which will contain a set of hotfixes, will be released in the near future.

Microsoft Application Virtualization 4.5 beta - Client impressions

2007-11-20T23:18:00.000+01:00

After publishing a set of first impressions on the two MAV 4.5 beta servers, now it is time to do some initial tests with the 4.5 beta client. Here are some things that I came up with...

1. Installation

The first thing that one notices during an installation or upgrade are the prerequisites that must be met before the MAV 4.5 client starts its installation:

Microsoft Visual C++ 2005 SP1 Redistributable package
Microsoft Core XML Services (MSXML) 6.0 SP1
Microsoft Application Error Reporting

These components are included in the MSI package that the client comes with, so there is no hassle of downloading components of the internet (in case all prerequisites are met, the installer automatically continues). Some remarks at this very first step of the client installation:

The MSXML 6.0 installation triggers the installation of Windows Installer 3.1 (KB893803). This is not explicitly listed as a prerequisite but is also installed along automatically.
My previous 4.2 client installation was detected and upgraded automatically. It was necessary to manually stop the SFTTray icon process (sfttray.exe) before the MSI installer allowed performing the upgrade.
A reboot is required after the installation or upgrade (as was the case with the older 4.1/4.2 clients).

The automatic detection of non-installed prerequisites is very convenient. Let's hope this becomes a standard practice :).

2. Client Management Console

The visual appearance of the Client Management Console did not change a lot, and by starting the "SoftGrid Client Management Console" (which I suppose will be renamed to remove the reference to SoftGrid), we are again greeted by the familiar list of applications, filetype associations and desktop configuration servers.

There are some striking changes that are not immediately noticed:

The remote management of MAV Clients is no longer possible. Yes, you read that correctly, you can no longer use a central Client Management Console to connect to another user's computer and to perform changes.

This has the very unfortunate side-effect that it is also no longer possible to connect back to a local computer using different credentials (this was a very convenient trick to perform administrative actions in a SoftGrid Client Management console when taking over a user's screen when working within the user's limited security context).
Logging can now also be done to the local event log (Application). The loglevel for the event log can be specified separately from the regular sftlog.txt file that existed already in earlier versions of the client, and is configured under the "System Log Level" dropdown that is shown in the figure above.

There are not other visual changes to the client management console.

3. New Client Functionality

There are some interesting new client functionalities that are available for configuration in the background:

Dynamic Suite Composition (DSC): this functionality allows to merge two bubbles on the fly such that the two virtualized applications can communicate with eachother.

This is very convenient for applications that depend on middleware such as Java, Oracle Client or Office; now you no longer need to include in each of your bubbles a copy of the Java or Oracle client, but simply merge a single SFT file containing Java with the application in question.

At this moment, the DSC is limited to two bubbles (one bubble can be merged within one other). I will go into more detail on the DSC feature in a later post where I will explore the limits, how the merging is done, ...
ApplicationSourceRoot, IconSourceRoot, OSDSourceRoot: it is possible to overwrite the SFT, ICO and OSD locations that are specified by a desktop configuration server. This allows to redirect traffic intended for one MCVAS server to another (for example, when traveling to a branch office). The redirection can be done to another MCVAS server (using URL) or to a UNC path.

This redirection is done by changing the respective ApplicationSourceRoot, IconSourceRoot and OSDSourceRoot keys in the client's registry at HKLM\Software\Microsoft\SoftGrid\4.5\Configuration. For example, setting the ApplicationSourceRoot to "rtsp://localmcvas" will force the client to connect to "rtsp://localmvas/data.sft" when the original OSD file says that the SFT file that should be streamed is "rtsp://w2k3-mcvas/data.sft". For ICO/OSD files, one would typically use a UNC path or HTTP path for the redirection.

The SourceRoot redirection can be a very powerful tool for branch offices in combination with the lightweight streaming server discussed in a previous post on this blog. With some clever loginscripting or GPO setting, you can now make the SoftGrid client context aware.
Autoload functionality on clients: applications can be triggered to load FB2 data at a low priority in the background after the application has started and FB1 data has been loaded into the cache. This behavior is controlled by the AutoLoadTarget and AutoLoadTriggers registry keys in HKLM\Software\Microsoft\SoftGrid\4.5\Configuration.

(more functionalities will be added soon :))

4. Internals

Some nice other things to know about the new client:

By default, the MAV 4.5 client no longer supports streaming data from a local file, a trick mentioned before here. Attempting to do so will result in an error message

"The operation failed because you do not have sufficient permissions to stream from a file. Please report the following error code to your System Administrator.

Error code: 450260-14901604-0000180B",

as shown below:

In order to re-enable streaming from files, it is necessary to change the AllowIndependentFileStreaming registry value from 0 to 1 at HKLM\SOFTWARE\Microsoft\SoftGrid\4.5\Client\Configuration.

By the way, now you can also stream from local OSD files that contain spaces in their full path location, this used to give errors on earlier clients.

MAV 4.5 - Things to know

2007-11-18T17:40:00.000+01:00

Here are some small issues that I ran into while testing Microsoft Application Virtualization 4.5:

An important change is that the MCVAS service is now launched under NETWORK SERVICE credentials instead of the LOCAL SYSTEM credentials that were used in the 4.1/4.2 server. This means that the NETWORK SERVICE should have read & list rights on all files and folders of your content directory and on the content directory itself. If this is not the case, the MCVAS server cannot access the package files and this will result in "Launch Failed" errors at your clients.
The 4.5 beta sequencer has the same bug as the older 4.2/4.1 sequencers: when creating a new package and defining the server URL, do not select "RTSP" or "RTSPS" (or "FILE") in the package configuration wizard. This will write an URL with capital "RTSP" or "RTSPS" in the OSD file that is created with the package.

The server does not handle application requests from clients that use capital protocols well and will log an error "Unsupported method" and the client will give you a "Launch Failed" error message. Manually changing the URL inside the OSD file to have a lowercase "rtsp" or "rtsps" in it will fix the problem as was already mentioned in a previous post on this blog. (Ironically, all other XML tags should be in uppercase for them to work properly in an OSD file, the protocol should apparently be in lowercase).
The 4.5 beta sequencer still has the same bug as older 4.2/4.1 sequencers: when upgrading a package, changes made to the package are not shown in the "Files" tab of the sequencer until either you run the Application wizard again (to determine FB1) or until you save the package --- all changes are included but not just immediately shown.
The usage of BITS (Background Intelligent Transfer Service) is not supported or possible with a stand-alone installation of MAV 4.5. Probably the SCCM integration allows to download virtualized applications (MSI files) in the background using BITS, but this remains to be verified & tested.

(to be continued with more issues as they come up)

Microsoft Application Virtualization 4.5 beta - Lightweight Server impressions

2007-11-18T16:17:00.000+01:00

The next step in a first peek at the 4.5 beta of Microsoft Application Virtualization is a short investigation of the Lightweight Streaming Server (LWS).

0. Introduction

The Heavyweight Streaming Server offers full blown streaming and desktop configuration options with a dependency on SQL/AD backends. The LWS version on the other hand, does not depend on these Active Directory and SQL backends, and does not even require a SoftGrid management console to operate properly.

Its intended use is deployment in branch offices: it can stream FB1 and FB2 data to clients that require additional data, and it can be used to authorize application usage in branch offices. This means that a client can launch an application from an OSD file that points to a LWS. The LWS cannot perform desktop configuration, this functionality is only offered in a HWS installation.

The following picture shows a typical deployment scenario for a LWS server in a branch office.

The SoftGrid client is connected in the branch office. Here is the remainer of the configuration

The main office contains a HWS server, which uses the well-known SQL and AD backends, and uses a fileserver to host the SFT/OSD/ICO content.
The branch office uses a LWS server that uses a non-specified algorithm to keep its local content directory in sync with the content fileserver in the main office.
The client is configured to perform a desktop configuration refresh against the HWS server in the main office, which delivers the list of applications that the user has access to.
The ApplicationSourceRoot, IconSourceRoot and OSDSourceRoot point the client towards the local LWS server whenever it needs SFT, ICO or OSD data (more about these configuration options in a post on the 4.5 client). This means that the hefty load of transferring OSD/ICO files during a desktop configuration refresh is taken away from the WAN link.

This makes the advantage of having LWS servers in branch offices clear immediately: in the 4.1/4.2 versions of the SoftGrid server, a desktop configuration refresh could mean the transfer of several hundreds of kilobytes of data per client, per refresh over a WAN link when no SoftGrid server was present in the branch office (and even with a SoftGrid server present, there was still some chatty behaviour between the SQL backend and the SGVAS server). All these problems are remedied by pointing the clients towards the local LWS server that uses a WAN-bandwidth efficient mechanism to keep the content in sync (... do I hear anybody say "SMS/SSCM distribution point replication"?).

1. Installation

The LWS server comes in a separate MSI file that is included in the Microsoft Application Virtualization 4.5 package. The prerequisites of MMC3, .NET2.0 and IIS are not needed for the LWS server. The installation is straightforward:

The default installation path is "C:\Program Files\Microsoft SoftGrid\Microsoft System Center Virtual Application Server". This is the same path as the HWS so I suppose some care needs to be taken in case you want to mix a HWS & LWS installation on a single server (why would you want to do that anyway??).

As with the HWS, it is possible to configure a server certificate, default server port (554 for RTSP, 322 for RTSPS) during the installation.

The only thing needed during the configuration is the "content root" which is the location of the SFT files that you want to stream. This will typically be the folder that contains the replicated files of your main office's content folder.

The entire streaming server configuration is summarized in a single screen during the installation:

Most of these options are familiar from the HWS server's advanced configuration. Here's an educated guess at what the non-trivial settings mean:
- Client connection management
  - Enable User Authentication: when disabled, the user's credentials are not checked when attempting to launch an application.
  - Enable User Authorization: when disabled, the authorization to use a package offline is not passed along with the launch of an application.
- Package management
  - Cache Block Size: this could be the amount of memory that is cached per application to stream from, i.e., some sort of buffer that is filled for each application that is launched.
  - Max. Cache Size: the maximum amount of memory that is used for caching; this should in principle be the maximum amount of memory used by the LWS service (and core processes).
  - Package Detection Interval: the amount of seconds between each scan of the content directory; this scan is used to construct the package list that this LWS server offers.
I must stress that these are interpretations that I give from my personal experience and that they can be wrong; anybody with corrections or a better knowledge, please let me know!

After the installation is finished, no visual footprint of the LWS server can be found: there is no management console or an icon that allows to do "anything". What we find back is the "SoftGrid Lightweight Virtual Application Server" service that runs in the background.

2. Internals

First of all, let's immediately get over the myth that the LWS server does not require anything Active Directory related at all; the server that hosts the LWS must be a member of the domain that your users belong to. When you authenticate users, this is against the Active Directory.

Furthermore, some AD connection is needed because supposedly the LWS will grant users access to the applications that they have access to based on the NTFS permissions of the files in the content directory that you specified during the installation.

Some more things that you might find interesting:

From the moment a user has read permissions on the SFT file of the package that is being started, the LWS server will authorize that usage. This check can be bypassed by disabling the user authentication, either during the installation or by changing the registry key EnableAuthentication to zero.

I noticed during my tests that setting the EnableAuthorization to zero, would also automatically put the EnableAuthentication to zero (i.e., disabling authorization also disables the authentication checks).
The LWS server seems to require 4.5 clients; older client versions will give "Error parsing the XML file" messages when attempting a desktop configuration refresh against a LWS server. But even with 4.5 clients, a desktop configuration refresh is not possible, an error message saying "The server threw an exception" is listed. To me, it is not clear whether or not the LWS can perform a desktop configuration or not.

The configuration of the LWS is placed entirely in the registry at the following location: HKLM\Software\Microsoft\SoftGrid\4.5\DistributionServer. Interesting name, that hints at a future integration with SCCM distribution points if you'd ask me.
The application metering statistics of a LWS server are not stored anywhere. When launching an application that points directly to a LWS, or that was redirected there by using ApplicationSourceRoot, does not log the usage information (even if the HWS server is chosen as the desktop configuration server).

This means that your reports pulled from a HWS server will not include the launches that were allowed by LWS servers.

3. Roundup

The LWS server seems to be the long-awaited answer to the branch-office scalability issues that haunted the older 4.1/4.2 SoftGrid servers, without eliminating streaming completely as is the case with using a "stand-alone" mode of the SoftGrid infrastructure.

The Microsoft Application Virtualization client can be redirected towards the local LWS server by setting the appropriate ApplicationSourceRoot, IconSourceRoot and OSDSourceRoot registry keys, thus achieving the context-aware streaming that we have been waiting for (without the burden of setting up a replicated database or a full blown dedicated SoftGrid server in a branch office).

Microsoft Application Virtualization 4.5 beta - Heavyweight Server impressions

2007-11-18T13:08:00.000+01:00

In a first series of post regarding the recently released 4.5 beta version of Microsoft Application Virtualization, we're going to have a look at the standard server installation, refered to as the Heavyweight Streaming Server (HWS). The stand-alone mode and "Lightweight streaming server" (LWS) mode will be discussed in an upcoming series of posts.

0. Introduction

The new release of Microsoft Application Virtualization 4.5 introduces a set of deployment models that are available for delivering virtualized applications to clients. This post discusses the HWS model, which is closest to the "traditional" SoftGrid server that we know from the older 4.0/4.1 (SP1) releases of the SoftGrid server. The components for this set-up are as follows:

MCVAS server (Microsoft System Center Virtual Application Server) which performs the desktop configuration of the clients and performs the streaming.
SQL server and Active Directory backends for the MCVAS.
Fileserver for the content location
Management Web Service which communicates with the SQL backend.
Management Console for performing the MCVAS configuration; the management console connects to the management webservice.

1. Installation

The MCVAS setup comes in the form of an MSI installer, as was the case with the previous 4.x server installs.

After starting the installer, I was informed that I need to install MMC 3.0 before the MCVAS installation could complete. This is quite surprising since I already had installed MMC 3.0 on that particular machine (Windows 2003 Standard Edition SP1). Luckily, this was not a showstopper for the installation which continued after this error message.

An existing 4.1 SP1 server installation is properly detected and an upgrade path was proposed. This upgrade succeeded flawlessy.

Prerequisites for installing the MCVAS server are the same as for a 4.1 SP1 installation:
- Microsoft Management Console 3.0
- Microsoft .NET Framework 2.0
- Internet Information Services Web Service (presumably IIS 5.0 is sufficient?)
It is no longer possible to install the "SoftGrid Client Management Console" from the server installer.

The default installation path has changed from "C:\Program Files\Softricity\SoftGrid server" to "C:\Program Files\Microsoft SoftGrid\Microsoft System Center Virtual Application Server".

It is no longer possible to install MSDE --- aka "the SQL overlords will punish you if you still use that one" --- during the MCVAS installation. The availability of a database is now simply a prerequisite for the installation. I used SQL Server 2005 Express Edition SP2 without problems during my tests.

The default database name is now "SOFTGRID", which is irony at its best. The very release that they finally change the default database name from "SOFTRICITY" to "SOFTGRID", the product gets renamed.

It is no longer necessary to provide the a SQL built-in account with sufficient privileges during the installation; integrated authentication is used instead (no longer necessary to configure your SQL server to mixed authentication). The user that is installing the MCVAS needs sufficient privileges on this server to create a new database or to modify an already existing database.

Several new features that are asked during the installation include:
- The "SoftGrid Secure Communication Mode", which allows to assign a server certificate to the server to immediately allow secure communications (more on this later in this post). The server certificate must be installed before the setup is started.
- The port to use for the MCVAS server's communications (RTSP, default: 554)
- The port to use for the MCVAS server's secure communications (RTSPS, default port: 322). That is no typo, the default port number for RTSPS has indeed changed from 332 to 322.

A small "bug", or rather, discomfort, that I ran into is when trying to install the MCVAS on a server that is not joined to a domain. When prompted for the SoftGrid Administrators and SoftGrid Users group, this leads to an error in the installer log file "ADSOpenObject Error 0x80005000, no value for domain name". The default option of "Domain Admins" (for the SoftGrid Administrators) is listed instead, and only when trying to continue, an error message stating that the domain could not be found is listed. We will see later in this report that indeed it is no longer possible to install the MCVAS on a server that is not joined in the domain.

2. Management Console

A first thing I noticed (also throughout the installation) is that the term "SoftGrid" is still used in many locations; for example, the management console is still named "SoftGrid Management Console", the opening picture that has "Softricity" in it is still used, and the reports still contain the same old logo's.

Sure, no drama here, but it makes you wonder why 17 months after the purchase of Softricity by Microsoft, no-one took the time to replace the GIF images in "C:\Program Files\Softricity\SoftGrid Management Console\images" ?

At first sight, not many things seem to have changed in the MCVAS management console:

The good old options "Applications", "Filetype Associations", "Packages", "Application Licenses" and "SoftGrid Administrators" seem to be the same as in the previous versions with no visual differences.

The "Provider Policies" option has been simplified; in the provider pipeline, the "Basic Authentication" and "Anonymous Authentication" have been dropped, now you can only authenticate using "Windows Authentication".

This might have a consequence for those of you that use separate provider pipelines for anonymous authentication or PC's that are not joined in a domain.

An interesting development in the "Server Group" option is the easier management of certificates in the "Ports" tab of a SoftGrid server.

By clicking the "Server Certificate" button, it is possible to select a certificate that is stored in the Computer account's personal certificate store as the base certificate for RTSPS communications. This greatly simplifies the configuration of encrypted streaming (see a previous blog post of mine for the lengthy procedure in pre-4.5 servers).

In the advanced configuration of a SoftGrid server, the "Max. Block Size" option is still present, even though AFAIK, the entire 4.x range of servers simply ignores this option and automatically determines the maximum block size.
The greatest surprise is without doubt the absence of the "Account Authority" option which allowed to configure a SoftGrid AD browser account in previous versions of the SoftGrid server. In practice, this means that the reference domain for assigning rights to applications is now the domain that the MCVAS server is placed in. This is also a simplication that was to be expected after generic LDAP support was dropped.

3. In the background...

Of course the major changes implemented in version 4.5 are not in the graphical interface, but functionalities that are only visible when looking a bit deeper. Here are some nice changes that I discovered when poking around in version 4.5's internals:

Remember that pesky server.conf file that contained a database user and password in plaintext? That is finally fixed: now the machine account of the MCVAS server is used to read the configuration from the database (the machine account receives SFTeveryone and SFTread roles in the database). In order to make changes to the database, the default security on the database is such that you need to be in the SoftGrid Administrators group (SFTadmin SQL role). This also means that no longer the "sa" account is needed during the installation, and that mixed authentication can be disabled on your SQL database server.

The registry keys for the SFT content and logging have moved from "HKLM\Software\Softricity\SoftGrid Server" to "HKLM\Software\Microsoft\SoftGrid\4.5\Server". The only new value created there is a "Version" REG_SZ value which contains the complete build number of the 4.5 SoftGrid server, in casu "4.5.0.606".
Corrected: The older SoftGrid 4.1 / 4.2 clients seem to (partially?) work with the new 4.5 beta server. I was able to stream a pre-4.5-sequenced application but let it be clear that there is no guarantee whatsoever that a new 4.5-sequenced package will work on an older client. (This should not come as a surprise of course, how would an older client know how to interpret features that were introduced in a newer version of the sequencer/server?)

In fact, as Gene Ferioli pointed out in a comment, Microsoft does not support the connection of older clients to the 4.5 server; the recommended upgrade path is to first upgrade all the clients to 4.5 (since this client knows how to talk to the earlier 4.1/4.2 servers) and only then to upgrade the server to version 4.5.

4. Roundup

The functional improvements that are visible in this beta version of the HWS MCVAS surely have their benefit:

the integration with the domain is more tight (no need for browser accounts, direct Windows authentication, no database accounts).

the configuration of RTSPS is much easier and now directly compatible with a Microsoft Certification Authority.

Besides these small functional improvements, there have not been many changes to the Virtual Application Server. At this moment, after having investigated the main server component, I am a bit afraid that this new version is also mainly about improvements in the virtualization itself (i.e., at the client & sequencer side). I don't see any immediate improvements in management and scalability:

it is still not possible to have a delegated SoftGrid administrator permissions (as is possible, for example, in SMS and SCCM where you can have separate server and package administrators).

For centralized client management, it still seems necessary to revert to third-party or self-made administrative templates for GPO's.

The scalability issues should be tackled by introducing different deployment schemes as discussed in the beginning of this post. These different deployment schemes will be discussed in a series of other posts. To be continued!

Microsoft Application Virtualization 4.5

2007-11-13T21:38:00.001+01:00

The long awaited version 4.5 of "Microsoft SoftGrid" has been released in beta, and there are some remarkable surprises to be read in the release notes.

First of all, the name "SoftGrid" is dropped and now we should talk about "Microsoft Application Virtualization".
Secondly, the centralized management and integration with SCCM 2007 is now in effect. This was the major improvement in functionality that was expected previously. This is basically manifested in the different installation modes that are possible:
- Standalone mode: which allows the execution of virtual applications without the streaming but using SCCM or another electronic software distribution (ESD) tool to get the SoftGrid package at the client.
- "SoftGrid compatibility" mode: the infrastructure as we know it from version 4.0 - 4.2 where a central application server performs the desktop configuration, performs application tracking/licensing/access management and streaming.
- Streaming Server mode: which is a trimmed down version of the previous mode where only streaming is performed at such a server. I suppose that this is the component that will be placed on SCCM distribution points once SCCM2007 R2 (which would feature Microsoft Application Virtualization integration) is released to allow for distribution points to perform the streaming.
Finally, the big new feature is that we now can merge bubbles on the fly! The "Dynamic Suite Composition" feature allows to create separate packages for middleware such as Java Runtime Engines or Oracle clients.

The beta can be downloaded at the Microsoft Connect website. More details on this new release can be read at the SCCM site or at the SoftGrid team blog --- I wonder what we should call that blog from now on ;).

I am yet to test this new release but you can expect some detailed feature reports very soon.

Virtualized Domain Controller's time synchronization issues

2007-11-04T11:43:00.000+01:00

At home, I use the excellent VMWare Server to run my own little network with domain controller and SoftGrid server, like most of us do. Due to circumstances, I regularly reinstall the entire network, and a single problem keeps on recurring: the times on all my computers go nuts after about a week of running properly synchronized. Quite annoying if some of your client PC's have a TV-card with a thight recording schedule ;).

The problem is the bleeding obvious: the time on the domain controller (which holds the PDC role as it is the only domain controller in my miniature network) is not correctly synchronized anymore after a while. I noticed that it starts with small deviations at first, but very soon, this accumulates to very large deviations. The skew between the "real time" and the DC's time seems to increase in a nonlinear fashion. Also, the DC uses bridged networking so external timesources such as "time.windows.com" are reachable at all times.

I still have to figure out if this is due to the fact that this is a virtualized domain controller on VMWare Server; I cannot remember seeing this problem elsewhere (not on a VMWare ESX or on physical installations of a domain controller). Anyway, here is the solution to the problem, you might find it useful at a given point in time:

First of all, inside the virtual machine, ensure that the option "Time synchronization between the virtual machine and the host operating system" is disabled. In my setup, this could potentially lead to a loopback since the host OS is in fact a member of the domain of which it is hosting the domain controller (and PDC emulator) in a virtual machine.

Secondly, I had not configured the domain controller to use an external time source. The procedure is detailed at Microsoft's Knowledgebase article 816042, but in essence, it comes down to setting the following registry values under HKLM\SYSTEM\CurrentControlSet\Services\W32Time\:
- Parameters\Type = "NTP"
- Parameters\NtpServer = "time.windows.com,0x1"
- Config\AnnounceFlags = "5"
- TimeProviders\NtpServer\Enabled = "1"
- TimeProviders\NtpClient\SpecialPollInterval = "60"

Finally, you need to restart the W32Time service at the domain controller using net stop W32Time followed by net start W32Time.

On your clients, use w32tm /resync to reset the time according to your domain controller (you can also use this command at the domain controller to immediatelly poll the NTP server you specified).

What happens to Feature Block 1 when you upgrade a package?

2007-11-01T20:13:00.000+01:00

Recently, I had a discussion with a colleague around what happens to Feature Block 1 when you perform a package upgrade: does it get completely restreamed, when is the FB1 data erased when opening a package for upgrade, ... ? I decided to perform a set of small tests in order to figure out what happens exactly.

Here are the results: I used the SoftGrid Sequencer 4.2 build 303 for creating the packages, SFTExplorer 1.0 for finding out what the FB1/FB2 contents looks like and NetLimiter 2.0.10 Freeware to monitor network traffic.

When is the FB1/FB2 separation erased when opening a package for upgrade?

A first test consisted of finding out when the FB1/FB2 separation is erased when using the "installation wizard" for making modifications to the package:

When adding a new file to the package using the "installation wizard", and not running the "application wizard" (FB1 determination), results in a loss of a previous FB1/FB2 division. The entire package becomes FB1 data.
When adding a new file to the package using the "installation wizard", and then running the "application wizard" for FB1 determination, results in FB1 containing those (parts of the) files that are accessed during the FB1 monitoring. This is precisely the expected behaviour.

When opening the package for upgrade and immediately saving the package without making any modifications, retains the FB1/FB2 separation. As mentioned above, when you click the "Begin Monitoring" button in the "installation wizard", the FB1/FB2 separation vanishes, even when not changing anything in the package. This means that the "Begin Monitoring" click is (roughly speaking) the point where the previous FB1 is lost.

A second test consists of what happens when you use the "Encode Directory" option to modify the contents of the package instead of the "installation wizard". Something interesting here:

When you use the "encode directory" wizard, the FB1/FB2 separation is lost, which coincides with the behaviour of using the "installation wizard". Strangely enough, now the entire package becomes FB2 (!!) with only SoftGrid's osguard.cp file being in FB1.
When using the "encode directory" wizard, followed by the "application wizard", creates the usual FB1/FB2 separation that one would expect.

Here, the start of the "encode directory" wizard triggers the erase of the previous FB1/FB2 separation.

What is streamed to a client after a package was upgraded on the SoftGrid server?

This is very simple: the entire package needs to be restreamed to the client when the package is upgraded on the server. I performed the following three tests on a SoftGrid package that contained 3 files of 5 MB each:

The first time the package was created, I added one file of 5 MB to FB1 (by reading it into WordPad during the monitoring phase). After publishing this application on the SoftGrid server, the SoftGrid client pulls in 5 MB as expected ; this was monitored using NetLimiter and verified in the SoftGrid client management console.
Then I opened the package for upgrade, ran the "Application Wizard" and now touched the original 5 MB file that was already in FB1 and also a second 5 MB file. The sequencer correctly reports that the FB1 size then becomes 10 MB, as expected.

The client's total transfer size after placing this new package on the server (using "Add Version") came down to 10 MB, the entire FB1. Notice that the launch bar kept on saying "100%" at the client side, even though a restreaming was in fact being performed.
The third test consisted of opening the package with a 10 MB FB1 size and changing a single byte within one of the files that was in FB1. Again, it turned out that the entire FB1 was transferred to the client when I upgraded the package on the server.

I have heard several people say that SoftGrid will only stream the difference between an original package and the ActiveUpgrade that is performed on it; from these simple tests it turns out that this is not the case: upon every package upgrade, all previously cached data is invalidated at the client.

In a sense, this is not really surprising: at every save of a package, it is restructured and optimized for streaming according to the FB1/FB2 separation that was specified during the application wizard. This new SFT structure could in principle be completely different from the old SFT, making it nearly impossible or inpredictable on a 32 KB-block level to say which blocks need retransfering. I don't really see a practical way of implementing this when using a streaming protocol that only knows about block numbers and not about the contents of the package.

SoftGrid applications without a SoftGrid server

2007-11-01T17:46:00.000+01:00

A convenient trick to launch a SoftGrid application locally, without contacting a SoftGrid server, is to direct the SoftGrid client directly towards the SFT file. This can be done by editing the OSD file and replacing

HREF="rtsp://%SFT_SOFTGRIDSERVER%:554/content/appname/app.sft"

with

HREF="file://D:/SoftGrid/appname/app.sft"

(replace by the correct path) and then double clicking on the OSD file. This triggers SFTTray.exe to launch the specified application.

Some caveats and things to know:

The path that you specify may not contain any spaces or you will get a "Launch failed" error message.
The "file" prefix in the URI is case-sensitive (but this is also the case for the more traditional rtsp & rtsps prefixes).
You need the "Add Applications" privilege in order for this to work: double-clicking the OSD file of an application that is not known on the system will first add the application and then start loading FB1 into the cache. Only afterwards, the application is started.

This means that by the default configuration of the SoftGrid client, you need to be a local administrator before you can use this trick.

Package information leakage through the Q drive

2007-10-18T20:45:00.000+02:00

We all know that if you are a local administrator on a machine that has a SoftGrid client, then you can see all the applications that have data in the local sftfs.fsd cache file by opening the SoftGrid Client Management Console and finding all the applications that don't have a "Idle (0%)" package status. This is because local administrators can see all applications that are defined on a machine in the Client Management Console, whether or not they actually have the proper group memberships or SoftGrid access to be able to start the application.

A colleague (thanks Anthony) pointed me out to another interesting way of finding out what applications have cached data on a SoftGrid client, but that doesn't require administrative privileges. Simply go to the root of the Q drive inside a bubble and look at what asset directories you can see!

It turns out that a user can see all the asset directories of all applications that have some data cached at the client, even if the user doesn't have access to the applications themselves (note: of course you cannot see the contents of these directories since this is prohibited by the isolation; you can only see their existence).

As an illustration, notice on the figure below how the user only has access to Office 2000 and still can see the asset directories of Office 2003 and Office 97 on the Q drive.

You can do this test yourself as an end-user by modifying a locally cached OSD file (hey, they are in the All Users profile with write access for all!) and using the following script tag:


<SCRIPT TIMING="PRE" EVENT="LAUNCH" PROTECT="TRUE" WAIT="TRUE">
  <SCRIPTBODY>
    @echo off \n
    dir q: \n
    pause \n
  </SCRIPTBODY>
</SCRIPT>

Then (as a local administrator), perform an unload of one of the applications that you don't have access to as a regular user and witness how the next time you use the script above under the user's account, the asset directory has vanished!

Note on motivation:

So you might be thinking... big deal, so what? Who cares about that asset directory being visible or not?

It was important for one of our customers, because they wanted to prevent users from seeing what applications were installed on a machine (locally installed and virtualized). One reason for that was security: knowledge of applications installed might open ways to use certain exploits. Another reason was cost: they had a very peculiar licensing system for a set of applications that required them to cough up the license fee from the moment an end-user could see that the application was installed. I suppose they didn't want to enter a legal dispute whether the visibility off the asset directory would mean they have to start paying license fees ;).

Automatic starting of SoftGrid applications

2007-10-18T19:07:00.000+02:00

It is possible to publish applications to the "Startup" menu, i.e. by placing an icon in %CSIDL_PROGRAMS%\Startup location. However, it might not be a good idea.

The problem occurs when you have use the desktop configuration to perform refresh cycles of the shortcuts of the icons, and in this case, the problem is two-fold. First of all, the icon is added to the start menu only after the user has logged in (at least when you configured the SoftGrid client to perform a refresh upon login). This means that the first time the application is published in the Startup menu, the application will not execute since the logon moment has passed and Windows already processed all the Startup menu entries. On the other hand, from that moment onwards, the SoftGrid shortcut will remain in the Startup menu, it points to the locally cached OSD file and the next logoff/logon of the user will start the virtualized application from the Startup menu as expected.

The second problem occurs when retracting the user's rights to this application. The icon will be removed the next time the user logs on again (... if you enabled refresh upon logon, that is; the reasoning remains valid if you use periodic refreshes and the user logs off before that period refresh removed the icon). However, before the SoftGrid client removes the icon, Windows will already have processed the Startup menu and attempted to start the virtualized application, which will result in an error message because the user no longer has the proper rights.

If you manually publish your shortcut in the Start Menu's Startup folder (not relying on the desktop configuration of the SoftGrid client), you are not vulnerable to this error. You will notice that the SoftGrid application will be started upon the user login, as expected.

Terminate Children (yes/no/abort) ?

2007-10-18T18:27:00.000+02:00

When you specify the TERMINATECHILDREN="TRUE" value in the VIRTUALENV element, then all processes that are running in the bubble are killed when the main application closes. The name of the element might let you think that only child processes of the executable that you specify as FILENAME in the CODEBASE element are removed, but this appears not to be correct. This also means that any script that you still have running in the background can be killed, or a command prompt that you opened up for troubleshooting.

In my opinion you should be careful using TERMINATECHILDREN, since the child executables and other processes can also perform I/O operations that might get interrupted. If you want to alert the user of the killing of background processes and make them aware of this danger, you might want to check out the LaunchIT utility from Tim Mangan. This utility monitors all child processes that are launched by the main process and optionally alerts the user when child processes remain once the main process has closed.

For more information, see the OSD illustrated which also explains this, or Microsoft KB 931589.

Hiding the Q drive on SoftGrid clients

2007-10-17T20:43:00.001+02:00

So finally you have gone through the work of getting a SoftGrid server up and running, configuring your clients, sequencing your packages, ... to get a zero footprint of applications on your workstations. And right at that moment it turns out that the SoftGrid client leaves a very visible trace on the client computer: the Q drive is plainly visible there. And what is even worse, inside your bubbles, the end user has full write access on that Q drive (or at least, within the application's asset directory). Are you really ready to become your helpdesk team's new nightmare when all the calls come in, whining about documents saved to Q gone missing? ;)

I think not. So we simply hide the Q drive from the end-users who want to store their documents everywhere but their home drive!

First, we must hide the Q drive from the user's desktop. Thanks to the isolation, users cannot access the Q drive (unless they are in a bubble), so hiding the Q drive from the desktop is purely for aesthetic reasons. Edit your domain policy and set the Hide these specified drives in My Computer group policy to hide the Q drive. Since the default options do not include a "Q drive" option, you will need to edit your system.adm file to include such an option; all the details are in Microsoft Knowledgebase article 231289, but basically you should modify the system.adm file as follows:
```
POLICY !!NoDrives
  EXPLAIN !!NoDrives_Help
     PART !!NoDrivesDropdown          DROPDOWNLIST NOSORT REQUIRED
        VALUENAME "NoDrives"
         ITEMLIST
               NAME !!QOnly           VALUE NUMERIC 10000
                ;low 26 bits on (1 bit per drive)
               NAME !!RestNoDrives     VALUE NUMERIC 0 (Default)
         END ITEMLIST
    END PART              
  END POLICY

[strings]
QOnly="Restrict Q drive only"
RestNoDrives="Do not restrict drives"
```
Of course, you can create a separate SoftGrid.adm file that includes this registry change...

You can also use a registry file along the lines of
```
Windows Registry Editor Version 5.00

[HKEY_CURRENT_USER\Software\Microsoft\Windows\CurrentVersion\Policies\Explorer]
"NoDrives"=dword:00010000
"NoViewOnDrive"=dword:00010000
```
in the login script to prevent the user from seeing the Q drive.

Note: The value 0x00010000 displayed above is the value that hides ONLY the Q drive. If you want to hide another drive, or multiple drives, you need to recalculate the value there as described in the knowledgebase article!
We also want to hide the Q drive inside the bubble to avoid that a clever user overrides the OS registry values that we just defined, or that a sequencer accidently overwrites these values due to bad packaging. So, we must also include the above registry setting in the virtual registry! Microsoft knowledgebase article 931626 discusses three ways of doing this, but without doubt the cleanest way is to modify your OSD file and add a REGISTRY entry in the between the VIRTUALENV tags:
```
<VIRTUALENV>
 <REGISTRY>
   <REGKEY HIVE="HKCU" KEY="Software\Microsoft\Windows\CurrentVersion\Policies\Explorer">
     <REGVALUE REGTYPE="REG_DWORD" NAME="NoDrives">10000</REGVALUE>
     <REGVALUE REGTYPE="REG_DWORD" NAME="NoViewOnDrive">10000</REGVALUE>
   </REGKEY>
 </REGISTRY>
</VIRTUALENV>
```
This does not pollute the user's virtual registry.

Note: The virtual registry overwrites values from the underlying OS. If your group policy hides other drives than Q, they will reappear if you blindly use the "0x00010000" value shown above as an example!

Of course, this does not prevent the innovative user of accessing the Q drive from within bubbles (after all, it's only hidden). But if they really insist on saving their data on the Q drive, at least now you have a valid reason for applying all your voodoo on them :).

SoftGrid error 0060922C-800736B1

2007-10-17T09:03:00.001+02:00

This is a good one to know. If you get the following error message...

The SoftGrid Client could not launch (APPNAME)
This application has failed to start because the 
application configuration is incorrect. 
Reinstalling the application may fix this problem.

Error code: XXXXXX-0060922C-800736B1

then you might want to consider installing the Visual Studio Runtime Executables. It solved a problem for me where a package streamed ok, but the application itself simply refused to launch (Adobe Reader 8).

More information available at the Microsoft Knowledge Base in article 939084.

Sequencing... WinZIP

2007-10-17T09:00:00.000+02:00

There is no real problem in sequencing WinZIP: just follow the standard procedures, and you got yourself another SoftGrid package! Except for one annoying message that keeps on popping up: "WinZIP is currently not associated to...". In the sequencer, this error does not appear.

What goes wrong here is the good ol' File-Type-Association that is handled by SoftGrid and no longer by WinZIP itself. In fact, there is nothing you can do to prevent this message, except for making sure it does not appear anymore. This can be done by adding a virtual registry key AssocMsg at:


HKCU \ Software \ Nico Mak Computing \ WinZip \ WinZip

as a REG_SZ and setting its value to "0". This simply does not show that particular dialog window anymore to the user. This does not impact the functionality of WinZIP, since the filetype associations are handled by the SoftGrid client.

Configuring RTSPS (RTSP over TLS) in SoftGrid

2007-10-17T08:51:00.000+02:00

One of the convenient features of SoftGrid is that all the streaming and management traffic can be encapsulated in a TLS tunnel. This is very useful if you want to make SoftGrid available through a webinterface or to internet users. Some documentation is available (MSKB930870) on how you should configure the RTSPS protocol on the SoftGrid server, but very little is said on how you should obtain a server certificate that SoftGrid can understand.

It turns out that SoftGrid requires a cert.pem and key.pem file, and cannot read Microsoft's PKCS#12 certificate stores (where multiple certificates and keys are stored in a single file). So how can you obtain a separate certificate and private key file if you only have a Microsoft Certification Authority in your enterprise? Here's how...

I. Preparing to request the certificate

First of all, you need to create a new certificate template that allows the exporting of private keys from a Microsoft certificate store. If you don't do this, you cannot separate the private key and the actual certificate that are stored in the single PKCS#12 file. An additional complication is that your certification authority need to be running Windows 2003 Enterprise Edition in order to allow for custom templates! If that is all set, do the following steps:

Open a MMC and open the Certificate Templates snap-in.

Rightclick the Webserver template and choose to duplicate it.

Change the following settings:
- Template Display Name: Webserver (25 years and exportable)
- Validity period: 25 years
Also make sure that on the Request Handling tab, the Allow private key to be exported checkmark is checked!

Save the template & exit this MMC.

Next, open the Certification Authory MMC on your CA server, and go to the Certificate Templates branch in the navigation tree on the left. Rightclick and select New template to issue.

Select the template you just created. Afterwards, restart the CA service.

Done! You have just created a new certificate template that allows the private keys of the (entire) certificate to be exported.

II. Requesting the certificate

On your SGVAS server (... you need to do this on every SoftGrid server!), navigate to the CA's certificate webpage, which is typically available at http://ca_hostname/certsrv.

There, select to request a new certificate, select advanced certificate request. Then, select the option to Create and submit a request to this CA..

Enter the following information:
- Certificate Template: “Webserver (25 years and exportable)
- Name: use the FQDN of your SGVAS server.
- Ensure the Mark keys as exportable checkmark is checked.
- Enable Store certificate in the local computer certificate store.
- Attributes: can be used to specify additional hostnames for a particular server.
- Friendly name: specify a human readable name here for the certificate, e.g. SoftGrid SGVAS Certificate.

Click the Submit button when done, and install the certificate afterwards.

That concludes the requesting and installation of the certificate.

III. Preparing to convert the certificate

Before we can convert the certificate into a form that SoftGrid can understand, we must first extract it from the local certificate store again. This is done using by following these steps:

Open a MMC, and select the Certificates snap-in for Local Computer.

Find the certificate you requested under Personal / Certificates.

Rightclick the certificate, select All tasks and choose to export the certificate.

In the next screens, ensure that:
- The Yes, export private key option is selected.
- The export format is set to PKCS #12.

Enter a private key password and be sure to remember it!!

Select the filename to save the certificate and private key to, e.g. sgvas.pfx.

Now we're ready to convert the certificate once we have installed the OpenSSL tools which allow the manipulation of X.509 certificates (the Microsoft tools suck). Go to the OpenSSL site and find the page that gives a link to binary distributions. You will be pointed towards the Shining Light Productions Win32 port of OpenSSL. Download and install the light package, the current version is v0.9.8.e.

IV. Converting the certificate for SoftGrid

Follow this procedure to convert the PKCS#12 certificate to a separate private key and security certificate file:

Open a command prompt and go to the installation directory of OpenSSL (default is C:\OpenSSL).

Enter the following command (change the location of the input file sgvas.pfx and output file sgvas.pem as desired):
```
openssl pkcs12 -in sgvas.pfx -out sgvas.pem -nodes
```
You will be prompted for the import password, which is the private key password that you choose when exporting the certificate in the Microsoft management console.

Open the sgvas.pem file using a text editor like WordPad. Find the part that is enclosed by the BEGIN RSA PRIVATE KEY tags, and copy it entirely (including the begin/end declarations) to a separate file that you call key.pem. The resulting key.pem file should look like:

Back in the sgvas.pem file, find the part that is enclosed by the BEGIN CERTIFICATE declarations and copy it entirely (including the being/end declarations) to a separate text file that you call cert.pem. An example of how this file should look like:


-----BEGIN CERTIFICATE-----
MIIGCzCCA/OgAwIBAgIKF55bUgAAAAAABzANBgkqhkiG9w0BAQUFADA9MRUwEwYK
CZImiZPyLGQBGRYFbG9jYWwxFzAVBgoJkiaJk/IsZAEZFgdwcmV0bmV0MQswCQYD
.. .. ..
d+RlSH6zX6R20XG6sgv4icLxHVhXF3zF3CUnMfkGIQ8rfHYcOmD+mXju/uLXbFYJ
lid5NBZ7JoNgHoTY/sa9T1ckaoo1MTLL5Kc6X9NUx3fa/tiVPqzjbQPu/Zj/2El3
4uw1pPuC+x5rT1vsnil4
-----END CERTIFICATE-----

This gives you a cert.pem and key.pem file that you can use in SoftGrid.

V. Configuring SoftGrid

From this point on, you can follow the procedure described in Microsoft Knowledgebase article 930870. We briefly summarize that here:

Copy both the cert.pem and key.pem files to the following directory:
```
C:\Program Files\Softricity\SoftGrid Server\conf
```
Ensure that the files are deleted from all other locations to prevent compromising your security (the private key is not supposed to leak out!!)

Open the SoftGrid Management console, go to Server Groups and rightclick the SoftGrid server that the certificate was generated for. Add a new protocol (RTSPS) and fill out the certificate and security key values as follows:

where you use the password that you entered when exporting the key in the Microsoft Management console.

Modify all your OSD files to use RTSPS and port 332 instead of RTSP at port 554, i.e. replace all occurences of
```
rtsp://%SFT_SOFTGRIDSERVER:554/...
```
with
```
rtsps://%SFT_SOFTGRIDSERVER:332/...
```
Restart the SoftGrid server to enable RTSPS streaming at the server side.

That finishes the server-side configuration of enabling the RTSPS. Now do not forget to reconfigure your clients to use a Secure SoftGrid Virtual Application Server to let them use RTSPS.

Note: A bug in the SoftGrid client 4.1.0.56 breaks the TLS functionality in the client. You are thus forced to upgrade to 4.1 SP1 / 4.2 or roll back to the 3.x client if you want to use a TLS tunnel!

SoftGrid & Office 2007

2007-10-14T12:13:00.000+02:00

After a long while, I finally decided to attempt to sequence Office 2007. I had tried a couple of times in the past, but I always failed because of stupid things like...

using an older version of the sequencer; you need to use at least 4.1 SP1 since Office 2007 uses WinSxS.

not having enough memory in my virtual machine, which led to an out-of-memory error when downloading the virtual environment after the monitoring; solved it by using 768MB of RAM.

Yesterday, I finally got around to sequencing it under the right circumstances with version 4.1 SP1 of the sequencer and enough memory.

The sequencing went alright, but now the importing of the Office suite failed on me! The error message was a mysterious:

"Unable to create new application. Please report the following error code to your system administrator: 0000B003"

Some googling turns up Microsoft Knowledgebase article 929944, which informs us that the "friendly name" of a shell command associated to a file type must be unique. Consider the following example:


<SOFTPKG>
  <MGMT_FILEASSOCIATIONS>
    <PROGIDLIST>
      <PROGID>
        <SHELLCOMMANDLIST>          
           <SHELLCOMMAND NAME="open" FRIENDLYNAME="&Open DefaultApp"/>
           <SHELLCOMMAND NAME="edit" FRIENDLYNAME="&Open DefaultApp"/>
        </SHELLCOMMANDLIST>
      </PROGID>
    </PROGIDLIST>
  </MGMT_FILEASSOCIATIONS>
</SOFTPKG>

This will produce the same error since the friendly names are not unique...

So I went looking for errors in the OSD files that were generated, and sure enough, there were some duplicate "Print" friendly names in the Word 2007 OSD file. I solved the problem but still the application refused to import (note: you need to stop an existing import and start all over again after changing the OSD file before the changes are detected).

The problem got solved after I deleted the entire PROGID entry for "Word.Backup.8", which seems like a file type that I hope never to need using anyway...

Synchronizing Google Calendar with Outlook 2003/2007

2007-10-14T12:08:00.000+02:00

The recent discovery of Google Calendar has made the planning of my personal agenda a lot easier... now my wife can do that for me ;). However, what lacks is the synchronization with Outlook, which in turns allows to see your appointments on the PDA.

A very nice tool that tackles the problem is Remote Calendars, which is a COM plugin for Outlook to connect to iCalendars all over the internet, and perform a periodical synchronization between both. The tool is straightforward to install and use, and has all the features I was looking for:

One-way synchronization from Google Calendar to Outlook; two-way is also possible but I didn't want that ;).

Automatic categorizing of imported calendar events; optionally mark these events private.

All settings like "all day event" or "recurring" are copied from the Google Calendar.

In short: the tool does exactly what it needs to do. The only disadvantage in my opinion is that you see a small "iCalendar update" window appear every now and then, which seems to freeze for a moment when no internet connection is available. Besides that, great tool, check it out!

SoftGrid MSI Utility

2007-10-14T12:04:00.000+02:00

News regarding the automatic distribution of SoftGrid packages: Microsoft is working on a MSI utility that will take a SoftGrid package as input and deliver a MSI package as output, ready for distribution using ESD's such as SMS and SCCM. More information is available at the the SoftGrid team blog.

The article hints at how this will be done in practice: the SoftGrid package is loaded into the cache and published on the desktop. This probably means that the utility will roughly do this:

Use the sfttray utility to load the unpacked SFT file into the cache.

Publish the application using sftmime.

Write uninstall information in the registry.

which is pretty much what the SMS Connector VBScripts do at the moment. Appearantly the MSI utility is going to replace the (crappy) SMS connector, which sounds like a good step forward. Also, the availability of SoftGrid applications in MSI form allows a better integration with existing ESD's, making application virtualization available without the immediate need for a full blown SoftGrid infrastructure.

Contacting SoftGrid servers in untrusted domains

2007-10-14T12:00:00.000+02:00

In large environments, it is not uncommon to have different setups for a development/test/production environment which typically consist of three domains without trusts. Suppose that you are working on a production workstation and want to launch an application from the SoftGrid server in the test environment, what would you do?

The problem is that the desktop configuration service captures the security token upon the user login, and uses these NTLM credentials to authenticate against the desktop configuration server that was specified in the SoftGrid Client. When there is no trust between the production and test environment, this means that you are presenting unknown credentials to the SoftGrid server in test, and thus you cannot perform a desktop configuration refresh...

One option is to just live with this and start applications from the OSD file. When the SoftGrid client sees that it does not have the proper credentials to start an application, it will query the user for valid credentials...

This can be used to launch applications from a different domain that the SoftGrid server is placed in (or rather: the browser account of that SoftGrid server).

Recently, I read about another option on Justin Zarb's blog, which uses the Windows XP network password features for computers that are not joined to a domain: by opening the XP Users And Accounts control panel, then clicking the user account & then the option "Manage My Network Passwords", it is possible to supply the credentials that should be used to contact a certain server (given its FQDN). when supplying the FQDN of the SoftGrid server that you want to use, you can force the SoftGrid client to use the credentials specified there. Read a full howto on Justin's page.

SoftGrid scripting: hiding the DOS box

2007-10-14T11:56:00.000+02:00

So you have created your wonderful SoftGrid script that performs the most advanced features known to man, but there is always that annoying DOS box that keeps on popping up when you use SCRIPTBODY or even when you use a HREF tag to launch the script? (Yeah that's right, also with HREF you can get DOS boxes...).

Here is a workaround to launch an external script to hide the DOS box. The idea is very simple: you use a Visual Basic script that runs the batch file in the background, thus effectively hiding all output!

The VBScript code is a single line and as follows:

CreateObject("Wscript.Shell").Run WScript.Arguments(0), 0, False

Save this file as HideDOS.vbs on your content share, for example in the Tools folder or something. Then call your external script called G:\Doit.bat using the following syntax:

<SCRIPT TIMING="PRE" EVENT="LAUNCH" WAIT="TRUE" PROTECT="TRUE">
<HREF>wscript \\server\content\Tools\HideDOS.vbs G:\Doit.bat</HREF>
</SCRIPT>

There you go, gone is the DOS box!

Some SoftGrid gotcha's

2007-10-14T11:33:00.000+02:00

In a SoftGrid coaching project with a customer, I discovered some small things that you should know and are not really mentioned clearly in the documentation... Here it goes:

During the FB1 determination phase, when you can launch the applications that have been defined for that package, the changes to files and registries are included in the package. In other words: the monitoring is restarted during the construction of feature block one!

In version 4.1 SP1 and 4.2, a modification that was made to a local OSD file are not reset by just performing a refresh of the applications. Only when the OSD file is modified on the server, it is copied again to the client.

That is not really a disaster, but what is worse is that when you delete the OSD file from the cache (from the All Users profile), then 4.1 SP1 and 4.2 no longer are able to create the application!! The error message from the sftlog.txt indicates that the SoftGrid client can no longer find the OSD file (presumably because it wants to compare the local file with the server OSD file to detect changes on the server side). The same goes for ICO files by the way!

This is different from the behavior in 4.1, the OSD file was always copied there. The end-result is that while troubleshooting with SoftGrid 4.1 SP1 and higher, you first need to create a copy of the local OSD file and work from there.

There is no visual indication of whether the SoftGrid client has entered MDO (Minimal Disconnected Operation) mode, which typically occurs when the SoftGrid server is not available but where the application should be allowed to continue to run. However, it is possible to generate a log file that informs you when a certain application has entered the MDO state. More details are in MSKB 931252.