App-V 4.5 Certificate Galore

1) Setting
This weekend I finally found some time to delve a bit deeper into properly configuring an App-V 4.5 infrastructure for large scale deployments. One of the first things that I investigated was the usage of RTSPS for smoother firewall tunneling: as you know, when using RTSP a series of ports is dynamically chosen, which means that you need to open up entire portranges in your firewall. This is not something your firewall guys will like if you work in a larger environment.

Going for RTSPS means you need to use a server public certificate and a corresponding private key in order to let the App-V server sign and encrypt its communications. I have blogged before about how to configure this in SoftGrid 4.1/4.2 -- luckily the procedure for configuring an SSL certificate got a lot simpler. At least, that is what I thought. Some issues I ran into that might save you some valuable troubleshooting time:

• As always, when requesting a certificate from your Enterprise PKI, use the Virtual Application Server's FQDN as the subject. It is probably also a good idea to use the hostname as a subject alternate name for those people that still refer to servers by their shortnames.
  
  
• After the App-V 4.5 Web Management Service has been installed, don't forget to configure the certificate for the IIS Default Website. In IIS7, that requires adding a binding & selecting the proper certificate. It is not clear to me why the App-V installer cannot handle this automatically!?
  
  
• App-V 4.5 runs under the NETWORK SERVICE account by default and no longer under the SYSTEM account as SoftGrid 4.1/4.2 used to. This has some consequences when it comes to Windows PKI: you need to grant the NETWORK SERVICE account read permissions on the private key.
  

This later action is a lot harder than you think when reading them ;). Read on for more information.

2) Configuring permissions on private keys
You have three options to get this working:

• If you are using a Windows 2008 Enterprise CA and are using your own certificate templates, then you can modify the template to automatically grant the NETWORK SERVICE account read permissions on all certificates issued using that template.
  
  
  Since you will typically be creating a new certificate template for server deployment (to enable longer than 2 years validity & exporting of private keys), this is probably the easiest solution if you have a Windows Server 2008 Enterprise CA.
  
  
• In a pre-Windows 2008 CA world, you will have to use the WinHTTPcertcfg.exe tool, the Windows HTTP Services Certificate Configuration tool. In our situation, we need to modify the ACL of the certificate to grant read access to the service account of the Management Service (which is the NETWORK SERVICE by default).
  
  winhttpcertcfg -g -c LOCAL_MACHINE\My -s (subjectname) -a NetworkService
  
  Verify that everything went ok by listing the permissions:
  
  winhttpcertcfg –l –c LOCAL_MACHINE\My –s (subjectname)
  
  
• It is also possible to explicitly set the permissions on the private key file. This information is based on information obtained from the App-V blog, with some corrections below.
  
  
  • First, obtain the certificate thumbprint. You can find this in the details tab of the certificate:
    
    Copy/paste the thumbprint for the next commandline.
    
    
  • Next, use the FindPrivateKey.exe utility to locate the private key file on disk (compiled version available here -- download & use untrusted executables from the internet at your own risk). Use the following syntax:
    
    FindPrivateKey.exe My LocalMachine -t "your thumbprint"
    
    This will give you the full path. Read the caveat message below if this path looks awkward.
    
    
  • Grant the NETWORK SERVICE account read & execute permissions on the private key file.
    

CAVEAT: the location of the private key should be in a publicly accessible location. For WinXP/Win2K3 the default is: C:\Documents and Settings\All Users\Application Data\Microsoft\Crypto\RSA\MachineKeys For W2K8/Vista, this changed to: C:\ProgramData\Microsoft\Crypto\RSA\MachineKeys If you have a different location, then take actions to deplace the private key. I requested my certificate through the Web Enrollment pages of Active Directory Certificate Services on Windows 2008. This stores the public & private key in your user account's profile by default. I knew this and drog & dropped the public certificate from the "Certificates (My User)" to the "Certificates (My Computer)" MMC and when your private key was marked as exportable, this is indeed possible. However, this does not actually move the private key and leaves it in your user profile location (for example: C:\Users\Administrator\AppData\Roaming\Microsoft\Crypto\RSA). I fixed this by explicitly exporting the certificate & private key from my user account and then explicitly importing everything again. So huge warning for all you regular crypto-users: no more drag 'n dropping of public/private keypairs!

4) Conclusion
A bit messy... yet secure! The move towards the NETWORK SERVICE account for the App-V Management service (... and other Microsoft products as well) is obviously a good choice, yet it brings along some difficulties that probably can be streamlined from within the App-V Management Server's installer.

PS: You didn't forget to grant the NETWORK SERVICE account also read permissions on your content directory, since otherwise your streaming won't work?

VMware Tools without a reboot?

Every now and then, you see blogposts appearing on the "issue" that you need to reboot a guest operating system after you install or update the VMware Tools. Many people have pondered about whether a reboot is in fact really necessary and if it can be avoided all together. Recent posts about this can be read here and here, refering to this VMware community thread -- the question is still alive in multiple-year spanning threads like this one right here. I usually frown my eyebrowses when reading on these "no reboot" topics, yet I am interested in keeping up with the advancements in that subject for some of the large customers that I come in contact with professionaly.

The scripts and methods outlined in these blogposts sound a bit tricky at first if you ask me, and I feared they might not have the outcome you expected. I would think the VMware tools really require a reboot on some operating systems because you update parts of the virtual device drivers and those need to be reloaded by a reboot of the operating system (Note: strictly speaking you don't need a reboot for all types of device drivers, only under a specific set of circumstances documented by Microsoft. The VMware disk drivers host a boot device so that would fit under the "requires a reboot" category from that document). This means that just running the installer with a "Suppress Reboot" parameter on all your machines will place the new VMware Tools files on your harddisk, but will not actively load all of them... I am not sure if that is a state I would want my production virtual machines in!? And to be very clear: what these scripts do is request an automatic postpone of the reboot, not trigger some hidden functionality in VMware Tools not to really reboot after all!

To remove all suspicion, I did a little test on a Windows 2003 virtual machine and upgraded the tools from ESX 3.0.2 to ESX 3.5U2 without rebooting (using the commandline setup.exe /S /v"REBOOT=R /qb" on the VMware Tools ISO). This effectively updates the following services and drivers without rebooting:

• VMware services (bumped from build 63195 to build 110268)
• VMware SVGA II driver, VMware Pointing Device driver

It left the following drivers untouched:

• VMware Virtual disk SCSI Disk Device ("dummy" harddisk driver - Microsoft driver)
  
• NECVMWar VMware IDE CDR10 (virtual CD-ROM driver)
• Intel Pro/1000 MT Network Connection (vmnet driver - Microsoft driver)
• LSI Logic PCI-X Ultra320 SCSI Host Adapter (storage adapter - Microsoft driver)

It turned out that these drivers didn't require updating for my specific virtual machine (even after a reboot). In fact, I wasn't immediatelly able to find one machine in the test environment at work that required updating any bootdisk device drivers (and some still had 3.0.2 VMware Tools running!).

To conclude, I would say that in some circumstances it is safe to postpone the reboot of your virtual machine, if at minimum the boot disk device drivers are not touched. Postponing the reboot is very convenient if you use it in the context of a patch weekend where you want to postpone the restart to one big, single reboot at the end of all your patches.

Update: as Duncan Epping points out in a recent blogpost, be also advises that updating the network driver effectively drops all network connections. This is for all practical purposes maybe just as bad as actually rebooting your server, so beware with the "fake level of safety and comfort" that you might have by postponing a VMware Tools reboot!