Wednesday, April 30, 2008

Windows 2008 Certificate Authority and Windows 2000/XP/2003 clients

I was experimenting with Windows 2008 Certificate Services the other day in order to create certificates for WSUS 3.0 and for doing SSL tunneling of RDP towards the internet. I noticed that several of my clients were unable to automatically install the WSUS client, with vague errors in the event log (Win32HResult=0x00000000):


I had quickly discovered that the problem was related with the certificate that I had issued for the WSUS IIS server. It turned out that Windows 2008 WSUS clients could connect without any problem to the WSUS webpage, but Windows 2003 and Windows XP clients could not. What made it even more puzzling is that on a Windows XP system, connecting to the IIS homepage didn't succeed using Internet Explorer, but worked perfectly fine using Firefox.

Opening the certificate of my WSUS server gave the following result:


with a "This certificate has an nonvalid digital signature" error in the "Certification Path" details for both the issued certificate and my CA certificate.

Root cause:
The answer is the bleeding obvious: Windows 2008 has several new additions to the cryptography API, called Cryptography Next Generation (CNG), that are used in the V3 certificate templates for CA's and Webservers in Windows 2008. Amongst those new features is support for new certificate signing algorithms (in my case SHA512, a SHA-2 variant) which is not recognized by older clients. Windows XP SP3 adds support for XP, I suppose a future hotfix will add compatibility for Windows 2003.

Solution:
In absense of a worldwide XP SP3 deployment and a working hotfix for W2K3, the only option here is to ensure that the Windows 2008 CA certificate is created with a non-CNG cryptographic provider. If you already created a CA certificate using the new CNG features, the only option is to reinstall your CA and regenerate your CA certificate --- remember how mum always told you to think things over twice before just plainly installing a W2K8 CA... I bet you regret that now (just like I did :D) ? Reinstalling your CA could be messy, and make your PKI infrastructure go berserk, so this time do think twice before going down that road!

Step by Step plan of attack (POA)
So you have decided you want to proceed? First verify that you are indeed using a CNG CSP. To do this, open your registry editor and navigate to the following key:

[HKLM\SYSTEM\CurrentControlSet\Services\CertSvc\
Configuration\{CAname}\CSP]


If you find a CNGHashAlgorithm REG_SZ value, and the HashAlgorithm DWORD is set to 0xFFFFFFFF, then you are using a CNG CSP. If the HashAlgorithm is set to a value such as 0x00008003, then you are already using a "classic" CSP. You can also use the following command on the CA to retrieve the CSP:

certutil -getreg ca\csp\HashAlgorithm
certutil -getreg ca\csp\Provider


which will return the HashAlgorithm and the name of the CSP. For more information, I refer to the Microsoft whitepaper "Active Directory Certificate Server Enhancements in Windows Server Code Name Longhorn", you crypto-boys out there will love it.

Keep in mind that when you are adding the Certificate Services Role to your Windows 2008 server, that you need to specify the proper cryptographic service provider. The image below displays some of the options, what is important to remember here is that all the service providers that contain a hash sign ("#") are CNG providers and thus incompatible with Windows XP SP2/Windows 2003 and earlier clients.


The default cryptographic service provider for Windows 2003 is the "Microsoft Strong Cryptographic Provider", so that is what you want to use. Notice how selecting this provider reduces the number of certificate signing options... SHA-2 algorithms are no longer included! Proceed as usual to end up with a CA that produces certificates that can be handled by legacy clients.

14 comments:

Anonymous said...

Thanks for that Tim. 2008 Tripped me up nicely there!

Mark said...

Here is a hotfix from microsoft which may alleviate some problems in this area if not specifically this one

http://support.microsoft.com/kb/938397

Anonymous said...

Hi Tim,

Could you help me with my setup?
We could talk online.
Running this at home, with TS Remote on a single server, using ISA 2006 and internal CA, trying to issue public key server cert..stuck here..

raymond.scott@live.com

Anonymous said...

Very good hotfix,

thank you

Eyal Estrin said...

Check out my step-by-step guide for installing Windows 2008 R2 certificate authority
http://eyalestrin.blogspot.com/2010/07/windows-2008-r2-certification-authority.html

six sigma certification said...

Certificates for Microsoft are a bit difficult yo get. Its good that you published your research here will be of use for many a geeks.

Jared (jared.burr@digicert.com) said...

This tripped me up for a couple of hours with a customer trying to install a SHA-2 certificate to a Windows Server 2003 machine, because in Windows Server 2003 it sees a SHA-2 certificate the same as a self signed certificate. Anyway, if anybody receives the big red X that says the certificate has a nonvalid digital signature check to see if the certificate is SHA-2.

Anonymous said...

Thanks! This helped us out big time. We are running a Windows 2003 Terminal Services with IE8 and ran into this issue with a partners web site. Worked fine with other 3rd party browsers. Odd duck indeed.

Ralph said...

We had a similar issue today trying to use PKI certs for SCOM on 2003 servers. This hotfix fixed our issues:

http://support.microsoft.com/kb/968730

Anonymous said...

good post

celebrex 200mg said...

Nice post. This post is different from what I read on most blog. And it have so many valuable things to learn.Thank you for your sharing!

Anonymous said...

Wow .. this stumped me for a while.

Anonymous said...

no further need to reinstall.... .-) http://support.microsoft.com/kb/968730
Marco

Anonymous said...

People in comments seem confused.

This article is referring to v3 certificates, NOT sha2 or greater certificate support. none of the hotfixes you list will work for v3 certs on a 2k3 server.