Wednesday, October 17, 2007

Configuring RTSPS (RTSP over TLS) in SoftGrid

One of the convenient features of SoftGrid is that all the streaming and management traffic can be encapsulated in a TLS tunnel. This is very useful if you want to make SoftGrid available through a webinterface or to internet users. Some documentation is available (MSKB930870) on how you should configure the RTSPS protocol on the SoftGrid server, but very little is said on how you should obtain a server certificate that SoftGrid can understand.

It turns out that SoftGrid requires a cert.pem and key.pem file, and cannot read Microsoft's PKCS#12 certificate stores (where multiple certificates and keys are stored in a single file). So how can you obtain a separate certificate and private key file if you only have a Microsoft Certification Authority in your enterprise? Here's how...

I. Preparing to request the certificate

First of all, you need to create a new certificate template that allows the exporting of private keys from a Microsoft certificate store. If you don't do this, you cannot separate the private key and the actual certificate that are stored in the single PKCS#12 file. An additional complication is that your certification authority need to be running Windows 2003 Enterprise Edition in order to allow for custom templates! If that is all set, do the following steps:

  1. Open a MMC and open the Certificate Templates snap-in.

  2. Rightclick the Webserver template and choose to duplicate it.

  3. Change the following settings:

    • Template Display Name: Webserver (25 years and exportable)

    • Validity period: 25 years

    Also make sure that on the Request Handling tab, the Allow private key to be exported checkmark is checked!

  4. Save the template & exit this MMC.

  5. Next, open the Certification Authory MMC on your CA server, and go to the Certificate Templates branch in the navigation tree on the left. Rightclick and select New template to issue.

  6. Select the template you just created. Afterwards, restart the CA service.

Done! You have just created a new certificate template that allows the private keys of the (entire) certificate to be exported.

II. Requesting the certificate

On your SGVAS server (... you need to do this on every SoftGrid server!), navigate to the CA's certificate webpage, which is typically available at http://ca_hostname/certsrv.

  1. There, select to request a new certificate, select advanced certificate request. Then, select the option to Create and submit a request to this CA..

  2. Enter the following information:

    • Certificate Template: “Webserver (25 years and exportable)

    • Name: use the FQDN of your SGVAS server.

    • Ensure the Mark keys as exportable checkmark is checked.

    • Enable Store certificate in the local computer certificate store.

    • Attributes: can be used to specify additional hostnames for a particular server.

    • Friendly name: specify a human readable name here for the certificate, e.g. SoftGrid SGVAS Certificate.

  3. Click the Submit button when done, and install the certificate afterwards.

That concludes the requesting and installation of the certificate.

III. Preparing to convert the certificate

Before we can convert the certificate into a form that SoftGrid can understand, we must first extract it from the local certificate store again. This is done using by following these steps:

  1. Open a MMC, and select the Certificates snap-in for Local Computer.

  2. Find the certificate you requested under Personal / Certificates.

  3. Rightclick the certificate, select All tasks and choose to export the certificate.

  4. In the next screens, ensure that:

    • The Yes, export private key option is selected.
    • The export format is set to PKCS #12.

  5. Enter a private key password and be sure to remember it!!

  6. Select the filename to save the certificate and private key to, e.g. sgvas.pfx.

Now we're ready to convert the certificate once we have installed the OpenSSL tools which allow the manipulation of X.509 certificates (the Microsoft tools suck). Go to the OpenSSL site and find the page that gives a link to binary distributions. You will be pointed towards the Shining Light Productions Win32 port of OpenSSL. Download and install the light package, the current version is v0.9.8.e.

IV. Converting the certificate for SoftGrid

Follow this procedure to convert the PKCS#12 certificate to a separate private key and security certificate file:

  1. Open a command prompt and go to the installation directory of OpenSSL (default is C:\OpenSSL).

  2. Enter the following command (change the location of the input file sgvas.pfx and output file sgvas.pem as desired):

    openssl pkcs12 -in sgvas.pfx -out sgvas.pem -nodes

    You will be prompted for the import password, which is the private key password that you choose when exporting the certificate in the Microsoft management console.

  3. Open the sgvas.pem file using a text editor like WordPad. Find the part that is enclosed by the BEGIN RSA PRIVATE KEY tags, and copy it entirely (including the begin/end declarations) to a separate file that you call key.pem. The resulting key.pem file should look like:

    -----END RSA PRIVATE KEY-----

  4. Back in the sgvas.pem file, find the part that is enclosed by the BEGIN CERTIFICATE declarations and copy it entirely (including the being/end declarations) to a separate text file that you call cert.pem. An example of how this file should look like:

    .. .. ..
    -----END CERTIFICATE-----

This gives you a cert.pem and key.pem file that you can use in SoftGrid.

V. Configuring SoftGrid

From this point on, you can follow the procedure described in Microsoft Knowledgebase article 930870. We briefly summarize that here:

  1. Copy both the cert.pem and key.pem files to the following directory:
    C:\Program Files\Softricity\SoftGrid Server\conf

    Ensure that the files are deleted from all other locations to prevent compromising your security (the private key is not supposed to leak out!!)

  2. Open the SoftGrid Management console, go to Server Groups and rightclick the SoftGrid server that the certificate was generated for. Add a new protocol (RTSPS) and fill out the certificate and security key values as follows:

    where you use the password that you entered when exporting the key in the Microsoft Management console.

  3. Modify all your OSD files to use RTSPS and port 332 instead of RTSP at port 554, i.e. replace all occurences of


  4. Restart the SoftGrid server to enable RTSPS streaming at the server side.

That finishes the server-side configuration of enabling the RTSPS. Now do not forget to reconfigure your clients to use a Secure SoftGrid Virtual Application Server to let them use RTSPS.

Note: A bug in the SoftGrid client breaks the TLS functionality in the client. You are thus forced to upgrade to 4.1 SP1 / 4.2 or roll back to the 3.x client if you want to use a TLS tunnel!

1 comment:

C said...

Awsome! Thanks.