Certificator

Using the Certificator to replace the vCenter Server Virtual Appliance Certificates

Background

If you ever tried to replace the certificates on a VCSA, you would have most likely come across VMware’s KB2057223 article, Configuring Certificate Authority (CA) signed certificates for vCenter Server Appliance 5.5 (http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2057223). 

I have worked through this article manually many times and, like a good book, you spot something new on every read. Unfortunately, missing some key information on the previous reads has meant that the steps didn’t work and it caused errors such as:

Getting SSL certificates for https://vcenter55.company.co.za:7444/lookupservice/sdk

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate chain not verified

Return code is: SslHandshakeFailed

Or sometimes

Getting SSL certificates for https://vcenter55.company.co.za:7444/lookupservice/sdk

com.vmware.vim.vmomi.core.exception.CertificateValidationException: Server certificate assertion not verified and thumbprint not matched

Return code is: SslHandshakeFailed

After trying many things over and over again, I decided to scripts some of the parts to make it easier to repeat. Eventually, I coded the entire process in the hope that it will help the community that has helped me many times.

Generate-Certificates.ps1

The Certificator is a PowerShell script that uses a form to read data relevant during certificate creation, then build commands for OpenSSL to perform. Although, the main focus of the script is on VMware products, I tried to make the output of the functions as generic as possible so that it can be used in other implementations too. 

Download the code at Pastebin

http://pastebin.com/acsdLUQz

Pre-requisites

·   OpenSSL 0.9.8 - Obviously! OpenSSL will be used to sign and create all the certificates. The script will test for this installation and direct you to the correct website for the installation instructions. At this point, I have to take my hat off for the OpenSSL team for their product that has helped me many times.

WinSCP 5.2 or higher - This is needed to upload the newly created files to the relevant product.

PowerCLI - This is not strictly in use yet but in future versions it will be used to collect the host names from vCenter rather than typing them individually. I anticipate that this will be used for new implementation mostly, so the hosts won’t necessarily be in the vCenter anyway. 

PowerShell v4 - Most of the script will work with v.2 even, but the experience will be so much better with 4. 

Functions

Home Page

From the home page, there are various tasks that can be performed:

• Specify default values
• Generate a Certificate Authority
• Generate vCenter Server certificates
• Generate ESXi Host certificates
• Generate a custom certificate

As soon as the script is run, the first checks that are performed, are for OpenSSL and WinSCP. If you don’t have OpenSSL installed or the version is incorrect, you will be informed and will be directed to relevant web site. Remember, this is VMware focused, so the versions are specified by VMware.

With the correct software and versions installed, you can now click on the button, relevant to the task you want to perform. 

“Specify Default Values” Button

If you’ve run the script before and saved your settings, the script will retrieve those settings for you. I use this for my different customers so that I can keep their settings in case I need to run the script again later.

Select the file you want to import and click OK, or click Cancel to accept the default script settings

You will then be presented with defaults form where you can specify the values relevant to your implementation:

• Company name
• Root folder - This is where the certificates will be created. Actually, this is the folder where an ssl folder will be created if one does not already exist followed by the subdirectories. Each vCenter service, ESXi host or custom Web Server will have its own subdirectory for its files.
• Hostname - This is the non-FQDN name of the server. This will become one of the entries in the Subject Alternative Name field of the certificate.
• IP Address - This will also be an entry in the Subject Alternative Name field of the certificate.
• Company FQDN - This will be added to the hostname to make the third entry in the Subject Alternative Name field.
• Country Code - A two letter code (https://www.digicert.com/ssl-certificate-country-codes.htm)
• Province
• Location or City
• Number of bits - Although you can set it here on the defaults page, this value will only take effect when signing a CA or creating a CSR for a custom web server. VMware certificate keys are always 2048 bits.
• Signing Root CA - The next three boxes deal with the signing CA

• Root folder - This is where the OpenSSL related files for the CA is kept and contains various files that OpenSSL needs for the CA to function.
• CRT and Key file - This could be in the same folder as the CAs root or not.
• root Password - This is used for the VCSA and the ESXi hosts.
• SSO Administrator password - the default administrator@vsphere.local is assumed here. If you changed you VCSA account, you would have to change $strCertCommands.

Save to this Session - keeps the default values you entered in the session for as long as the Home form is running.

Close - takes you back to the Home form

Save to Profile - Puts the settings you entered into an XML file to recall later.

“Generate a Certificate Authority” button

From here you can create a Certificate Authority that you can store for reuse. Where relevant, fields from the defaults form are used

Generate - Creates the relevant files required in order to be a CA and creates the .crt and .key file that will be used to sign other certificates with.

Close - Takes you back to the home form

Trust this Root CA - Adds the certificate to the “Trusted Root Certification Authorities” list (requires PowerShell version 4)

“Generate vCenter Server Certificates” Button

Apart the number of bits that can be specified on the Defaults screen, the vCenter Certificate screen looks very much the same. In this case though, the buttons are different.

• Generate - Creates a folder and all the relevant files for all the vCenter services: vpxd, inventoryservice, logbrowser and autodeploy

• Upload to vCenter - This assumes the vCenter Server Virtual Appliance, since these days that is all I use, but I hope to add the Windows server certificates upload to the same button in the future. For now, it requires WinSCP, which then uploads the files to the appliance and runs all the scripts in the required order. Below are some screenshots of the process. 

As you can see from the screen above (Giving up on security...), the connection is made to the server regardless of the certificate thumbprint it finds. This is OK, since we know the certificate is a self-signed out-of-box certificate. 

The next thing you want to keep an eye out for are results that are returned. We want this to be 0 or success. 

The next important point to note is that you might see the warning messages as shown above. This is normal and nothing to worry about, except if it times out. If it does, you would have to start the process all over again after you  have regenerated the certificates on the appliance again. For instructions on how this is done, see William Lam’s blog at http://www.virtuallyghetto.com/2013/04/automating-ssl-certificate-regeneration.html.

Once all the steps are complete, WinSCP will exit and you will be taken back to the form. At this point you need make absolutely sure of two things:

• Certificate regeneration is turned off in the VAMI
• The vCenter Server Virtual Appliance is rebooted

When it comes back online, and assuming you trusted the signing CA, you should have valid certificate on the VCSA.

“Generate ESXi Host Certificates” button

This page is a work in progress, but it will mostly give you what you need, which are the certificates for the hosts. Adding hosts and their IP is a manual process, which I hope to automate in future by querying the vCenter for the hosts and their IPs. This will be good for current implementations, but since most of my requirement is around new implementations, I’ve not been in too much of a hurry to add this. 

 Again, most of the fields are populated by the defaults excpet for the host names and IP addresses. To spcify these, click on the “Specify Hostname and IP” button for the entry box to appear, as shown above.

The rest is self-explanatory. Once you click “Done”, the list-view box will be populated. 

Generate - As before, this button creates the certificates in a folder espcially for the host, which will be inside the ssl folder. 

As shown below, any folder should contain the following:

• Config file (.cfg) - Contains all the settings that are used when creating the signing request and when signing the certificate.
• Certificate Signing Request (.csr) - normally sent to the CA in return for a certificate. In this case, you are the CA.
• Private key (.key) - the secret part of your certificate, which you should share with nobody.
• Certificate (.crt) - This is what it is all about.
• Chain file (.pem) - This file is a combination of the servers’ certificate and the CA’s certificate in PEM format. 

Upload to ESXi Hosts - Watch this space.

“Generate a customer certificate” button

The final button allows you to create your own custome certificate for a web server. The entry form accepts most of the entries that you’ll find on the defaults form, but two entries are worth mentioning:

• Path to .pfx file - You don’t need to complete this field, unless you already have a .pfx that you want to import. The field is popluated automatically to prepare for the certificate import, discuss shortly.
• Passphrase - The password to unlock you key. 

Generate - This button does mostly what the other “Generate” buttons did, except that it adds a .pfx file, which is handy in some web server implementations. I’ve tested it in various setup already including basic IIS servers as well as with Citrix Secure Gateway.

Import the .pfx - this does what you’d expect, which is to import the certificate into the computer’s personal certificate store. For that reason it would good to run this part of the script on the actual server.

I thought of, but haven’t tested it, to put the CA certificate on a shared folder with the necessary NTFS permissions so that you can reference it anywhere. This is specifically so that you can generate the request and sign it from any server and also to upload it later.
 Alternatively, you can generate the .pfx on one machine and simply browse to the .pfx file later on. 

Servers that require a PEM certificate that includes the key

Two products come to mind where you might want to create a PEM certificate that also contains the KEY:

• vCenter Operations Manager
• Citrix NetScaler (not tested)

To do this, you will notice that, after running the custom certificate creation, the following line appears:

Creating WEB01 files

       Creating the custom config file

       Creating the certificate signing request

       Checking if the key is in PKCS1 format

              C:\Customers\A Test Company\ssl\WEB01\WEB01.key is in PKCS1 format

       Checking if the certificate is in PEM format...

              C:\Customers\A Test Company\ssl\WEB01\WEB01.crt is in PEM format

       Creating a certificate chain

       Creating the PFX File C:\Customers\A Test Company\ssl\WEB01\WEB01.pfx

To create a PEM file for vCOPS, copy and paste the text below to the command prompt using no password

C:\OpenSSL\bin\openssl.exe pkcs12 -in "C:\Customers\A Test Company\ssl\WEB01\WEB01.pfx" -nodes -out "C:\Customers\A Test Company\ssl\WEB01\WEB01.pem"

All you need to do now is copy and paste the text into a command prompt window to run the command. 

When prompted for the password, enter the same password that was used on the form

Note that, if this certificate is meant for vCenter Operations Manager, the password field should be empty before you generate the files. Then, when prompted for the password, you again use no password. The result is an additional file in the folder associated with the server you are busy with. 

And that is that! I hope this is of much use to you as it is to me. 

Happy scripting!

Andre Combrinck

No Network Adapters

No Network Adapters

I ran into this problem today while trying to set up my lab:

Looking through various forums, the most obvious way to avoid this is to make sure your hardware is on the VMware Hardware Compatibility List before you start. Since this is a lab environment and I want to make best use of what I have, I'm going to ignore this. 

The next solution to this problem is to get an ESXi ISO from the hardware vendor that already contains all the drivers you need for the hardware. This is heading in the right direction, but since this is a custom built PC, I don't expect that the vender will have such an ISO. I will have to create it myself.

My NIC is an onboard Atheros AR8151 PCI-E Gigabit Ethernet Controller (NDIS 6.20)

While trying to find a solution on how to do this, I came accross Derek Seaman's blog that does a good job of explaining how you can go about creating an ISO such as this for your Cisco and HP servers. He uses PowerCLI, which is great.

So what do you need?

1. You will need the ESXi drivers for your particular device stored inside a .tgz file. If you need help in creating your own .tgz file, have a look at Andreas Peetz's blog under Step 1 of the Usage Instructions. I tried this, but couldn't get Linux installed at all. (The installation stops at the kernel helper, which is probably because of the NIC driver)
2. Get the Community Packaging Tools (again from Andreas's blog), which you'll use to convert the .tgz file into a VIB file. All the instructions are on Andreas's blog.
3. Get the ESXi Customizer from Andreas Peetz (consider a donation for this great tool!)
4. Lastly, you will need the ISO file for the ESXi installation. 

After about 2 minutes, my ISO file was ready for me to use.

Next, I used unetbootin-windows-581.exe to create a bootable USB key from my new ISO.

My Lab

The Host Hardware - a PC

Gigabyte X1 Chassis - R300

Gigabyte 585W Power Supply - R450

Gigabyte Z77-D3H Motherboard - R1400

Intel Core i7 3770 3.4GHz CPU (socket 1155) - R2700

Transcend DDR RAM 8GB - 4 x R400

Seagate Barracuda 500GB SATA3 hard disk drive - R600

UPDATE: This Motherboard comes with the Atheros AR8151 PCI-E Gigabit Ethernet Controller, which is not supported OOB on ESXi or Linux for that matter. Make sure your NIC is supported before you buy. 

For more advice on building a lab, check out Boelowie's blog. I opted out of the SSD disk and decided to double my RAM instead. I wanted to give the SATA3 disk a go first, which doubles the SATA2 transfer speed from 3 Gbit/s to 6 Gbits/s. The spindle speed is still only 7200RPM but it remains to be seen if there is any improvement. 

The base OS - ESXi

Rather than installing Windows 7, then VMware Workstation, then ESXi, then my VMs, I opted for a nested ESXi and then my VMs. The PC is only for a lab and I can easily set up the restart order of the VMs so that I can get back to my lab whenever I need to. The host is connected to a WiFi switch so that I can access my lab from anywhere in the house.