Many of Citrix’s US and Canadian (and some EU) government customers are mandated to abide by the FIPS encryption standards published by the National Institute of Standards and Technology (NIST). This standard defines various levels of compliance (140-2 Level 2, Level 3, etc.) that organizations may be mandated, or wish, to meet. For Citrix customers, this most notably means utilizing our line of FIPS NetScalers which include FIPS 140-2 Level 3 compliant, tamper-resistant, cryptographic hardware modules for storage of private encryption keys.
Backup and recovery of these keys is imperative for security and availability in any environment. By instituting proper backup procedures, you will ensure that any catastrophic loss of the cryptographic module will not bring down your critical production services. In this blog I will discuss the available backup options, pitfalls to avoid, and our consulting leading practices.
The Recommended Backup Method
There are two main ways of generating and installing server certificates on a FIPS NetScaler device:
- Generate the Private Key and Certificate Signing Request (CSR) on the NetScaler directly.
- Generate the key and CSR on an outside entity (such as IIS) and import the private key into the NetScaler cryptographic module.
In general, I always recommend the first method if possible. This ensures that the private key only exists secured within a FIPS-compliant hardware module and that access to it is restricted and tracked.
Unfortunately, this also makes the private key extremely hard to backup due to FIPS restrictions. Luckily, the NetScaler is able to export these keys, while maintaining FIPS-compliance, to a secondary NetScaler through a process called Secure Information Management (SIM).
This process allows a NetScaler to securely encrypt, export, and then import a FIPS key onto another NetScaler FIPS device and is the only recommended method for doing so. For instructions on how to complete this process, see CTX200441 and CTX130199. This method also works even if the respective NetScalers are not in an HA pair! This means you can backup your private keys across datacenters by either directly using SIM between NetScaler management IPs, or by exporting and importing keys manually though the SIM process.
Another Backup Method
The other options when creating certificates is to generate the CSR and private key outside of the NetScaler. This may be necessary in some organizations which require a specific certificate request process.
In this case, you will need to import a PFX certificate file into the NetScaler. Since the PFX contains both the public key\certificate as well as the private key, the NetScaler will need to know to import both. NetScaler 12 makes this much easier as a PFX can be imported natively. For older versions of NetScaler firmware, see the instructions posted here.
After this has been completed, I would highly recommend that you delete the original PFX and complete a SIM configuration with a secondary NetScaler as recommended above. This will ensure that your critical certificate keys only reside within the FIPS modules on each NetScaler.
If you cannot use SIM, then backing up your PFX files in a secured network location is an option. However, I don’t recommend it for reasons I’ll discuss below.
What to Avoid
As I mentioned previously, the recommended (and fully Citrix supported, as well as FIPS-compliant) method to making your private certificate keys highly available, is with two NetScalers through our built-in SIM functionality.
However, you may have noticed that there is an Export function under our FIPS settings that seems to act as a backup mechanism. While this will export a private key (encrypted of course) to the local flash drive on the NetScaler, it is not a valid backup method! This key cannot be imported onto another NetScaler and will be invalid if the NetScaler cryptographic module fails in the future. I admit that our documentation on this function is slightly vague, but now you should understand that this method of backup is not recommended.
I also want to reiterate that using the second certificate generation and backup option above (generating keys outside the NetScaler), can come with its own set of issues. I often see customers “backup” their sensitive PFX files containing private keys to shared network locations. They are often secured with a well-known or weak internal password and have been copied to various other network locations by members of the IT administration team. This creates a scenario where their organizations critical, public facing, private certificate keys are completely unsecured, have been duplicated, with no accountability or ability to audit access! This sounds like a huge security risk that I would try to avoid at all costs! Instead, these keys should always be securely copied to another NetScaler through the SIM process.
Summary
In summary, always utilize pairs of FIPS NetScalers to ensure your private certificate keys are highly available through SIM. It’s also good operational practice to have a pair of NetScalers in an HA configuration anyway, so win-win! If you must generate your keys outside of the NetScaler itself, ensure that you are taking steps to either delete or protect them afterwards and avoid having your organizations name in the papers due to a future security breach.
Nick Czabaranek
Lead Architect, US Public Sector
