Renewal of FMC Sftunnel CA Certificate for FTD Connectivity

Introduction

This document describes the renewal of Firepower Management Center (FMC) sftunnel Certificate Authority (CA) certificate in relationship with the Firepower Threat Defense (FTD) connectivity.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Firepower Threat Defense
• Firepower Management Center
• Public Key Infrastructure (PKI)

Components used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

FMC and FTD communicate with eachother over sftunnel (Sourcefire tunnel). This communication uses certificates to make the conversation secure over a TLS session. More information on the sftunnel and how it does get established can be found on this link.

From the packet capture, you can see that the FMC (10.48.79.232 in this example) and FTD (10.48.79.23) are exchanging certificates with eachother. They do this in order to validate that they talk with the correct device and there is no eavesdropping or Man-In-The-Middle (MITM) attack. The communication is encrypted using those certificates and only the party that has the associated private key for that certificate is able to decrypt it again.

Certificate_exchange_server_cert

Certificate_exchange_client_cert

You can see the certificates are signed by the same InternalCA (Issuer) Certificate Authority (CA) which is set up on the FMC system. The configuration is defined on the FMC on /etc/sf/sftunnel.conf file which contains something like:

proxyssl {
  proxy_cert   /etc/sf/keys/sftunnel-cert.pem;                ---> Certificate provided by FMC to FTD for the sftunnel communication (signed by InternalCA)
  proxy_key    /etc/sf/keys/sftunnel-key.pem;
  proxy_cacert /etc/sf/ca_root/cacert.pem;                    ---> CA certificate (InternalCA)
  proxy_crl    /etc/sf/ca_root/crl.pem;
  proxy_cipher 1;
  proxy_tls_version TLSv1.2;
}; 

This indicates the CA that is used to sign all certificates for sftunnel (both the FTD and FMC one) and the certificate used by the FMC to send to all of the FTDs. This certificate is signed by the InternalCA.

When FTD registers to the FMC, the FMC also creates a certificate to push to the FTD device that is used for the further communication on the sftunnel. This certificate is also signed by the same Internal CA certificate. On FMC, you can find that certificate (and private key) under /var/sf/peers/<UUID-FTD-device> and potentially under certs_pushed folder and is called sftunnel-cert.pem (sftunnel-key.pem for private key). On FTD, you can find those under /var/sf/peers/<UUID-FMC-device> with same naming convention.

However each certificate also has a validity period for security purposes. When inspecting the InternalCA certificate, we can see as well the validity period which is 10 years for the FMC InternalCA as shown from the packet capture.

FMC-InternalCA_validity

Problem

The FMC InternalCA certificate is only valid for 10 years. After the expiry time, the remote system does not trust this certificate anymore (as well as certificates signed by it) and this leads to sftunnel communication issues between FTD and FMC devices. This means as well that several key functionalities like connection events, malware lookups, identity based rules, policy deployments and many other things are not working.

The devices do show up as disabled on the FMC UI under the Devices > Device Management tab when the sftunnel is not connected. The issue that relates to this expiry is tracked on Cisco bug ID CSCwd08098. Note though that all systems are affected, even when you run a fixed release of the defect. More information on this fix is found in the Solution section.

Disabled-devices

The FMC does not automatically refresh the CA and republish the certificates to the FTD devices. And there is also no FMC health alert which indicates that the certificate expires. Cisco bug ID CSCwd08448 is tracked in this regards to provide a health alert on the FMC UI in the future.

What happens after the expiry date?

Initially nothing happens and the sftunnel communication channels continue to operate as before. However when the sftunnel communication between FMC and FTD devices gets broken and it tries to re-establish the connection, it does fail and you can observe log lines on the messages log file that point to the certificate expiry.

Log lines from FTD device from /ngfw/var/log/messages:

Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [INFO] Initiating IPv4 connection to 10.10.200.31:8305/tcp
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [INFO] Wait to connect to 8305 (IPv4): 10.10.200.31
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [INFO] Connected to 10.10.200.31 from resolved_ip_list (port 8305) (IPv4)
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR] -Error with certificate at depth: 1
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR]   issuer   = /title=InternalCA/OU=Intrusion Management System/CN=06f5f3ca-c77b-11e2-81bf-884d9d11f3ef/O=Sourcefire, Inc.
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR]   subject  = /title=InternalCA/OU=Intrusion Management System/CN=06f5f3ca-c77b-11e2-81bf-884d9d11f3ef/O=Sourcefire, Inc.
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR]   err 10:certificate has expired
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR] SSL_renegotiate error: 1: error:00000001:lib(0):func(0):reason(1)
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [ERROR] Connect:SSL handshake failed
Sep 20 04:10:47 FTD-hostname SF-IMS[50792]: [51982] sftunneld:sf_ssl [WARN] SSL Verification status: certificate has expired

Log lines from FMC device from /var/log/messages:

Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [INFO] VERIFY ssl_verify_callback_initial ok=1!
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [ERROR] SSL_renegotiate error: 1: error:00000001:lib(0):func(0):reason(1)
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL Verification status: ok
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [WARN] establishConnectionUtil: SSL handshake failed: error:14094415:SSL routines:ssl3_read_bytes:sslv3 alert certificate expired
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [INFO] establishConnectionUtil: Failed to connect using SSL to: '10.10.21.10'
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [ERROR] establishSSLConnection: Unable to connect with both threads:
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [ERROR] establishSSLConnection: ret_accept status : Too many levels of symbolic links
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [ERROR] establishSSLConnection: iret_connect status: Too many levels of symbolic links
Sep 20 03:14:23 FMC-hostname SF-IMS[1504]: [4171] sftunneld:sf_ssl [ERROR] establishSSLConnection: Failed connecting with SSL to using to: '10.10.21.10' rval[40] *CA cert from FMC

The sftunnel communication can be broken due to various reasons:

• Communication loss because of loss of network connectivity (potentially only temporary)
• Reboot of FTD or FMC
  • Expected ones: manual reboot, upgrades, manual restart of sftunnel process on FMC or FTD (for example by pmtool restartbyid sftunnel)
  • Unexpected ones: tracebacks, power outage

Because there are so many possibilities that can break the sftunnel communication, it is highly advised to correct on the situation as quickly as possible, even when currently all FTD devices are properly connected despite the expired certificate.

How to quickly verify if the certificate is expired or when it does expire?

The easiest way is to run these commands on the FMC SSH session:

expert
sudo su
cd /etc/sf/ca_root
openssl x509 -dates -noout -in cacert.pem

This shows you the Validity elements of the certificate. The main relevant part here is the "notAfter" which shows that the certificate here is valid till 5th of October 2034.

NotAfter

If you prefer a single command to be ran that immediately gives you the amount of days that the certificate is still valid for, you can use this:

CERT_PATH="/etc/sf/ca_root/cacert.pem"; EXPIRY_DATE=$(openssl x509 -enddate -noout -in "$CERT_PATH" | cut -d= -f2); EXPIRY_DATE_SECONDS=$(date -d "$EXPIRY_DATE" +%s); CURRENT_DATE_SECONDS=$(date +%s); THIRTY_DAYS_SECONDS=$((30*24*60*60)); EXPIRY_THRESHOLD=$((CURRENT_DATE_SECONDS + THIRTY_DAYS_SECONDS)); DAYS_LEFT=$(( (EXPIRY_DATE_SECONDS - CURRENT_DATE_SECONDS) / (24*60*60) )); if [ "$EXPIRY_DATE_SECONDS" -le "$CURRENT_DATE_SECONDS" ]; then DAYS_EXPIRED=$(( (CURRENT_DATE_SECONDS - EXPIRY_DATE_SECONDS) / (24*60*60) )); echo -e "\nThe certificate has expired $DAYS_EXPIRED days ago.\nIn case the sftunnel communication with the FTD is not yet lost, you need to take action immediately in renewing the certificate.\n"; elif [ "$EXPIRY_DATE_SECONDS" -le "$EXPIRY_THRESHOLD" ]; then echo -e "\nThe certificate will expire within the next 30 days!\nIt is ONLY valid for $DAYS_LEFT more days.\nIt is recommended to take action in renewing the certificate as quickly as possible.\n"; else echo -e "\nThe certificate is valid for more than 30 days.\nIt is valid for $DAYS_LEFT more days.\nThere is no immediate need to perform action but this depends on how far the expiry date is in the future.\n"; fi

An example of a setup where the certificate is still valid for multiple years is shown.

Certificate_expiry_validation_command

How do I get notified in the future about an upcoming certificate expiry?

With recent VDB updates (399 or higher), you are alerted automatically when your certificate expires within 90 days. Therefore you do not need to manually track on this yourself as you are alerted when you are close to the expiry time. This then shows up on the FMC web page in two forms. Both ways refer to the field notice page.

The first method is through Task Tab. This message is sticky and available to the user unless explicitly closed. The notification pop up also shows up and is available until explicitly closed by the user. It does always show up as an error.

Expiry Notification on Task Tab

The second method is through Health Alert. This shows up in the Health tab however this is not sticky and replaces or removes when health monitor is run which by default is every 5 minutes. It also shows up a notification pop up which needs to be explicitly closed by the user. This can show up both as error (when expired) as a warning (when going to expire).

Expiry notification on Health tab

Warning notification on Health Alert Pop Up

Error notification on Health Alert Pop Up

Solution 1 - Certificate has not yet expired (ideal scenario)

This is the best situation as then depending on the certificate expiry, we still have time. Either we take the fully automated approach (recommended) that has a dependency on the FMC version or we take on a more manual approach which requires TAC interaction.

Recommended approach

This is the situation where no down time and least amount of manual operations is expected in normal circumstances.

Before proceeding, you must install the hotfix for your particular version as listed here. The benefit here is that those hotfixes do not require a reboot of the FMC and thus potential broken sftunnel communication when the certificate is expired already. The available hotfixes are:

• 7.0.0 - 7.0.6 : Hotfix FK - 7.0.6.99-9
• 7.1.x : no fixed release as end of software maintenance
• 7.2.0 - 7.2.9 : Hotfix FZ - 7.2.9.99-4
• 7.3.x : Hotfix AE - 7.3.1.99-4
• 7.4.0 - 7.4.2 : Hotfix AO - 7.4.2.99-5
• 7.6.0 : Hotfix B - 7.6.0.99-5

Once the hotfix is installed, the FMC now contains the generate_certs.pl script that:

1. Regenerates the InternalCA
2. Recreates the sftunnel certificates signed by this new InternalCA
3. Pushes the new sftunnel certificates and private keys over to the respective FTD devices (when the sftunnel is operational)

Note: The generate_certs.pl script currently checks whether critical operations are running. If not, then it fails to run.

Critical operations can be: Smart agent not registered or registration in progress, Backup/Restore task in progress, SRU update task in progress, VDB update task in progress, Domain task in progress, HA Operation in progress or Upgrade is running.

Therefore you cannot run this script when you only use Classic Licenses on your FMC (or any of the listed operations need to complete first) in which case you need to contact Cisco TAC to bypass this check, regenerate the certificates and then undo the bypass again.

Therefore it is recommended (if possible) to:

1. Install the applicable hotfix for your version train
2. Take a backup on the FMC
3. Validate all current sftunnel connections using sftunnel_status.pl script on the FMC (from expert mode)
4. Run the script from expert mode using generate_certs.pl
5. Inspect the outcome to validate if any manual operations are required (when devices are not connected to FMC) [explained further below]
6. Run sftunnel_status.pl from the FMC to validate that all of the sftunnel connections are running fine

Generate_certs.pl script

Note: When you have FMC running in High-Availability (HA), you need to perform the operation first on the primary node and then on the secondary node as it uses those certificates as well to communicate between the FMC nodes. The InternalCA on both FMC nodes is different.

On the example here you see that it creates up a log file on /var/log/sf/sfca_generation.log, indicates to use sftunnel_status.pl, indicates the expiry time on the InternalCA and indicates for any failures on it. Here for example it failed to push the certificates over to device BSNS-1120-1 and EMEA-FPR3110-08 device, which is expected because the sftunnel was down for those devices.

In order to correct the sftunnel for the failed connections, you run the next steps:

1. On FMC CLI, open the FAILED_PUSH file using cat /var/tmp/certs/1728303362/FAILED_PUSH (number value represents unix time, so check the output of previous command in your system) which has the next format: FTD_UUID FTD_NAME FTD_IP SOURCE_PATH_ON_FMC DESTINATION_PATH_ON_FTD
   
   FAILED_PUSH
   
   
2. Transfer over those new certificates (cacert.pem / sftunnel-key.pem / sftunnel-cert.pem) from the FMC to the FTD devices
   ===Automatic approach===
   
   The hotfix installation also provides the copy_sftunnel_certs.py and copy_sftunnel_certs_jumpserver.py scripts that automate the transfer of the various certificates to systems for which the sftunnel was not up while the certificates were regenerated. This can also be used for systems that had a broken sftunnel connection because the certificate was expired already.
   
   You can use the copy_sftunnel_certs.py script when the FMC itself has SSH access to the various FTD systems. If that is not the case, you can download the script (/usr/local/sf/bin/copy_sftunnel_certs_jumpserver.py) from the FMC to a jump server that has SSH access to both the FMC(s) and FTD devices and run the Python script from there. If that is also not possible, then suggest to run the manual approach shown next. The next examples show the copy_sftunnel_certs.py script being used, but the steps are the same for the copy_sftunnel_certs_jumpserver.py script.
   
   A. Create a CSV file on the FMC (or jump server) that contains the device info (device_name, IP address, admin_username, admin_password) that is used to make the SSH connection.
   
   When you run this from a remote server like a jump server for Primary FMC, make sure to add in the primary FMC details as the first entry followed by all managed FTD and secondary FMC. When you run this from a remote server like a jump server for Secondary FMC, make sure to add in the secondary FMC details as the first entry followed by all managed FTD.
   
   i. Create a file using vi devices.csv.vi devices.csv
   
   ii. This opens up the empty file (not shown) and you fill in the details as shown after you use i letter on keyboard to go into INTERACTIVE mode (seen at bottom of the screen).
   devices.csv example
   
   iii. When fully done, you close and save the file by using ESC followed by :wq and then Enter.
   Save the devices.csv
   
   B. Run the script (from root using sudo) with copy_sftunnel_certs.py devices.csv and it shows you the outcome. Here it shows that the certificate to FTDv was correctly pushed and that for BSNS-1120-1 it could not make the SSH connection to the device.
   
   copy_sftunnel_certs.py devices.csv
   
   ===Manual approach===
   
   
   1. Print (cat) the output each of the files for each FTD impacted (cacert.pem / sftunnel-key.pem (not shown completely for security purposes) / sftunnel-cert.pem) on the FMC CLI by copying the file location from previous output (FAILED_PUSH file).
      
      cacert.pem
      sftunnel-key.pem
      sftunnel-cert.pem
   2. Open FTD CLI of each respective FTD on expert mode with root privileges through sudo su and renew the certificates with the next procedure.
      1. Browse to the location seen on the light blue highlight from FAILED_PUSH output (cd /var/sf/peers/cdb123c8-4347-11ef-aca1-f3aa241412a1 here for example but this is different for each FTD).
      2. Take backups of the existing files.
         

         cp cacert.pem cacert.pem.backup
         cp sftunnel-cert.pem sftunnel-cert.pem.backup
         cp sftunnel-key.pem sftunnel-key.pem.backup​

         
         Take backups of the current certificates
      3. Empty the files so we can write up new content in them.
         

         > cacert.pem
         > sftunnel-cert.pem
         > sftunnel-key.pem​

         
         Empty contents of existing certificate files
      4. Write the new content (from FMC output) in each of the files individually using vi cacert.pem / vi sftunnel-cert.pem / vi sftunnel-key.pem (separate command per file - screenshots only show this for cacert.pem but needs to be repeated for sftunnel-cert.pem and sftunnel-key.pem).vi cacert.pem
         1. Press i to go into interactive mode (after vi command is entered and you see an empty file).
         2. Copy paste the entire content (including -----BEGIN CERTIFICATE----- and -----END CERTIFICATE-----) in the file.
            Copy content in vi (INSERT mode)
         3. Close and write to the file with ESC followed by :wq and then enter.
            ESC followed by :wq to write to file
      5. Validate that the correct permissions (chmod 644) and owners (chown root:root) are set for each of the files using ls -hal. This is correctly set actually when we update the existing file.
         All certificate files updated with right owners and permissions
         
         
   3. Restart the sftunnel on each respective FTD where the sftunnel was not operational for the changes in the certificate to take effect with the command pmtool restartbyid sftunnel
      
      pmtool restartbyid sftunnel
      
      
3. Validate that all FTDs are correctly connected now using sftunnel_status.pl output

Solution 2 - Certificate has already expired

In this situation, we have two different scenarios. Either all of the sftunnel connections are still operational or they are not anymore (or partial).

FTDs still connected through sftunnel

We can apply the same procedure as indicated in the Certificate has not yet expired (ideal scenario) - Recommended approach section.

However do NOT upgrade or reboot the FMC (or any FTD) in this situation as it disconnects all of the sftunnel connections and we need to manually run all of the certificate updates on each FTD. The only exception to this one, are the listed Hotfix releases as they do not require a reboot of the FMC.

The tunnels remain connected and the certificates are replaced on each of the FTDs. In case some certificates would fail to populate, it does prompt you with the ones that failed and you need to take the manual approach as indicated earlier on the previous section.

FTDs not connected anymore through sftunnel

Recommended approach

We can apply the same procedure as indicated in the Certificate has not yet expired (ideal scenario) - Recommended approach section. In this scenario, the new certificate will be generated on the FMC but cannot be copied to the devices as the tunnel is already down. This process can be automated with the copy_sftunnel_certs.py / copy_sftunnel_certs_jumpserver.py scripts

If all of the FTD devices are disconnected from the FMC, we can upgrade the FMC in this situation as it does not have an impact on the sftunnel connections. If you still have some devices connected through sftunnel, then be aware that the upgrade of the FMC closes all of the sftunnel connections and they do not come up again due to the expired certificate. The benefit of the upgrade here would be that it does provide you a good guidance on the certificate files that need to be transferred to each of the FTDs.

Manual approach

In this situation, you can then run the generate_certs.pl script from the FMC which generates the new certificates but you still need to push them over to each of the FTD devices manually. Depending on the amount of devices, this is doable or can be a tedious task. However when using the copy_sftunnel_certs.py / copy_sftunnel_certs_jumpserver.py scripts this is highly automated.