Prime Infrastructure 3.5+ Integration Issues due to TOFU Certificate

Available Languages

Download Options

  • PDF     (189.9 KB)
    View with Adobe Reader on a variety of devices
  • ePub     (201.4 KB)
    View in various apps on iPhone, iPad, Android, Sony Reader, or Windows Phone
  • Mobi (Kindle)     (104.9 KB)
    View on Kindle device or Kindle app on multiple devices
Updated:March 18, 2020
Document ID:215335

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

    Introduction

    This document describes the integration issue that occurs due to Trust-on-first-use (TOFU) certificate mismatch after a new Certificate Signing Request (CSR) is generated in Cisco Prime Infrastructure (primary/secondary), how to troubleshoot and resolve it.

    Prerequisites

    Requirements

    Cisco recommends that you have knowledge of these topics:

    • Cisco Prime Infrastructure
    • High Availability

    Components Used

    The information in this document is based on Cisco Prime Infrastructure version 3.5 and above.

    The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

    Background Information

    These are the reference documents that provide information on High Availability and certificate generation in Cisco Prime Infrastructure.

    High Availability Guide: https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-6/admin/guide/bk_CiscoPrimeInfrastructure_3_6_AdminGuide/bk_CiscoPrimeInfrastructure_3_6_AdminGuide_chapter_01011.html

    Administrator Guide: https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/infrastructure/3-6/admin/guide/bk_CiscoPrimeInfrastructure_3_6_AdminGuide/bk_CiscoPrimeInfrastructure_3_6_AdminGuide_chapter_0100.html

    Problem

    TOFU - The certificate received from the remote host is trusted when the connection is made for the first time.

    TOFU certificate on prime infrastructure or the remote host that prime is connected to can change if a new certificate is generated or if the server is deployed again on VM host.

    Generating and importing a new CSR on prime infrastructure server (primary/secondary) sends the new TOFU certificate information to remote servers when the connectivity is re-initiated after a service restart.

    If the remote host sends a different certificate for any sub-sequent connection after the first, the connection will be rejected.

    Remote host could be (Primary or Secondary server in HA deployment, Integrated Service Engine (ISE) server) where the old TOFU is still present.

    This causes registration failure between Primary and Secondary servers, Prime and ISE server.

    The troubleshoot section describes the error messages that can be found in the health monitor logs in such scenarios.

    Troubleshoot

    In Primary health monitor log, these error messages pointing the mismatch in the secondary certificate can be found.

    [system] [HealthMonitorThread] TOFU failed. 
    Check local trust Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=prime-sec, OU=Prime Infra, O=Cisco Systems, L=SJ, ST=CA, C=US 
    javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: 
    Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=prime-sec

    These error messages can be found on the prime infrastructure logs pointing the mismatch in ISE server certificate.

    [system] [seqtaskexecutor-3069] TOFU failed. 
    Check local trust Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=ISE-server
    javax.net.ssl.SSLHandshakeException: java.security.cert.
    CertificateException: Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=ISE-server

    In Secondary health monitor log, these error messages pointing the mismatch in the primary certificate can be found.

    [system] [HealthMonitorThread] TOFU failed. 
    Check local trust Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=prime-pri, OU=Prime Infra, O=Cisco Systems, L=SJ, ST=CA, C=US 

    javax.net.ssl.SSLHandshakeException: java.security.cert.CertificateException: 
    Trust-on-first-use is configure for this connection. 
    Current certificate of the remote host is different from what was used earlier 
    - CN=prime-pri

    Solution

    The current TOFU certificates on prime needs to be listed, from that the old certificate entry for the corresponding remote host should be identified and removed before you attempt the integration from prime again.

    Configuration

    View Certificate Validation List

    The command ncs certvalidation tofu-certs listcerts can be used to view the certificate validation list.

    This output is from the Cisco Prime Infrastructure primary server [IP=1XX.XX.XX.XX]:

    prime-pri/admin# ncs certvalidation tofu-certs listcerts

    Host certificate are automatically added to this list on first connection, 
    if trust-on-first-use is configured - ncs certvalidation certificate-check ...

    host=1X.XX.XX.XX_8082;  subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=prime-pri
    host=1Z.ZZ.ZZ.ZZ_443;  subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=ISE-server
    host=1YY.YY.YY.YY_8082;  subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=prime-sec

    prime-pri/admin#

    This output is from the Cisco Prime Infrastructure secondary server [IP=1YY.YY.YY.YY]

    prime-sec/admin# ncs certvalidation tofu-certs listcerts

    Host certificate are automatically added to this list on first connection, 
    if trust-on-first-use is configured - ncs certvalidation certificate-check ...

    host=1YY.YY.YY.YY_8082;  subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=prime-sec
    host=127.0.0.1_8082;  subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=prime-sec
    host=1X.XX.XX.XX_8082; subject= /C=US/ST=CA/L=SJ/O=Cisco Systems/OU=Prime Infra/CN=prime-pri

    prime-sec/admin#

    Delete Certificate

    Use command ncs certvalidation tofu-certs deletecert host <host> in order to delete to certificate validation.

    From primary server check and delete the old entries for ISE and secondary server's TOFU certificates respectively.

    • ncs certvalidation tofu-certs deletecert host 1YY.YY.YY.YY_8082
    • ncs certvalidation tofu-certs deletecert host 1Z.ZZ.ZZ.ZZ_443

    From secondary server check and delete the old entries for primary server's tofu certificate with the use of command ncs certvalidation tofu-certs deletecert host 1X.XX.XX.XX_8082.

    Re-initialize HA from Primary to Secondary

    Step 1. Log in to Cisco Prime Infrastructure with a user ID and password that has administrator privileges.

    Step 2. From the menu, navigate to Administration > Settings > High Availability. Cisco Prime Infrastructure displays the HA status page. 

    Step 3. Select HA Configuration and then complete the fields as follows:

    1. Secondary Server: Enter the IP address or the host name of the secondary server.
    2. Authentication Key: Enter the authentication key password you set during the secondary server installation.
    3. Email Address: Enter the address (or comma-separated list of addresses) to which notification about HA state changes should be mailed. If you have already configured email notifications using the Mail Server Configuration page (see “Configure Email Server Settings”), the email addresses you enter here will be appended to the list of addresses already configured for the mail server.
    4. Failover Type: Select either Manual or Automatic. It is recommended that you select Manual.

    It is recommended to use DNS server in order to resolve the host name to an IP address. If you use /etc/hosts file instead of DNS server, you should enter the secondary IP address instead of the host name.

    Step 4. If you use the virtual IP feature, select the Enable Virtual IP checkbox, then complete the additional fields as follows:

    1. IPV4 Virtual IP: Enter the virtual IPv4 address you want both HA servers to use.
    2. IPV6 Virtual IP: (Optional) Enter the IPv6 address you want both HA servers to use.

    Virtual IP addressing will not work unless both the servers are on the same subnet. You should not use IPV6 address block fe80, it has been reserved for link-local unicast addressing.

    Step 5. Click Check Readiness in order to ensure if the HA related environmental parameters are ready for the configuration.

    Step 6. Click Register in order to view the Milestone progress bar, to check the 100% completion of Pre-HA Registration, Database Replication and Post HA Registration as shown here. Cisco Prime Infrastructure initiates the HA registration process. When registration completes successfully, the Configuration Mode will display the value of Primary Active.

    Re-configure ISE Servers

    Step 1. Navigate to Administration > Servers > ISE Servers

    Step 2. Navigate to Select a command > Add ISE Server, then click Go
    Step 3. Enter the ISE server’s IP address, user name, and password

    Step 4. Confirm the ISE server password.

    Step 5. Click Save.

    Verify

    The command ncs certvalidation tofu-certs listcerts can be used to verify the new certificate. 

    Related Information

    TAC Authored

    Contributed by Cisco Engineers

    • Annangarachari R
      Cisco TAC Engineer

    Was this Document Helpful?

    FeedbackFeedback

    Contact Cisco

    This Document Applies to These Products