Troubleshoot an Secure Firewall ASA Device
ASA Fails to Reconnect to CDO After Reboot
If CDO and your ASA do not connect after an ASA reboot, it may be because the ASA has fallen back to using an OpenSSL cipher suite that is not supported by CDO's Secure Device Connector (SDC). This troubleshooting topic tests for that case and provides remediation steps.
Symptoms
-
ASA reboots and CDO and the ASA fail to reconnect. CDO displays the message, "Failed to reconnect."
-
When attempting to onboard an ASA, CDO displays the message: Certificate could not be retrieved for <ASA_IP_Address>.
Cannot onboard ASA due to certificate error
Environment: ASA is configured with client-side certificate authentication.
Solution: Disable client-side certificate authentication.
Details: ASAs support credential-based authentication as well as client-side certificate authentication. CDO cannot connect to ASAs that use client-side certificate authentication. Before onboarding your ASA to CDO, make sure it does not have client-certificate authentication enabled by using this procedure:
Procedure
Step 1 |
Open a terminal window and connect to the ASA using SSH. |
Step 2 |
Enter global configuration mode. |
Step 3 |
At the hostname (config)# prompt, enter this command: no ssl certificate-authentication interfaceinterface-nameport 443 The interface name is the name of the interface CDO connects to. |
Determine the OpenSSL Cipher Suite Used by your ASA
Use this procedure to identify the OpenSSL cipher suite being used by your ASA. If the cipher suite named in the command output is not in the list of supported cipher suites, the SDC doesn't support that cipher suite and you will need to update the cipher suites on your ASA.
Procedure
Step 1 |
Open a console window on a computer that can reach the SDC. |
||
Step 2 |
Connect to your SDC using SSH. You can log in as a regular user such as CDO or SDC or some other user you created. You don't need to be logged in as root.
|
||
Step 3 |
At the command prompt enter: openssl s_client -showcerts -connect ASA_IP_Address:443 |
||
Step 4 |
Look for these lines in the command output.
In this example, the cipher suite being used by the ASA is |
Cipher Suites Supported by CDO's Secure Device Connector
CDO's Secure Device Connector uses node.js which only accepts the latest and most secure ciphers. As a result, CDO's SDC only supports this list of ciphers:
-
ECDHE-RSA-AES128-GCM-SHA256
-
ECDHE-ECDSA-AES128-GCM-SHA256
-
ECDHE-RSA-AES256-GCM-SHA384
-
ECDHE-ECDSA-AES256-GCM-SHA384
-
DHE-RSA-AES128-GCM-SHA256
-
ECDHE-RSA-AES128-SHA256
-
DHE-RSA-AES128-SHA256
-
ECDHE-RSA-AES256-SHA384
-
DHE-RSA-AES256-SHA384
-
ECDHE-RSA-AES256-SHA256
-
DHE-RSA-AES256-SHA256
If the cipher suite you use on your ASA is not in this list, SDC does not support it and you will need to update the cipher suite on your ASA.
Updating your ASA's Cipher Suite
To update the TLS cipher suites on an ASA:
Procedure
Step 1 |
Connect to the ASA using SSH. |
||
Step 2 |
Once connected to the ASA, elevate your privileges to global configuration mode. Your prompt should look like this: asaname(config)# |
||
Step 3 |
At the prompt, enter a command similar to this:
|
||
Step 4 |
After you submit the command, enter write memory at the prompt to save the local configuration. For example: |
Troubleshoot ASA using CLI commands
This section discusses some of the important commands you may want to use to troubleshoot the ASA and test basic connectivity. See CLI Book 1: Cisco ASA Series General Operations CLI Configuration Guide to learn about other troubleshooting scenarios and CLI commands. In the 'System Administration' section, navigate to the 'Testing and Troubleshooting' chapter.
You can use the CDO CLI interface available for each ASA device to execute these commands. See Using the CDO Command Line Interface to learn about how to use the CLI interface in CDO.
NAT Policy Settings
Some of the important commands to determine the NAT settings are as follows:
-
To determine NAT policy statistics, use show nat.
-
To determine the NAT pools, including the addresses and ports allocated, and how many times they were allocated, use show nat pool.
For more commands related to NAT, see CLI Book 2: Cisco ASA Series Firewall CLI Configuration Guide, and navigate to the 'Network Address Translation (NAT)' chapter.
Test Basic Connectivity: Pinging Addresses
You can ping the ASA device using the ping <IP address> command using the ASA CLI interface. To learn about
Display the Routing Table
Use the show route command to view the entries in the routing table.
ciscoasa# show route
Example output for a routing table of an ASA:
Codes: L - local, C - connected, S - static, R - RIP, M - mobile, B - BGP
D - EIGRP, EX - EIGRP external, O - OSPF, IA - OSPF inter area
N1 - OSPF NSSA external type 1, N2 - OSPF NSSA external type 2
E1 - OSPF external type 1, E2 - OSPF external type 2, V - VPN
i - IS-IS, su - IS-IS summary, L1 - IS-IS level-1, L2 - IS-IS level-2
ia - IS-IS inter area, * - candidate default, U - per-user static route
o - ODR, P - periodic downloaded static route, + - replicated route
SI - Static InterVRF
Gateway of last resort is 192.168.0.254 to network 0.0.0.0
S* 0.0.0.0 0.0.0.0 [1/0] via 192.168.0.254, management
C 10.0.0.0 255.0.0.0 is directly connected, Outside
L 10.10.10.1 255.255.255.255 is directly connected, Outside
C 192.168.0.0 255.255.255.0 is directly connected, management
L 192.168.0.118 255.255.255.255 is directly connected, management
Monitor Switch Ports
-
show interface
Displays interface statistics.
-
show interface ip brief
Displays interface IP addresses and status.
-
show arp
Shows dynamic, static, and proxy ARP entries. Dynamic ARP entries include the age of the ARP entry in seconds.
Example output of ARP entries:
management 10.10.32.129 0050.568a.977b 0
management 10.10.32.136 0050.568a.5387 21
LANFAIL 20.20.21.1 0050.568a.4d70 96
outsi 10.10.16.6 0050.568a.e6d3 3881
outsi 10.10.16.1 0050.568a.977b 5551
Troubleshoot ASA Remote Access VPN
This section discusses some of the troubleshooting issues that may occur when configuring remote access VPN on an ASA device.
Missing Information on the RA VPN Monitoring Page
This issue may occur if the outside interface is not enabled for Webvpn.
Resolution:
-
In the navigation pane, click Inventory.
-
Click the Devices tab and then click the ASA tab.
-
Select the RA VPN headend ASA device that is having issues.
-
In the Management pane on the right, click Configuration.
-
Click Edit and search for ‘webvpn’.
-
Press Enter and add
enable
interface_name. Here, the interface_name is the name of the outside interface to which users connect when making the remote access VPN connection. Although this is normally the outside (internet-facing) interface, choose whichever interface is between the device and the end-users you are supporting with this connection profile.For example:
webvpn
enable outside
-
Click Save.
-
Preview and deploy the configuration to the device.
ASA Real-time Logging
Use real-time logging to display the last 20 seconds of logged data or the last 10 KB of logged data, whichever limit is reached first. When CDO retrieves the real-time data, it reviews the existing logging configuration on your ASDM, changes it to request debugging-level data, and then returns the logging configuration to your configuration. The logging CDO displays reflects any logging filters you may have set in ASDM.
You can see the commands that CDO sends to perform logging by reviewing the change log. Below is an example of a change log entry. The first entry (on the bottom) indicates that CDO "turned on" logging with the logging enable command and changed the ASDM logging level to debugging. The second entry (on the top) shows that the logging configuration was returned to its previous state. Logging was "turned off" with the no logging enable command and the ASDM logging level was returned to informational.
View ASA Real-time Logs
Procedure
Step 1 |
On the Inventory page, click the Devices tab. |
Step 2 |
Click the appropriate device type tab and select the device for which you want to view real-time data. |
Step 3 |
Click Troubleshoot . |
Step 4 |
(Optional) Before clicking View Real-time Log, you can define a filter in the left pane to refine the results of your logging search. |
Step 5 |
Click View Real-time Log. CDO retrieves the real-time logging data based on your filterin criteria and displays it. |
Step 6 |
To see an additional 20 seconds of logged data or the last 10 KB of logged data. click View Real-Time Log again. |
ASA Packet Tracer
Packet tracer allows you to send a synthetic packet into the network and evaluate how the existing routing configuration, NAT rules, and policy configurations, affect that packet. Use this tool to troubleshoot these kinds of issues:
-
Users report that they cannot reach resources that they should be able to.
-
Users report that they can reach resources they should not be able to.
-
Test a policy to determine if it works as you expect.
Packet tracer can be used on a live, online, ASA device either physical or virtual. Packet Tracer does not work on ASA model devices. Packet tracer evaluates packets based on the saved configuration on the ASA. Staged changes on CDO are not evaluated by packet tracer.
We consider it a best-practice to run packet tracer on an ASA that is in the synced state. Though packet tracer will run if the device is not synced, you could encounter some unexpected results. For example, if you deleted a rule in the staged configuration on CDO, and this same rule was triggered on the ASA during packet tracing, CDO won't be able to show you the result of the packet's interaction with that rule.
Troubleshooting with ASA Packet Tracer
As packet tracer sends the packet through the routing configuration, NAT rules, and security policies of your ASA, it displays the packet's status at each step. If the packet is allowed by the policy it receives a green checkmark . If a packet is denied and dropped, CDO displays a red X .
Packet tracer also displays a real time log of the result of the packet trace. In the example below, you can see where a rule denied a tcp packet.
Troubleshoot an ASA Device Security Policy
Procedure
Step 1 |
From the Inventory page, select your ASA, and click Troubleshoot in the Actions pane. |
Step 2 |
In the pane, select the interface and packet type you want to send virtually through your ASA. |
Step 3 |
(Optional) If you want to trace a packet where the security group tag value is embedded in the Layer 2 CMD header (Trustsec), check SGT number and enter the security group tag number, 0-65535. |
Step 4 |
Specify the source and destination. You can specify IPv4 or IPv6 addresses, fully-qualified domain names (FQDN), or security group names or tags if you use Cisco Trustsec. For the source address, you can also specify a username in the format Domain\username. |
Step 5 |
Specify other protocol characteristics:
|
Step 6 |
Click Run Packet Tracer. |
Step 7 |
Continue with Analyze Packet Tracer Results. |
Troubleshoot an Access Rule
Procedure
Step 1 |
Select . |
Step 2 |
Select a policy that is associated with your ASA. |
Step 3 |
Select a rule in the network policy to troubleshoot and click Troubleshoot in the details pane. Notice that in the values panel of the troubleshoot page, many of the fields are pre-populated with the attributes of the rule you chose. |
Step 4 |
Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer button becomes active. |
Step 5 |
Click Run Packet Tracer. |
Step 6 |
Continue with Analyze Packet Tracer Results. |
Troubleshoot a NAT Rule
Procedure
Step 1 |
From the Inventory page, select your ASA, and click View NAT Rules in the Action pane. |
Step 2 |
Select your ASA, and click View NAT Rules in the Action pane. |
Step 3 |
Select the rule from the NAT Rules table that you want to troubleshoot and click Troubleshoot in the details pane. Notice that in the values panel of the Troubleshoot page, many of the fields are pre-populated with the attributes of the rule you chose. |
Step 4 |
Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer becomes active. |
Step 5 |
Click Run Packet Tracer. |
Step 6 |
Continue with Analyze Packet Tracer Results. |
Troubleshoot a Twice NAT Rule
Procedure
Step 1 |
From the Inventory page, select your ASA, and click View NAT Rules in the Action pane. |
Step 2 |
Select the rule from the NAT Rules table that you want to troubleshoot and click Troubleshoot in the details pane. For a bi-directional Twice NAT rule, this opens a dropdown where you choose to troubleshoot the source packet translation or the destination packet translation. |
Step 3 |
Enter information in the remaining required fields. Once you have completed all the required fields the Run Packet Tracer becomes active. |
Step 4 |
Click Run Packet Tracer. |
Analyze Packet Tracer Results
Whether the packet is dropped or allowed, you can learn why by expanding a row in the packet trace table and reading the rule or logging information related to that action. In the example below, packet tracer identified an access list policy that included a rule to deny an IP packet coming from any source and going to any destination. If this is not the action you want, you can click the View in Network Policies link and edit that rule immediately. After you edit the rule, be sure to deploy that configuration change to the ASA and then re-run packet tracer to ensure that you get the access results you expect.
Along with the packet tracer results, CDO displays the real-time logs from the ASA.
Cisco ASA Advisory cisco-sa-20180129-asa1
The Cisco Product Security Incident Response Team (PSIRT) published the security advisory cisco-sa-20180129-asa1 which describes a critical-severity ASA and Firepower security vulnerability. Read the entire PSIRT team advisory for a full explanation of what ASA and Firepower hardware, software, and configurations are affected.
If you determine that your ASAs are impacted by the advisory, you can upgrade your ASAs to the patched version using CDO. Use this process:
Procedure
Step 1 |
Configure a DNS server on each ASA that is affected. |
Step 2 |
Return to the advisory to determine which software patch you need. |
Step 3 |
See Upgrade ASA and ASDM Images on a Single ASA for topics that describe how to use CDO to upgrade your ASAs to the fixed releases listed in the ASA advisory. Start with the upgrade prerequisites and then read about upgrading individual ASAs, upgrading ASAs in an active-standby configuration, or upgrading ASAs in bulk. For your convenience, here is the summary of the security advisory that Cisco reported: |
Confirming ASA Running Configuration Size
To confirm the size of your running configuration file, follow this procedure:
Procedure
Step 1 |
Access the ASA's command line interface in one of these ways:
|
Step 2 |
At the prompt type |
Step 3 |
When prompted for the Source filename, don't type anything and press <Enter> |
Step 4 |
When prompted for the destination filename, enter a name for the output file. After ASA copies the running configuration the file you specified, it returns you to the privileged EXEC prompt. |
Step 5 |
At the prompt, type |
Step 6 |
Look in the length column. If your file is over 4718592 bytes, it is larger than 4.5 MB. Here is a sample set of commands and output:
|
Container Privilege Escalation Vulnerability Affecting Secure Device Connector: cisco-sa-20190215-runc
The Cisco Product Security Incident Response Team (PSIRT) published the security advisory cisco-sa-20190215-runc which describes a high-severity vulnerability in Docker. Read the entire PSIRT team advisory for a full explanation of the vulnerability.
This vulnerability impacts all CDO customers:
-
Customers using CDO's cloud-deployed Secure Device Connector (SDC) do not need to do anything as the remediation steps have already been performed by the CDO Operations Team.
-
Customers using an SDC deployed on-premise need to upgrade their SDC host to use the latest Docker version. They can do so by using the following instructions:
Updating a CDO-Standard SDC Host
Use these instructions if you deployed an SDC using the CDO image.
Procedure
Step 1 |
Connect to your SDC host using SSH or the hypervisor console. |
||
Step 2 |
Check the version of your Docker service by running this command:
|
||
Step 3 |
If you are running one of the latest virtual machines (VMs) you should see output like this: It's possible you may see an older version here. |
||
Step 4 |
Run the following commands to update Docker and restart the service:
|
||
Step 5 |
Run the docker version command again. You should see this output:
|
||
Step 6 |
You are done. You have now upgraded to the latest, and patched, version of Docker. |
Updating a Custom SDC Host
If you have created your own SDC host you will need to follow the instructions to update based on how you installed Docker. If you used CentOS, yum and Docker-ce (the community edition) the preceding procedure will work.
If you have installed Docker-ee (the enterprise edtion) or used an alternate method to install Docker, the fixed versions of Docker may be different. You can check the Docker page to determine the correct versions to install: Docker Security Update and Container Security Best Practices.
Bug Tracking
Cisco is continuing to evaluate this vulnerability and will update the advisory as additional information becomes available. After the advisory is marked Final, you can refer to the associated Cisco bug for further details:
Large ASA Running Configuration Files
Behavior in CDO
You may see behavior such as the ASA failing to onboard, CDO not displaying all of the configuration defined in the ASA's running configuration file, or CDO failing to write to the change log.
Possible Cause
The running configuration file of your ASA may be "too large" for CDO.
When you an onboard an ASA to CDO, CDO stores a copy of the ASA's running configuration file in its database. Generally, if that running configuration file is too large (4.5 MB or larger), or it contains too many lines (approximately 22,000 lines), or there are too many access-list entries for a single access group, CDO will not be able to predictably manage that device.
To confirm the size of your running configuration file, see Confirming ASA Running Configuration Size.
Workaround or Solution
Contact your Cisco account team for help safely reducing the size of your configuration file without disrupting your security policies.