Troubleshoot Common GETVPN Issues

Introduction

This document describes what debugs to collect for most of the common Group Encrypted Transport VPN (GETVPN) issues.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• GETVPN
• Syslog Server Use

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information - GETVPN Troubleshooting Tools

GETVPN provides an extensive set of troubleshooting tools in order to ease the troubleshoot process. It is important to understand which of these tools are available, and when they are appropriate for each troubleshooting task. When troubleshooting, it is always a good idea to start with the least intrusive methods, so that the production environment is not negatively impacted. In order to assist that process, this section describes some of the commonly used tools available:

Control Plane Debugging Tools

Show Commands

Show commands are commonly used in order to show runtime operations in a GETVPN environment.

Syslogs

GETVPN has an enhanced set of syslog messages for significant protocol events and error conditions. This should always be the first place to look before you run any debugs.

Group Domain of Interpretation (GDOI) Event Trace

This feature was added in Version 15.1(3)T. Event tracing offers lightweight, always-on tracing for significant GDOI events and errors. There is also exit-path tracing with traceback enabled for exception conditions.

GDOI Conditional Debugs

This feature was added in Version 15.1(3)T. It allows filtered debugs for a given device based on the peer address, and should always be used when possible, especially on the Key Server.

Global Crypto and GDOI Debugs

These are the all of the various GETVPM debugs. Admins must use caution when debugging in large-scale environments. With GDOI debugs, five debug levels are provided for further debugging granularity:

GM1#debug crypto gdoi gm rekey ?
  all-levels  All levels
  detail      Detail level
  error       Error level
  event       Event level
  packet      Packet level
  terse       Terse level

Debug Level	What You Will Get
Error	Error conditions
Terse	Important messages to the user and protocol issues
Event	State transitions and events such as send and receive rekeys
Detail	Most detailed debug message information
Packet	Includes dump of detailed packet information
All	All of the above

Data Plane Debugging Tools

Here are some data plane debugging tools:

• Access Lists
• IP Precedence Accounting
• Netflow
• Interface Counters
• Crypto Counters
• IP Cisco Express Forwarding (CEF) Global and Per-feature Drop Counters
• Embedded Packet Capture (EPC)
• Data Plane Debugs (IP packet and CEF debugs)

Troubleshoot

Logging Facility Preparation and Other Best Practices

Before you begin to troubleshoot, ensure that you have prepared the logging facility as described here. Some best practices are also listed here:

• Check the router amount of free memory, and configure logging buffered debugging to a large value (10 MB or more if possible).
  
  
• Disable logging to the console, monitor, and syslog servers.
  
  
• Retrieve the logging buffer content with the show log command at regular intervals, every 20 mins to an hour, in order to prevent log loss due to buffer reuse.
  
  
• Whatever happens, enter the show tech command from affected Group Members (GMs) and Key Servers (KSs), and examine the output of the show ip route command in global and each Virtual Routing and Forwarding (VRF) involved, if any are required.
  
  
• Use Network Time Protocol (NTP) in order to sync the clock between all devices that are debugged. Enable millisecond (msec) timestamps for both debug and log messages:
  

  service timestamps debug datetime msec
  service timestamps log datetime msec

  
  
  
• Make sure show command outputs are timestamped.
  

  Router#terminal exec prompt timestamp

  
  
  
• When you collect show command outputs for control plane events or data plane counters, always collect multiple iterations of the same output.

Troubleshoot IKE Establishment

When the registration process first begins, GMs and KSs negotiate Internet Key Exchange (IKE) Sessions in order to protect GDOI traffic.

• On the GM, check that IKE is successfully established:
  

  gm1#show crypto isakmp sa
  IPv4 Crypto ISAKMP SA
  dst             src             state          conn-id status
  172.16.1.9      172.16.1.1      GDOI_REKEY        1068 ACTIVE
  172.16.1.1      172.16.1.9      GDOI_IDLE         1067 ACTIVE

  
  

  Note: The GDOI_IDLE state, which is the base of the registration, times out quickly and disappears, because it is not needed anymore after the initial registration.

  
  
  
• On the KS, you should see:
  

  ks1#show crypto isakmp sa
  IPv4 Crypto ISAKMP SA
  dst             src             state          conn-id status
  172.16.1.1      172.16.1.9      GDOI_IDLE         1001 ACTIVE

  
  

  Note: The rekey session only appears when needed on the KS.

  
  
  

Complete these steps if you do not reach that state:

• For insight about the cause of the failure, check the output from this command:

  router# show crypto isakmp statistics

• If the previous step is not helpful, you can get protocol-level insights if you enable the usual IKE debugs:

  router# debug crypto isakmp

  Notes:
  * Even though IKE is used, it is not used on the usual UDP/500 port, but rather on UDP/848.
  * If you encounter an issue at this level, provide the debugs for both KS and the affected GM.

  
  
  
• Due to the dependence on Rivest-Shamir-Adleman (RSA) sigs for the group rekeys, the KS must have an RSA key configured, and it must have the same name as the one specified in the group configuration.
  
  In order to check this, enter this command:
  
  

  ks1# show crypto key mypubkey rsa

Troubleshoot the Initial Registration

On the GM, in order to check the registration status, examine the output of this command:

gm1# show crypto gdoi | i Registration status
       Registration status  : Registered
gm1#

If the output indicates anything other than Registered, enter these commands:

On the GMs:

• Shut down crypto-enabled interfaces.
  

  Caution: It is expected that out-of-band management is enabled.

  
  
  
• Enable these debugs:
  

  gm1# debug crypto gdoi infra packet
  gm1# debug crypto gdoi gm packet

  
  
  
• Enable debugs on the KS side (see the next section).
  
  
• When the KS debugs are ready, unshut crypto-enabled interfaces, and wait for registration (in order to accelerate the process, issue the clear crypto gdoi command on the GM).

On the KSs:

• Verify the presence of the RSA key on the KS:
  

  ks1# show crypto key mypubkey rsa

  
  
  
• Enable these debugs:
  

  ks1# debug crypto gdoi infra packet
  ks1# debug crypto gdoi ks packet 

Troubleshoot Policy-Related Issues

Policy Issue Occurs PRIOR to Registration (Related to Fail-close Policy)

This issue only affects GMs, so collect this output from the GM:

gm1# show crypto ruleset

Note: In Cisco IOS-XE^?, this output is always empty since packet classification in not done in the software.

The show tech command output from the affected device provides the rest of the required information.

Policy Issue Occurs POST Registration, and Pertains to the Global Policy that is Pushed

There are usually two ways that this problem manifests:

• The KS cannot push the policies to the GM.
• There is a partial application of the policy amongst the GMs.

In order to help troubleshoot either issue, complete these steps:

1. On the affected GM, collect this output:
   

   gm1# show crypto gdoi acl 
   gm1# show crypto ruleset

   
   
   
2. Enable these debugs on GM:
   

   gm1# debug crypto gdoi infra packet 
   gm1# debug crypto gdoi gm acls packet

   
   
   
3. On the KS to which the affected GM registers, collect this output:
   

   ks1# show crypto gdoi ks members
   ks1# show crypto gdoi ks policy

   
   
   

   Note: In order to identify which KS the GM connects to, enter the show crypto gdoi group command.

   
   
   
4. On the same KS, enable these debugs:
   

   ks1# debug crypto gdoi infra packet
   ks1# debug crypto gdoi ks acls packet

   
   
   
5. Force the GM to register with this command on the GM:
   

   clear crypto gdoi

Policy Issue Occurs POST Registration, and Pertains to the Merge of Global Policy and Local Overrides

This issue usually manifests itself in the form of messages that indicate that an encrypted packet was recieved for which the local policies indicate that it is not supposed to be encrypted and vice versa. All of the data requested in the previous section and the show tech command output is required in this case.

Troubleshoot Rekey Issues

On the GMs:

• Collect these debugs:
  

  gm1# debug crypto gdoi infra packet 
  gm1# debug crypto gdoi gm packet 
  gm1# debug crypto gdoi gm rekey packet 

  
  
  
• Enter this command in order to verify that the GM still has an IKE Security Association (SA) of type GDOI_REKEY:
  

  gm1# show crypto isakmp sa

On the KSs:

• Collect the show crypto key mypubkey rsa command output from EACH KS. The keys are expected to be identical.
  
  
• Enter these debugs in order to view what occurs on the KS:
  

  ks1# debug crypto gdoi infra packet 
  ks1# debug crypto gdoi ks packet 
  ks1# debug crypto gdoi ks rekey packet 

Troubleshoot Time-based Anti-replay (TBAR)

The TBAR feature requires time-keeping across groups, and therefore requires the GMs pseudo-time clocks to be constantly resynced. This is performed during rekey or every two hours, whichever comes first.

Note: All output and debugs must be collected at the same time from both GMs and KS so that they can be correlated appropriately.

In order to investigate issues that occur at this level, collect this output.

• On the GMs:
  

  gm1# show crypto gdoi 
  gm1# show crypto gdoi replay

  
  
  
• On the KS:
  

  ks1# show crypto gdoi ks members 
  ks1# show crypto gdoi ks replay

In order to investigate TBAR time-keeping in a more dynamic way, enable these debugs:

• On the GM:
  

  gm1# debug crypto gdoi gm rekey packet 
  gm1# debug crypto gdoi replay packet (verbosity might need to be lowered)

  
  
  
• On the KS:
  

  ks1# debug crypto gdoi ks rekey packet
  ks1# debug crypto gdoi replay packet (verbosity might need to be lowered)

As of Cisoc IOS Version 15.2(3)T, the ability to record TBAR errors has been added, which makes it easier to spot these errors. On the GM, use this command in order to check if there are any TBAR errors:

R103-GM#show crypto gdoi gm replay
Anti-replay Information For Group GETVPN:
  Timebased Replay:
    Replay Value             : 512.11 secs
    Input Packets            : 0        Output Packets           : 0
    Input Error Packets    : 0        Output Error Packets   : 0
    Time Sync Error         : 0        Max time delta            : 0.00secs

TBAR Error History (sampled at 10pak/min):
    No TBAR errors detected

For more information about how to troubleshoot TBAR issues, refer to Time Based Anti-Replay Failure.

Troubleshoot KS Redundancy

Cooperative (COOP) establishes an IKE session in order to protect interKSs communication, so the troubleshooting technique previously described for IKE establishment  is applicable here as well.

COOP-specific troubleshooting comprises output checks of this command on all KSs involved:

ks# show crypto gdoi ks coop

Note: The most common mistake made with deployment of COOP KSs is to forget to import the same RSA key (both private and public) for the group on all KSs. This causes problems during rekeys. In order to check and compare public keys amongst KSs, compare the output of the show crypto key mypubkey rsa command from each KS.

If protocol-level troubleshooting is required, enable this debug on all KSs involved:

ks# debug crypto gdoi ks coop packet

FAQ

Why do you see this error message "% Setting rekey authentication rejected"?

You see this error message when you configure the KS after this line is added:

KS(gdoi-local-server)#rekey authentication mypubkey rsa GETVPN_KEYS
% Setting rekey authentication rejected.

The reason for this error message is usually because the key labelled GETVPN_KEYS does not exist. In order to fix this, create a key with the correct label using the command:

crypto key generate rsa mod <modulus> label <label_name> 

Note: Add the exportable keyword at the end if this is a COOP deployment and then import the same key in the other KS

Can a router configured as KS for one GETVPN group also function as a GM for the same group?

No. All GETVPN deployments require a dedicated KS which cannot participate as a GM for the same groups. This feature is not supported, because adding GM functionality to KS with all possible interactions like encryption, routing, QoS, etc., is not optimal for the health of this crucial network device. It must be available at all times for the entire GETVPN deployment to work.

Related Information

• Group Encrypted Transport VPN (GET VPN) - Cisco Systems
• Technical Support & Documentation - Cisco Systems