Nexus 7000 Series Switch Problem with Remote User Authentication via SSH with a TACACS account

Available Languages

Updated:September 11, 2013
Document ID:116335

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

This document provides the steps required in order to troubleshoot and confirm that a Cisco Nexus 7000 Series switch is affected by the known software defect Cisco bug ID CSCud02139.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

The information in this document is based on these software and hardware versions:

  • Cisco Nexus 7000 Series Switches
  • Cisco Nexus Operating System (NX-OS) Versions 5.2(5) to 5.2(7) Inclusive
  • Cisco NX-OS Versions 6.0(1) to 6.1(3) Inclusive

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Problem

Symptoms

Users are unable to remotely log in to a Nexus 7000 Series switch Virtual Device Context (VDC) with TACACS authentication.

Additionally, these messages are seen in the logs:

n7k-vdc-1# show log last 200 | grep TACACS
2013 May 13 17:17:31 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:17:46 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:18:06 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:18:12 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:18:16 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:20:26 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:20:39 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:21:50 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
2013 May 13 17:22:09 n7k-vdc-1  TACACS-3-TACACS_ERROR_MESSAGE  All servers 
failed to respond
n7k-vdc-1#

Conditions

This problem is encountered on Nexus 7000 Series switches that run Cisco NX-OS Versions between 5.2(5) and 5.2(7), as well as between 6.0.1 to 6.1(3).

The VDC must use TACACS authentication, like this example:

n7k-vdc-1# show run tacacs+

!Command: show running-config tacacs+
!Time: Mon May 13 17:20:57 2013

version 6.1(2)
feature tacacs+

ip tacacs source-interface mgmt0
tacacs-server timeout 30
tacacs-server host 192.0.2.9 key 7 "keypassword"
aaa group server tacacs+ default
    server 192.0.2.9
    use-vrf management


n7k-vdc-1# show run aaa

!Command: show running-config aaa
!Time: Mon May 13 17:21:30 2013

version 6.1(2)
aaa authentication login default group default
aaa authorization config-commands default group default
aaa authorization commands default group default
aaa accounting default group default
no aaa user default-role
aaa authentication login error-enable
tacacs-server directed-request

Troubleshoot

  1. Confirm the TACACS Server Status

    • Confirm that the Nexus 7000 Series switch is successfully able to ping the TACACS server via the correct Virtual Routing and Forwarding (VRF).
    • Confirm that the TACACS server still successfully authenticates users on other devices.
  2. Check the Authentication, Authorization, and Accounting (AAA) Process Error Logs

    Use this command in order to check the AAA process error logs:

    n7k-vdc-1# show system internal aaa event-history errors

    1) Event:E_DEBUG, length:54, at 786852 usecs after Mon May 13 17:22:09 2013
        [102] All Configured methods failed for default:default

    2) Event:E_DEBUG, length:53, at 786796 usecs after Mon May 13 17:22:09 2013
        [102] protocol TACACS failed with server group default

    3) Event:E_DEBUG, length:54, at 379206 usecs after Mon May 13 17:22:09 2013
        [102] All Configured methods failed for default:default

    4) Event:E_DEBUG, length:53, at 379172 usecs after Mon May 13 17:22:09 2013
        [102] protocol TACACS failed with server group default

    5) Event:E_DEBUG, length:54, at 89083 usecs after Mon May 13 17:21:51 2013
        [102] All Configured methods failed for default:default

    6) Event:E_DEBUG, length:53, at 89051 usecs after Mon May 13 17:21:51 2013
        [102] protocol TACACS failed with server group default


  3. Check the TACACS+ Process Error Logs

    Use this command in order to check the TACACS+ process error logs:

    n7k-vdc-1# show system internal tacacs+ event-history errors

    1) Event:E_DEBUG, length:88, at 786728 usecs after Mon May 13 17:22:09 2013
        [100] switch_tac_server:  Unreachable servers case .setting error code  for
        aaa session 0

    2) Event:E_DEBUG, length:77, at 786726 usecs after Mon May 13 17:22:09 2013
        [100] switch_tac_server:  no more server in the server group for 
         aaa session 0

    3) Event:E_DEBUG, length:103, at 786680 usecs after Mon May 13 17:22:09 2013
        [100] connect_tac_server: non blocking connect failed, switching server for
        aaa session id(0) rtvalue(3)

    4) Event:E_DEBUG, length:97, at 786677 usecs after Mon May 13 17:22:09 2013
        [100] non_blocking_connect(171): getaddrinfo(DNS cache fail) with retcode:-1
        for server:192.0.2.9

    5) Event:E_DEBUG, length:62, at 786337 usecs after Mon May 13 17:22:09 2013
        [100] tplus_encrypt(655):key is configured for this aaa session.

    6) Event:E_DEBUG, length:95, at 786287 usecs after Mon May 13 17:22:09 2013
        [100] tplus_make_acct_request(1343):Not calling the name-resolution routine
        as rem_addr is empty

    7) Event:E_DEBUG, length:63, at 786285 usecs after Mon May 13 17:22:09 2013
        [100] tplus_make_acct_request(1308):Accounting userdata:console0

    8) Event:E_DEBUG, length:63, at 786266 usecs after Mon May 13 17:22:09 2013
        [100] init_tplus_req_state_machine:Global source-interface mgmt0

    9) Event:E_DEBUG, length:48, at 785842 usecs after Mon May 13 17:22:09 2013
        [100] is_intf_up_with_valid_ip(1129):Port is up.

    10) Event:E_DEBUG, length:57, at 785812 usecs after Mon May 13 17:22:09 2013
        [100] is_intf_up_with_valid_ip(1126):Proper IOD is found.

    11) Event:E_DEBUG, length:52, at 785799 usecs after Mon May 13 17:22:09 2013
        [100] Exiting function: get_if_index_from_global_conf

    12) Event:E_DEBUG, length:66, at 785797 usecs after Mon May 13 17:22:09 2013
        [100] Function get_if_index_from_global_conf: found interface mgmt0

    13) Event:E_DEBUG, length:53, at 785783 usecs after Mon May 13 17:22:09 2013
        [100] Entering function: get_if_index_from_global_conf

    14) Event:E_DEBUG, length:68, at 785781 usecs after Mon May 13 17:22:09 2013
        [100] init_tplus_req_state_machine:Falling to globally configured one

    15) Event:E_DEBUG, length:79, at 785779 usecs after Mon May 13 17:22:09 2013
        [100] init_tplus_req_state_machine:No source-interface configured for this group


  4. Debug TACACS+ Authentication Requests

    • Turn on debugging for TACACS+ Authentication requests.
    • AAA debugging outputs these logs:

      n7k-vdc-1# debug tacacs+ aaa-request
      n7k-vdc-1# show logging logfile last 5
      2013 May 13 18:20:26.077572 tacacs: tplus_encrypt(655):key is configured 
      for this aaa session.
      2013 May 13 18:20:26.077918 tacacs: non_blocking_connect(171): getaddrinfo
      DNS cache fail) with retcode:-1 for server:192.0.2.9
      2013 May 13 18:20:26.077938 tacacs: connect_tac_server: non blocking connect
      failed, switching server for aaa session id(0) rtvalue(3)
      2013 May 13 18:20:26.077978 tacacs: switch_tac_server:  no more server in the
      server group for aaa session 0
      2013 May 13 18:20:26.077993 tacacs: switch_tac_server:  Unreachable servers
      case .setting error code  for aaa session 0


  5. Perform a Packet Capture on the TACACS Server

    A packet capture on the TACACS server shows that no packets arrive from the VDC.

  6. Perform a Ethanalyzer Capture on the Nexus 7000 Series Switch

    An Ethanalyzer capture shows that no packets egress towards the TACACS server.

  7. Check the Running Processes on the VDC

    The show proc cpu sort command shows 33 instances (32 defunct) of the TACACSD process running.
    n7k-vdc-1# show proc cpu sort | include tacacs
    1538           16        16   1014    0.0%  tacacsd
    1855           16        10   1625    0.0%  tacacsd
    2163           16        10   1678    0.0%  tacacsd
    2339           15        23    676    0.0%  tacacsd
    3820           15        10   1595    0.0%  tacacsd
    3934           16        13   1272    0.0%  tacacsd
    4416           25         8   3211    0.0%  tacacsd
    4470           16        23    734    0.0%  tacacsd
    5577           26        12   2191    0.0%  tacacsd
    6592       969767  14589069     66    0.0%  tacacs
    6934           16        13   1297    0.0%  tacacsd
    8878           16        13   1252    0.0%  tacacsd
    8979           16        12   1345    0.0%  tacacsd
    10153           26        11   2453    0.0%  tacacsd
    10202           15         8   1888    0.0%  tacacsd
    10331           26        11   2368    0.0%  tacacsd
    10482           16        14   1190    0.0%  tacacsd
    14148           15        11   1433    0.0%  tacacsd
    14385           14        10   1496    0.0%  tacacsd
    14402           15         9   1775    0.0%  tacacsd
    20678           16         9   1785    0.0%  tacacsd
    20836           16        13   1246    0.0%  tacacsd
    21257           15        13   1212    0.0%  tacacsd
    21617           15         9   1749    0.0%  tacacsd
    22159           15        12   1328    0.0%  tacacsd
    23776           15        12   1320    0.0%  tacacsd
    24017           25         9   2788    0.0%  tacacsd
    29496           15         8   1990    0.0%  tacacsd
    29972           15        11   1368    0.0%  tacacsd
    30111           25         9   2847    0.0%  tacacsd
    30204           15         9   1721    0.0%  tacacsd
    30409           16        13   1254    0.0%  tacacsd
    32410           15         8   1876    0.0%  tacacsd


Solution

The VDC encounters the known software defect Cisco bug ID CSCud02139.

The TACACSD process spawns child processes that get stuck. This reaches a maximum of 32 processes, and it is unable to spawn any more in order to pass the authentication.

Confirmation

  1. Confirm that there are 33 instances of TACACSD. You can use the command show proc cpu sort | grep -c 'tacacsd' in order to count the instances.
  2. Perform an ethanalyzer capture, and confirm that the request does not leave the Nexus 7000 Series switch.
  3. Match the previous log messages.

Workarounds

There are three possibilities. Remove all of the TACACS configuration, and remove and readd the feature and the configuration. Another option is to perform a supervisor switchover. Or you can reload the VDC.

Resolved Versions

  • NX-OS Versions 5.2(9) and Later in the 5.2 Train
  • NX-OS Versions 6.1(3) and Later in the 6.1 Train

Related Information

Revision History

Revision Publish Date Comments
1.0
20-Jun-2013
Initial Release
TAC Authored

Contributed by Cisco Engineers

  • Scott Laffer
    Cisco TAC Engineer.

Was this Document Helpful?

FeedbackFeedback

Contact Cisco

This Document Applies to These Products