Contents

Introduction

Problem

Solution

Introduction

This document describes the issue on a Cisco IOS® router when Secure Shell (SSH) to the router sometimes fails with a reported user authentication failure in the SSH debugs. This issue occurs even though the user credentials entered are correct and the same credentials work correctly for Telnet.

Note: Cisco bug ID CSCum19502  has been filed in order to make the behavior between SSH and Telnet consistent.

Problem

Notice in these debugs that even though "debug aaa authentication" is enabled, there are no Authentication, Authorization, and Accounting (AAA) debugs being printed to show AAA actually is invoked and returns the failure.

Router#show debug

General OS:

AAA Authentication debugging is on

SSH:

Incoming SSH debugging is on

ssh detail messages debugging is on

Router#

*Sep 30 20:28:57.172: SSH2 2: MAC compared for #8 :ok

*Sep 30 20:28:57.172: SSH2 2: input: padlength 15 bytes

*Sep 30 20:28:57.172: SSH2 2: Using method =

keyboard-interactive

*Sep 30 20:28:57.172: SSH2: password authentication failed

for cisco

*Sep 30 20:28:59.172: SSH2 2: send:packet of  length 64

(length also includes padlen of 14)

*Sep 30 20:28:59.172: SSH2 2: computed MAC for sequence

no.#8 type 51

*Sep 30 20:29:01.751: SSH2 2: ssh_receive: 144 bytes received

*Sep 30 20:29:01.751: SSH2 2: input: total packet length of

128 bytes

*Sep 30 20:29:01.751: SSH2 2: partial packet length(block size)

16 bytes,needed 112 bytes,

Sometimes the syslog shown here is also observed when SSH is attempted, but it does not get printed consistently:

*Sep 30 20:23:27.598: %AAA-3-ACCT_LOW_MEM_UID_FAIL: AAA unable to create UID for incoming

calls due to insufficient processor memory

The root cause of the problem is low memory conditions on the router. When AAA fails to allocate memory to create the Unique ID (UID) for the incoming SSH session, it reports the same failure as an AAA authentication failure even though AAA is not attempted. This condition occurs when the processor free memory falls below the AAA "Authentication low-memory threshold", which by default is set to 3% of the total memory and can be checked with the show aaa memory command. This problem is often seen on an Aggregation Services Router (ASR) 1001 platform where there is limited memory on the router that can be exhausted with heavy control plane usage, such as a full Border Gateway Protocol (BGP) table. On the ASR 1001 there is 4GB of DRAM installed, but after all of the other CPUs and Linux processors boot Cisco IOS gets the 1.1 GB left over. Once the memory is exhausted to the point that AAA can no longer allocate memory for UID, SSH fails to work.

Consider this memory data from two ASRs:

SSH Not Working:

----------------

ASR1#show memory summary

Head       Total(b)     Used(b)      Free(b)   Lowest(b)  Largest(b)

Processor  7FE150387010  1160982064  1146067400     14914664    14225352    13918620

lsmpi_io  7FE14FB7E1A8      6295128     6294304          824         824         412

SSH Working:

------------

ASR2#show memory summary

Head       Total(b)     Used(b)      Free(b)   Lowest(b)  Largest(b)

Processor  7FFB6ACB0010  1160982064  1120122056     40860008    29163912    24132068

lsmpi_io  7FFB6A4A71A8      6295128     6294304          824         824         412

From a simple calculation, on the nonworking ASR the percentage of  free memory is 1.28% (14914664 / 1160982064 * 100) of total available memory. On the working ASR it is 3.51% (40860008 / 1160982064 * 100), which is just above the authentication low-memory threshold.

This problem is difficult to identify because the %AAA-3-ACCT_LOW_MEM_UID_FAIL message often does not get printed when this error occurs due to the low memory condition. Moreover, the way that AAA calculates memory threshold does not depend on the raw amount of processor memory available on the route processor (RP), but rather a percentage of the total memory. Therefore, there might still be seemingly plenty of processor memory shown as free in the show memory summary command output when this occurs with no malloc failures reported.

Note: Cisco bug ID CSCuj50368 has been filed in order to make SSH error messages more explicit about the real reason for the authentication failure.

One way to to verify if this is indeed the problem is to look at the AAA memory statistics:

Router#show aaa memory

Allocator-Name                  In-use/Allocated            Count

----------------------------------------------------------------------------

AAA AttrL Hdr             :          0/65888      (  0%) [      0] Chunk

AAA AttrL Sub             :          0/65888      (  0%) [      0] Chunk

AAA DB Elt Chun           :        544/65888      (  0%) [      4] Chunk

AAA Unique Id Hash Table  :       8196/8288       ( 98%) [      1]

AAA chunk                 :          0/16936      (  0%) [      0] Chunk

AAA chunk                 :          0/16936      (  0%) [      0] Chunk

AAA Interface Struct      :       1600/1968       ( 81%) [      4]

Total allocated: 0.230 Mb, 236 Kb, 241792 bytes

AAA Low Memory Statistics:

__________________________

Authentication low-memory threshold      : 3%

Accounting low-memory threshold          : 2%

AAA Unique ID Failure                    : 96

Local server  Packet dropped             : 0

CoA  Packet dropped                      : 0

PoD  Packet dropped                      :

If the "AAA Unique ID Failure" count increments with each failed SSH attempt, the problem is caused by this low memory condition.

In order to troubleshoot this issue, standard ASR 1000 memory troubleshooting steps should be taken in order to isolate the cause. For more information on how to troubleshoot memory issues on the ASR, see Memory Usage Overview.

Solution

In order to troubleshoot this issue, standard router memory troubleshooting steps should be taken. The steps isolate whether the problem is due to normal usage, in which case a platform/memory upgrade might be warranted; or a memory leak where additional memory monitoring and troubleshooting might be required. See Memory Leak Detector and common memory troubleshooting techniques for more details.

For versions that do not have the fix from Cisco bug ID CSCum19502 , the most obvious workaround is to enable Telnet or console access to the router, since only SSH is affected by this threshold.

Tip: The aaa memory threshold command allows you to reduce the threshold values to a minimum of 1%. However, while this provides a temporary way to SSH to the router, it can lead to other implications such as the allowance of processor memory utilization to drop really low before admins are alerted. This might cause more important processes, such as BGP that uses up large amounts of memory, to no longer work. Hence this is something that should be used with caution.

As explained earlier, it is completely plausible that the router does not leak memory but is just oversubscribed for the features enabled. In this case a platform/memory upgrade might be warranted.