Troubleshoot LDP Neighborship Issue on ASR9000

Introduction

This document describes the Label Distribution Protocol (LDP) Neighbor issue due to low memory on ASR9000 routers.

Background Information

LDP is a label distribution protocol that helps generate and share the label information for IGP routes. But before exchanging the label, the router first forms the LDP neighborship. As you know, LDP works on UDP and TCP protocols. For neighbour discovery, LDP uses UDP protocol and to create the neighborship it uses TCP protocol.

The router must have a good amount of memory and CPU to make a TCP connection.

Issue Summary

As evident, you can ping and traceroute.

RP/0/RSP0/CPU0:R1#ping 192.168.12.6 so l14

Thu Jun 27 07:01:31.192 UTC

Type escape sequence to abort.

Sending 5, 100-byte ICMP Echos to 192.168.12.6 timeout is 2 seconds:

!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 2/2/3 ms




RP/0/RSP0/CPU0:R1# traceroute 192.168.12.6 so l14 numeric

Thu Jun 27 07:01:56.748 UTC

Type escape sequence to abort.

Tracing the route to 192.168.12.6

1  192.168.99.220 [MPLS: Labels 26664/75671 Exp 0] 3 msec  2 msec  2 msec

2  192.168.96.8     [MPLS: Label 75671 Exp 0] 1 msec  1 msec  1 msec

3  192.168.62.151 [MPLS: Label 24201 Exp 0] 2 msec  2 msec  2 msec

4  192.168.12.6     2 msec  2 msec  2 msec

All ping and traceroute are working, but no LDP membership. 

RP/0/RSP0/CPU0:R1#  show mpls ldp discovery 192.168.12.6 detail 

Thu Jun 27 07:05:43.503 UTC




Local LDP Identifier: 192.168.248.84:0

Discovery Sources:

  Targeted Hellos:

    192.168.248.84 -> 192.168.12.6 (active), xmit/recv

      Hello interval: 5 sec (due in 978 msec)

      Quick-start: Enabled

      LDP Id:192.168.248.84:0

          Hold time: 45 sec (local:90 sec, peer:45 sec)

                     (expiring in 41.3 sec)

          Established: Jun 26 12:02:16.216 (18:58:47 ago)

          Session bringup fail reason:

             waiting for tcp incall

You can see the TCP Fail reason.

Analysis of the Issue

When Show mpls ldp trace error reverse | in 192.168.12.6  command is checked, it is noticed that there are no LDP neighbors due to low memory.

Jun 27 07:08:29.742 mpls/ldp/err 0/RSP0/CPU0 t1  [ERR][MISC]:8201: VRF(0x60000000): 
ldp_nbr_ok_to_connect: Call from 192.168.12.6 rejected - state=2, reason=''MPLS_LDP' 
detected the 'resource not available' condition 'A low memory condition prevents new 
LDP sessions'

Also, it is noticed that there is minor memory on RSP0 but RSP1 is normal.

RP/0/RSP0/CPU0:R1#show watchdog memory-state 

Thu Jun 27 06:16:13.033 UTC

Memory information:

    Physical Memory: 6144     MB

    Free Memory:      554.988 MB

    Memory State:          Minor

RP/0/RSP0/CPU0:R1#show watchdog memory-state location 0/RSP1/CPU0

Thu Jun 27 06:17:06.110 UTC

Memory information:

    Physical Memory: 6144     MB

    Free Memory:     1208.164 MB

    Memory State:         Normal

Because this issue is related to memory, you can verify the memory consumption of different components of the systems and ltrace utilizing the highest memory, even when compared to other RSPs and devices. It is found that RSP0 is utilizes the highest amount of memory for ltrace.

RP/0/RSP0/CPU0:R1#show shmem summary location 0/rSP0/CPU0 

Thu Jun 27 07:24:48.601 UTC

Total Shared memory: 2345M

 ShmWin: 349M

 Image: 71M

 LTrace: 690M

 AIPC: 73M

 SLD: 3M

 SubDB: 528K

 CERRNO: 148K

 GSP-CBP: 165M

 EEM: 0

 XOS: 15M

 CHKPT: 10M

 CDM: 9M

 XIPC: 4M

 DLL: 64K

 SysLog: 10M

 Miscellaneous: 940M




LTrace usage details:

 Used: 690M, Max: 3366M

 Current: default(dynamic)

 Configured: dynamic with scale-factor: 16 (changes take effect after reload)

You can also see that Ltrace is utilizing a high amount of memory.

Workaround:

1. Reload the standby RP.
2. Wait for it to sync. Confirm that on  sh shmem summary loc 0/RSP1/CPU0, the current profile is scale-factor 16.
3. Switchover.
4. Confirm that memory is normal. Confirm that LDP is back up.
5. Confirm that scale-factor 16 is applied to RSP0 with  sh shmem summary loc 0/RSP0/CPU0. 

Collect Logs:

show mpls ldp neighbor 

show mpls ldp neighbor detail

show mpls ldp discovery 192.168.12.6 detail

show watchdog memory-state 

show watchdog memory-state location 0/RSP1/CPU0

show shmem summ loc 0/RSP1/CPU0

show shmem summ loc 0/RSP0/CPU0

show redundancy

show tech-support mpls ldp

Note: Check the memory usage on another router with the same configuration.