Introduction
This document describes how the hardware ip glean throttle feature works with examples and the intention of this feature.
Prerequisites
Requirements
Cisco recommends that you have basic knowledge of Nexus 7000 Series Switches configuration.
Components Used
The information in this document is based on these software and hardware versions:
- Nexus 7000 with Release 6.2.x and later
- F2e series line card
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Background Information
When you forward an incoming IP packet in a line card, if the Address Resolution Protocol (ARP) request for the next hop is not resolved, the line card forwards the packets to the supervisor in order to generate an ARP request. Once the ARP request responds to the supervisor, it resolves the MAC address for the next hop and programs the hardware.
If the supervisor cannot resolve the ARP entry, then the linecard sends all packets destined to that address to the supervisor. The supervisor generates ARP requests indefinitely until the ARP entry is resolved. There is a hardware rate limiter called glean placed in order to protect the supervisor's processor (CPU) from excessive traffic.
An issue that can arise is a single destination IP drops off the network due to maintenance or a hardware problem, and suddenly all traffic destined to it is being sent to the CPU. Since the rate limiter is in place, the CPU does not go high but this single destination IP can consume the entire rate limiter and not give other legitimate IP's access to the CPU. It is for this scenario that hardware ip glean throttle was created.
With the hardware ip glean throttle configuration, routed traffic for each unknown destination IP reaches the CPU post Hardware Rate Limiter (HWRL) action for ARP resolution. Unreachable destination will result in a /32 drop adjacency to be created in hardware. This prevents additional packets to the same next-hop IP address to be forwarded to the supervisor. While this drop adjacency is added, subsequent packets are dropped yet the supervisor continues to generate ARP requests until the next-hop is resolved. The drop adjacency is installed for a short period of time, which is configurable. Once the timer expires, one packet is again sent to the CPU and the process repeats. The number of entries that is installed in this fashion is limited to 1000 by default, but is configurable to a larger number of desired. This is to limit the impact on the Routing Information Base (RIB) table size.
Lab Test
In this case, you have a server, 172.28.191.200, which is down due to a hardware failure, and is currently unavailable to service traffic.
Note: There is no ARP entry for the host and no adjacency is created.
N7K# show ip route vrf VRF_ABC 172.28.191.200
IP Route Table for VRF "VRF_ABC"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
172.28.191.192/28, ubest/mbest: 1/0, attached >>> There is no /32 entry
*via 172.28.191.195, Vlan1601, [0/0], 02:01:17, direct
Traffic is sent to the supervisor in order to generate an ARP request:
N7K# show system internal forwarding vrf VRF_ABC ipv4 route 172.28.191.200 detail
slot 1
=======
RPF Flags legend:
S - Directly attached route (S_Star)
V - RPF valid
M - SMAC IP check enabled
G - SGT valid
E - RPF External table valid
172.28.191.192/28 , sup-eth2
Dev: 0 , Idx: 0x65fb , Prio: 0x8487 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x5a , LIFB: 0 , LIF: sup-eth2 (0x1fe1 ), DI: 0xc01
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.192/28 , sup-eth2
Dev: 1 , Idx: 0x65fb , Prio: 0x8487 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x5a , LIFB: 0 , LIF: sup-eth2 (0x1fe1 ), DI: 0xc01
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.192/28 , sup-eth2
Dev: 2 , Idx: 0x65fb , Prio: 0x8487 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x5a , LIFB: 0 , LIF: sup-eth2 (0x1fe1 ), DI: 0xc01
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.192/28 , sup-eth2
Dev: 5 , Idx: 0x65f1 , Prio: 0x84f2 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x5a , LIFB: 0 , LIF: sup-eth2 (0x1fe1 ), DI: 0xc01
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
The glean rate limiter for the specific module throttles the traffic to 100 packets per second, per module. You can see that some of the packets gets dropped.
N7K# show hardware rate-limiter
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
rl-1: STP and Fabricpath-ISIS
rl-2: L3-ISIS and OTV-ISIS
rl-3: UDLD, LACP, CDP and LLDP
rl-4: Q-in-Q and ARP request
rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic
Module: 1
R-L Class Config Allowed Dropped Total
+------------------+--------+---------------+---------------+-----------------+
L3 mtu 500 0 0 0
L3 ttl 500 0 0 0
L3 control 10000 0 0 0
L3 glean 100 3326 3190 6516
L3 mcast dirconn 3000 0 0 0
L3 mcast loc-grp 3000 0 0 0
L3 mcast rpf-leak 500 0 0 0
L2 storm-ctrl Disable
access-list-log 100 0 0 0
copy 30000 1877 0 1877
receive 30000 318 0 318
When the hardware ip glean throttle command is configured:
N7K(config)#hardware ip glean throttle
An adjacency is installed in the RIB:
N7K# show ip route 172.28.191.200 vrf VRF-ABC
IP Route Table for VRF "VRF-ABC"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
172.28.191.200/32, ubest/mbest: 1/0, attached
*via 172.28.191.200, Vlan1601, [250/0], 00:01:37, am
When you look at the hardware programming, a drop index is installed:
N7K# show system internal forwarding vrf VRF_ABC ipv4 route 172.28.191.200 detail
slot 1
=======
RPF Flags legend:
S - Directly attached route (S_Star)
V - RPF valid
M - SMAC IP check enabled
G - SGT valid
E - RPF External table valid
172.28.191.200/32 , Drop
Dev: 0 , Idx: 0x1a5 , Prio: 0x8b61 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x8913 , LIFB: 0 , LIF: Drop (0x0 ), DI: 0x0
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.200/32 , Drop
Dev: 1 , Idx: 0x1a5 , Prio: 0x8b61 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x8913 , LIFB: 0 , LIF: Drop (0x0 ), DI: 0x0
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.200/32 , Drop
Dev: 2 , Idx: 0x1a5 , Prio: 0x8b61 , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x8913 , LIFB: 0 , LIF: Drop (0x0 ), DI: 0x0
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
172.28.191.200/32 , Drop
Dev: 5 , Idx: 0x1e1 , Prio: 0x88ee , RPF Flags: VS , DGT: 0 , VPN: 9
RPF_Intf_5: Vlan1601 (0x19 )
AdjIdx: 0x8914 , LIFB: 0 , LIF: Drop (0x0 ), DI: 0x0
DMAC: 0000.0000.0000 SMAC: 0000.0000.0000
You can now see that the hardware rate-limiter does not see any drops.
N7K# show hardware rate-limiter
Units for Config: packets per second
Allowed, Dropped & Total: aggregated since last clear counters
rl-1: STP and Fabricpath-ISIS
rl-2: L3-ISIS and OTV-ISIS
rl-3: UDLD, LACP, CDP and LLDP
rl-4: Q-in-Q and ARP request
rl-5: IGMP, NTP, DHCP-Snoop, Port-Security, Mgmt and Copy traffic
Module: 1
R-L Class Config Allowed Dropped Total
+------------------+--------+---------------+---------------+-----------------+
L3 mtu 500 0 0 0
L3 ttl 500 0 0 0
L3 control 10000 0 0 0
L3 glean 100 0 0 0
L3 mcast dirconn 3000 0 0 0
L3 mcast loc-grp 3000 0 0 0
L3 mcast rpf-leak 500 0 0 0
L2 storm-ctrl Disable
access-list-log 100 0 0 0
copy 30000 1877 0 1877
receive 30000 318 0 318
Related Information
Feedback