Introduction
This document describes the routing behaviour in ACI leaf switch when it receives same route via EIGRP and eBGP.
Prerequisites
The reader must have good understanding of the ACI components, terminologies & operations along with the routing protocols (EIGRP & BGP).
Setup and Topology
- This set up has been done using 2 different ACI fabrics connected as:
- Direct link between both DC Border Leaf switches (BGP).
- Extended via WAN network (EIGRP). SW1 & SW2 are WAN switches.
2. 192.168.10.0/24 is internal ACI subnet connected at Fabric-1 & advertised to Fabric-2 via eBGP as well as EIGRP.
Problem Statement
Fabric-2 Border Leaf switch is receiving the same route via EIGRP and eBGP where eBGP route gets installed in routing table of the switch as expected. When the eBGP session goes down, EIGRP routes gets installed in routing table of the switch. Switch retains EIGRP route even when eBGP comes up. Expectations here is that the eBGP route must get installed in routing table as soon as the eBGP session comes up since eBGP has lesser AD value [ 20 ] than EIGRP [ 90 ].
Issue Summary
- Fabric-1 & Fabric-2 data centres are connected via WAN network (EIGRP) and direct link between both sites BL switches running eBGP.
- Fabric-1 Border Leaf switch is advertising subnet 192.168.10.0/24 to Fabric-2 via eBGP and EIGRP.
- Both the L3Out are in same VRF.
- BGP route gets installed in the routing table of Fabric-2 Border Leaf switch on the basis of AD value.
- When eBGP session between both Fabric-1 & Fabric-2 goes down, EIGRP route gets installed in the routing table of Fabric-2_BL switch which is expected.
- When eBGP comes up, the expectation is that eBGP route must get re-installed & EIGRP route is to be removed from routing table which is not happening.
- Fabric-2 Border Leaf switch is retaining the EIGRP route in its routing table instead.
Troubleshooting and Verification
- Verify the eBGP neighborship between Fabric-1 & Fabric-2 Border Leaf switches.
Fabric-2_BL# show bgp sessions vrf snTn:snTn_VRF
Total peers 3, established peers 3
ASN 100
VRF snTn:snTn_VRF, local ASN 100
peers 1, established peers 1, local router-id 172.16.2.100
State: I-Idle, A-Active, O-Open, E-Established, C-Closing, S-Shutdown
Neighbor ASN Flaps LastUpDn|LastRead|LastWrit St Port(L/R) Notif(S/R)
10.10.10.3 65001 2 1d23h |never |never E 179/26051 45/6
- Verify EIGRP neighborship at Fabric-2.
Fabric-2_BL# show ip eigrp neighbors vrf snTn:snTn_VRF
EIGRP neighbors for process 500 VRF snTn:snTn_VRF
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 10.10.20.3 vlan7 13 2d00h 1 50 0 8
SW-2# show ip eigrp neighbors VRF default
IP-EIGRP neighbors for process 500 VRF default
H Address Interface Hold Uptime SRTT RTO Q Seq
(sec) (ms) Cnt Num
0 10.10.20.2 Vlan776 14 2d00h 6 50 0 9
- Initially, BGP route gets installed in the routing table and same route is present in EIGRP topology table of Fabric-2 Border Leaf switch.
Fabric-2_BL# show ip route 192.168.10.0/24 vrf snTn:snTn_VRF
IP Route Table for VRF "snTn:snTn_VRF"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.10.0/24, ubest/mbest: 1/0
*via 10.10.10.3%snTn:snTn_VRF, [20/0], 00:00:17, bgp-100, external, tag 65005
recursive next hop: 10.10.10.3/32%snTn:snTn_VRF
Fabric-2_BL# show ip eigrp topology 192.168.10.0/24 vrf snTn:snTn_VRF
EIGRP (AS 500): VRF: snTn:snTn_VRF , Topology entry for 192.168.10.0/24
State is Passive, Query origin: Local origin, 0 Successor(s), FD is Infinity
Routing Descriptor Blocks:
10.10.20.3(vlan7), from 10.10.20.3
Urib State: in-rib,up-to-date
Composite metric is (128576/128320), Route is Internal
Vector metric:
Minimum bandwidth is 8000000 Kbit
Total delay is 5010 microseconds
Reliability is 255/255
Load is 1/255
Minimum MTU is 1500
Hop count is 1
Internal tag is 0
- EIGRP route gets installed in routing table of Fabric-2 Border Leaf switch when eBGP session goes down between Fabric-1 & Fabric-2 Border Leaf switches and retains the EIGRP route even when the eBGP comes up.
Fabric-2_BL# show ip route 192.168.10.0/24 vrf snTn:snTn_VRF
IP Route Table for VRF "snTn:snTn_VRF
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.10.0/24, ubest/mbest: 1/0
*via 10.10.20.3, vlan7, [90/128576], 2d00h, eigrp-default, internal
- The expectation here is that, eBGP route must get re-installed in routing table as soon as eBGP session comes up. But Fabric-2_BL switch keeps EIGRP route only.
Why EIGRP route is preferred over eBGP route?
- When eBGP session goes down, Fabric-2_BL switch installs EIGRP route in routing table and same gets redistributed into MP-BGP to forward it to other service Leaf switches in the Fabric-2.
- Since Fabric-2_BL switch is redistributing it, becomes an origin for that route with default weight value 32768. Whereas, route coming from eBGP holds weight 0.
- Since higher weight is the preferred one, Fabric-2_BL switch considers redistributed route as best route and does not install eBGP route.
- The output shown beneath is when the eBGP session came back up.
Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF
BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast
BGP routing table entry for 192.168.10.0/24, version 28 dest ptr 0xa0fe0328
Paths: (2 available, best #1)
Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported
vpn: version 371, (0x100002) on xmit-list
Multipath: eBGP iBGP
Advertised path-id 1, VPN AF advertised path-id 1
Path type (0xa961d880): redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path
AS-Path: NONE, path locally originated
Tx Domain path attribute Flag 0xc0,Code 36, Length 8, segment length 1
domain path: { <1:5345:128>}
0.0.0.0 (metric 0) from 0.0.0.0 (172.16.0.10)
Origin incomplete, MED 128576, localpref 100, weight 32768 tag 0, propagate 0
Extcommunity:
RT:100:2129921
VNID:2129921
COST:pre-bestpath:128:128576
COST:pre-bestpath:162:90
0x8800:32768:0 (Flags = 32768, Tag = 0)
0x8801:500:128256 (ASN = 500, Delay = 128256)
0x8802:65281:320 (Reliability = 255, Hop = 1, Bandwidth = 320)
0x8803:1:1500 (Reserve = 0, Load = 1, MTU = 1500)
0x8804:0:0 (Remote ASN = 0, Remote ID = 0)
0x8805:0:0 (Remote Prot = 0, Remote Metric = 0)
VPN AF advertised path-id 2
Path type (0xa961e0bc): external 0x28 0x0 ref 0 adv path ref 1, path is valid, not best reason: Weight
AS-Path: 65001 , path sourced external to AS
Source Domain: <1:16:128>
Tx Domain path attribute Flag 0xc0,Code 36, Length 15, segment length 2
domain path: { <1:5345:128>,<1:16:128>}
10.10.10.3 (metric 0) from 10.10.10.3 (172.16.1.100)
Origin IGP, MED not set, localpref 100, weight 0 tag 0, propagate 0
Extcommunity:
RT:100:2129921
VNID:2129921
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 advertised to peers:
10.0.152.65 10.0.152.66
Path-id 2 not advertised to any peer
Solution
There are 2-ways to fix this issue:
- LPM is one of the solution:
- Advertise the same subnet with /23 mask under EIGRP & /24 mask via eBGP so that both routes are present in routing table of Fabric-2_BL switch.
SW-2# show run interface vlan 776
!Command: show running-config interface Vlan776
!Time: Sun Jun 23 06:30:43 2024
version 7.0(3)I7(5) Bios:version 07.66
interface Vlan776
no shutdown
ip address 10.10.20.3/24
ip router eigrp 500
ip summary-address eigrp 500 192.168.10.0/23 >>>>>> Advertised /23 via EIGRP
Fabric-2_BL# show ip route vrf snTn:snTn_VRF
IP Route Table for VRF "snTn:snTn_VRF"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.10.0/23, ubest/mbest: 1/0
*via 10.10.20.3, vlan20, [90/128576], 00:24:11, eigrp-default, internal >>>>>>>>> EIGRP Route
192.168.10.0/24, ubest/mbest: 1/0
*via 10.10.10.3%snTn:snTn_VRF, [20/0], 00:04:12, bgp-100, external, tag 65005 >>>>>>>> BGP Route
b. When, eBGP session goes down, EIGRP route is still present in routing table for redundancy.
c. As soon as BGP session comes up, BGP route gets re-installed in routing table and preferred for traffic forwarding.
- Applying Weight on eBGP route:
- If you need to advertise the subnet with same subnet mask via both EIGRP & BGP, higher weight (than 32768) can be applied on eBGP route to be the preferred route always.
- How to apply weight on ACI:
- Create Route-map policy.
Tenant ----> Policies ----> Route Maps for Route Control (Right click and create new policy, fill all the required details)----> Create "Set Rule" policy ---> Select "Weight" attribute policy and enter value
ii. Apply Route Map to L3Out:
Tenant ---> Networking ---> L3Out ----> Logical Node Profiles ---> Node Profile ----> Logical Interface Profile ---> Interface Profile ---> Peer Profile ---> Click on "+" under "Route Control Profile" and select created new Route Map
Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF
BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast
BGP routing table entry for 192.168.10.0/24, version 61 dest ptr 0xa0fa3f70
Paths: (1 available, best #1)
Flags: (0x80c001a 00000000) on xmit-list, is in urib, is best urib route, is in HW, exported
vpn: version 79, (0x100002) on xmit-list
Multipath: eBGP iBGP
Advertised path-id 1, VPN AF advertised path-id 1
Path type (0xa95a2d5c): external 0x28 0x0 ref 0 adv path ref 2, path is valid, is best path
AS-Path: 65005 65001 , path sourced external to AS
Source Domain: <1:16:128>
Tx Domain path attribute Flag 0xc0,Code 36, Length 15, segment length 2
domain path: { <1:5345:128>,<1:16:128>}
10.10.10.3 (metric 0) from 10.10.10.3 (172.16.0.10)
Origin IGP, MED not set, localpref 100, weight 32769 tag 0, propagate 0
Extcommunity:
RT:100:2129921
VNID:2129921
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 advertised to peers:
10.0.152.65 10.0.152.66
c. The catch here is, you do not see redistributed EIGRP route in BGP table when BGP session is up. Reason is FD is set to Infinity for EIGRP external route.
Fabric-2_BL# show ip eigrp topology vrf snTn:snTn_VRF
EIGRP Topology Table for AS(500)/ID(172.16.2.100) VRF snTn:snTn_VRF
Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status
P 192.168.10.0/24, 0 Successors, FD is Infinity
via 10.10.20.3(128576/128320), vlan20
d. The "FD is Infinity" message is actually an indicator within EIGRP that the RIB rejected the route due to a lower admin distance route already being present.
e. EIGRP route only gets redistributed into MP-BGP and installed in routing table of fabric-2_BL switch when BGP session goes down.
Fabric-2_BL# show ip bgp summary vrf snTn:snTn_VRF
BGP summary information for VRF snTn:snTn_VRF, address family IPv4 Unicast
BGP router identifier 172.16.2.100, local AS number 100
BGP table version is 65, IPv4 Unicast config peers 1, capable peers 0
6 network entries and 6 paths using 1248 bytes of memory
BGP attribute entries [4/704], BGP AS path entries [0/0]
BGP community entries [0/0], BGP clusterlist entries [2/8]
Neighbor V AS MsgRcvd MsgSent TblVer InQ OutQ Up/Down State/PfxRcd
10.10.10.3 4 65001 18530 18554 0 0 0 00:04:25 Idle
Fabric-2_BL# show ip eigrp topology vrf snTn:snTn_VRF
IP-EIGRP Topology Table for AS(500)/ID(172.16.2.100) VRF snTn:snTn_VRF
Codes: P - Passive, A - Active, U - Update, Q - Query, R - Reply,
r - reply Status, s - sia Status
P 192.168.10.0/24, 1 successors, FD is 128576
via 10.10.20.3 (128576/128320), Vlan20
Fabric-2_BL# show ip route vrf snTn:snTn_VRF
IP Route Table for VRF "snTn:snTn_VRF"
'*' denotes best ucast next-hop
'**' denotes best mcast next-hop
'[x/y]' denotes [preference/metric]
'%<string>' in via output denotes VRF <string>
192.168.10.0/24, ubest/mbest: 1/0
*via 10.10.20.3, Vlan20, [90/128576], 02:31:52, eigrp-default, internal >>>>>>> EIGRP Route
Fabric-2_BL# show ip bgp 192.168.10.0/24 vrf snTn:snTn_VRF
BGP routing table information for VRF snTn:snTn_VRF, address family IPv4 Unicast
BGP routing table entry for 192.168.10.0/24, version 65 dest ptr 0xa0fa3f70
Paths: (1 available, best #1)
Flags: (0x80c0002 00000000) on xmit-list, is not in urib, exported
vpn: version 83, (0x100002) on xmit-list
Multipath: eBGP iBGP
Advertised path-id 1, VPN AF advertised path-id 1
Path type (0xa95a2c64): redist 0x408 0x1 ref 0 adv path ref 2, path is valid, is best path
AS-Path: NONE, path locally originated
Tx Domain path attribute Flag 0xc0,Code 36, Length 8, segment length 1
domain path: { <1:5345:128>}
0.0.0.0 (metric 0) from 0.0.0.0 (172.16.0.10)
Origin incomplete, MED 128576, localpref 100, weight 32768 tag 0, propagate 0
Extcommunity:
RT:100:2129921
VNID:2129921
COST:pre-bestpath:128:128576
COST:pre-bestpath:162:90
0x8800:32768:0 (Flags = 32768, Tag = 0)
0x8801:500:128256 (ASN = 500, Delay = 128256)
0x8802:65281:320 (Reliability = 255, Hop = 1, Bandwidth = 320)
0x8803:1:1500 (Reserve = 0, Load = 1, MTU = 1500)
0x8804:0:0 (Remote ASN = 0, Remote ID = 0)
0x8805:0:0 (Remote Prot = 0, Remote Metric = 0)
VRF advertise information:
Path-id 1 not advertised to any peer
VPN AF advertise information:
Path-id 1 advertised to peers:
10.0.152.65 10.0.152.66