The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes how to troubleshoot Overlay Management Protocol (OMP) routes and explains vSmart route selection order of operations.
Cisco recommends that you have knowledge of Cisco Software Defined Wide Area Network (SDWAN) solution.
This document is not restricted to hardware platforms. This article depects a problem seen in a lab with vSmart on 20.6.3 and cEdge routers on 17.6.3, but can also been seen on other software versions.
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
The scenario described in this document shows that two sites advertise a default route to vSmart. The vSmart chooses the best path and advertises that out to Edge devices. In this case, the AMER_DC is chosen due to a control policy which blocks the EMEAR_DC default route for the remote Edge, the Cloud Edge. The vSmart only has control connections on biz-internet with AMER_DC Edges. When there is an outage on the AMER_DC biz-internet, all control connections are lost and vSmart places all the routes learned from AMER_DC into a "stale" state. This causes vSmart not to consider these a best path.
At this point, vSmart would normally choose the EMEA_DC as the best-path and advertise that. However, the control policy blocks the default route from EMEA_DC, and this is applied to the Cloud Edge. Without the OMP configuration of send-backup-paths, the vSmart does not send the stale routes which are the only default routes that are not rejected by the control policy. It is also necessary to increase the send-path-limit in vSmart OMP configuration in order to send more than the number of non-stale routes.
In order to understand the problem better, here is a simple topology diagram that depicts the setup:
A brief summary of the configuration:
hostname | site-id | system-ip |
EMEA_DC1 | 2016 | 10.4.4.1 |
EMEA_DC2 | 2016 | 10.4.4.6 |
AMER_DC1 | 2034 | 10.5.5.3 |
AMER_DC2 | 2034 | 10.5.5.4 |
Cloud | 202 | 10.10.20.2 |
vSmart | 10 | 10.3.3.1 |
The vSmart is configured with the default OMP configuration.
vsmart1# show running-config omp omp no shutdown graceful-restart ! vsmart1# show running-config omp | details omp no shutdown send-path-limit 4 no send-backup-paths no discard-rejected graceful-restart timers holdtime 60 advertisement-interval 1 graceful-restart-timer 43200 eor-timer 300 exit
!
Tip: In order to see the full configuration, default and non-default, include | details at the end of the show running-config command.
The vSmart has a centralized control policy configured. The control policy is applied to the Cloud site, and has action reject configured for the default route from the EMEA_DC site.
Here is the policy configuration:
policy control-policy Rej_Remote_Default sequence 1 match route site-list EMEA_DC prefix-list default_route ! action reject ! ! default-action accept ! lists prefix-list default_route ip-prefix 0.0.0.0/0 ! site-list Cloud site-id 202 ! site-list EMEA_DC site-id 2016 ! ! ! apply-policy site-list Cloud control-policy Rej_Remote_Default out ! !
Under normal operating conditions, the default route from the AMER_DC site is received by the Cloud site. This can be verified with the command show sdwan omp routes vpn 1 0.0.0.0/0. If your service vpn is not vpn 1, replace the number 1 with your service vpn number.
Cloud#show sdwan omp routes vpn 1 0.0.0.0/0
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH PSEUDO
FROM PEER ID LABEL STATUS KEY TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------
10.3.3.1 81 1003 C,I,R 1 10.5.5.3 biz-internet ipsec -
10.3.3.1 97 1003 C,I,R 1 10.5.5.3 private1 ipsec -
10.3.3.1 98 1003 C,I,R 1 10.5.5.3 private2 ipsec -
10.3.3.1 99 1003 C,I,R 1 10.5.5.4 biz-internet ipsec -
The vSmart receives the default route on 3 TLOCs from all 4 DC routers. The vSmart has a total of 12 routes.
vsmart1# show omp routes vpn 1 0.0.0.0/0 received | tab
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
10.4.4.1 68 1002 C,R installed 10.4.4.1 biz-internet ipsec -
10.4.4.1 81 1002 C,R installed 10.4.4.1 private1 ipsec -
10.4.4.1 82 1002 C,R installed 10.4.4.1 private2 ipsec -
10.4.4.6 68 1003 C,R installed 10.4.4.6 biz-internet ipsec -
10.4.4.6 81 1003 C,R installed 10.4.4.6 private1 ipsec -
10.4.4.6 82 1003 C,R installed 10.4.4.6 private2 ipsec -
10.5.5.3 68 1003 C,R installed 10.5.5.3 biz-internet ipsec -
10.5.5.3 81 1003 C,R installed 10.5.5.3 private1 ipsec -
10.5.5.3 82 1003 C,R installed 10.5.5.3 private2 ipsec -
10.5.5.4 68 1003 C,R installed 10.5.5.4 biz-internet ipsec -
10.5.5.4 81 1003 C,R installed 10.5.5.4 private1 ipsec -
10.5.5.4 82 1003 C,R installed 10.5.5.4 private2 ipsec -
In the event that there is an outage on the biz-internet circuit at the AMER_DC site, the Cloud Edge device no longer receives a default route. You would expect it to lose the biz-internet route but keep the private1 and private2 routes. You can verify this with the commands show sdwan omp routes vpn 1 0.0.0.0/0 and show sdwan omp routes vpn 1.
Note: If the prefix is not received by the device, the show sdwan omp routes command shows the CLI command error as shown here.
Cloud#show sdwan omp routes vpn 1 0.0.0.0/0
Generating output, this might take time, please wait ...
show omp best-match-route family ipv4 entries vpn 1 0.0.0.0 | tab
show omp best-match-route family ipv4 entries vpn
syntax error: unknown argument
Error executing command: CLI command error -
Cloud#show sdwan omp routes
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
VPN PREFIX FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------------------------------------
1 10.5.100.0/24 10.3.3.1 72 1003 Inv,U installed 10.5.5.3 biz-internet ipsec -
10.3.3.1 73 1003 C,I,R installed 10.5.5.3 private1 ipsec -
10.3.3.1 74 1003 C,I,R installed 10.5.5.3 private2 ipsec -
The vSmart goes into a graceful restart state with the AMER_DC Edge devices, which you can see from show omp peers command.
vsmart1# show omp peers
R -> routes received
I -> routes installed
S -> routes sent
DOMAIN OVERLAY SITE
PEER TYPE ID ID ID STATE UPTIME R/I/S
------------------------------------------------------------------------------------------
172.16.4.1 vedge 1 1 101 up 0:13:13:02 9/0/22
172.16.5.1 vedge 1 1 104 up 0:13:13:03 3/0/28
10.4.4.1 vedge 1 1 2016 up 0:01:45:10 6/0/27
10.4.4.6 vedge 1 1 2016 up 0:02:13:27 6/0/27
10.5.5.3 vedge 1 1 2034 down-in-gr 6/0/0
10.5.5.4 vedge 1 1 2034 down-in-gr 6/0/0
10.10.20.2 vedge 1 1 202 up 0:12:40:09 3/0/24
The vSmart continues to receive all 12 routes, 3 for each DC device. This can be verified with the show omp routes vpn 1 0.0.0.0/0 received | tab command. The routes from the AMER_DC sites show in stale state.
Tip: In order to see the output in a user friendly tabular format, include | tab at the end. Without it, the output of the command is in a different format.
vsmart1# show omp routes vpn 1 0.0.0.0/0 received | tab
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH ATTRIBUTE
FROM PEER ID LABEL STATUS TYPE TLOC IP COLOR ENCAP PREFERENCE
-----------------------------------------------------------------------------------------------------------
10.4.4.1 68 1002 C,R installed 10.4.4.1 biz-internet ipsec -
10.4.4.1 81 1002 C,R installed 10.4.4.1 private1 ipsec -
10.4.4.1 82 1002 C,R installed 10.4.4.1 private2 ipsec -
10.4.4.6 68 1003 C,R installed 10.4.4.6 biz-internet ipsec -
10.4.4.6 81 1003 C,R installed 10.4.4.6 private1 ipsec -
10.4.4.6 82 1003 C,R installed 10.4.4.6 private2 ipsec -
10.5.5.3 68 1003 R,S installed 10.5.5.3 biz-internet ipsec -
10.5.5.3 81 1003 R,S installed 10.5.5.3 private1 ipsec -
10.5.5.3 82 1003 R,S installed 10.5.5.3 private2 ipsec -
10.5.5.4 68 1003 R,S installed 10.5.5.4 biz-internet ipsec -
10.5.5.4 81 1003 R,S installed 10.5.5.4 private1 ipsec -
10.5.5.4 82 1003 R,S installed 10.5.5.4 private2 ipsec -
In order to verify what routes the vSmart sends to the Edge device, you can run the command show omp routes vpn 1 0.0.0.0/0 advertised detail | tab.
A few things to make note of from the output:
vsmart1# show omp routes vpn 1 0.0.0.0/0 advertised detail | tab
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
ULTIMATE
PATH UNKNOWN TLOC
TO PEER ADVERTISE ID ID LABEL TLOC IP COLOR ENCAP PROTOCOL METRIC DOMAIN ID SITE ID PREFERENCE TAG ATTRIBUTE LEN ORIGINATOR TLOC IP COLOR ENCAP ACTION OVERLAY ID AS PATH COMMUNITY
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
172.16.4.1 67 50 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
78 56 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 57 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 61 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
172.16.5.1 67 56 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
78 62 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 63 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 67 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
10.4.4.1 67 53 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
68 54 1003 10.4.4.6 private1 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
69 55 1003 10.4.4.6 private2 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
10.4.4.6 78 97 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 98 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 102 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
This is the expected behavior given the current configuration in this environment. It happens because of the order of operations for OMP route selection on the vSmart.
To correct the issue, the vSmart OMP configuration needs to be modified to send more than 4 routes and more than the best routes.
vsmart1# show running-config omp
omp
no shutdown
send-path-limit 16
send-backup-paths
graceful-restart
!
In order to verify that the vSmart OMP configuration change had the intended effect, you can run the command show omp routes vpn 1 0.0.0.0/0 advertised detail | tab.
A few things to make note of from the output:
vsmart1# show omp routes vpn 1 0.0.0.0/0 advertised detail | tab
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
ULTIMATE
PATH UNKNOWN TLOC
TO PEER ADVERTISE ID ID LABEL TLOC IP COLOR ENCAP PROTOCOL METRIC DOMAIN ID SITE ID PREFERENCE TAG ATTRIBUTE LEN ORIGINATOR TLOC IP COLOR ENCAP ACTION OVERLAY ID AS PATH COMMUNITY
----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
172.16.4.1 22 64 1003 10.5.5.3 biz-internet ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
23 65 1003 10.5.5.3 private1 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
24 66 1003 10.5.5.3 private2 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
28 67 1003 10.5.5.4 biz-internet ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
29 68 1003 10.5.5.4 private1 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
30 69 1003 10.5.5.4 private2 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
67 50 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
68 62 1003 10.4.4.6 private1 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
69 63 1003 10.4.4.6 private2 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
78 56 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 57 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 61 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
172.16.5.1 22 70 1003 10.5.5.3 biz-internet ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
23 71 1003 10.5.5.3 private1 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
24 72 1003 10.5.5.3 private2 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
28 73 1003 10.5.5.4 biz-internet ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
29 74 1003 10.5.5.4 private1 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
30 75 1003 10.5.5.4 private2 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
67 56 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
68 68 1003 10.4.4.6 private1 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
69 69 1003 10.4.4.6 private2 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
78 62 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 63 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 67 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
10.4.4.1 22 57 1003 10.5.5.3 biz-internet ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
23 58 1003 10.5.5.3 private1 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
24 59 1003 10.5.5.3 private2 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
28 60 1003 10.5.5.4 biz-internet ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
29 61 1003 10.5.5.4 private1 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
30 62 1003 10.5.5.4 private2 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
67 53 1003 10.4.4.6 biz-internet ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
68 54 1003 10.4.4.6 private1 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
69 55 1003 10.4.4.6 private2 ipsec static 0 - 2016 - - - 10.4.4.6 - - - - 1 - -
10.4.4.6 22 103 1003 10.5.5.3 biz-internet ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
23 104 1003 10.5.5.3 private1 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
24 105 1003 10.5.5.3 private2 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
28 106 1003 10.5.5.4 biz-internet ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
29 107 1003 10.5.5.4 private1 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
30 108 1003 10.5.5.4 private2 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
78 97 1002 10.4.4.1 biz-internet ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
79 98 1002 10.4.4.1 private2 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
81 102 1002 10.4.4.1 private1 ipsec static 0 - 2016 - - - 10.4.4.1 - - - - 1 - -
10.10.20.2 22 112 1003 10.5.5.3 biz-internet ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
23 113 1003 10.5.5.3 private1 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
24 114 1003 10.5.5.3 private2 ipsec static 0 - 2034 - - - 10.5.5.3 - - - - 1 - -
28 115 1003 10.5.5.4 biz-internet ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
29 116 1003 10.5.5.4 private1 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
30 117 1003 10.5.5.4 private2 ipsec static 0 - 2034 - - - 10.5.5.4 - - - - 1 - -
The Cloud Edge router receives the default route from the AMER_DC site. This can be verified with the show sdwan omp routes vpn 1 0.0.0.0/0 command. The biz-internet routes are in a Inv, U state as that circuit experienced an outage at the AMER_DC site.
Cloud#show sdwan omp routes vpn 1 0.0.0.0/0
Generating output, this might take time, please wait ...
Code:
C -> chosen
I -> installed
Red -> redistributed
Rej -> rejected
L -> looped
R -> resolved
S -> stale
Ext -> extranet
Inv -> invalid
Stg -> staged
IA -> On-demand inactive
U -> TLOC unresolved
PATH PSEUDO
FROM PEER ID LABEL STATUS KEY TLOC IP COLOR ENCAP PREFERENCE
--------------------------------------------------------------------------------------------------------
10.3.3.1 112 1003 Inv,U 1 10.5.5.3 biz-internet ipsec -
10.3.3.1 113 1003 C,I,R 1 10.5.5.3 private1 ipsec -
10.3.3.1 114 1003 C,I,R 1 10.5.5.3 private2 ipsec -
10.3.3.1 115 1003 Inv,U 1 10.5.5.4 biz-internet ipsec -
10.3.3.1 116 1003 C,I,R 1 10.5.5.4 private1 ipsec -
10.3.3.1 117 1003 C,I,R 1 10.5.5.4 private2 ipsec -
Only the private1 and private2 are installed into the routing table since they are in C,I,R state. The routes are installed into the table based on the output of the show ip route vrf 1 0.0.0.0 command.
Note: In the show sdwan omp commands, the vpn keyword is used to see the service side routers. In the show ip route commands, the vrf keyword is used to see the service side routers.
Cloud# show ip route vrf 1 0.0.0.0
Routing Table: 1
Routing entry for 0.0.0.0/0, supernet
Known via "omp", distance 251, metric 0, candidate default path, type omp
Last update from 10.5.5.4 on Sdwan-system-intf, 00:17:07 ago
Routing Descriptor Blocks:
10.5.5.4 (default), from 10.5.5.4, 00:17:07 ago, via Sdwan-system-intf
Route metric is 0, traffic share count is 1
* 10.5.5.3 (default), from 10.5.5.3, 00:17:07 ago, via Sdwan-system-intf
Route metric is 0, traffic share count is 1
Revision | Publish Date | Comments |
---|---|---|
2.0 |
30-Aug-2022 |
Initial Release |
1.0 |
24-Aug-2022 |
Initial Release |