The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes steps to understand and troubleshoot ACI Security Policies, known as Contracts.
The material from this document was extracted from the Troubleshooting Cisco Application Centric Infrastructure, Second Edition book, specifically the Security Policies - Overview, Security Policies - Tools, Security Policies - EPG to EPG, Security Policies - Preferred group and Security Policies - vzAny to EPG chapters.
The fundamental security architecture of the ACI solution follows a permitlist model. Unless a VRF is configured in unenforced mode, all EPG to EPG traffic flows are implicitly dropped. As implied by the out-of-the-box permitlist model, the default VRF setting is in enforced mode. Traffic flows can be allowed or explicitly denied by implementing zoning-rules on the switch nodes. These zoning-rules can be programmed in a variety of different configurations depending on the desired communication flow between endpoint groups (EPG) and the method used to define them. Note that zoning-rule entries are not stateful and will typically allow/deny based on port/socket given two EPGs once the rule has been programmed.
The main methods to program zoning-rules within ACI are as follows:
The following diagram can be used to reference the granularity of zoning-rule that each of the above methods allows for control:
While utilizing the contract method of programming zoning-rules, there is an option for defining the contract scope. This option must be given careful consideration if any route leaking/shared service design is required. If the wish is to get from one VRF to another within the ACI fabric, contracts are the method to do so.
The scope values can be the following:
Once the zoning-rule is programmed, it will appear as the following on a leaf:
+---------+--------+--------+----------+----------------+---------+---------+-----------------+----------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+---------+-----------------+----------+----------------------+
As each zoning rule gets programmed, a matrix of the zoning-rule entry mapped against filter entries will begin to consume Policy CAM on the switches. While designing allowed flows through an ACI fabric, special care should be taken when re-using contracts, as opposed to creating new ones, depending on the end design. Haphazardly re-using the same contract across multiple EPGs without understanding the resulting zoning-rules can quickly cascade into multiple flows being allowed unexpectedly. At the same time, these unintentional flows will continue to consume Policy CAM. When Policy CAM becomes full, the zoning-rule programming will begin to fail which can result in unexpected and intermittent loss depending on configuration and endpoint behaviors.
This is a special callout for the shared services use case which requires contracts to be configured. Shared services typically imply inter-VRF traffic within an ACI fabric which relies on the usage of either a 'tenant' or 'global' scoped contract. To fully understand this, one must first reinforce the idea that the typical pcTag value assigned to EPGs are not globally unique. pcTags are scoped to a VRF and the same pcTag could potentially be re-used within another VRF. When the discussion of route leaking comes up, start to enforce requirements on the ACI fabric including the need for globally unique values including subnets and pcTags.
What makes this a special consideration is the directionality aspect tied to an EPG being a consumer vs a provider. In a shared services scenario, the provider is typically expected to drive a global pcTag to get a fabric unique value. At the same time, the consumer will retain its VRF-scoped pcTag which puts it in a special position to be able to now program and understand the usage of the global pcTag value to enforce policy.
For reference, the pcTag allocation range is as follows:
In each VRF it is possible to define the enforcement direction setting.
Understanding where the policy is enforced depends on several different variables.
The table below helps to understand where the security policy is enforced at leaf level.
Scenario |
VRF enforcement mode |
Consumer |
Provider |
Policy enforced on |
Intra-VRF |
Ingress/egress |
EPG |
EPG |
· If destination endpoint is learned: ingress leaf* · If destination endpoint is not learned: egress leaf |
Ingress |
EPG |
L3Out EPG |
Consumer leaf (non-border leaf) |
|
Ingress |
L3Out EPG |
EPG |
Provider leaf (non-border leaf) |
|
Egress |
EPG |
L3Out EPG |
Border leaf -> non-border leaf traffic · If destination endpoint is learned: border leaf · If destination endpoint is not learned: non-border leaf Non-border leaf-> border leaf traffic · Border leaf |
|
Egress |
L3Out EPG |
EPG |
||
Ingress/egress |
L3Out EPG |
L3Out EPG |
Ingress leaf* |
|
Inter-VRF |
Ingress/egress |
EPG |
EPG |
Consumer leaf |
Ingress/egress |
EPG |
L3Out EPG |
Consumer leaf (Non-border leaf) |
|
Ingress/egress |
L3Out EPG |
EPG |
Ingress leaf* |
|
Ingress/egress |
L3Out EPG |
L3Out EPG |
Ingress leaf* |
*Policy enforcement is applied on the first leaf hit by the packet.
The figure below illustrates an example of contract enforcement where EPG-Web as consumer and L3Out EPG as provider have an intra-VRF contract. If VRF is set to Ingress enforcement mode, policy is enforced by the leaf nodes where EPG-Web resides. If VRF is set to Egress enforcement mode, policy is enforced by the border leaf nodes where L3Out resides if VM-Web endpoint is learned on the border leaf.
There are a variety of tools and commands that can be used to help in the identification of a policy drop. A policy drop can be defined as a packet drop due to a contract configuration or lack thereof.
The following tools and commands can be used to explicitly validate the zoning-rules that are programmed on leaf switches as a result of completed contract consumer/provider relationships.
A switch level command showing all zoning rules in place.
leaf# show zoning-rule
+---------+--------+--------+----------+----------------+---------+----------+-----------------+----------+----------------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+----------+-----------------+----------+----------------------------+
| 4156 | 25 | 16410 | 425 | uni-dir-ignore | enabled | 2818048 | external_to_ntp | permit | fully_qual(7) |
| 4131 | 16410 | 25 | 424 | bi-dir | enabled | 2818048 | external_to_ntp | permit | fully_qual(7) |
+---------+--------+--------+----------+----------------+---------+----------+-----------------+----------+----------------------------+
A filter that contains the sport/dport information that the zoning rule is acting on. The filter programming can be verified with this command.
leaf# show zoning-filter
+----------+----------+-------------+-------------+-------------+----------+-------------+-------------+-------------+-------------+----------+
| FilterId | Name | EtherT | Prot | ApplyToFrag | Stateful | SFromPort | SToPort | DFromPort | DToPort | Prio |
+----------+----------+-------------+-------------+-------------+----------+-------------+-------------+-------------+-------------+----------+
| implarp | implarp | arp | unspecified | no | no | unspecified | unspecified | unspecified | unspecified | dport |
| implicit | implicit | unspecified | unspecified | no | no | unspecified | unspecified | unspecified | unspecified | implicit |
| 425 | 425_0 | ip | tcp | no | no | 123 | 123 | unspecified | unspecified | sport |
| 424 | 424_0 | ip | tcp | no | no | unspecified | unspecified | 123 | 123 | dport |
+----------+----------+-------------+-------------+-------------+----------+-------------+-------------+-------------+-------------+----------+
This command can be run to verify the number of hits per zoning-rule. This is useful to determine whether an expected rule is being hit as opposed to another, such as an implicit drop rule that may have a higher priority.
leaf# show system internal policy-mgr stats
Requested Rule Statistics
Rule (4131) DN (sys/actrl/scope-2818048/rule-2818048-s-16410-d-25-f-424) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0
Rule (4156) DN (sys/actrl/scope-2818048/rule-2818048-s-25-d-16410-f-425) Ingress: 0, Egress: 0, Pkts: 0 RevPkts: 0
A switch level command that can be run at iBash level which reports ACL (contract) related drops and flow-related information including:
leaf# show logging ip access-list internal packet-log deny
[ Tue Oct 1 10:34:37 2019 377572 usecs]: CName: Prod1:VRF1(VXLAN: 2654209), VlanType: Unknown, Vlan-Id: 0, SMac: 0x000c0c0c0c0c, DMac:0x000c0c0c0c0c, SIP: 192.168.21.11, DIP: 192.168.22.11, SPort: 0, DPort: 0, Src Intf: Tunnel7, Proto: 1, PktLen: 98
[ Tue Oct 1 10:34:36 2019 377731 usecs]: CName: Prod1:VRF1(VXLAN: 2654209), VlanType: Unknown, Vlan-Id: 0, SMac: 0x000c0c0c0c0c, DMac:0x000c0c0c0c0c, SIP: 192.168.21.11, DIP: 192.168.22.11, SPort: 0, DPort: 0, Src Intf: Tunnel7, Proto: 1, PktLen: 98
An on-device Python script which produces an output that correlates the zoning-rules, filters and hit statistics while performing name lookups from IDs. This script is extremely useful in that it takes a multi-step process and turns it into a single command which can be filtered to specific EPGs/VRFs or on other contract related values.
leaf# contract_parser.py
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[7:4131] [vrf:common:default] permit ip tcp tn-Prod1/ap-Services/epg-NTP(16410) tn-Prod1/l3out-L3Out1/instP-extEpg(25) eq 123 [contract:uni/tn-Prod1/brc-external_to_ntp] [hit=0]
[7:4156] [vrf:common:default] permit ip tcp tn-Prod1/l3out-L3Out1/instP-extEpg(25) eq 123 tn-Prod1/ap-Services/epg-NTP(16410) [contract:uni/tn-Prod1/brc-external_to_ntp] [hit=0]
[12:4169] [vrf:common:default] deny,log any tn-Prod1/l3out-L3Out1/instP-extEpg(25) epg:any [contract:implicit] [hit=0]
[16:4167] [vrf:common:default] permit any epg:any tn-Prod1/bd-Services(32789) [contract:implicit] [hit=0]
An ASIC level report used to check forwarding details which indicates, in the case of a dropped packet, the drop reason. Relevant to this section, the reason can be a SECURITY_GROUP_DENY (contract policy drop).
A Python-based utility on the APIC which can track end-to-end packet flow with ELAM.
An APIC App that abstracts the complexity of various ASICs to make forwarding decision inspection much more convenient and user friendly.
Please refer to the "Intra-Fabric Forwarding" section for additional details on the ELAM, fTriage and ELAM Assistant Tools
Policy CAM usage on a per leaf basis is an important parameter to monitor to ensure the fabric is in a healthy status. The quickest way to monitor that is to use the 'Capacity Dashboard' within the GUI and explicitly check the 'Policy Cam' column.
This command is useful for validating a variety of resource limits and usage, including Policy CAM. Note that this command can only be run in vsh_lc, so pass it in using the '-c' flag if being run from iBash.
leaf8# vsh_lc -c "show platform internal hal health-stats"
|Sandbox_ID: 0 Asic Bitmap: 0x0
|-------------------------------------
...
Policy stats:
=============
policy_count : 96
max_policy_count : 65536
policy_otcam_count : 175
max_policy_otcam_count : 8192
policy_label_count : 0
max_policy_label_count : 0
=============
There are numerous ways to troubleshoot a connectivity issue between two endpoints. The following methodology provides a good starting point to quickly and effectively isolate whether the connectivity issue is the result of a policy drop (contract induced).
Some high-level questions worth asking before diving in:
With the various tools available, there are some that are more appropriate and convenient to start with than others, depending on the level of information already known about the affected flow.
Is the full path of the packet in the ACI fabric known (ingress leaf, egress leaf...)?
Please note that the fTriage tool will not be discussed in detail in this section. Refer to the chapter "Intra-Fabric Forwarding" for more detail on using this tool.
Consider that while Visibility & Troubleshooting can help to quickly visualize where packets are dropped between two endpoints, fTriage shows more in-depth information for further troubleshooting. i.e. fTriage will help identify interface, drop reason, and other low-level details about the affected flow
.
This example scenario will show how to troubleshoot a policy drop between two endpoints: 192.168.21.11 and 192.168.23.11
Assuming packet drops are experienced between those two endpoints, the following troubleshooting workflow will be used to identify the root cause of the problem:
Identify the src/dst leaf(s) involved in the traffic flow:
The above steps are described further in the next paragraph.
This example scenario will show how to troubleshoot a policy drop between two endpoints: 192.168.21.11 in EPG-Web and 192.168.23.11 in EPG-DB.
The Visibility & Troubleshooting tool will help to visualize the switch where the packet drop occurred for a specific EP-to-EP flow and identify where packets are possibly dropped.
Configure a Session Name, Source, and Destination endpoint. Then click 'Submit' or 'Generate Report'.
The tool will automatically find the endpoints in the fabric and provide information about the Tenant, Application Profile and EPG those EP belong to.
In this case, it will discover that the EPs belong to the tenant Prod1, they belong to the same Application Profile 'AppProf' and are assigned to different EPGs: 'Web' and 'DB'.
The tool will automatically visualize the topology of the troubleshooting scenario. In this case, the two endpoints happen to be connected to the same leaf switch.
By navigating to the Drop/Stats submenu, the user can view general drops on the leaf or spine in question. Refer to the "Interface Drops" section in the chapter "Intra-Fabric Forwarding" of this book for more information about understanding which drops are relevant.
Many of these drops are expected behavior and can be ignored.
By drilling down to drop detail using the yellow 'Packets dropped' button on the switch diagram, the user can view details about the dropped flow.
By navigating to the Contracts submenu, the user can identify which contract is causing policy drop off between the EPGs. In the example, it is Implicit to Deny Prod1/VRF1 which shows some hits. This does not necessarily mean the specified flow (192.168.21.11 and 192.168.23.11) is hitting this implicit deny. If the Hits of Context Implicit deny rule is increasing, it implies there is traffic between Prod1/DB and Prod1/Web that do not hit any of contracts, hence are dropped by the Implicit deny.
In the Application Profile Topology view at Tenant > select the Application Profile name on the left > Topology , it is possible to verify which contracts are applied to the DB EPG. In this case, no contract is assigned to the EPG:
Now that the source and destination EPGs are known, it is also possible to identify other relevant information such as the following:
Class ID and scope can be easily retrieved from the APIC GUI by opening the Tenant > select the Tenant name on the left > Operational > Resource IDs > EPGs
In this case the Class ID and Scopes are:
An interesting tool to verify the packet dropped on an ACI leaf is the iBash command line: 'show logging ip access-list internal packet-log deny':
leaf5# show logging ip access-list internal packet-log deny | grep 192.168.21.11
[2019-10-01T14:25:44.746528000+09:00]: CName: Prod1:VRF1(VXLAN: 2654209), VlanType: FD_VLAN, Vlan-Id: 114, SMac: 0xf6f26c4ec8d0, DMac:0x0022bdf819ff, SIP: 192.168.21.11, DIP: 192.168.23.11, SPort: 0, DPort: 0, Src Intf: Ethernet1/19, Proto: 1, PktLen: 126
[2019-10-01T14:25:44.288653000+09:00]: CName: Prod1:VRF1(VXLAN: 2654209), VlanType: FD_VLAN, Vlan-Id: 116, SMac: 0x3e2593f0eded, DMac:0x0022bdf819ff, SIP: 192.168.23.11, DIP: 192.168.21.11, SPort: 0, DPort: 0, Src Intf: Ethernet1/19, Proto: 1, PktLen: 126
As per the previous output, it can be seen that on the leaf switch, numerous ICMP packets sourced by EP 192.168.23.11 towards 192.168.21.11 have been dropped.
The contract_parser tool will help to verify the actual policies applied to the VRF where the Endpoints are associated with:
leaf5# contract_parser.py --vrf Prod1:VRF1
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[7:5159] [vrf:Prod1:VRF1] permit ip tcp tn-Prod1/ap-App1/epg-App(32771) eq 5000 tn-Prod1/ap-App1/epg-Web(32772) [contract:uni/tn-Prod1/brc-web_to_app] [hit=0]
[7:5156] [vrf:Prod1:VRF1] permit ip tcp tn-Prod1/ap-App1/epg-Web(32772) tn-Prod1/ap-App1/epg-App(32771) eq 5000 [contract:uni/tn-Prod1/brc-web_to_app] [hit=0]
[16:5152] [vrf:Prod1:VRF1] permit any epg:any tn-Prod1/bd-Web(49154) [contract:implicit] [hit=0]
[16:5154] [vrf:Prod1:VRF1] permit arp epg:any epg:any [contract:implicit] [hit=0]
[21:5155] [vrf:Prod1:VRF1] deny,log any epg:any epg:any [contract:implicit] [hit=38,+10]
[22:5153] [vrf:Prod1:VRF1] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
This can also be verified through the zoning rule programmed in the leaf the policies enforced by the switch.
leaf5# show zoning-rule scope 2654209
+---------+--------+--------+----------+----------------+---------+----------+------------+----------+-------------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+----------+------------+----------+-------------------------+
| 5155 | 0 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | any_any_any(21) |
| 5159 | 32771 | 32772 | 411 | uni-dir-ignore | enabled | 2654209 | web_to_app | permit | fully_qual(7) |
| 5156 | 32772 | 32771 | 410 | bi-dir | enabled | 2654209 | web_to_app | permit | fully_qual(7) |
+---------+--------+--------+----------+----------------+---------+----------+------------+----------+-------------------------+
As already seen by the Visibility & Troubleshooting tool, the contract_parser tool, and the zoning rules, output confirms there is no contract between the source and destination EPGs in troubleshooting. It is easy to assume that the packets dropped are matching the implicit deny rule 5155.
ELAM capture provides an ASIC level report used to check forwarding details which indicates, in the case of a dropped packet, the drop reason. When the reason of a drop is a policy drop, as in this scenario, the output of the ELAM capture will look like the following.
Please note that details of setting up an ELAM capture will not be discussed in this chapter, please see the chapter "Intra-Fabric Forwarding".
leaf5# vsh_lc
module-1# debug platform internal tah elam asic 0
module-1(DBG-elam)# trigger init in-select 6 out-select 0
module-1(DBG-elam)# trigger reset
module-1(DBG-elam-insel6)# set outer ipv4 src_ip 192.168.21.11 dst_ip 192.168.23.11
module-1(DBG-elam-insel6)# start
module-1(DBG-elam-insel6)# status
ELAM STATUS
===========
Asic 0 Slice 0 Status Triggered
Asic 0 Slice 1 Status Armed
module-1(DBG-elam-insel6)# ereport | grep reason
RW drop reason : SECURITY_GROUP_DENY
LU drop reason : SECURITY_GROUP_DENY
pkt.lu_drop_reason: 0x2D
The ELAM report above shows clearly that the packet was dropped due to a policy drop: 'SECURITY_GROUP_DENY'
The very same result of the ELAM capture can be shown through the ELAM Assistant App on the APIC GUI.
Typically, the user will configure both source and destination details for the flow of interest. In this example, src IP is used to capture traffic towards endpoint in destination EPG that does not have a contract relationship to the source EPG.
There are three levels of output that can be viewed with ELAM Assistant. These are Express, Detail, and Raw.
Under the Express Result, the Drop Code reason SECURITY_GROUP_DENY indicates that the drop was a result of a contract hit.
There are two types of policy enforcements available for EPGs in a VRF with a contract preferred group configured:
The contract preferred group feature enables greater control of communication between EPGs in a VRF. If most of the EPGs in the VRF should have open communication, but a few should only have limited communication with the other EPGs, configure a combination of a contract preferred group and contracts with filters to more precisely control inter-EPG communication.
EPGs that are excluded from the preferred group can only communicate with other EPGs if there is a contract in place to override the source-any-destination-any-deny default rule.
Essentially, Contract Preferred Groups are an inverse of regular contracts. For regular contracts, explicit permit zoning-rules are programmed with an implicit deny zoning-rule with the VRF Scope. For Preferred Groups, an implicit PERMIT zoning-rule is programmed with the highest numeric priority value and specific DENY zoning-rules are programmed to disallow traffic from EPGs which are not Preferred Group members. As a result, the deny rules are evaluated first and if the flow isn't matched by these rules, then the flow is implicitly permitted.
There's always a pair of explicit deny zoning-rules for every EPG outside of the preferred group:
The figure below shows a logical topology in which EPGs App, App2 and App3 are all configured as Preferred Group Members.
VM-App is part of EPG-App and VM-App2 is part of EPG-App2. Both App and App2 EPG should be part of the preferred and hence communicate freely.
VM-App initiates a traffic flow on TCP port 6000 to VM-App2. Both EPG-App and EPG-App2 are Preferred Group Members as part of VRF1. VM-App2 never receives any packets on TCP port 6000.
1. Look up the pcTag of EPG APP and its VRF VNID/Scope
EPG and VRF pcTags
2. Verify contract programming using contract_parser.py on the ingress leaf
Use contract_parser.py and/or the 'show zoning-rule' command and specify the VRF
fab3-leaf8# show zoning-rule scope 2654209
+---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------------+
| 4165 | 0 | 0 | implicit | uni-dir | enabled | 2654209 | | permit | grp_any_any_any_permit(20) |
| 4160 | 0 | 0 | implarp | uni-dir | enabled | 2654209 | | permit | any_any_filter(17) |
| 4164 | 0 | 15 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_any_dest_any_deny(19) |
| 4176 | 0 | 16386 | implicit | uni-dir | enabled | 2654209 | | permit | any_dest_any(16) |
| 4130 | 32770 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_src_any_any_deny(18) |
| 4175 | 49159 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_src_any_any_deny(18) |
| 4129 | 0 | 49159 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_any_dest_any_deny(19) |
| 4177 | 32778 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_src_any_any_deny(18) |
| 4128 | 0 | 32778 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_any_dest_any_deny(19) |
| 4178 | 32775 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_src_any_any_deny(18) |
| 4179 | 0 | 32775 | implicit | uni-dir | enabled | 2654209 | | deny,log | grp_any_dest_any_deny(19) |
+---------+--------+--------+----------+---------+---------+---------+------+----------+----------------------------+
fab3-leaf8# contract_parser.py --vrf Prod1:VRF1
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[16:4176] [vrf:Prod1:VRF1] permit any epg:any tn-Prod1/bd-App(16386) [contract:implicit] [hit=0]
[16:4160] [vrf:Prod1:VRF1] permit arp epg:any epg:any [contract:implicit] [hit=0]
[18:4130] [vrf:Prod1:VRF1] deny,log any tn-Prod1/vrf-VRF1(32770) epg:any [contract:implicit] [hit=?]
[18:4178] [vrf:Prod1:VRF1] deny,log any epg:32775 epg:any [contract:implicit] [hit=?]
[18:4177] [vrf:Prod1:VRF1] deny,log any epg:32778 epg:any [contract:implicit] [hit=?]
[18:4175] [vrf:Prod1:VRF1] deny,log any epg:49159 epg:any [contract:implicit] [hit=?]
[19:4164] [vrf:Prod1:VRF1] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
[19:4179] [vrf:Prod1:VRF1] deny,log any epg:any epg:32775 [contract:implicit] [hit=?]
[19:4128] [vrf:Prod1:VRF1] deny,log any epg:any epg:32778 [contract:implicit] [hit=?]
[19:4129] [vrf:Prod1:VRF1] deny,log any epg:any epg:49159 [contract:implicit] [hit=?]
[20:4165] [vrf:Prod1:VRF1] permit any epg:any epg:any [contract:implicit] [hit=65]
Examining the above output, the implicit permit entry — ruleId 4165 — with the highest priority of 20, is observed. This implicit permit rule will cause all traffic flows to be allowed unless there's an explicit deny rule with a lower priority disallowing the traffic flow.
In addition, there are two explicit deny rules observed for pcTag 32775 which is the pcTag of EPG App2. These two explicit deny zoning-rules disallow traffic from any EPG to EPG App2, and vice versa. Those rules have priority 18 and 19, so they will take precedence on the default permit rule.
The conclusion is that EPG App2 is not a Preferred Group Member as the explicit deny rules are observed.
3. Verify EPG preferred Group Member Configuration
Navigate the APIC GUI and check EPG App2 and EPG App Preferred Group Member Configuration, In the following figure, see EPG App2 is not configured as a Preferred Group Member.
EPG App2 — Preferred Group Member setting excluded
EPG App — Preferred Group Member setting included
4. Set EPG App2 to be a Preferred Group Member
Changing the configuration of App2 EPG enables the preferred group to communicate freely as part of the preferred group.
EPG App2 — Preferred Group Member setting included
5. Re-verify contract programming using contract_parser.py on the leaf where the src EP resides
Use contract_parser.py again and specify the VRF name to verify whether the explicit deny rules for EPG App2 are now gone.
fab3-leaf8# contract_parser.py --vrf Prod1:VRF1
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[16:4176] [vrf:Prod1:VRF1] permit any epg:any tn-Prod1/bd-App(16386) [contract:implicit] [hit=0]
[16:4160] [vrf:Prod1:VRF1] permit arp epg:any epg:any [contract:implicit] [hit=0]
[18:4175] [vrf:Prod1:VRF1] deny,log any epg:16390 epg:any [contract:implicit] [hit=0]
[18:4167] [vrf:Prod1:VRF1] deny,log any epg:23 epg:any [contract:implicit] [hit=0]
[18:4156] [vrf:Prod1:VRF1] deny,log any tn-Prod1/vrf-VRF1(32770) epg:any [contract:implicit] [hit=0]
[18:4168] [vrf:Prod1:VRF1] deny,log any epg:49159 epg:any [contract:implicit] [hit=0]
[19:4164] [vrf:Prod1:VRF1] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
[19:4169] [vrf:Prod1:VRF1] deny,log any epg:any epg:16390 [contract:implicit] [hit=0]
[19:4159] [vrf:Prod1:VRF1] deny,log any epg:any epg:23 [contract:implicit] [hit=0]
[19:4174] [vrf:Prod1:VRF1] deny,log any epg:any epg:49159 [contract:implicit] [hit=0]
[20:4165] [vrf:Prod1:VRF1] permit any epg:any epg:any [contract:implicit] [hit=65]
The explicit deny rules for EPG App2 and its pcTag 32775 are no longer observed in the above output. This means that traffic between EPs in EPG App and EPG App2 will now match the implicit permit rule — ruleId 4165 — with the highest priority of 20.
When configuring contracts between one or multiple EPGs, contracts can either be configured as a consumed or provided relation. When the number of EPGs grows, so can the amount of contract relations between them. Some common use cases require all EPGs to exchange traffic flows with another specific EPG. Such a use case could be an EPG containing EPs providing services that need to be consumed by all other EPGs inside the same VRF (NTP or DNS for example). vzAny allows for lower operational overhead in configuring contract relations between all EPGs and specific EPGs providing services to be consumed by all other EPGs. In addition, vzAny allows for a much more efficient Security Policy CAM usage on leaf switches as only 2 zoning-rules are added for each vzAny contract relation.
The figure below describes such a use case whereby VM-Web and VM-App in EPGs Web and App respectively need to consume NTP services from VM-NTP in EPG-NTP. Instead of configuring a provided contract on EPG NTP, and subsequently having that same contract as a consumed contract on EPGs Web and App, vzAny allows each EPG in VRF Prod:VRF1 to consume NTP services from EPG NTP.
vzAny — Any EPG in VRF Prod:VRF1 can consume NTP services from EPG NTP
Consider a scenario where drops are observed between EPGs that consume the NTP services when there is no contract between them.
1. Look up the pcTag of EPG NTP and its VRF VNID/Scope
'Tenant > Operational > Resource IDs > EPGs' allows finding the pcTag and scope
EPG NTP pcTag and its VRF VNID/Scope
2. Verify if a contract is configured as a vzAny consumed contract as part of the VRF
Navigate to the VRF and check if there's a consumed contract configured as vzAny under the 'EPG Collection for VRF'.
Contract configured as a consumed vzAny contract on the VRF
3. Verify if the same contract is applied as a provided contract on EPG NTP
In order to establish a contract relation, the same contract needs to be applied as a provided contract on EPG NTP which is providing NTP services to the other EPGs in its VRF.
4. Zoning-rule verification on ingress leaf using contract_parser.py or 'show zoning-rule'
The ingress leaf should have 2 zoning-rules to allow bi-directional traffic flows (if the contract subject is set to allow both directions) between any EPG and EPG NTP. 'Any EPG' is denoted as pcTag 0 in zoning-rule programming.
Using contract_parser.py or the 'show zoning-rule' commands on the ingress leaf whilst specifying the VRF allows to ensure the zoning-rule are programmed.
Using contract_parser.py and 'show zoning-rule' to check the presence of the vzAny based zoning-rules.
Here two types of rules are evident:
Given that lowest priority has precedence, all EPGs of the VRF will have access NTP EPG.
fab3-leaf8# contract_parser.py --vrf Prod1:VRF
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[13:4156] [vrf:Prod1:VRF1] permit ip tcp tn-Prod1/ap-Services/epg-NTP(49161) eq 123 epg:any [contract:uni/tn-Prod1/brc-any_to_ntp] [hit=0]
[14:4168] [vrf:Prod1:VRF1] permit ip tcp epg:any tn-Prod1/ap-Services/epg-NTP(49161) eq 123 [contract:uni/tn-Prod1/brc-any_to_ntp] [hit=0]
[16:4176] [vrf:Prod1:VRF1] permit any epg:any tn-Prod1/bd-App(16386) [contract:implicit] [hit=0]
[16:4174] [vrf:Prod1:VRF1] permit any epg:any tn-Prod1/bd-Services(32776) [contract:implicit] [hit=0]
[16:4160] [vrf:Prod1:VRF1] permit arp epg:any epg:any [contract:implicit] [hit=0]
[21:4165] [vrf:Prod1:VRF1] deny,log any epg:any epg:any [contract:implicit] [hit=65]
[22:4164] [vrf:Prod1:VRF1] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
fab3-leaf8# show zoning-rule scope 2654209
+---------+--------+--------+----------+---------+---------+---------+------------+----------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+---------+---------+---------+------------+----------+----------------------+
| 4165 | 0 | 0 | implicit | uni-dir | enabled | 2654209 | | deny,log | any_any_any(21) |
| 4160 | 0 | 0 | implarp | uni-dir | enabled | 2654209 | | permit | any_any_filter(17) |
| 4164 | 0 | 15 | implicit | uni-dir | enabled | 2654209 | | deny,log | any_vrf_any_deny(22) |
| 4176 | 0 | 16386 | implicit | uni-dir | enabled | 2654209 | | permit | any_dest_any(16) |
| 4174 | 0 | 32776 | implicit | uni-dir | enabled | 2654209 | | permit | any_dest_any(16) |
| 4168 | 0 | 49161 | 424 | uni-dir | enabled | 2654209 | any_to_ntp | permit | any_dest_filter(14) |
| 4156 | 49161 | 0 | 425 | uni-dir | enabled | 2654209 | any_to_ntp | permit | src_any_filter(13) |
+---------+--------+--------+----------+---------+---------+---------+------------+----------+----------------------+
Shared Layer 3 Out is a configuration which allows having an L3Out in one VRF providing some services (external access) and one or more other VRFs consume this L3Out. More detail on shared L3Out can be found in the "External routing" chapter.
When doing shared L3Out it is recommended to have the provider of the contract being the shared L3Out and the EPG being the consumer of the contract. This scenario will be illustrated in this section.
It is not recommended to do the opposite, which is L3Out consuming a service provided by an EPG. The reason for this has to do with scalability since for shared services, the zoning-rules are only installed on consumer VRF. The principles of consuming and providing denote where traffic flows are initiated. With default ingress policy enforcement, this means policy enforcement will be applied on the consumer side and more specifically on the ingress leaf (non-border leaf). For the ingress leaf to enforce policy it requires the pcTag of the destination. In this scenario the destination is the external EPG pcTag. The ingress leaf thus performs policy enforcement and forwards the packets to the border leaf. The border leaf receives the packet on its fabric link which performs a route lookup (LPM) and forwards the packet onto the adjacency for the destination prefix.
The border leaf however does NOT perform any policy enforcement when sending traffic onto the destination EP nor does it do so on the return traffic flow back to the source EP.
As a result, only the Policy CAM of the ingress non-BL leaf has entries installed (in the consumer VRF) and the BL's Policy CAM is not affected.
1. Verify EPG pcTag and VRF VNID/Scope for the consumer EPG
With shared L3Out, the zoning-rules are only installed in the consumer VRF. The provider must have a global pcTag (below 16k) which allows this pcTag to be used in all consumer VRFs. In our scenario, the provider is the external EPG and will have a global pcTag. The consumer EPG will have a local pcTag as usual.
pcTag of consumer EPG
2. Verify the pcTag and VRF VNID/Scope for the provider L3Out EPG
As noted in Step 1, the provider L3Out EPG has a global range pcTag as prefixes from L3Out which are leaked into the consumer VRF. As a result, the L3Out EPG pcTag is required to not overlap with pcTags in the consumer VRF, and so it is within the global pcTag range.
pcTag of provider external EPG
3. Verify the consumer EPG has either an imported tenant scoped contract or global contract configured
The consumer EPG NTP with subnet defined under the EPG/BD is consuming the 'tenant' or 'global' scoped contract
Contract consumed by EPG
4. Verify whether the BD of the consumer EPG has a subnet configured with its scope set to 'Shared between VRFs'
The subnet of the EPG is configured under the bridge domain but must have the 'shared between VRF' flag (to allow routed leaking) and the 'advertised externally' flag (to allow to advertise to L3Out)
5. Verify the provider L3Out EPG has either an imported tenant scoped contract or global contract configured
The L3Out EPG should either have a tenant scoped contract or global contract configured as a provided contract.
Contract on provider L3Out
6. Verify if the provider L3Out EPG has a subnet configured with the necessary scopes checked
The provider L3Out EPG should have the to-be-leaked prefix configured with the following scopes:
For more detail on subnet flag in L3Out EPG refer to the "External forwarding" chapter.
External EPG subnet settings
External EPG subnet settings expanded
7. Verify the pcTag of L3Out EPG subnet on the non-BL for the consumer VRF
When traffic destined to the external EPG subnet ingresses the non-BL, a lookup is performed against the destination prefix to determine the pcTag. This can be checked using the following command on the non-BL.
Note this output is took in the scope of the VNI 2818048 which is the consumer VRF VNID. By looking at the table the consumer can find the pcTag of the destination, even though it is not in the same VRF.
fab3-leaf8# vsh -c 'show system internal policy-mgr prefix' | egrep 'Vrf-Vni|==|common:default'
Vrf-Vni VRF-Id Table-Id Table-State VRF-Name Addr Class Shared Remote Complete
======= ====== =========== ======= ============================ ================================= ====== ====== ====== ========
2818048 19 0x13 Up common:default 0.0.0.0/0 15 False False False
2818048 19 0x80000013 Up common:default ::/0 15 False False False
2818048 19 0x13 Up common:default 172.16.10.0/24 25 True True False
The above output shows the combination of the L3Out EPG subnet and its global pcTag 25.
8. Verify the programmed zoning-rules on the non-BL for the consumer VRF
Use either 'contract_parser.py' or the 'show zoning-rule' command and specify the VRF.
Below command outputs display two zoning-rules are installed to allow traffic from the consumer EPG local pcTag 16410 to the L3Out EPG global pcTag 25. This is in the scope 2818048, which is the scope of the consumer VRF.
fab3-leaf8# show zoning-rule scope 2818048
+---------+--------+--------+----------+----------------+---------+---------+-----------------+----------+-----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+---------+-----------------+----------+-----------------------+
| 4174 | 0 | 0 | implarp | uni-dir | enabled | 2818048 | | permit | any_any_filter(17) |
| 4168 | 0 | 15 | implicit | uni-dir | enabled | 2818048 | | deny,log | any_vrf_any_deny(22) |
| 4167 | 0 | 32789 | implicit | uni-dir | enabled | 2818048 | | permit | any_dest_any(16) |
| 4159 | 0 | 0 | implicit | uni-dir | enabled | 2818048 | | deny,log | any_any_any(21) |
| 4169 | 25 | 0 | implicit | uni-dir | enabled | 2818048 | | deny,log | shsrc_any_any_deny(12)|
| 4156 | 25 | 16410 | 425 | uni-dir-ignore | enabled | 2818048 | external_to_ntp | permit | fully_qual(7) |
| 4131 | 16410 | 25 | 424 | bi-dir | enabled | 2818048 | external_to_ntp | permit | fully_qual(7) |
+---------+--------+--------+----------+----------------+---------+---------+-----------------+----------+-----------------------+
fab3-leaf8# contract_parser.py --vrf common:default
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[7:4131] [vrf:common:default] permit ip tcp tn-Prod1/ap-Services/epg-NTP(16410) tn-Prod1/l3out-L3Out1/instP-extEpg(25) eq 123 [contract:uni/tn-Prod1/brc-external_to_ntp] [hit=0]
[7:4156] [vrf:common:default] permit ip tcp tn-Prod1/l3out-L3Out1/instP-extEpg(25) eq 123 tn-Prod1/ap-Services/epg-NTP(16410) [contract:uni/tn-Prod1/brc-external_to_ntp] [hit=0]
[12:4169] [vrf:common:default] deny,log any tn-Prod1/l3out-L3Out1/instP-extEpg(25) epg:any [contract:implicit] [hit=0]
[16:4167] [vrf:common:default] permit any epg:any tn-Prod1/bd-Services(32789) [contract:implicit] [hit=0]
[16:4174] [vrf:common:default] permit arp epg:any epg:any [contract:implicit] [hit=0]
[21:4159] [vrf:common:default] deny,log any epg:any epg:any [contract:implicit] [hit=0]
[22:4168] [vrf:common:default] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
9. Verify the programmed zoning-rules on the BL for the provider VRF
Use either 'contract_parser.py' or the 'show zoning-rule' command and specify the VRF. The following command outputs show that there are NO specific zoning-rules in the provider VRF as outlined multiple times before.
It is in the scope 2719752 which is the scope of provider VRF.
border-leaf# show zoning-rule scope 2719752
+---------+--------+--------+----------+----------------+---------+---------+--------------+----------+----------------------+
| Rule ID | SrcEPG | DstEPG | FilterID | Dir | operSt | Scope | Name | Action | Priority |
+---------+--------+--------+----------+----------------+---------+---------+--------------+----------+----------------------+
| 4134 | 10937 | 24 | default | uni-dir-ignore | enabled | 2719752 | vrf1_to_vrf2 | permit | src_dst_any(9) |
| 4135 | 24 | 10937 | default | bi-dir | enabled | 2719752 | vrf1_to_vrf2 | permit | src_dst_any(9) |
| 4131 | 0 | 0 | implicit | uni-dir | enabled | 2719752 | | deny,log | any_any_any(21) |
| 4130 | 0 | 0 | implarp | uni-dir | enabled | 2719752 | | permit | any_any_filter(17) |
| 4132 | 0 | 15 | implicit | uni-dir | enabled | 2719752 | | deny,log | any_vrf_any_deny(22) |
+---------+--------+--------+----------+----------------+---------+---------+--------------+----------+----------------------+
border-leaf# contract_parser.py --vrf Prod1:VRF3
Key:
[prio:RuleId] [vrf:{str}] action protocol src-epg [src-l4] dst-epg [dst-l4] [flags][contract:{str}] [hit=count]
[9:4134] [vrf:Prod1:VRF3] permit any tn-Prod1/l3out-L3Out1/instP-extEpg2(10937) tn-Prod1/l3out-L3Out2/instP-extEpg2(24) [contract:uni/tn-Prod1/brc-vrf1_to_vrf2] [hit=0]
[9:4135] [vrf:Prod1:VRF3] permit any tn-Prod1/l3out-L3Out2/instP-extEpg2(24) tn-Prod1/l3out-L3Out1/instP-extEpg2(10937) [contract:uni/tn-Prod1/brc-vrf1_to_vrf2] [hit=0]
[16:4130] [vrf:Prod1:VRF3] permit arp epg:any epg:any [contract:implicit] [hit=0]
[21:4131] [vrf:Prod1:VRF3] deny,log any epg:any epg:any [contract:implicit] [hit=0]
[22:4132] [vrf:Prod1:VRF3] deny,log any epg:any pfx-0.0.0.0/0(15) [contract:implicit] [hit=0]
Revision | Publish Date | Comments |
---|---|---|
1.0 |
05-Aug-2022 |
Initial Release |