VXLAN EVPN Multi-Site, Release 12.2.2

The following table provides an overview of the significant changes up to this current release. The table does not provide an exhaustive list of all changes or of the new features up to this release.

A VXLAN Multi-Site is a multifabric container that is created to manage multiple member fabrics. A VXLAN Multi-Site is a single point of control for definition of overlay networks and VRFs that are shared across member fabrics. When you move fabrics that are designated to be a part of the multifabric overlay network domain under the VXLAN Multi-Site as member fabrics, the member fabrics share the networks and VRFs created at the VXLAN Multi-Site-level. This way, you can consistently provision network and VRFs for different fabrics, at one go. It significantly reduces the time and complexity involving multiple fabric provisioning.

You can add a Data Center VXLAN EVPN fabric or a VXLAN Multi-Site External Network fabric as member fabrics in a VXLAN Multi-Site. With Release 12.1.3, Cisco Nexus Dashboard Fabric Controller provides support for adding a Campus EVPN VXLAN fabric also as a member fabric in a VXLAN Multi-Site.

As server networks and VRFs are shared across the member fabrics as one stretched network, provisioning new networks and VRFs is provided at the VXLAN Multi-Site fabric level. You can create new networks and VRFs only for the VXLAN Multi-Site. All the member fabrics inherit any new network and VRF created for the VXLAN Multi-Site.

The topology view for the VXLAN Multi-Site fabric displays all member fabrics, and how they are connected to each other, in one view. You can deploy overlay networks and VRFs on member fabrics from a single topology deployment screen, instead of deploying each member fabric separately.

A VXLAN Multi-Site has multiple sites and hence has multiple member fabrics under a VXLAN Multi-Site. VRFs and networks are created for the VXLAN Multi-Site and get inherited by the member fabrics.

A high-level flow chart of the VXLAN Multi-Site and member fabric creation and VXLAN Multi-Site-to-member fabric inheritance process is depicted in the following figures:

The sample flow explains the inheritance from the VXLAN Multi-Site to one member. The following figure illustrates a sample flow from a VXLAN Multi-Site to multiple members:

In this example, VRF-50000 (and L3 network with ID 50000), and L2 networks with IDs 30000 and 30001 are created in one go. Networks and VRFs are deployed on the member fabric switches, one after another, as depicted in the image.

You can provision overlay networks through a single VXLAN Multi-Site deployment screen.

If you move a standalone fabric with existing networks and VRFs to a VXLAN Multi-Site, NDFC validates for any conflicts. This is explained in detail in an upcoming section.

Upcoming sections in the document explain the following:

The process is explained in two steps:

Creating the VXLAN EVPN Multi-Site Fabric

1. In Cisco Nexus Dashboard Fabric Controller, choose Manage > Fabrics.

2. From the Actions drop-down list, choose Create Fabric.

   The Create Fabric page appears.

3. Enter a unique name for the fabric and click Choose Fabric.

   A list of all the available fabric templates are listed.

4. From the available list of fabric templates, choose VXLAN EVPN Multi-Site and click Select.

5. Enter the necessary field values to create a fabric.

   The tabs and their fields in the screen are explained in the subsequent sections. The overlay and underlay network parameters are included in these tabs.

   • General Parameters

   • DCI

   • Resources

   • Configuration Backup

6. When you have completed the necessary configurations, click Save.

   A message appears at the bottom right part of the screen, indicating that you have created a new VXLAN EVPN Multi-Site fabric.

   The newly created VXLAN EVPN Multi-Site fabric instance appears in the fabrics table. It contains the member fabric names as a branch. When no member fabric is created, it is displayed as a standalone fabric.

7. Click on the fabric to display a summary in the slide-out pane. Click the Launch icon to display the Fabric Overview page.

General Parameters

All mandatory fields in the General Parameters tab are prefilled. Update the relevant fields as needed.

Field	Description
Layer 2 VXLAN VNI Range	Specifies the Layer 2 VXLAN segment identifier range.
Layer 3 VXLAN VNI Range	Specifies the Layer 3 VXLAN segment identifier range.
Enable IPv6 Underlay for VXLAN Fabric	Enables using an IPv6 underlay when creating a VXLAN Multi-Site fabric. By default, this field is not enabled. All VXLAN Multi-Site fabrics use IPv4 addresses for underlay traffic if this field is not enabled.
VRF Template	Specifies the default VRF template for leaf devices.
Network Template	Specifies the default network template for leaf devices.
VRF Extension Template	Specifies the default VRF extension template for border devices.
Network Extension Template	Specifies the default network extension template for border devices.
Enable Private VLAN (PVLAN)	Enables private VLAN on VXLAN Multi-Site and its child fabrics.
PVLAN Secondary Network Template	Specifies the default secondary PVLAN network template.
Anycast-Gateway-MAC	Specifies the anycast gateway MAC address.
Multi-Site VTEP VIP Loopback Id	Specifies the multisite routing loopback ID.
Border Gateway IP TAG	Routing tag associated with IP address of loopback and DCI interfaces
ToR Auto-deploy Flag	Enables automatic deployment of the networks and VRFs in VXLAN EVPN fabric connected to ToR switches in an External Fabric. The system enables this configuration on performing Recalculate and Deploy in the VXLAN EVPN Multi-Site fabric.

What’s next: Complete the configurations in another tabs, if required, or click Save when you have completed the necessary configurations for this fabric.

DCI

Field	Description
Multi-Site Overlay IFC Deployment Method	Defines how the data centers connect through the border gateways - manually, directly to the border gateways or through a route server. The Multi-Site IFCs can be created between the border gateways in the VXLAN EVPN fabrics and router server in an external fabric, or back-to-back between border gateways in two VXLAN EVPN fabrics. If you configured the deployment method for Multi-Site Overlay IFC Deployment Method as Centralized_To_Route_Server, you need to provide an IPv4 or an IPv6 address depending on if you configured an IPv4 or an IPv6 underlay on the General Parameters tab. DCI parameters need to be consistent with the General Parameters settings depending on if you configured an IPv4 or an IPv6 underlay.
Multi-Site Route Server List	Specifies the IP addresses of the route server. If you specify more than one, separate the IP addresses using a comma.
Multi-Site Route Server BGP ASN List	Specifies the BGP AS Number of the router server. If you specify more than one route server, separate the AS Numbers using a comma.
Enable 'redistribute direct' on Route Servers	Enables auto-creation of multi-site overlay IFCs on route servers. This field is applicable only when you have configured the deployment method for Multi-Site Overlay as Centralized_To_Route_Server.
Route Server IP TAG	Specifies the routing tag associated with the route server IP for redistribute direct. This is the IP used in eBGP EVPN peering.
Multi-Site Underlay IFC Auto Deployment Flag	Enables auto configuration. Uncheck the check box for manual configuration.
BGP Send-community on Multi-Site Underlay IFC	Enables the Enable BGP Send-Community both setting in auto-created multi-site underlay IFC, which will generate the send-community both in the eBGP session for a multi-site underlay.
BGP log neighbor change on Multi-Site Underlay IFC	Configures the Enable BGP log neighbor change setting in auto created multi-site underlay IFC.
BGP BFD on Multi-Site Underlay IFC	Configures the Enable BGP BFD setting in an auto-created multi-site underlay IFC.
Delay Restore Time	Specifies the Multi-Site underlay and overlay control planes convergence time. The minimum value is 30 seconds and the maximum value is 1000 seconds.
Multi-Site CloudSec	Enables CloudSec configurations on border gateways. If you enable this field, the remaining three fields for CloudSec are editable. For more information, see Support for CloudSec in Multi-Site Deployment.
CloudSec Key String	Specifies the Cisco Type 7 encrypted octet key string.
CloudSec Cryptographic Algorithm	Choose AES_128_CMAC or AES_256_CMAC for the encryption type.
CloudSec Enforcement	Specifies whether the CloudSec enforcement should be strict or loose. strict - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that don’t support CloudSec, then an error message is generated, and the configuration isn’t pushed to any switch. If you select strict, the tunnel-encryption must-secure CLI is pushed to the CloudSec enabled gateways within VXLAN Multi-Site. loose - Deploys the CloudSec configuration to all the border gateways in fabrics in VXLAN Multi-Site. If there are any border gateways that do not support CloudSec, then a warning message is generated. In this case, the CloudSec configuration is only deployed to the switches that support CloudSec. If you select loose, the tunnel-encryption must-secure CLI is removed, if available. There should be at least two fabrics in VXLAN Multi-Site with border gateways that support CloudSec. If there is only one fabric with a CloudSec capable device, then the following error message is generated: CloudSec needs to have at least 2 sites that can support CloudSec. To remove the error, make sure you have at least two sites that can support CloudSec or disable CloudSec.
CloudSec Status Report Timer	Specifies the CloudSec Operational Status periodic report timer in minutes. This value specifies how often the NDFC polls the CloudSec status data from the switch. The default value is 5 minutes and the range is from 5 to 60 minutes.
Enable Multi-Site eBGP Password	Enables eBGP password for Multi-Site underlay/overlay IFCs.
eBGP Password	Specifies the encrypted eBGP password hex string.
eBGP Authentication Key Encryption Type	Specifies the BGP key encryption type. It is 3 for 3DES and 7 for Cisco.
Enable IPv4 and/or IPv6 Tenant Routed Multicast across sites	After enabling this field, MVPN VRI IDs are tracked in an MSD fabric to ensure uniqueness within the MSD.

Click the Resources tab.

Resources

Field	Description
Multi-Site VTEP VIP Loopback IP Range	Specifies the Multi-Site loopback IP address range used for the EVPN Multi-Site function. A unique loopback IP address is assigned from this range to each member fabric because each member site must have a Multi-site Routing Loopback IP address assigned for overlay network reachability. The per-fabric loopback IP address is assigned on all the BGWs in a specific member fabric.
DCI Subnet IP Range	Specifies the Data Center Interconnect (DCI) subnet IP address.
Subnet Target Mask	Specifies the DCI subnet mask.

Configuration Backup

Field	Description
Scheduled Fabric Backup	Enables daily backup. This backup tracks changes in the running configuration of the fabric devices that are not tracked by configuration compliance.
Scheduled Time	Specifies the scheduled backup time in 24-hour format. This field is enabled if you check the Scheduled Fabric Backup check box. Select both the check boxes to enable both back up processes.

What’s next: Create a new standalone fabric and move it under the VXLAN EVPN Multi-Site fabric as a member fabric.

In standalone fabrics, networks and VRFs are created for each fabric. In a VXLAN Multi-Site fabric, networks and VRFs should be created at the VXLAN Multi-Site fabric level. The networks and VRFs are inherited by all the member networks. You cannot create or delete networks and VRFs for member fabrics. However, you can edit them.

For example, consider a VXLAN Multi-Site fabric with two member fabrics. If you create three networks in the VXLAN Multi-Site fabric, then all three networks will automatically be available for deployment in both the member fabrics.

Though member fabrics inherit the VXLAN Multi-Site fabric’s networks and VRFs, you have to deploy the networks and VRFs distinctly, for each fabric.

A deployment view is introduced for the VXLAN Multi-Site, in addition to the per-fabric deployment view. In this view, you can view and provision overlay networks for all member fabrics within the VXLAN Multi-Site, at once. However, you still have to apply and save network and VRF configurations distinctly, for each fabric.

Networks and VRFs are the common identifiers (represented across member fabrics) that servers (or end hosts) are grouped under so that traffic can be sent between the end hosts based on the network and VRF IDs, whether they reside in the same or different fabrics. Since they have common representation across member fabrics, networks and VRFs can be provisioned at one go. As the switches in different fabrics are physically and logically distinct, you have to deploy the same networks and VRFs separately for each fabric.

For example, if you create networks 30000 and 30001 for a VXLAN Multi-Site that contains two member fabrics, the networks are automatically created for the member fabrics and are available for deployment.

You can deploy 30000 and 30001 on the border devices of all member fabrics through a single (VXLAN Multi-Site fabric) deployment screen. Prior to this, you had to access the first member fabric deployment screen, deploy 30000 and 300001 on the fabric’s border devices, and then access the second member fabric deployment screen and deploy again.

Networks and VRFs are created in the VXLAN Multi-Site and deployed in the member fabrics. The steps are explained below:

If you move a standalone fabric with existing networks and VRFs to a VXLAN Multi-Site fabric as a member, ensure that common networks (L2 VNI and L3 VNI), anycast gateway MAC, and VRF and network templates are the same across the fabric and the VXLAN Multi-Site. NDFC validates the standalone fabric with the network and VRF information of the VXLAN Multi-Site fabric to avoid conflict entries. An example of conflict entries is two common network names with a different network ID. After validation, the standalone fabric is moved to the VXLAN Multi-Site fabric as a member fabric only if there are no conflicts.

The following are the different points to consider while moving a fabric under a VXLAN Multi-Site:

CloudSec feature allows secured data center interconnect in a multi-site deployment by supporting source-to-destination packet encryption between border gateway devices in different fabrics.

CloudSec feature is supported on Cisco Nexus 9000 Series FX2 platform with Cisco NX-OS Release 9.3(5) or later. The border gateways, border gateway spines, and border gateway super spines that are FX2 platforms, and run Cisco NX-OS Release 9.3(5) or later are referred as CloudSec capable switches.

You can enable CloudSec while creating a VXLAN Multi-Site fabric.

The CloudSec session is point to point over DCI between border gateways (BGWs) on two different sites. All communication between sites uses Multi-Site PIP instead of VIP. Enabling CloudSec requires a switch to move from VIP to PIP, which could cause traffic disruption for data flowing between sites. Therefore, it is recommended to enable or disable CloudSec during a maintenance window.

Refer to the "Guidelines and Limitations for Secure VXLAN EVPN Multi-Site Using CloudSec" section in the latest Cisco Nexus 9000 Series NX-OS VXLAN Configuration Guide for the guidelines and limitations for Secure VXLAN EVPN Multi-Site using CloudSec.

feature tunnel-encryption
evpn multisite border-gateway 600
  dci-advertise-pip
tunnel-encryption must-secure-policy
tunnel-encryption policy CloudSec_Policy1
tunnel-encryption source-interface loopback20
key chain CloudSec_Key_Chain1 tunnel-encryption
  key 1000
    key-octet-string 7 075e731f1a5c4f524f43595f507f7d73706267714752405459070b0b0701585440
    cryptographic-algorithm AES_128_CMA
tunnel-encryption peer-ip 192.168.0.6
  keychain CloudSec_Key_Chain1 policy CloudSec_Policy1

interface ethernet1/13
  no switchport
  ip address 192.168.1.14/24 tag 54321
  evpn multisite dci-tracking
  tunnel-encryption
  mtu 9216
  no shutdown

THE SPECIFICATIONS AND INFORMATION REGARDING THE PRODUCTS IN THIS MANUAL ARE SUBJECT TO CHANGE WITHOUT NOTICE. ALL STATEMENTS, INFORMATION, AND RECOMMENDATIONS IN THIS MANUAL ARE BELIEVED TO BE ACCURATE BUT ARE PRESENTED WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED. USERS MUST TAKE FULL RESPONSIBILITY FOR THEIR APPLICATION OF ANY PRODUCTS.

THE SOFTWARE LICENSE AND LIMITED WARRANTY FOR THE ACCOMPANYING PRODUCT ARE SET FORTH IN THE INFORMATION PACKET THAT SHIPPED WITH THE PRODUCT AND ARE INCORPORATED HEREIN BY THIS REFERENCE. IF YOU ARE UNABLE TO LOCATE THE SOFTWARE LICENSE OR LIMITED WARRANTY, CONTACT YOUR CISCO REPRESENTATIVE FOR A COPY.

The Cisco implementation of TCP header compression is an adaptation of a program developed by the University of California, Berkeley (UCB) as part of UCB’s public domain version of the UNIX operating system. All rights reserved. Copyright © 1981, Regents of the University of California.

NOTWITHSTANDING ANY OTHER WARRANTY HEREIN, ALL DOCUMENT FILES AND SOFTWARE OF THESE SUPPLIERS ARE PROVIDED “AS IS" WITH ALL FAULTS. CISCO AND THE ABOVE-NAMED SUPPLIERS DISCLAIM ALL WARRANTIES, EXPRESSED OR IMPLIED, INCLUDING, WITHOUT LIMITATION, THOSE OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.

IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THIS MANUAL, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

Any Internet Protocol (IP) addresses and phone numbers used in this document are not intended to be actual addresses and phone numbers. Any examples, command display output, network topology diagrams, and other figures included in the document are shown for illustrative purposes only. Any use of actual IP addresses or phone numbers in illustrative content is unintentional and coincidental.

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product.

Cisco and the Cisco logo are trademarks or registered trademarks of Cisco and/or its affiliates in the U.S. and other countries. To view a list of Cisco trademarks, go to this URL: http://www.cisco.com/go/trademarks. Third-party trademarks mentioned are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (1110R)

© 2017-2024 Cisco Systems, Inc. All rights reserved.