Configuring Infra for Cisco DCNM Sites

Prerequisites and Guidelines

The following sections describe the steps necessary to configure the general as well as site-specific fabric Infra settings.

Before you proceed with Infra configuration, you must have added the sites as described in previous sections.

In addition, keep in mind the following:

  • Adding or removing border gateway switches requires a Nexus Dashboard Orchestrator fabric connectivity information refresh described in the Refreshing Site Connectivity Information as part of the general Infra configuration procedures.

Configuring Infra: General Settings

This section describes how to configure general Infra settings for all the sites.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left sidebar, select General Settings.

Step 5

Configure Control Plane Configuration.

  1. Select the Control Plane BGP tab.

  2. Choose BGP Peering Type.

    • full-mesh—All border gateway switches in each site will establish peer connectivity with remote sites' border gateway switches.

    • route-server—The route-server option allows you to specify one or more control-plane nodes to which each site establishes MP-BGP EVPN sessions. The route-server nodes perform a function similar to traditional BGP route-reflectors, but for EBGP (and not iBGP) sessions. The use of route-server nodes avoids creating MP-BGP EVPN full mesh adjacencies between all the VXLAN EVPN sites managed by NDO.

  3. If you set the BGP Peering Type to route-server, click +Add Route Server to add one or more route servers.

    In the Add Route Server window that opens:

    • From the Site dropdown, select the site you want to connect to the route server.

    • The ASN field will be auto-populated with the site's ASN.

    • From the Core Router Device dropdown, select the route server to which you want to connect.

    • From the Interface dropdown, select the interface on the core router device.

    You can add up to 4 route servers. If you add multiple route servers, every site will establish MP-BGP EVPAN adjacencies to every route server.

  4. Leave the Keepalive Interval (Seconds), Hold Interval (Seconds), Stale Interval (Seconds), Graceful Helper, Maximum AS Limit, and BGP TTL Between Peers fields at default values as they are relevant for Cisco ACI fabrics only.

  5. Skip the OSPF Area ID and External Subnet Pool fields at default values as they are relevant for Cisco Cloud ACI fabrics only.

Step 6

Skip the IPN Devices tab settings.

The settings under the IPN Devices tab are for Cisco ACI inter-site connectivity between on-premises APIC and Cloud APIC sites. You can skip these settings when managing Cisco DCNM sites only.

Step 7

Configure DCNM Settings.

  1. Select the DCNM Settings tab.

  2. Provide the L2 VXLAN VNI Range.

  3. Provide the L3 VXLAN VNI Range.

  4. Provide the Multi-Site Routing Loopback IP Range.

    This field is used to auto-populate the Multi-Site TEP field for each fabric, which is described in Configuring Infra: DCNM Site Settings.

    For sites that were previously part of a Multi-Site Domain (MSD) in DCNM, this field will be pre-populated with the previously defined value.

  5. Provide the Anycast Gateway MAC.

Refreshing Site Connectivity Information

Infrastructure changes, such as adding and removing border gateway switches, require a Nexus Dashboard Orchestrator fabric connectivity refresh. This section describes how to pull up-to-date connectivity information directly from each site's controller.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Site Connectivity.

Step 3

In the top right of the main pane, click Configure.

Step 4

In the left sidebar, under Sites, select a specific site.

Step 5

In the main window, click the Refresh button to pull fabric information from the controller.

Step 6

(Optional) In the Confirmation dialog, check the box if you want to remove configuration for decommissioned border gateway switches.

If you choose to enable this checkbox, all configuration info for any currently decommissioned border gateway switches will be removed from the database.

Step 7

Finally, click Yes to confirm and load the connectivity information.

This will discover any new or removed spines and all site-related fabric connectivity will be re-imported from the site's controller.

Configuring Infra: DCNM Site Settings

This section describes how to configure site-specific Infra settings for on-premises sites.

Procedure

Step 1

Log in to the Cisco Nexus Dashboard Orchestrator GUI.
Step 2

In the left navigation menu, select Infrastructure > Infra Configuration.

Step 3

In the main pane, click Configure Infra.

Step 4

In the left pane, under Sites, select a specific DCNM.

Step 5

In the right <Site> Settings sidebar, specify the Multi-Site VIP.

This address is used for the inter-site L2 BUM and L3 multicast traffic. This IP address is deployed on all border gateway switches that are part of the same fabric.

Note 

If the site you are configuring is part of the DCNM Multi-Site Domain (MDS), this field will be pre-populated with the information imported from DCNM. In this case, changing the value and re-deploying the infra configuration, will impact traffic between the sites that are part of the MDS.

You can choose to Auto Allocate this field, which will allocate the next available address from the Multi-Site Routing Loopback IP Range you defined in previous section.

Step 6

Within the <fabric-name> tile, select the border gateway.

Step 7

In the right <border-gateway> setting sidebar, specify the BGP-EVPN ROUTER-ID and BGW PIP.

For border gateways that are part of a vPC domain, you must also specify a VPC VIP
Step 8

Click Add Port to configure the port that connects to the IPN.

Note 

This release does not support importing the port configuration from the DCNM. If the site you are configuring is already part of the DCNM Multi-Site Domain (MDS), you must use the same values that are already configured in DCNM.

Provide the following information specific to your deployment for the port that connects this border gateway to a core switch or another border gateway:

  • From the Ethernet Port ID dropdown, select the port that connects to the IPN.

  • In the IP Address field, enter the IP address and netmask.

  • In the Remote Address field, provide the IP address of the remote device to which the port is connected.

  • In the Remote ASN field, provide the remote site's ID.

  • In the MTU field, enter the port's MTU.

    MTU of the spine port should match MTU on IPN side.

    You can specify either inherit or a value between 576 and 9000.

  • For BGP Authentication, you can pick either None or Simple (MD5).

    If you select Simple, you must also provide the Authentication Key.

Deploying Infra Configuration

This section describes how to deploy the Infra configuration to each DCNM site.

Before you begin

You must have the general and site-specific infra configurations completed as described in previous sections of this chapter.

Procedure

Step 1

Ensure that there are no configuration conflicts or resolve them if necessary.

The Deploy button will be disabled and a warning will be displayed if there are any configuration conflicts from the already configured settings in each site. For example, if a VRF or network with the same name exists in multiple sites but uses different VNI in each site.

In case of configuration conflicts:

  1. Click Click to View link in the conflict notification pop-up.

  2. Note down the specific configurations that are causing the conflicts.

    For example, in the following report, there are ID mismatches between VRFs and networks in fab1 and fab2 sites.

  3. Click the X button to close the report, then exit Infra configuration screen.

  4. Unmanage the site in NDO, as described in Removing Sites.

    You do not need to remove the site from the Nexus Dashboard, simply unmanage it in NDO GUI.

  5. Resolve the existing configuration conflicts.

  6. Manage the site again, as described in Adding Cisco DCNM Sites.

    Since the site is already added in Nexus Dashboard, simply enable it for management in NDO.

  7. Verify that all conflicts are resolved and the Deploy button is available.

Step 2

Deploy configuration.

  1. In the top right of the Fabric Connectivity Infra screen, choose the appropriate Deploy option to deploy the configuration.

    If you are configuring only DCNM sites, simply click Deploy to deploy the Infra configuration.

  2. Wait for configuration to be deployed.

    When you deploy infra configuration, NDO will signal the DCNM to configure the underlay and the EVPN overlay between the border gateways.

    When configuration is successfully deployed, you will see a green checkmark next to the site in the Fabric Connectivity Infra screen: