Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Terminology

What is VXLAN

Why VXLAN

Configure

Network Diagram

Configurations

3172-A

9396-A

9396-B

Verify

Example Outputs

3172-A

9396-A

9396-B

VXLAN Packet Capture

VXLAN PCAP

Troubleshoot

Related Information

Introduction

This document describes a high level overview of Virtual Extensible LAN (VXLAN) and configuration examples with verification commands and output.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Multicast routing concepts such as Rendezvous Point (RP) and Platform Independent Multicast (PIM).

- Virtual Port Channel (vPC) concepts.

This document assumes that the IP routing and multicast routing has been established prior to VXLAN configuration.

Components Used

The information in this document is based on these software and hardware versions:

- Nexus 9396s as vPC Virtual Tunnel Endpoints (VTEPs) that run Version 7.0(3)I1(1b)

- Nexus 3172 that runs Version 6.0(2)U5(1)

- LAN_ENTERPRISE_SERVICES_PKG license installed

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Terminology

VXLAN (Virtual Extensible LAN) - The technology that provides the same Ethernet Layer 2 network services as VLAN does today, but with greater extensibility and flexibility.

VNID (Vxlan Network Identifier) - 24 bit segment ID that defines the broadcast domain. Interchangeable with "VXLAN Segment ID".

VTEP (Virtual Tunnel Endpoint) - This is the device that does the encapsulation and de-encapsulation.

NVE (Network Virtual Interface) - Logical interface where the encapsulation and de-encapsulation occur.

What is VXLAN

- VXLAN is a technology that allows overlaying a Layer 2 (L2) network over a Layer 3 (L3) underlay with use of any IP routing protocol.

- It uses MAC-in-UDP Encapsulation.

VXLAN solves three main problems:

1. 16M VNIs (broadcast domains) versus the 4K offered by traditional VLANs.

2. Allows L2 to be extended anywhere in an IP network.

3. Optimized flooding.

Why VXLAN

- VLAN Scalability - VXLAN extends the L2 Segment ID field to 24-bits, which potentially allows up to 16 million unique L2 segments over the same network.

- L2 Segment Elasticity over L3 Boundary - VXLAN encapsulates an L2 frame in an IP-UDP header, which allows L2 adjacency across router boundaries.

- Leverages multicast in the transport network in order to simulate flooding behavior for broadcast, unknown unicast, and multicast in the L2 segment.

- Leverage Equal Cost Multi-pathing (ECMP) in order to achieve optimal path usage over the transport network.

Configure

Network Diagram

Configurations

These configurations are specific to the VXLAN portion of configuration. Notice that 9396-A and B are in a vPC domain while 3172-A is not. These configurations assume full reachability to all L3 interfaces in the topology with the routing protocol of your choice. Open Shortest Path First (OSPF) was used in this example. It also assumes the multicast routing has been established over these same L3 interfaces.

3172-A

feature ospf

feature pim

feature vn-segment-vlan-based

feature nv overlay

vlan 10

vn-segment 160010

vlan 20

vn-segment 160020

interface nve1

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

no shutdown

interface Ethernet1/3

no switchport

ip address 192.168.1.10/30

ip router ospf 2 area 0.0.0.0

ip pim sparse-mode

interface loopback1

ip address 192.168.2.5/32

ip router ospf 2 area 0.0.0.0

ip pim sparse-mode

9396-A

Note: When vPCs are used as VTEPs, the secondary IP of the loopback interface is used and shared between the two peers. This is how both peers represent themselves as a single VTEP to the remote NVE peers.

feature ospf

feature pim

feature vn-segment-vlan-based

feature nv overlay

ip pim rp-address 192.168.1.100 group-list 224.0.0.0/4

vlan 1,10,20

vlan 10

vn-segment 160010

vlan 20

vn-segment 160020

vpc domain 1

peer-switch

peer-keepalive destination 10.122.140.99

peer-gateway

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel48

switchport mode trunk

vpc 48

interface nve1

mtu 9216

no shutdown

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

interface Ethernet1/7

no switchport

ip address 192.168.1.2/30

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface loopback1

ip address 192.168.2.2/32

ip address 192.168.2.1/32 secondary

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

9396-B

Note: When vPCs are used as VTEPs, the secondary IP of the loopback interface is used and shared between the two peers. This is how both peers represent themselves as a single VTEP to the remote NVE peers.

feature ospf

feature pim

feature vn-segment-vlan-based

feature nv overlay

ip pim rp-address 192.168.1.100 group-list 224.0.0.0/4

vlan 1,10,20

vlan 10

vn-segment 160010

vlan 20

vn-segment 160020

vpc domain 1

peer-switch

peer-keepalive destination 10.122.140.98

peer-gateway

interface port-channel1

switchport mode trunk

spanning-tree port type network

vpc peer-link

interface port-channel48

switchport mode trunk

vpc 48

interface nve1

mtu 9216

no shutdown

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

interface Ethernet1/7

no switchport

ip address 192.168.1.6/30

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

no shutdown

interface loopback1

ip address 192.168.2.3/32

ip address 192.168.2.1/32 secondary

ip router ospf 1 area 0.0.0.0

ip pim sparse-mode

Verify

Use this section to confirm that your configuration works properly.

The Cisco CLI Analyzer (registered customers only) supports certain show commands. Use the Cisco CLI Analyzer in order to view an analysis of show command output.

- show nve peers < --- you can not see any output for this until traffic is initiated from both sides of the overlay

- show nve vni

- show run interface nve1

- show nve internal platform interface detail (9K only)

- show mac address-table

- show ip mroute detail

Example Outputs

These outputs are in a steady state. The VTEP peers have discovered each other and traffic has passed between both in the encap and decap directions.

3172-A

3172-A# show nve peers

Interface          Peer-IP           Peer-State

----------------   ---------------   -------------

nve1               192.168.2.1        Up

3712-A# show nve vni

Interface          VNI        Multicast-group   VNI State

----------------   --------   ---------------   ---------

nve1               160010     203.0.113.1         Up

nve1               160020     203.0.113.1         Up

3172-A# show run interface nve1

!Command: show running-config interface nve1

!Time: Sat Apr 25 15:09:13 2015

version 6.0(2)U5(1)

interface nve1

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

no shutdown

3172-A# show nve internal platform interface detail

3172-A# show mac address-table vlan 10

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since first seen,+ - primary entry using vPC Peer-Link

VLAN     MAC Address      Type      age     Secure NTFY   Ports/SWID.SSID.LID

---------+-----------------+--------+---------+------+----+------------------

* 10       0000.1111.1111    dynamic   5030       F    F  Eth1/48

* 10       0000.2222.2222    dynamic   5010       F    F  nve1(192.168.2.1)

3172-A# show ip mroute detail

IP Multicast Routing Table for VRF "default"

Total number of routes: 3

Total number of (*,G) routes: 1

Total number of (S,G) routes: 1

Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 3w3d, static(1) pim(0) ip(0)

Stats: 15/1539 [Packets/Bytes], 0.000   bps

Incoming interface: Ethernet1/3, RPF nbr: 192.168.1.9, uptime: 1w0d

Outgoing interface list: (count: 1)

loopback1, uptime: 3w3d, static

(192.168.2.5/32, 231.1.1.1/32), uptime: 3w3d, ip(0) mrib(1) pim(1)

Stats: 142751/9136064 [Packets/Bytes], 34.133  bps

Incoming interface: loopback1, RPF nbr: 192.168.2.5, uptime: 3w3d

Outgoing interface list: (count: 2)

Ethernet1/3, uptime: 1w0d, pim

loopback1, uptime: 3w3d, mrib, (RPF)

(*, 232.0.0.0/8), uptime: 3w3d, pim(0) ip(0)

Stats: 0/0 [Packets/Bytes], 0.000   bps

Incoming interface: Null, RPF nbr: 0.0.0.0, uptime: 3w3d

Outgoing interface list: (count: 0)

9396-A

9396-A# show nve peers

Interface Peer-IP          State LearnType Uptime   Router-Mac

--------- ---------------  ----- --------- -------- -----------------

nve1      192.168.2.5      Up    DP        2d20h    n/a

9396-A# show nve vni

Codes: CP - Control Plane        DP - Data Plane

UC - Unconfigured         SA - Suppress ARP

Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags

--------- -------- ----------------- ----- ---- ------------------ -----

nve1      160010   203.0.113.1         Up    DP   L2 [10]

nve1      160020   203.0.113.1         Up    DP   L2 [20]

9396-A# show run interface nve1

!Command: show running-config interface nve1

!Time: Sat Apr 25 15:20:45 2015

version 7.0(3)I1(1a)

interface nve1

mtu 9216

no shutdown

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

9396-A# show nve internal platform interface detail

Printing details of all NVE Interfaces

|======|=========================|===============|===============|=====|=====|

|Intf  |State                    |PriIP          |SecIP          |Vnis |Peers|

|======|=========================|===============|===============|=====|=====|

|nve1  |UP                       |192.168.2.2    |192.168.2.1    |2    |1    |

|======|=========================|===============|===============|=====|=====|

SW_BD/VNIs of interface nve1:

================================================

|======|======|=========================|======|====|======|

|Sw BD |Vni   |State                    |Intf  |Type|Vrf-ID|

|======|======|=========================|======|====|======|

|10    |160010|UP                       |nve1  |DP  |0

|20    |160020|UP                       |nve1  |DP  |0

|======|======|=========================|======|====|======|

Peers of interface nve1:

============================================

peer_ip: 192.168.2.5, peer_id: 1, state: UP MAC-learning: Enabled

active_swbds:

add_pending_swbds:

rem_pending_swbds:

9396-A# show mac address-table vlan 10

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN     MAC Address      Type      age     Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

+   10     0000.1111.1111   dynamic  0         F      F    nve1(192.168.2.5)

*   10     0000.2222.2222   dynamic  0         F      F    Po48

G    -     7c0e.ceca.f177   static   -         F      F    sup-eth1(R)

9396-A# show ip mroute detail

IP Multicast Routing Table for VRF "default"

Total number of routes: 4

Total number of (*,G) routes: 1

Total number of (S,G) routes: 2

Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 2d21h, nve(1) ip(0) pim(0)

Data Created: No

Stats: 1/64 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.1

Outgoing interface list: (count: 1)

nve1, uptime: 2d21h, nve

(192.168.2.1/32, 203.0.113.1/32), uptime: 2d21h, nve(0) ip(0) mrib(0) pim(0)

Data Created: Yes

VXLAN Flags

VXLAN Encap

Stats: 1/51 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: loopback1, RPF nbr: 192.168.2.1

Outgoing interface list: (count: 0)

(192.168.2.5/32, 203.0.113.1/32), uptime: 2d21h, ip(0) mrib(0) nve(1) pim(0)

Data Created: Yes

Stats: 16474/1370086 [Packets/Bytes], 13.600  bps

Stats: Active Flow

Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.1

Outgoing interface list: (count: 1)

nve1, uptime: 2d21h, nve

(*, 232.0.0.0/8), uptime: 2d21h, pim(0) ip(0)

Data Created: No

Stats: 0/0 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: Null, RPF nbr: 0.0.0.0

Outgoing interface list: (count: 0)

9396-A# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1

Peer status                       : peer adjacency formed ok

vPC keep-alive status             : peer is alive

Configuration consistency status  : success

Per-vlan consistency status       : success

Type-2 consistency status         : success

vPC role                          : secondary

Number of vPCs configured         : 1

Peer Gateway                      : Enabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Auto-recovery status              : Disabled

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans

--   ----   ------ --------------------------------------------------

1    Po1    up     1,10,20

vPC status

----------------------------------------------------------------------

id   Port   Status Consistency Reason                     Active vlans

--   ----   ------ ----------- ------                     ------------

48   Po48   up     success     success                    1,10

9396-B

9396-B# show nve peers

Interface Peer-IP          State LearnType Uptime   Router-Mac

--------- ---------------  ----- --------- -------- -----------------

nve1      192.168.2.5      Up    DP        1w0d     n/a

9396-B# show nve vni

Codes: CP - Control Plane        DP - Data Plane

UC - Unconfigured         SA - Suppress ARP

Interface VNI      Multicast-group   State Mode Type [BD/VRF]      Flags

--------- -------- ----------------- ----- ---- ------------------ -----

nve1      160010   203.0.113.1         Up    DP   L2 [10]

nve1      160020   203.0.113.1         Up    DP   L2 [20]

9396-B# show run interface nve1

!Command: show running-config interface nve1

!Time: Sat Apr 25 15:23:25 2015

version 7.0(3)I1(1b)

interface nve1

mtu 9216

no shutdown

source-interface loopback1

member vni 160010 mcast-group 203.0.113.1

member vni 160020 mcast-group 203.0.113.1

9396-B# show nve internal platform interface detail

Printing details of all NVE Interfaces

|======|=========================|===============|===============|=====|=====|

|Intf  |State                    |PriIP          |SecIP          |Vnis |Peers|

|======|=========================|===============|===============|=====|=====|

|nve1  |UP                       |192.168.2.3    |192.168.2.1    |2    |1    |

|======|=========================|===============|===============|=====|=====|

SW_BD/VNIs of interface nve1:

================================================

|======|======|=========================|======|====|======|

|Sw BD |Vni   |State                    |Intf  |Type|Vrf-ID|

|======|======|=========================|======|====|======|

|10    |160010|UP                       |nve1  |DP  |0

|20    |160020|UP                       |nve1  |DP  |0

|======|======|=========================|======|====|======|

Peers of interface nve1:

============================================

peer_ip: 192.168.2.5, peer_id: 1, state: UP MAC-learning: Enabled

active_swbds:

add_pending_swbds:

rem_pending_swbds:

9396-B# show mac address-table vlan 10

Legend:

* - primary entry, G - Gateway MAC, (R) - Routed MAC, O - Overlay MAC

age - seconds since last seen,+ - primary entry using vPC Peer-Link,

(T) - True, (F) - False

VLAN     MAC Address      Type      age     Secure NTFY Ports

---------+-----------------+--------+---------+------+----+------------------

*   10     0000.1111.1111   dynamic  0         F      F    nve1(192.168.2.5)

+   10     0000.2222.2222   dynamic  0         F      F    Po48

G    -     58f3.9ca3.64dd   static   -         F      F    sup-eth1(R)

9396-B# show ip mroute detail

IP Multicast Routing Table for VRF "default"

Total number of routes: 4

Total number of (*,G) routes: 1

Total number of (S,G) routes: 2

Total number of (*,G-prefix) routes: 1

(*, 231.1.1.1/32), uptime: 2w1d, nve(1) ip(0) pim(0)

Data Created: No

VXLAN Flags

VXLAN Decap

VPC Flags

RPF-Source Forwarder

Stats: 1/64 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.5

Outgoing interface list: (count: 1)

nve1, uptime: 2w1d, nve

(192.168.2.1/32, 203.0.113.1/32), uptime: 2w1d, nve(0) ip(0) mrib(0) pim(1)

Data Created: Yes

VXLAN Flags

VXLAN Encap

VPC Flags

RPF-Source Forwarder

Stats: 5/511 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: loopback1, RPF nbr: 192.168.2.1

Outgoing interface list: (count: 1)

Ethernet1/7, uptime: 1w0d, pim

(192.168.2.5/32, 203.0.113.1/32), uptime: 2w1d, ip(0) mrib(0) pim(0) nve(1)

Data Created: Yes

VXLAN Flags

VXLAN Decap

VPC Flags

RPF-Source Forwarder

Stats: 86621/7241564 [Packets/Bytes], 13.600  bps

Stats: Active Flow

Incoming interface: Ethernet1/7, RPF nbr: 192.168.1.5

Outgoing interface list: (count: 1)

nve1, uptime: 2w1d, nve

(*, 232.0.0.0/8), uptime: 2w1d, pim(0) ip(0)

Data Created: No

Stats: 0/0 [Packets/Bytes], 0.000   bps

Stats: Inactive Flow

Incoming interface: Null, RPF nbr: 0.0.0.0

Outgoing interface list: (count: 0)

9396-B# show vpc

Legend:

(*) - local vPC is down, forwarding via vPC peer-link

vPC domain id                     : 1

Peer status                       : peer adjacency formed ok

vPC keep-alive status             : peer is alive

Configuration consistency status  : success

Per-vlan consistency status       : success

Type-2 consistency status         : success

vPC role                          : primary

Number of vPCs configured         : 1

Peer Gateway                      : Enabled

Dual-active excluded VLANs        : -

Graceful Consistency Check        : Enabled

Auto-recovery status              : Disabled

vPC Peer-link status

---------------------------------------------------------------------

id   Port   Status Active vlans

--   ----   ------ --------------------------------------------------

1    Po1    up     1,10,20

vPC status

----------------------------------------------------------------------

id   Port   Status Consistency Reason                     Active vlans

--   ----   ------ ----------- ------                     ------------

48   Po48   up     success     success                    1,10

VXLAN Packet Capture

The packet capture (PCAP) is from the previous topology and contains the OSPF hellos, the PIM Joins/Registers, and the VXLAN encapsulated traffic for the toplogy shown in the network diagram. You can notice some Internet Control Message Protocol (ICMP) flags such as 'no response'. This is due to the nature of the monitor session completed on the RP.

The monitor session included interfaces Eth4/17-18 and Eth4/20, so it throws off Wireshark some. The important information is the format and flags.

Note: All encapsulated packets (BUM, or known unicast) are sourced from the VTEP loopback IP destined to the remote VTEP loopback IP. This is the secondary loopback IP on any vPC VTEPs.

BUM (Broadcast, Unknown unicast, Multicast) traffic can be destined to the mcast-group.

Unicast traffic is then destined to the remote VTEP loopback IP.

Troubleshoot

There is currently no specific information available to troubleshoot this configuration.

Related Information

- VXLAN Overview: Cisco Nexus 9000 Series Switches

- Technical Support & Documentation - Cisco Systems