FCoE Configuration Between VIC Adapter on UCS Rack Server and Nexus 5500 Switch

Introduction

This document describes how to configure Fibre Channel over Ethernet (FCoE) connectivity between Virtual Interface Card (VIC) adapters installed on Cisco Unified Computing System (UCS) rack servers and Cisco Nexus 5500 (Nexus 5K) Series Switches. Two sample topologies include Cisco Fabric Extender (FEX) devices, and one sample topology does not.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Cisco Nexus 5500 Series Switches
• Cisco UCS rack server - Cisco Integrated Management Controller (CIMC) configuration
• Basic understanding of FCoE

This configuration exists in the environment:

• The storage area network (SAN) array is configured in order to provide access to logical unit numbers (LUNs) for virtual host bus adapters (vHBAs) on the Cisco VIC card.
• Cisco Nexus Version 5.1(3)N1(1a) or later is installed on the Cisco Nexus 5K switches.
• The necessary licenses are installed on the Nexus 5K switches.

Refer to the configuration guide of each specific hardware device for limitations and compatibility information.

Components Used

The information in this document is based on these software and hardware versions:

• Cisco UCS C220 M3 server with VIC 1225
• CIMC 1.5.4 firmware bundle
• Nexus 5548UP with Version 6.0(2)N2(1)
• FEX model - Nexus 2232PP

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Overview

This is an overview of the configuration process:

1. For a new setup, perform the basic configuration on Nexus 5K switches. You can skip this step if you are adding a server to an existing network.
2. Configure the VIC adapter on the UCS server.
3. Configure the Nexus 5K switches based on your topology.

The server can be deployed in one of these three topologies:

• Topology 1 - Virtual port channel (vPC) configured on Nexus 5K switches, and converged network adapter (CNA) ports directly connected to a Nexus 5K switch
• Topology 2 - FEX connected to a Nexus 5K switch in straight-through mode, and CNA connected to FEX ports
• Topology 3 - FEX connected to a Nexus 5K switch in enhanced vPC mode, and CNA connected to FEX ports

The document provides sample configurations for FCoE connectivity for each topology.

Notes:

Use the Command Lookup Tool (registered customers only) in order to obtain more information on the commands used in this section.

The Output Interpreter Tool (registered customers only) supports certain show commands. Use the Output Interpreter Tool in order to view an analysis of show command output.

Step 1: Configure New Setup

Even though this document focuses on the FCoE configuration on VIC adapters and Nexus 5K switches, it also includes the required basic configuration on the vPC and of the FCoE features for completeness.

The topology in this example uses Virtual Storage Area Network (VSAN) 100 and VSAN 200 as two SAN fabrics. These VSANs are created on the A side and the B side of the Nexus 5K switch and are mapped to FCoE VLANS 100 and 200, respectively.

For a new setup, this procedure describes how to perform the basic configuration on the Nexus 5K switches:

1. Enable the required features (Link Layer Discovery Protocol [LLDP], Link Aggregation Control Protocol [LACP], FCoE, FEX, vPC) on the Nexus 5K switches. See Required Features Configuration for an example of this step.
2. Complete the basic vPC configuration:
   
   1. Create the vPC domain.
   2. Create the vPC peer keepalive links.
   3. Configure the vPC peer links. See Basic vPC Configuration for an example of how to perform Steps 1-3 and how to enable the vPC on both Nexus 5K switches.
   4. Create the vPC, or re-use the port channels.
   5. Verify the vPC.
3. Configure the VSANs. See VSAN Configuration for an example of this step.

Note: You can skip these steps if you are adding a server to an existing network; proceed to Step 2: Configure UCS Server and VIC Adapter.

Required Features Configuration

5548-switch# configure terminal
5548-switch(config)# feature lldp
5548-switch(config)# feature lacp 
5548-switch(config)# feature fcoe
5548-switch(config)# feature fex
5548-switch(config)# feature vpc
5548-switch(config)# end

Basic vPC Configuration

5548-switch# configure terminal
5548-switch(config)# vpc domain 150
5548-switch (config-vpc-domain)# peer-keepalive destination <peer-switch-ip> 
   source <source-ip> vrf management
5548-switch(config-vpc-domain)# interface port-channel 150
5548-switch(config-if)# switchport mode trunk
5548-switch(config-if)# vpc peer-link
5548-switch(config-if)# no shut

VSAN Configuration

5548-switch-A # configure terminal
5548-switch-A(config)# vsan database
5548-switch-A(config-vsan-db)#vsan 100
5548-switch-A(config)#exit

5548-switch-A(config)#vlan 100
5548-switch-A(config-vlan)#fcoe vsan 100

5548-switch-B # configure terminal
5548-switch-B(config)#vsan database
5548-switch-B(config-vsan-db)#vsan 200
5548-switch-B(config)#exit

5548-switch-B(config)#vlan 200
5548-switch-B(config-vlan)#fcoe vsan 200

Step 2: Configure UCS Server and VIC Adapter

Configure the VIC adapter on the UCS server. This procedure describes how to apply this configuration to all Cisco VIC (P81E or VIC 1225 or VIC 1225T) adapters:

1. Use the Host Update Utility in order to update the firmware of the server hardware components to the latest version. Be sure to update the VIC firmware.
2. By default, the VIC adapter operates in Classical Ethernet (CE) mode and is configured with two virtual network interface controllers (vNICs) and two vHBAs. Each of these vNICs and vHBAs are pinned to an uplink DCE physical interface; that is, fc0 is pinned to uplink DCE port 0, and fc1 is pinned to uplink DCE port 1 of the adapter:
   
   
3. Power on the server, and note that the VIC adapter details are populated in the inventory section.
4. Click the vHBAs tab, and select a vHBA interface (fc0 or fc1).
5. Click properties, then configure the FCoE properties on the VIC adapter. Be sure to define the FCoE VLAN ID in the Default VLAN text field:
   
   
   
   If you plan to boot from the SAN, check the FC SAN Boot checkbox, and add the Target WWPN to the boot table:
   
   
6. Reboot the server for the configuration changes to take effect.

Step 3: Configure Nexus 5K Switches for Different Topologies

The server can be deployed in one of three topologies. This section describes how to configure the Nexus 5K switches based on your topology.

Common configuration guidelines for all three topologies are:

1. The Virtual Fibre Channel (VFC) interface is bound to a member port (physical Ethernet interface) of the port channel.
2. Switchports are in trunk mode, and the FCoE VLAN is configured as a non-native VLAN.
3. The spanning-tree type of switch ports are configured so they connect to the server as an edge trunk.

This document provides sample configurations for Fabric A; you should perform a similar configuration on the other switch for Fabric B.

Topology 1: CNA Ports Directly Connected to Nexus 5K Switch

Overview

In this scenario, the Nexus 5K switches are in the vPC, and the server CNA ports are directly connected to both Nexus 5K switches. This procedure describes the configuration for this topology:

1. Associate the Ethernet interface to the new port-channel interface.
2. Configure the port channel as a trunk with the FCoE VLAN tagged, and associate the port channel with the vPC ID.
3. Create the VFC interface, and bind it to the port-channel member (Ethernet interface).
4. Associate the VFC interface to the corresponding VSAN ID. See Sample Configuration on A Side of Nexus 5K Switch for an example of Steps 1-4.
5. Repeat Steps 1-4 in order to configure the B side of the Nexus 5K switch.
6. Verify the configuration.

Sample Configuration on A Side of Nexus 5K Switch

N5548UP-A(config)# int eth 1/1
N5548UP-A(config-if)# channel-group 1
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit
N5548UP-A# configure terminal 
N5548UP-A(config)# interface port-channel 1
N5548UP-A(config-if)# switchport mode trunk
N5548UP-A(config-if)# switchport trunk native vlan 10
N5548UP-A(config-if)# switchport trunk allowed vlan 10,100
N5548UP-A(config-if)# spanning-tree port type edge trunk
N5548UP-A(config-if)# vpc 1
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit
N5548UP-A(config)# int vfc 1
N5548UP-A(config-if)# bind interface ethernet 1/1
N5548UP-A(config-if)# switchport trunk allowed vsan 100
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exitN5548UP-A(config)# vsan database
N5548UP-A(config-vsan-db)# vsan 100 interface vfc 1
N5548UP-A(config-vsan-db)# end 

Verification

N5548UP-A# show run int port-channel 1 membership

interface port-channel 1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 spanning-tree port type edge trunk
 speed 10000
 vpc 1
 interface Ethernet1/1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 channel-group 1
 
N5548UP-A# show run int vfc 1
interface vfc1
 bind interface Ethernet1/1
 switchport trunk allowed vsan 100
 no shutdown

Topology 2: FEX Connected to Nexus 5K Switch in Straight-Through Mode and CNA Connected to FEX Ports

Overview

In this scenario, the FEX is connected to the Nexus 5K switch in straight-through mode, and the CNA is connected to the FEX ports. This procedure describes the configuration for this topology:

Straight-through FEX configuration:

1. Assign a FEX ID.
2. Associate the Ethernet interface with the new port-channel interface ID, and configure the new port channel with the FEX fabric parameters. See Sample Configuration for an example of Steps 1-2.
   
   Host vPC configuration:
   
   
3. Associate the FEX host interfaces (HIF) with the new port-channel interface ID, and configure the port-channel parameters.
4. Associate the HIF port channel to the vPC interface.
5. Create the VFC interface, and bind it to the member (Ethernet) port of the host port channel. See Host vPC Configuration for an example of Steps 3-5.
6. Verify the configuration.

Repeat Steps 1-4 on the B side of the Nexus 5K switch, and use the same vPC interface number under the HIF port channel.

Sample Configuration

N5548UP-A# configure terminal 
N5548UP-A(config)# fex 101
N5548UP-A(config-fex)# exit

N5548UP-A(config)# interface ethernet 1/5 - 6
N5548UP-A(config-if-range)# channel-group 210
N5548UP-A(config-if-range)#no shut
N5548UP-A(config-if-range)# exit

N5548UP-A(config)# interface port-channel 210
N5548UP-A(config-if)# switchport mode fex-fabric
N5548UP-A(config-if)# fex associate 101 
N5548UP-A(config-if)#no shut
N5548UP-A(config-if)# exit

Host vPC Configuration

<---Create HIF port-channel-->

N5548UP-A(config-if)# interface ethernet 101/1/1
N5548UP-A(config-if)# channel-group 1
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

<---Configure port-channel parameters-->

N5548UP-A(config)# interface port-channel 1
N5548UP-A(config-if)#switchport mode trunk
N5548UP-A(config-if)# switchport trunk native vlan 10
N5548UP-A(config-if)# switchport trunk allowed vlan 10,100
N5548UP-A(config-if)# spanning-tree port type edge trunk 
N5548UP-A(config-if)# vpc 10
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

<---Create vfc interfvace and bind to port-channel (Ethernet interface)-->

N5548UP-A(config)# int vfc 1
N5548UP-A(config-if)# bind interface ethernet 101/1/1
N5548UP-A(config-if)# switchport trunk allowed vsan 100
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

<---Associate vfc interface to corresponding VSAN ID-->

N5548UP-A(config)# vsan database
N5548UP-A(config-vsan-db)# vsan 100 interface vfc 1
N5548UP-A(config-vsan-db)# end

Verification

N5548UP-A# show run int port-channel 210 membership

interface port-channel210
 switchport mode fex-fabric
 fex associate 101

interface Ethernet1/5
 switchport mode fex-fabric
 fex associate 101
 channel-group 210

interface Ethernet1/6
 switchport mode fex-fabric
 fex associate 101
 channel-group 210

N5548UP-A# show int fex
 Fabric Fabric Fex FEX 
Fex Port Port State Uplink Model Serial 
---------------------------------------------------------------
101 Eth1/5 Active 1 N2K-C2232PP-10GE SSI142602QN
101 Eth1/6 Active 2 N2K-C2232PP-10GE SSI142602QN

N5548UP-A# show run int port-channel 1 membership

interface port-channel 1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 spanning-tree port type edge trunk
 speed 10000
 vpc 10

interface Ethernet101/1/1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 channel-group 1

N5548UP-A# show run int vfc 1
interface vfc1
 bind interface Ethernet101/1/1
 switchport trunk allowed vsan 100
 no shutdown

Topology 3: FEX Connected to Nexus 5K Switch in Enhanced vPC Mode and CNA Connected to FEX Ports

Overview

In this scenario, the FEX is connected to the Nexus 5K switch in enhanced vPC mode, and the CNA is connected to the FEX ports.

SAN Fabric A traffic (FCoE VLAN 100) is associated with FEX 101 / A side of the Nexus 5K switch. SAN Fabric B traffic (FCoE VLAN 200) is associated with FEX 102 / B side of the Nexus 5K switch.

For a two member host port channel, you must configure HIF ports on both FEX devices on both Nexus 5K switches; that is, a total of four HIF port configurations are required.

This procedure describes the configuration for this topology:

Active - Active FEX configuration:

1. Assign the FEX ID.
2. Add the 'fcoe' keyword under a single FEX on each Nexus 5K switch in order to enable SAN fabric isolation.
3. Create the FEX fabric port channel.
4. Repeat Steps 1 and 3 for the other FEX on the same Nexus 5K switch.
5. Repeat Steps 1-4 on the B side of the Nexus 5K switch. See Active - Active FEX Configuration and FCoE Keyword on FEX 102 on B Side of Nexus 5K Switch for an example.
   
   Host vPC configuration:
   
   
6. Create the port-channel interface, and configure the necessary VLANs.
7. Configure the HIF port (the FEX interface connected to the CNA) on both FEXs, and allow the FCoE VLAN and Ethernet VLANs, then associate the HIF port with the host port-channel ID created in Step 6.
8. Create the VFC interface, and bind it to the member (Ethernet) port of the host port channel.
9. Repeat Steps 6-8 on the B side of the Nexus 5K switch, and use the same host port-channel interface ID. See Host vPC Configuration for an example.
   
   VFC interface configuration:
   
   
10. Create the VC interface, and bind it to the port-channel member (HIF) port.
11. Assign the VFC interface to the appropriate VSAN.
12. Repeat Steps 10-11 on the B side of the Nexus 5K switch. See VFC Configuration for an example.
13. Verify the configuration.

Active - Active FEX Configuration

N5548UP-A# configure terminal 
N5548UP-A(config)# fex 101
N5548UP-A(config-fex)# fcoe
N5548UP-A(config-fex)# exit

N5548UP-A(config)# interface ethernet 1/3 - 4
N5548UP-A(config-if-range)# channel-group 201
N5548UP-A(config-if-range)#no shut
N5548UP-A(config-if-range)# exit
N5548UP-A(config)# interface port-channel 201
N5548UP-A(config-if)# switchport mode fex-fabric
N5548UP-A(config-if)# fex associate 101 
N5548UP-A(config-if)# vpc 201
N5548UP-A(config-if)#no shut
N5548UP-A(config-if)# exit

N5548UP-A# configure terminal 
N5548UP-A(config)# fex 102
N5548UP-A(config-fex)# exit

N5548UP-A(config)# interface ethernet 1/5 - 6
N5548UP-A(config-if-range)# channel-group 202
N5548UP-A(config-if-range)#no shut
N5548UP-A(config-if-range)# exit

N5548UP-A(config)# interface port-channel 202
N5548UP-A(config-if)# switchport mode fex-fabric
N5548UP-A(config-if)# fex associate 102 
N5548UP-A(config-if)# vpc 202
N5548UP-A(config-if)#no shut
N5548UP-A(config-if)# exit

FCoE Keyword on FEX 102 on B Side of Nexus 5K Switch

N5548UP-B# configure terminal 
N5548UP-B(config)# fex 102
N5548UP-B(config-fex)# fcoe
N5548UP-B(config-fex)# exit

Host vPC Configuration

N5548UP-A(config-if)# interface ethernet 101/1/1
N5548UP-A(config-if)# channel-group 1
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

N5548UP-A(config)# interface ethernet 102/1/1
N5548UP-A(config-if)# channel-group 1
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

N5548UP-A(config)# interface port-channel 1
N5548UP-A(config-if)#switchport mode trunk
N5548UP-A(config-if)# switchport trunk native vlan 10
N5548UP-A(config-if)# switchport trunk allowed vlan 10,100
N5548UP-A(config-if)# spanning-tree port type edge trunk 
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

VFC Configuration

<---Create vfc interface and bind to port-channel member (ethernet interface)-->

N5548UP-A(config)# int vfc 1
N5548UP-A(config-if)# bind interface ethernet 101/1/1
N5548UP-A(config-if)# switchport trunk allowed vsan 100
N5548UP-A(config-if)# no shut
N5548UP-A(config-if)# exit

<---Associate vfc interface to corresponding VSAN ID-->

N5548UP-A(config)# vsan database
N5548UP-A(config-vsan-db)# vsan 100 interface vfc 1
N5548UP-A(config-vsan-db)# end

Verification

N5548UP-A# show fex
 FEX FEX FEX FEX 
Number Description State Model Serial 
------------------------------------------------------------------------
101 FEX0101 Online N2K-C2232PP-10GE SSI142602QN
102 FEX0102 Online N2K-C2232PP-10GE SSI14260EVV

N5548UP-A# show interface fex
 Fabric Fabric Fex FEX 
Fex Port Port State Uplink Model Serial 
---------------------------------------------------------------
101 Eth1/3 Active 2 N2K-C2232PP-10GE SSI142602QN
102 Eth1/4 Active 1 N2K-C2232PP-10GE SSI14260EVV

N5548UP-A# show fex 101 detail 
FEX: 101 Description: FEX0101 state: Online
 FEX version: 6.0(2)N2(1) [Switch version: 6.0(2)N2(1)]
 FEX Interim version: 6.0(2)N2(1)
 Switch Interim version: 6.0(2)N2(1)
 Extender Serial: SSI142602QN
 FCoE Admin: true
 FCoE Oper: true
 FCoE FEX AA Configured: true

N5548UP-A# show run interface port-channel 1 membership

interface port-channel1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 spanning-tree port type edge trunk
 speed 10000

interface Ethernet102/1/1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 spanning-tree port type edge trunk
 channel-group 1

interface Ethernet101/1/1
 switchport mode trunk
 switchport trunk native vlan 10
 switchport trunk allowed vlan 10,100
 spanning-tree port type edge trunk
 channel-group 1

N5548UP-A# show int vfc 1
vfc1 is trunking
 Bound interface is Ethernet101/1/1
 Port WWN is 20:00:54:7f:ee:e1:6a:bf
 Admin port mode is F, trunk mode is on
 Port mode is TF
 Port vsan is 100
 Trunk vsans (admin allowed and active) (100)
 Trunk vsans (up) (100)
 Trunk vsans (isolated) ()
 Trunk vsans (initializing) ()

N5548UP-A# show run int port-channel 201 membership

interface port-channel201
 switchport mode fex-fabric
 fex associate 101
 vpc 201

interface Ethernet1/3
 switchport mode fex-fabric
 fex associate 101
 channel-group 201

interface Ethernet1/4
 switchport mode fex-fabric
 fex associate 101
 channel-group 201

N5548UP-A# show run int port-channel 202 membership

interface port-channel202
 switchport mode fex-fabric
 fex associate 102
 vpc 202

interface Ethernet1/5
 switchport mode fex-fabric
 fex associate 102
 channel-group 202

interface Ethernet1/6
 switchport mode fex-fabric
 fex associate 102
 channel-group 202

This screen shot from the server boot sequence validates the switch configuration as the VIC firmware successfully detects the LUN and updates the boot table:

Verify

Verification procedures are included within the configuration examples.

Troubleshoot

This section provides information you can use in order to troubleshoot your configuration.

Perform these checks in order to determine if the VFC is coming up:

1. Verify that the cables are connected correctly between adapter ports mapped to vHBA and that the Nexus 5K switch ports have the necessary VSANs.
2. Make sure the interfaces are not administratively shut down.
3. Verify that the FCoE VLAN is configured on the Ethernet interface bounded by the VFC.

Use these commands in order to verify the configuration:

• show run fcoe
• show vsan
• show vlan fcoe
• show interface vfc ID
• show vpc
• show flogi database

Related Information

• Managing Network Adapters section of Cisco UCS C-Series Servers Integrated Management Controller GUI Configuration Guide, Release 1.5
• Configuring Enhanced Virtual Port Channels section of Cisco Nexus 5500 Series NX-OS Layer 2 Switching Configuration Guide, Release 6.x
• Configuring the Fabric Extender section of Cisco Nexus 5500 Series NX-OS Layer 2 Switching Configuration Guide, Release 6.x
• Cisco Nexus 5500 Series NX-OS Fibre Channel over Ethernet Configuration Guide, Release 6.x
• Technical Support & Documentation - Cisco Systems