Introduction
This document describes how the 3rd generation Cisco Unified Computing System Fabric Interconnect (UCS-FI-6332-16UP) contains 16x10GE unified ports,18x40GE breakout capable ports, and 6x40GE ports. Certain configuration changes require a reboot of the fabric interconnect. Multiple reboots consume valuable time during initial system setup. It explains a process to complete all configuration changes with a single reboot of each fabric interconnect. Specifically this document covers configuring System QoS, fibre channel interfaces, and 4x10GE breakout interfaces. You may as well wish to add additional steps of setting Ethernet and Fibre Channel (FC) switching modes.
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- Cisco Unified Computing System (UCS) Manager
- Cisco Unified Computing System (UCS) Manager Command Line Interface (CLI)
Components Used
The information in this document is based on these software and hardware versions:
- UCS Manager version 3.1(1) or later
- UCS-FI-6332-16UP
- UCS-FI-6332 (Does not support Unified Ports)
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.
Configure
Network Diagram
These are the configurations of the document:
- Set the best-effort class for MTU 9216.
- Configure ports 1/1-1/6 as Fibre Channel.
- Configure ports 1/31-1/34 as 4x10GE Breakout Ports.
System QoS Policy
Configure the best-effort system QoS policy for jumbo framing. Unlike previous fabric interconnect generations the 6300 series requires a reload when modifying global QoS policies.
UCS-6332-A# scope eth-server
UCS-6332-A /eth-server # scope qos
UCS-6332-A /eth-server/qos # scope eth-best-effort
UCS-6332-A /eth-server/qos/eth-best-effort # set mtu 9216
UCS-6332-A /eth-server/qos/eth-best-effort *# top
Unified Ports for Fibre Channel
Unified Port are numbered 1/1-16 on the 6332-16UP. Fibre Channel port modes must be configured such that the last FC port occupies port id 6,12, or 16.This results in a minimum of six FC ports and a maximum of sixteen.
Valid port combinations are:
- Ethernet: 1/1-1/16
- FC: 1/1-1/6 Ethernet: 1/7-1/16
- FC: 1/1-1/12 Ethernet: 1/13-1/16
- FC: 1/1-16
UCS-6332-A *# scope fc-uplink
UCS-6332-A /fc-uplink *# scope fabric a
UCS-6332-A /fc-uplink/fabric* # create interface 1 1
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 2
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 3
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 4
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 5
UCS-6332-A /fc-uplink/fabric/interface* # up
UCS-6332-A /fc-uplink/fabric* # create interface 1 6
UCS-6332-A /fc-uplink/fabric/interface* # top
Breakout Ports 4x10GE Connectivity
Breakout ports are used to connect 40GE interfaces to 10GE capable equipment. These can be used as Uplink ports connecting to a 10G switch or as Server ports connecting to a 22XX IOM module or as FCoE ports.
- Ethernet 1/17-1/34 are capable of breakout configuration.
- Ethernet 1/35-1/40 are fixed at 40GE.
- An individual 1x10GE interface in a breakout port is capable of FCoE.
Note: If jumbo QoS is configured globally then only 4 physical ports can be configured for breakout.
UCS-6332-A *# scope cabling
UCS-6332-A /cabling *# scope fabric a
UCS-6332-A /cabling/fabric *# create breakout 1 31
Warning: Port breakout create action reboots FI and any existing configurations on 40G port will be erased.!
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 32
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 33
UCS-6332-A /cabling/fabric/breakout* # up
UCS-6332-A /cabling/fabric # create breakout 1 34
UCS-6332-A /fc-uplink/fabric/interface* # top
Save the Configuration
Warning: The Fabric Interconnect will immediately reload when the changes are committed.
UCS-6332-A* # commit-buffer
*The switch will now reboot.
Repeat the changes on Fabric Interconnect B.
Verify
Confirm ports FC 1/1-6 are configured for Fibre Channel and ports Ethernet 1/31-34 are in Breakout mode. Notice the breakout ports now have 4 subinterfaces. In this example, ports 1/33/1 and 1/33/2 are configured for FCoE and 1/33/3-4 as Uplink interfaces.
UCS-6332-A# scope fabric-interconnect a
UCS-6332-A /fabric-interconnect # show port
Ether Port:
Slot Aggr Port Port Oper State Mac Role Xcvr
----- ---------- ----- ---------------- -------------------- ------- ----
[...]
1 0 30 Sfp Not Present 8C:60:4F:BC:C4:D0 Unknown N/A
1 0 35 Sfp Not Present 8C:60:4F:BC:C4:E4 Unknown N/A
1 0 36 Sfp Not Present 8C:60:4F:BC:C4:E5 Unknown N/A
1 0 37 Sfp Not Present 8C:60:4F:BC:C4:E6 Unknown N/A
1 0 38 Sfp Not Present 8C:60:4F:BC:C4:E7 Unknown N/A
1 0 39 Sfp Not Present 8C:60:4F:BC:C4:E8 Unknown N/A
1 0 40 Sfp Not Present 8C:60:4F:BC:C4:E9 Unknown N/A
1 31 1 Sfp Not Present 8C:60:4F:BC:C4:D4 Unknown N/A
1 31 2 Sfp Not Present 8C:60:4F:BC:C4:D5 Unknown N/A
1 31 3 Sfp Not Present 8C:60:4F:BC:C4:D6 Unknown N/A
1 31 4 Sfp Not Present 8C:60:4F:BC:C4:D7 Unknown N/A
1 32 1 Sfp Not Present 8C:60:4F:BC:C4:D8 Unknown N/A
1 32 2 Sfp Not Present 8C:60:4F:BC:C4:D9 Unknown N/A
1 32 3 Sfp Not Present 8C:60:4F:BC:C4:DA Unknown N/A
1 32 4 Sfp Not Present 8C:60:4F:BC:C4:DB Unknown N/A
1 33 1 Up 8C:60:4F:BC:C4:DC Fcoe Uplink
QSFP 40G SR4
1 33 2 Up 8C:60:4F:BC:C4:DD Fcoe Uplink
QSFP 40G SR4
1 33 3 Up 8C:60:4F:BC:C4:DE Network N/A
1 33 4 Up 8C:60:4F:BC:C4:DF Network N/A
1 34 1 Sfp Not Present 8C:60:4F:BC:C4:E0 Unknown N/A
1 34 2 Sfp Not Present 8C:60:4F:BC:C4:E1 Unknown N/A
1 34 3 Sfp Not Present 8C:60:4F:BC:C4:E2 Unknown N/A
1 34 4 Sfp Not Present 8C:60:4F:BC:C4:E3 Unknown N/A
FC Port:
Slot Port Oper State Wwn
----- ----- ---------------- ---
1 1 Up 20:01:8C:60:4F:BC:C4:80
1 2 Up 20:02:8C:60:4F:BC:C4:80
1 3 Sfp Not Present 20:03:8C:60:4F:BC:C4:80
1 4 Sfp Not Present 20:04:8C:60:4F:BC:C4:80
1 5 Sfp Not Present 20:05:8C:60:4F:BC:C4:80
1 6 Sfp Not Present 20:06:8C:60:4F:BC:C4:80
In NXOS, breakout ethernet ports are named as br-ethernet x/y/z.
UCS-6332-A# # connect nxos a
UCS-6332-A(nxos)# show int br-ethernet 1/33/1
Br-Ethernet1/33/1 is up
Dedicated Interface
Hardware: 10000 Ethernet, address: 8c60.4fbc.c4dc (bia 8c60.4fbc.c4dc)
Description: C: FcoeUplink
MTU 1500 bytes, BW 10000000 Kbit, DLY 10 usec
reliability 255/255, txload 1/255, rxload 1/255
[...]
Troubleshoot
There is currently no specific troubleshooting information available for this configuration.