Migrate to Cisco UCS 6400 Fabric Interconnect with 10- to 25-Gbps Ports White Paper

 

Cisco Unified Computing System overview

The Cisco Unified Computing System^™ (Cisco UCS^®) is a next-generation data center platform that unites computing, networking, storage access, and virtualization resources into a cohesive system designed to reduce Total Cost of Ownership (TCO) and increase business agility.

The Cisco UCS 6400 series FI is the replacement for the UCS 6200 series FI to deliver higher speeds for both Ethernet and Fibre Channel. The Cisco UCS 6400 series FI provides high performance, consistent latency, and lossless fabric to address the needs of customers deploying today’s modern data centers.

Product overview

The Cisco UCS 6400 series uses a cut-through architecture, supporting deterministic, low-latency, line-rate 10/25/40/100 Gigabit Ethernet ports.

The fourth-generation Cisco UCS 6454 Fabric Interconnect is a one-rack-unit device with high-density, nonblocking, low-power 10- and 25-Gbps ports. The switch has twenty-eight 10/25-Gbps Ethernet ports, four 1/10/25-Gbps Ethernet ports, six 40/100-Gbps Ethernet uplink ports, and sixteen unified ports that can support 10/25-Gbps Ethernet or 8/16/32-Gbps Fibre Channel ports. All Ethernet ports are capable of supporting FCoE. (See Figure 1.)

Cisco UCS 6454 Fabric Interconnect (rear view)

Cisco UCS 6454 Fabric Interconnect have two power supplies (hot swappable, 1+1) and four variable-speed fans (hot swappable, 3+1) that insert into the front of the chassis with fan-side intake and port-side exhaust. Server-facing ports are at the rear of the chassis to simplify connections to servers and fabric extenders. Serial, management, and USB ports are provided at the front of the chassis for easy access at the front of the rack. (See Figure 2.)

Cisco UCS 6454 Fabric Interconnect (front view)

The Cisco UCS 64108 Fabric Interconnect is a two-rack-unit device with seventy-two 10/25-Gbps Ethernet ports, eight 1/10/25- Gbps Ethernet ports, twelve 40/100-Gbps Ethernet uplink ports, and sixteen unified ports that can support 10/25-Gbps Ethernet or 8/16/32-Gbps Fibre Channel ports. All Ethernet ports are capable of supporting FCoE. (See Figure 3.)

 

Cisco UCS 64108 Fabric Interconnect (rear view)

Cisco UCS 64108 Fabric Interconnect has two power supplies (hot swappable, 1+1) and three variable-speed fans (hot swappable, 2+1) that insert into the front of the chassis with front-to-back cooling. Server-facing ports are at the rear of the chassis to simplify connections to servers and fabric extenders. Serial, management, and USB ports are provided at the front of the chassis for easy access at the front of the rack. (See Figure 4.)

Cisco UCS 64108 Fabric Interconnect (front view)

Audience

The audience for this document consists of system architects, system engineers, and any other technical staff who are responsible for planning or upgrading the fabric interconnects. Although every effort has been made to make this document appeal to the widest possible audience, the document assumes that readers have an understanding of Cisco UCS hardware, terminology, and configurations.

Objective

The Cisco UCS 6400 Series Fabric Interconnects are hardware switches that use the latest Cisco silicon Switch-on-Chip (SoC) ASIC technology. Because the hardware is new, you may need to make changes in port and software configurations to experience a successful migration.

This document discusses the process for upgrading from Cisco UCS 6200 Series Fabric Interconnects to Cisco UCS 6400 Series Fabric Interconnects. After reading this document, you should understand the upgrade process and any precautions you should consider to achieve a successful migration.

Planning: precheck steps

This document only describes the process of migrating from the Cisco UCS 6200 series FI to the Cisco UCS 6400 series FI. (Table 1) It summarizes various upgrade paths between fabric Interconnect, fabric extenders and IOMs, and Cisco Nexus^® 2000 Series Fabric Extenders.

Table 1.        Migration support matrix for infrastructure hardware

Precheck 1: Back up and export Cisco UCS Manager configuration.

Precheck 2: Take an inventory of the Cisco UCS domain and remove any unsupported hardware.

Cisco UCS 6200 Series Fabric Interconnect migration to Cisco UCS 6454 Fabric Interconnect is supported from Cisco UCS Manager (UCSM) Release 4.0(1), and the 6200 series migration to the 64108 Fabric Interconnect is supported with UCSM Release 4.1(1). The focus of this white paper is from the perspective of UCSM Release 4.1(1).

The migration procedure is applicable for any release above 4.0(1) for FI 6454 and 4.1(1) onward for FI 64108. Note that there are many new UCS hardware products that were launched post UCSM Release 4.0(1), including the VIC 1400 in UCSM Release 4.0(2) and IOM 2408 in UCSM Release 4.0(4e). Hence the recommendation for a migration from FI 6200 to FI 6400 would be to use the latest UCSM release.

Tables 2 and 3 list the supported hardware for Cisco UCSM Release 4.1(1). Note that any server prior to the M3 platform will cause the upgrade to fail on the Cisco UCS 6200 Series Fabric Interconnects.

Table 2.        Server migration support for Cisco UCSM 4.1(1)

 

Table 3.        I/O and fabric extender migration support for Cisco UCSM 4.1(1)

 

Note:     Cisco UCS Virtual Interface Card (VIC) 1400 series and IOM 2408 are supported only from UCSM releases 4.0(2) and 4.0(4e), respectively. The VIC 1495/1497 cannot be connected to 6400 series fabric interconnects and hence is not part of the adapter list.

Precheck 3: Confirm that the data VLAN does not overlap with the VLANs reserved for the Cisco UCS 6400 Series Fabric Interconnects. If it does, then identify a block of 128 contiguous VLANs. With the 6400 series, the number of active VLANs increases from 2000 in the predecessor switches to 3000 in the new switches. This document describes how to change the reserved VLAN range in a later step (step 8). Table 4 lists the reserved VLAN ranges.

Table 4.        Cisco UCS and fabric interconnect reserved VLANs

Figure 5 shows an example from the Cisco UCS Manager installer in which active VLANs overlapped with the default VLANs reserved for the Cisco UCS 6400 series.

Example showing active VLANs overlapping with the default VLANs reserved for the Cisco UCS 6400 Series Fabric Interconnects

Precheck 4: Validate software features and configurations. Cisco UCS 6400 Series Fabric Interconnects do not support some software features that were allowed with the 6200 series. Some of these features will become available at a later software release.

Table 5.        Features that need special attention prior to upgrading

Failure to comply with these remediation steps will result in a migration warning alert during the migration process and prevent the fabric interconnects from synchronizing.

The Cisco UCS 6400 Series Fabric Interconnect supports only port-channel mode for chassis-discovery. On changing the “Chassis/FEX discovery policy” on the 6200 series FI to port-channel, the UCS 5108 chassis needs to be reacknowledged before proceeding with the migration. If the chassis is not reacknowledged, the migration will fail. This is one of the most common reasons for migration failure seen by customers; so it is important to ensure that IOM fabric ports are in port-channel toward the UCS 6200 series FI before starting the migration and that the UCS 5108 chassis’s are reacknowledged. Figure 4 shows an example of a migration warning.

Sample migration warning

Precheck 5: The Cisco UCS 6200 Series Fabric Interconnects are modular, whereas the 6400 series are fixed, as shown in Figures 7 and 8. Thus, the port schemes change when you migrate from the 6200 series to the 6400 series. Tables 6 and 7 list the supported port capabilities of the 6400 Series. Use the information in this table to plan the port migration.

Note:     As a best practice, you should label the cables.

Ports on the Cisco UCS 6454 Fabric Interconnect

Table 6.        Port capabilities for the Cisco UCS 6454 Series Fabric Interconnects in Cisco UCSM 4.1(1)

Ports on the Cisco UCS 64108 Fabric Interconnect

Table 7.        Port capabilities for the Cisco UCS 64108 Series Fabric Interconnects in Cisco UCSM 4.1(1)

Precheck 6: Unified ports on the 6400 series fabric interconnect (FI) are the first 1-16 ports; this is different from the unified port ordering on the 6200 series FI. Hence, unconfigure the fiber channel ports on the migrating subordinate 6200 and plan them to be remapped/reconfigured to port 1-16 on the 6400. Also remove port configuration for the required ports in the 1-16 range on the migrating 6200 to ensure these ports can be converted to FC ports on the 6400.

FC ports on 6400 series FI support speeds for 8/16/32G connectivity, but note that FC ports at 8 Gbps speed require IDLE fill pattern.

The 6400 series FI requires the IDLE fill pattern for FC uplink ports and FC storage ports at 8 Gbps speed, so ensure that corresponding peers FC switch ports or that the direct-attach FC storage array ports are also set to IDLE fill pattern at 8 Gbps. Without IDLE fill pattern on both ends, the links would not be functional at 8 Gbps. There is no such restriction for FC ports on 6400 series FIs at 16 and 32Gbps speeds.

Precheck 7: All the components need to be upgraded to Cisco UCSM Release 4.1(1) or a later release – Fabric Interconnect, IOM, and servers. Ensure that both the 6200 series FI and 6400 series FI are on the same UCSM build before migration.

Upgrade steps

After the prerequisite steps are complete, the upgrade process can commence. Ensure the prechecks are completed before attempting the upgrade.

Step 1: Move the traffic to the primary fabric interconnect. This change can be accomplished in several ways. This document describes two of these approaches: fabric evacuation and uplink disablement. Use the flowchart in Figure 6 to choose between the two based on your Cisco UCS domain server and connectivity.

Flowchart for choosing the best method for moving traffic

Figure 10 shows the fabric evacuation method.

Fabric evacuation

Figure 11 shows the uplink disablement method.

Uplink disablement

Step 2: Verify that all traffic has failed over to the primary fabric interconnect. Unconfigure all the server ports on the subordinate fabric interconnect.

Step 3: Power down the old subordinate fabric interconnect and disconnect the power and the L1/L2 cables.

Step 4: If the new fabric interconnect is being racked in the same location as the old one, disconnect the server cables. Un-rack the old subordinate fabric interconnect and rack the new subordinate fabric interconnect.

Note:     As a best practice, you should label the cables.

Step 5: Connect the L1/L2 cables and the server ports according to your port mapping plan.

Step 6: Power up the new fabric interconnect. If it is connected correctly, the new subordinate fabric interconnect will detect that it is being added to an existing cluster (Figure 9).

Powering up the new fabric interconnect

Step 7: Enter the IP address information for the new subordinate fabric interconnect (Figure 12).

Entering the IP address information

Step 8: After the management IP address is configured for the new subordinate fabric interconnect, the installer will perform a configuration and VLAN check.

If any misconfiguration is found (incorrect switch mode, VM-FEX, non-port-channel IOM, etc.), the installer will provide a list of objects that are out of compliance, as shown in the example in Figure 14.

Checking for misconfigurations

Any objects that are out of compliance will need to be fixed as described in Table 5 before you can proceed with the migration. Corrections can be performed while in a mixed fabric interconnect connection.

If the active VLANs overlapped with the default reserved VLANs for the Cisco UCS 6400 Series Fabric Interconnect, the warning message shown in Figure 15 will be displayed.

Warning message for overlapping VLANs

If VLANs overlap, change the reserved VLANs after the new subordinate fabric interconnect joins the cluster. The change can be implemented through the GUI or the Command-Line Interface (CLI), as shown in Figures 16 and 17.

Changing the reserved VLANs in the GUI

Changing the reserved VLANs in the CLI

After you change the reserved VLAN block, you will need to reboot the new subordinate fabric interconnect. If you need to configure Fibre Channel, then wait for reboot until after you have configured the Fibre Channel ports. Otherwise, reboot the new subordinate fabric interconnect and skip step 9 and proceed to step 10.

Note:     Do not reboot the primary fabric interconnect. The change in the reserved VLAN range is a global configuration, and when the primary fabric interconnect is replaced, the fabric interconnect will inherit the changes.

Step 9: Convert Ethernet ports to Fibre Channel ports and reboot (Figure 18). The default speed will be 16 Gbps. If 8-Gbps Fibre Channel optics are used, then the port will be error-disabled. Manually set the speed to 8 Gbps and disable and reenable the port. If the system is connected to a Cisco MDS 9000 Family switch, then you need to configure the “fill-pattern IDLE speed 8000” setting.

Note:     When you convert from Ethernet ports to Fibre Channel ports, a reboot is required. Fibre Channel ports are converted in blocks of four.

Note:     Converting the ports

Converting the ports

Step 10: After bootup, verify that the new subordinate fabric interconnect joined the cluster successfully.

Step 11: Configure the ports based on your port-mapping worksheet. If you disabled the uplinks (Ethernet or Fibre Channel) in step 1, then make sure to keep the uplinks disabled on the secondary fabric interconnect until the configuration is complete, to help ensure that the traffic is still flowing on the primary fabric interconnect.

For the server ports that were unconfigured in step 2, make sure to configure the server ports at this time on the subordinate fabric interconnect.

If IOM 2204/2208 migration to IOM 2408 is also planned, this would be the time to replace the IOMs toward the subordinate fabric interconnect. Ensure that the server ports are unconfigured on the Cisco UCS 6400 Series Fabric Interconnect before proceeding with replacing the IOMs. Note that IOM 2408 connects to a 6400 series fabric interconnect over 25 GbE links. After connecting the IOM 2408, reconfigure the server ports on the 6400 series fabric interconnect and wait for the IOM 2408 auto upgrade to complete.

Note:     If the Cisco UCS domain is configured for disjointed networks, then validate the VLANs to uplink.

Step 12: For FI migration, if port-mapping is maintained across migration, UCSM attempts to bring up the internal virtual interfaces (VIFs) from the UCS servers to the fabric interconnects without any disruption after migration. Thus reacknowledgement would not be needed if port-mapping is maintained. If you have changed port mappings, you may need to reacknowledge the IOM, FEX, or direct-connect rack server connected to the subordinate fabric interconnect.

In most cases, an IOM or a fabric extender would not require reacknowledgement, but if they do not become operational, reacknowledgement may be required. You can perform this step on a per-device basis, so reacknowledge just the IOMs and fabric extenders that were migrated to the 6400 series fabric interconnect (Figure 19). Reacknowledgement of IOM/FEX does not result in server reboot.

For directly connected rack servers, if port-mapping is maintained across the FI migration, reacknowledgement can mostly be avoided. If port-mapping is changed across migration, or if the vNICs/VIF paths in UCSM for the server are down, then reacknowledgement is needed to help ensure that the virtual interfaces (VIFs) have been built correctly. Reacknowledgement for rack servers is disruptive because it reboots the server.

Ensure there are no new active UCSM faults after bringing up server ports on the 6400 series FI after migration; meanwhile, note that there will be faults due to uplinks being down. If there are new faults in UCSM, reacknowledgement of IOM, FEX, or direct-connect rack-server would be required to clear the faults.

Re-acknowledging fabric extender and IOM

Step 13: Move the traffic back to the new subordinate fabric interconnect. If you used the fabric evacuation method, then deselect fabric evacuation. If you disabled the uplinks (Ethernet and Fibre Channel), then reenable the uplinks. Verify that traffic is flowing correctly on the new subordinate fabric interconnect.

Verify that the FI cluster state is “HA ready” using the CLI command “show cluster extended-state,” as shown in Figure 20. Additionally from UCSM, check that there are no new active faults after the uplinks are UP.  You can also check that no vNICs are DOWN that were previously UP, from UCSM Equipment >> Server >> VIF Paths. Also follow the traffic verification steps outlined in the UCSM firmware management guide.¹ Reacknowledgement of the corresponding IOM, FEX, or direct-connect rack servers would be required if these checks fail.

Step 14: After verifying that traffic is flowing on the subordinate fabric interconnect, promote the new subordinate fabric interconnect to make it the primary fabric interconnect. This can be done using the “cluster lead <a|b>” command under the “connect local-mgmt” context in the primary fabric interconnect CLI.

Verify that the primary fabric interconnect is now the 6400 series (Figure 20).

Verifying that the Cisco UCS 6400 series is the primary fabric interconnect

Step 15: Repeat these steps for the other fabric-interconnect replacements to complete the migration.

Steps 1–15 detailed the migration of Cisco UCS 6200 series to 6400 series fabric interconnects and the same steps can be followed to migrate from 6454 fabric interconnect to the higher-port-count 64108 fabric interconnect.

For 6454 to 64108 migration, one additional check needs to be taken care at Step 2, unconfigure ports 49–54 on the 6454 fabric interconnects before migration and then reconfigure these ports on the 64108s after completing the migration of each fabric interconnect.

For more information

Read more about Cisco UCS products:
https://www.cisco.com/site/us/en/products/computing/servers-unified-computing-systems/index.html.

Read more about the Cisco UCS 6400 Series Fabric Interconnect:
https://www.cisco.com/c/en/us/products/servers-unified-computing/fabric-interconnects.html.

Read more about the Cisco UCS VIC 1400:
https://www.cisco.com/c/en/us/products/interfaces-modules/unified-computing-system-adapters/index.html.

¹ Cisco UCS Manager Firmware Management Guide, Release 4.1, “Verifying the Ethernet data path”