Cisco MDS NX-OS Configuration Limits
This document discusses the configuration and scalability limits for various SAN switching parameters:
Change History
Date | Description |
---|---|
August 18, 2023 | Added scale values for Cisco MDS 9396V Multilayer Fabric switch.
Added Zone Fanout scale limits. |
October 20, 2021 |
Updated zone member footnotes in the Fabric-Level Configurations Limits table. |
August 24, 2021 |
Added scale values for Fabric Performance Monitor (FPM). |
March 29, 2021 |
Added scale values for Cisco MDS 9220i switch. |
July 1, 2020 |
Cisco MDS 9718 switches support 8,000 FLOGIs per switch. |
May 1, 2020 | The following scale values were updated:
|
Switch-Level Fibre Channel Configuration Limits for Cisco MDS 9000 Series Switches
System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.
The following table lists the switch-level configuration limits for Cisco MDS 9000 Series switches:
Feature Name |
Parameter |
MDS 9700 Series |
MDS 9250i |
MDS 9148S |
MDS 9396S |
MDS 9132T |
MDS 9148T |
MDS 9396T |
MDS 9220i |
MDS 9124V |
MDS 9148V |
MDS 9396V |
---|---|---|---|---|---|---|---|---|---|---|---|---|
Physical Fabric Logins |
FLOGIs or FDISCs per port |
2561 |
2561 |
2561 |
2561 |
2561 |
2561 |
2561 |
2561 |
2561
|
2561 |
2561 |
FLOGIs or FDISCs per module |
1,0002
2,0003
|
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) |
None (refer to chassis limits) | |
FLOGIs or FDISCs per switch |
4,0004
8,0005
|
400 |
1,000 |
1,000 |
2,000 |
2,000 |
2,0006
4,0007
|
2,000 |
2,000 |
2,000 |
4,000 | |
N-Port Virtualization (NPV) |
NPV switches per NPIV core switch |
105 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
8 |
Inter-Switch Links (ISLs) |
ISL instances per switch 8
|
Up to 200 ISLs, each ISL with 16 VSANs, for a total number of 3200-port VSAN instances |
40 ISLs, each with 29 VSANs, for a total number of 1160 port VSAN instances |
48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances |
96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances |
32 ISLs, each with 29 VSANs, for a total number of 928 port VSAN instances |
48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances |
96 ISLs, each with 15 VSANs, for a total number of 1440 port VSAN instances |
12 ISLs, each with 29 VSANs, for a total number of 348 port VSAN instances |
24 ISLs, each with 29 VSANs, for a total number of 696 port VSAN instances |
48 ISLs, each with 29 VSANs, for a total number of 1392 port VSAN instances |
96 ISLs, each with 29 VSANs, for a total number of 2784 port VSAN instances |
Port channels (PC)9
|
Fibre Channel port channels |
256 |
40 |
48 |
96 |
32 |
48 |
96 |
12 |
24 |
48 |
96 |
Members in each Fibre Channel port channel |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
12 |
16 |
16 |
16 | |
Ethernet port channels |
128 |
8 |
None |
None |
None |
None |
None |
None |
None |
None |
None | |
Members in each Ethernet port channel |
16 |
8 |
None |
None |
None |
None |
None |
None |
None |
None |
None | |
SSH 10
|
Simultaneous SSH sessions per switch |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
16 |
SAN Analytics and SAN Telemetry Streaming |
Active Initiator Target LUNs (ITLs) or Initiator Target Namespace ID (ITNs) per sampling interval per switch |
20,00011 40,00012
100,00013
|
None |
None |
None |
20,000 |
20,000 |
20,000 |
None |
20,000 |
40,000 |
None |
Active ITLs or ITNs per sampling interval per module |
DS-X9648-1536K9: 20,000 |
None |
None |
None |
20,000 |
20,000 |
20,000 |
None |
None |
None |
None | |
Telemetry receivers per switch |
2 |
None |
None |
None |
2 |
2 |
2 |
None |
2 |
2 |
None | |
Fabric Performance Monitor (FPM)14 |
Slow devices15 per port |
200 |
None |
None |
None |
200 |
200 |
200 |
200 |
200 |
200 |
200 |
Slow devices per module |
200 |
None |
None |
None |
None |
None |
None |
None |
None |
None |
None | |
Slow devices per switch |
800 |
None |
None |
None |
200 |
200 |
200 |
200 |
200 |
200 |
200 | |
DIRL Dynamically rate limited ports per module |
10 |
None |
None |
None |
None |
None |
None |
None |
None |
None |
None | |
DIRL Dynamically rate limited ports per switch |
80 |
None |
None |
None |
10 |
10 |
10 |
10 |
10 |
10 |
10 |
This is the recommended limit. FLOGIs or FDISCs are not rejected until 512 logins have been accepted on the port.
Supported on DS-X9448-768K9.
Supported on DS-X9648-1536K9 and DS-X9748-3072K9.
Supported in Cisco MDS 9706 switches.
Supported in Cisco MDS 9710 switches in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.
Supported in Cisco MDS 9710 switches with Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2) and later releases.
Supported in Cisco MDS 9718 switches with Cisco MDS 9700 Series Supervisor-1 Module (DS-X97-SF1E-K9) or Cisco MDS 9700 Series Supervisor-4 Module (DS-X97-SF4-K9) from Cisco MDS NX-OS Release 8.4(2a) and later releases.
Supported in Cisco MDS NX-OS Release 8.4(1a) and earlier releases.
Supported from Cisco MDS NX-OS Release 8.4(2) and later releases.
ISLs can be an individual FC link, VFC link, FC port channel, or VFC port channel. One port channel interface with up to 16 member links is considered as one ISL instance.
A system message appears in the user’s session when the maximum supported limit of port channels and member ports is exceeded.
A system message appears on the user’s session when the maximum supported number of SSH sessions is exceeded.
Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.
When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.
Beginning with Cisco MDS NX-OS Release 9.3(2), the number of ITLs or ITNs distributed across multiple modules in multilayer directors is increased from 40,000 to 100,000
Supported from Cisco MDS NX-OS Release 8.3(2) and later releases.
When there are more than 40,000 ITLs or ITNs distributed across multiple modules in multilayer directors or 20,000 ITLs or ITNs in fabric switches, a random set of ITL or ITN data is reported. The ITL or ITN data may change frequently and becomes unpredictable. Hence, we recommend that you limit the number of ITLs or ITNs in a switch to less than the maximum limit or query only view types. For example, a single query configured can display up to the maximum active ITLs or ITNs. If there are two queries configured, the queries together can display up to the maximum active ITLs or ITNs.
Beginning with Cisco MDS NX-OS Release 9.3(2), the number of ITLs or ITNs distributed across multiple modules in multilayer directors is increased from 40,000 to 100,000
Physical Fabric Configuration Limits for Cisco MDS 9000 Series Switches
The following table lists the configuration limits for Cisco MDS 9000 fabrics.
Feature Name |
Parameter |
Fabric Composition (lowest number is the limit for the entire fabric unless noted) |
|
---|---|---|---|
MDS 9700 (Supervisor 1, Supervisor 1E, Supervisor 4) MDS 9396V, 9124V, MDS 9148V, MDS 9396T, MDS 9148T, MDS 9132T, MDS 9220i |
MDS 9396S, MDS 9250i, MDS 9148S |
||
VSANs |
VSANs per physical fabric |
80 |
3216
|
Logins and aliases |
FCNS entries in physical fabric |
20,00017
|
10,00018
|
Device alias entries in physical fabric |
20,000 |
8,000 |
|
FC alias entries per VSAN |
2,048 |
2,048 |
|
Domains and hops |
Domains per physical fabric |
80 |
80 |
Switch hops from server to storage |
7 |
7 |
|
Zones and Smart Zones |
Configured zone sets |
1,000 |
1,000 |
Active zones |
16,00019
|
8,00020
|
|
Active zone members |
32,00021
|
16,00022 |
|
Active zone database size23
|
3.8 MB24
|
2 MB |
|
Inter-VSAN Routing (IVR) |
Active IVR zone sets |
32 |
32 |
Active IVR zones25
|
2,000 |
2,000 |
|
Active IVR zone members |
4,000 |
4,000 |
|
Active IVR service groups |
16 |
16 |
|
CFS |
CFS peers26
|
80 |
80 |
CFS static peers over IP27
|
100 |
100 |
In a physical fabric with mixed VSAN limits, all switches may be in the first 32 VSANs. The remaining VSANs may only exist on the switches that support more than 32 VSANs.
An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.
A warning message is logged to the configuration session when this limit is exceeded; however, further increases are not blocked.
An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.
A warning message is logged to the configuration session when this limit is exceeded; however, further increases are not blocked.
This limit has two components. The first limit is the FCNS maximum of 20,000 online entries in the fabric. The second limit is a maximum of 32,000 zone members in the zone database. Up to the FCNS limit of these may be online. The remaining may be unique offline members or reused online members. This limit was tested with 20,000 unique online devices and 12,000 reused online zone members.
This limit is supported when all switches in the fabric are running Cisco MDS NX-OS Release 6.2(7) or later. If there are any switches in the fabric that are running Cisco MDS NX-OS Release 6.2(5) or earlier, then the lower 2 MB zone database limit prevails and must not be exceeded for reliable zone operation.
An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.
Total across all VSANs. Any single VSAN may not have more than 1,000 IVR zones.
An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.
An error message is logged to the configuration session when this limit is exceeded and further increases are blocked.
Example: Checking the Size of the Zone Database
The following example shows how to check the size of the zone database in VSAN 310:
(config)# show zone status vsan 310 | Inc "Db Size" P 1
Full Zoning Database :
Db Size: 1040524 Bytes
--
Active Zoning Database :
Db Size: 962156 Bytesc
Now, Add 962156 with 1040524 = 2002680
2002680 / (1024 * 1024) = 1.9 MB.
Note |
This database size does not include the pending changes in a session. |
Configuration Limits for Port Line Rate Throughput on Cisco MDS 9700 40 Gbps 24-Port FCoE Modules
For ports to operate at full-line rate on Cisco MDS 9706, MDS 9710, and MDS 9718 switches, the switch should have a specific configuration. The following table lists the maximum number of line rate ports for given Cisco MDS 9700 switch configurations.
No. of Fabric 1 Cards |
Front Panel FCoE Bandwidth/Slot |
Maximum No. of Ports/Speed |
---|---|---|
3 |
660 Gbps |
16 ports/40 Gbps |
4 |
880 Gbps |
20 ports/40 Gbps |
5 |
1100 Gbps |
24 ports/40 Gbps |
No. of Fabric 3 Cards |
Front Panel FCoE Bandwidth/Slot |
Maximum No. of Ports/Speed |
3 |
1320 Gbps |
33 ports/40 Gbps |
4 |
1760 Gbps |
44 ports/40 Gbps |
5 |
2200 Gbps |
55 ports/40 Gbps |
Note |
If the number of ports that you configure exceeds the maximum limit for the number of ports, it results in all ports operating in oversubscribed mode. |
Fibre Channel Topology Combinations and Scaling
From Cisco MDS NX0-OS Release 6.2(7), the following configurable features that support scale enhancements are supported:
-
Fibre Channel Name Server (FCNS) bulk notification
-
Coalesce switch Registered State Change Notification (SW-RSCN)
For more information about the FCNS and SW-RSCN features, see the Cisco MDS 9000 Series NX-OS Fabric Configuration Guide and the Cisco MDS 9000 Family Command Reference documents.
Note |
In multidimensional scale configurations, supervisors with 1-GB memory may experience the system manager hap-reset error. |
The following table provides the configuration limits with and without the FCNS and SW-RSCN optimizations.
Scale Topology |
FCNS Bulk Notification |
Coalesce SW-RSCN |
Configuration Limits |
Scale Optimizations |
---|---|---|---|---|
Cisco MDS NX-OS Release 6.2(9) and later on Cisco MDS 9700 Series-only Fabric |
On |
On |
MDS NX-OS Release 6.2(7) and later |
Enabled |
FLOGI per module: 1,000 FLOGI node: 4000 FCNS: 20,000 Zone: 16,000 Zone members: 32,000 |
||||
Off |
Off |
MDS NX-OS Release 6.2(5) and earlier |
Disabled (Disabled by default in Cisco MDS NX-OS Release 6.2(7)) |
|
FLOGI per module: 500 FLOGI node: 2500 FCNS: 10,000 Zone: 8,000 Zone members: 16,000 |
||||
Cisco MDS NX-OS Release 6.2(9) and later on MDS Mixed Fabric28 (Cisco MDS 9700 Series, 9500, 9250i, 9222i, and 9148)
|
On or off |
On or off |
FLOGI per module: 400 FLOGI node: 2000 FCNS: 10,000, 13,000 29
Zone: 8,000, 10,400 30
Zone members: 16,000, 20,800 31
|
Enabled or disabled |
The scale enhancements introduced in Cisco MDS NX-OS 6.2(7) and later are available on Cisco MDS 9700 Series Switches (only networks with Cisco MDS NX-OS Release 6.2(7) and later, running on all the switches and the configurable optimizations): FCNS bulk notification and RSCN coalesce enabled. These enhancements are not available in mixed fabrics regardless of whether FCNS and RSCN are enabled. In Cisco MDS NX-OS Release 6.2(9) and later, FCNS bulk notification is enabled by default. (To disable FCNS bulk notification, use the fcns no-bulk-notify command.) Coalesce SW-RSCN is disabled by default.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
This increase in scale is applicable only to the Cisco MDS 9500 Series Supervisor-2A Module.
Fibre Channel Scale Limitations
The various Fibre Channel scale limitations are described in this section.
Limitation 1
From Cisco MDS NX-OS Release 6.2(7), the fcns bulk-notify and rscn coalesce swrscn vsan commands are available on all Cisco MDS switches. We recommend that you use these commands only on Cisco MDS 9700 Series switches with 48-port 16-Gbps Fibre Channel switching module because the higher configuration limits are currently not supported on any other MDS platform.
Note |
The FCNS bulk notification and Coalesce SW-RSCN features are supported only in NX-OS Release 6.2(7) and later releases. Enable the Coalesce SW-RSCN only if all the switches in a physical fabric are Cisco MDS switches that are running Cisco MDS NX-OS Release 6.2(7) or later. |
Limitation 2
The maximum zone database size is increased from 2 to 3.8 MB in Cisco MDS Release 6.2(7) and later for all MDS 9700 Series switches to enable zone scale enhancements. The new limit of 16000 zones is supported only on the Cisco MDS 9700 only fabric. Physical fabrics with Cisco MDS 9500, 9200, or 9100 Series switches continue to have the 2-MB zone database limit, supporting only up to 8000 zones.
Limitation 3
In a three-node serial topology, traffic imbalance may occur if the number of port channels that are configured between the switches are the same. We recommend that you have a single port channel between two switches with any number of member ports. If you configure more than one port channel, ensure that the count of port channels between the switches varies.
Limitation 4
The maximum latency (round-trip time) and the TCP retransmits that are supported on FCIP links are 250 ms and 0.05 percent receptively.
Note |
The maximum limit is the same regardless whether latency and packet drop conditions exist together or only one of the conditions exists. |
Syslog and Warnings for Network Scale
The following table lists the syslogs and warnings for a network.
Syslog and Warnings |
Cisco MDS 9700 Series |
Cisco MDS 9500, 9200, and 9100 |
---|---|---|
Zone limit |
16,000 zones – Hard limit: Syslog warning that states no more zones can be configured:
|
8,000 zones – Hard limit: Syslog warning that states no more zones can be configured:
10,400 zones – Hard limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning that states no more zones can be configured:
|
FCNS limit |
20,000 FCNS entries – Hard limit: Syslog warning stating that no more name server entries are supported:
|
10,000 FCNS entries – Soft limit: Syslog warning about validated limit:
13,000 FCNS entries – Soft limit for Cisco MDS 9500 Series Supervisor-2A Module: Syslog warning about validated limit:
20,000 FCNS entries – Hard limit: Syslog warning:
|
Switch-Level Configuration Limits for Fibre Channel over Ethernet
System messages are generated whenever the configuration limits are exceeded. For more information, see the Cisco MDS 9000 Series and Nexus 7000 Series NX-OS System Messages Reference document.
The following table lists the switch-level configuration limits for Fibre Channel over Ethernet (FCoE) on Cisco MDS.
Feature Name |
MDS 9710 with 48-Port 10-Gigabit FCoE |
---|---|
FLOGIs per Port |
256 |
FLOGIs per Module |
1,000 |
FLOGIs per Switch |
4,000 |
VSAN |
80 |
VSAN-VLAN Mapping |
80 |
Virtual Fibre Channel Port Channel and Member Ports |
128 virtual fibre channel port channel and a maximum number of 16 members in one virtual fibre channel port channel |
Physical Fabric Configuration Limits for Fibre Channel over IP
The following table lists the fabric-level configuration limits for Fibre Channel over IP (FCIP):
Feature Name |
Cisco MDS 9250i and MDS 9220i Fabric Switches |
Maximum latency (round-trip time) and TCP retransmits supported on FCIP links |
250-ms round trip and 0.05% packet drop |
Feature Name |
Cisco MDS 9700 Series Multilayer Directors with Cisco MDS 24/10-Port SAN Extension Module |
Maximum latency (round-trip time) and TCP retransmits supported on FCIP links |
250-ms round trip and 0.05% packet drop |
Cisco MDS 9000 I/O Accelerator Configuration Limits
The following table lists the Cisco MDS 9000 I/O Accelerator configurations and the corresponding limits.
Parameter |
MSM-18/4 or SSN-16 Module on MDS 9222i and MDS 9500 Modular Chassis and MDS 9250i Fabric Switch, 24/10-Port SAN Extension Module on MDS 9710 Chassis |
||
---|---|---|---|
Number of switches in a cluster |
4 |
||
Number of clusters per switch |
16 |
||
Number of switches in a SAN fabric for FC-Redirect |
34 |
||
Number of hosts per target |
128 |
||
Number of concurrent flows per Cisco MDS 9000 I/O Accelerator service engine |
128 |
||
Number of flows per Cisco MDS 9000 I/O Accelerator service engine (hard limit) |
128 - Release 4.2(1) on MDS 9222i and MDS 9500 512 - Release 4.2(7) or later on MDS 9222i and MDS 9500 512 - Release 6.2(5) or later on MDS 9250i 512 - Release 8.2(1) or later on MDS 9710 |
||
Number of flows per Cisco MDS 9000 I/O Accelerator service engine (soft limit) |
64 - Release 4.2(1) on MDS 9222i/MDS 9500 256 - Release 4.2(7) or later on MDS 9222i/MDS 9500 256 - Release 6.2(5) or later on MDS 9250i
256 - Release 8.2(1) or later on MDS 9710 |
||
Number of flows in a cluster |
1,024 - Release 4.2(7d) 1,248 - Release 5.2(6b) or later |
Fibre Channel Flow Configuration Limits
The following table lists the Fibre Channel flow configurations limits.
Cisco MDS Device |
Aggregate Flow and Flow Statistics Limit |
Flow Statements per Module |
---|---|---|
Cisco MDS 24/10 Port SAN Extension Module |
512 |
512 |
Cisco MDS 9700 48-Port 16-Gbps Fibre Channel Switching Module (DS-X9448-768K9) |
512 |
512 |
Cisco MDS 9700 48-Port 32-Gbps Fibre Channel Switching Module (DS-X9648-1536K9) |
512 |
512 |
Cisco MDS 9700 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) |
512 |
512 |
Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line card SAN Analytics ITL limit
The SAN Analytics ITL limit for Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line cards is 20,000. Beginning with Cisco MDS NX-OS Release 9.3(2), the limit is increased to 40,000.
Configuring ITL Scale Limits for Cisco MDS 48-Port 64-Gbps Fibre Channel Switching Module (DS-X9748-3072K9) line cards
To configure the ITL scale limits, perform these steps:
Procedure
Step 1 |
Enter configuration mode: switch# configure |
||||||
Step 2 |
switch(config)# analytics scale version {default |1|2}
|
Zone Fanout Scale Limits
Fanout Ratio (Target:Initiator) | Initiator | Target | Plaftforms Tested |
---|---|---|---|
1:200 | 20 port each with 10 logins per port | 15 port each with 3 logins per port | MDS 9700 for Line Cards DS-X9448-768K9, DS-X9648-1536K9, and DS-X9748-3072K9 |
For DS-X9448-768K9 it was tested on DS-X97-SF1-K9
For DS-X9648-1536K9 and DS-X9748-3072K9 it was tested on DS-X97-SF4-K9