Mux Overview

The mux object can be thought of as a specialized switch which is optimized for many-to-one connectivity. The advantage of using a mux over a switch is that no packet inspection or routing decisions are required. This results in significant latency reductions.

A mux object has many downstream ports which are connected to one upstream port. The upstream port is typically connected to a shared connection - for example, an order line to an exchange. The downstream ports are usually connected to devices that need to share a connection, for example local trading servers.

The supported number of mux objects, upstream ports, and downstream ports varies depending on the currently running firmware mode (see Multiple mux objects for more details). The mux object requires the switch, switch-1g, mux, or fastmux firmware to be loaded and running.

note.gif

Noteblank.gifA switch license is not required to create mux objects in the switch or switch-1g firmware mode. A license is only required to create switch objects. Refer to Licensing for further details.

459688.jpg 459689.jpg

A typical Mux setup, showing a single upstream WAN connection (A1) to be shared amongst three local downstream servers (B5, B9, C5)

note.gif

Noteblank.gifTraffic cannot be sent between downstream ports within a mux object. In the example depicted above B5 is not able to communicate with B9 or C5.

The primary function of a mux is to forward traffic from downstream ports up to the upstream port. However, most use-cases require a bidirectional connection, which means that traffic must be returned from the upstream to the downstream ports. Mux objects can be configured in several different modes that affect how return traffic is routed from the upstream port to the downstream ports.

  • In layer2 mode, traffic that arrives at the upstream port is routed to a downstream port according to the destination MAC address. The upstream port to downstream port latency in this configuration is approximately 107ns. In this mode downstream hosts will only see traffic destined for their MAC address, as well as broadcast traffic.
  • In raw mode, traffic that arrives at the upstream port is broadcast via the layer 1 cross-point device to the downstream ports. This allows return traffic to be routed to hosts in around 5 ns. However, all downstream hosts will see all return data from the upstream connection. Usually the network card on the downstream host will perform this filtering transparently, so raw mode can provide significant latency improvements. One consideration in the use of raw mode is network security. In shared setups, where multiple independent parties are sharing an Cisco Nexus 3550-F Fusion (formerly ExaLINK Fusion), it may be undesirable for all local hosts to have visibility into all return data from the upstream port. In such setups, layer2 mode should be used. Note: when using a mux object with fastmux firmware, the layer2 forwarding is not supported.
  • The fast-vlan mode can be used on VLAN enabled mux objects to slightly improve the latency by not inspecting the VLAN tag of packets entering a downstream port. Refer to VLAN support for more details.
note.gif

Noteblank.gifWhen raw mode is used, all ports within the mux object operate at the same link speed (e.g. 10Gb/s). This is because all there is no opportunity to do rate conversion at layer 1. The CLI will issue warnings if ports of different speeds are added to a mux object in raw mode.

Configuring a Mux Object

A mux object must first be created and assigned a name. This must be done from within the configuration mode, or by prefixing the command with config.

admin@N3550-F(config)> mux my_mux
Mux "my_mux" created

A common next step is to add front panel ports to the mux. Any front panel port can be assigned as the upstream port, but currently only one upstream port can be assigned per mux. As an example, to assign port A1 as an upstream port:

admin@N3550-F(config-mux:my_mux)> port up A1
Added upstream port "A1" to mux "my_mux"

Any number of downstream ports can then be assigned to the mux. To add a downstream port simply omit the 'up' modifier to the port command:

admin@N3550-F(config-mux:my_mux)> port A2
Added downstream port "A2" to mux "my_mux"

Both upstream and downstream ports can be removed from the mux using the no form of the port command. It is recommended to remove unused ports from a mux when they are no longer required.

admin@N3550-F(config-mux:my_mux)> no port A2
Removed port "A2" from mux "my_mux"

The mode of the mux can be changed between layer2 (the default setting) and raw using the mode command:

admin@N3550-F(config-mux:my_mux)> mode raw
Set mux "my_mux" to raw mode

The mode can be changed back to layer2 using:

admin@N3550-F(config-mux:my_mux)> mode layer2
Set mux "my_mux" to layer2 mode
note.gif

Noteblank.gifA mux object using fastmux firmware can only operate in raw mode.

To display the current configuration of a mux object use the show command:

admin@N3550-F(config-mux:my_mux)> show
Mux name      : my_mux
Mode          : layer2
VLAN tagging  : disabled
IGMP snooping : disabled

Port Side
---- ----------
A1   upstream
A2   downstream

The mux object should be deleted when no longer in use. This is done using the no form of the mux command:

admin@N3550-F(config)> no mux my_mux
Mux "my_mux" deleted

Multiple Mux Objects

Mux objects can be created when running any of the three standard firmware images (switch, switch-1g, mux and fastmux). The supported number of mux objects, upstream ports, and downstream ports varies depending on the firmware mode. These differences are summarized in the table below:

Firmware Mux Objects Upstream Ports Downstream Ports
Switch Unlimited Unlimited Unlimited
Mux Unlimited 4 Unlimited
FastMux 4 4 15, 11, 11, 7

Switch firmware

When running switch or switch-1g firmware, there is no restriction on the number of mux objects that can be created. The number of downstream ports for each mux object is also unlimited. For example, a single mux with 47 downstream ports can be created (48 ports total). Similarly, one can create 6 muxes each with 7 downstream ports (48 ports total).

Mux firmware

The mux firmware also supports any number of mux objects, however, at most 4 individual upstream ports are supported. Multiple mux objects can use the same upstream port if the VLAN ID of the upstream port is different. Please refer to VLAN support for further details. As with the switch firmware, all mux objects in mux firmware support an unlimited number of downstream ports. For example, a single mux with 47 downstream ports can be created (48 ports total). Similarly one could create 3 muxes each with 15 downstream ports (48 ports total). However, 6 muxes with 6 different upstream ports is not supported in the mux firmware mode.

Fastmux firmware

In the fastmux firmware mode the underlying FPGA firmware has 4 different fixed sized mux resources in it. This means that the fastmux firmware can create up to 4 mux objects each with its own upstream port. The number of downstream ports supported by each mux object depends on the hardware resource allocated to it. It may be:

  • 15 ports to 1
  • 11 ports to 1
  • 11 ports to 1
  • 7 ports to 1

The management processor will attempt to automatically map mux objects to one of the 4 muxes within the FPGA, depending on the number of ports in the configuration. For example, if a 14:1 mux and a 5:1 mux are created, the software will map the 14:1 mux to the 15:1 FPGA mux resource and the 5:1 mux to the 7:1 FPGA mux resource.

The allocation logic will wait until an upstream port is specified before making a decision on which FPGA mux to use.

If additional ports are added to a mux object, causing it to exceed the number of ports that the underlying hardware can support, the following error will be displayed: WARNING: Failed to allocate switch resources for port C3. In this case, config optimize needs to be run to reallocate resources within the system. If any mux object has more than 15 downstream ports, this command will not succeed.

warn.gif

Warningblank.gifRunning the optimize command may cause traffic flows to be momentarily interrupted.

As an example, assume that a single 5:1 mux is required and there are no other mux objects defined. In this case, the following steps would be taken:

admin@N3550-F> config mux my_mux
Created mux "my_mux"
WARNING: Mux object "my_mux" disabled: Mux objects must be set to raw mode when using fastmux module function
admin@N3550-F(config-mux:my_mux)> mode raw
Set mux "my_mux" to raw mode
admin@N3550-F(config-mux:my_mux)> port a1-a5
Added downstream port "A1" to mux "my_mux"
Added downstream port "A2" to mux "my_mux"
Added downstream port "A3" to mux "my_mux"
Added downstream port "A4" to mux "my_mux"
Added downstream port "A5" to mux "my_mux"
admin@N3550-F(config-mux:my_mux)> port up b1
Added upstream port "B1" to mux "my_mux"

This mux has 5 downstream ports so the allocation logic will map this mux to the 7:1 hardware mux resource within the FPGA.

If, at a later stage, an additional 3 ports needs to be added:

admin@N3550-F(config-mux:my_mux)> port c1-c3
Added downstream port "C1" to mux "my_mux"
Added downstream port "C2" to mux "my_mux"
Added downstream port "C3" to mux "my_mux"
WARNING: Failed to allocate switch resources for port C3
WARNING: The "optimize" command may help to resolve allocation failures
admin@N3550-F(config-mux:my_mux)>

When the 8th port is added (ie C3), an "allocation" error results. The optimize command can then be issued to instruct the software to swap ports and objects around to fit the requested config.

admin@N3550-F(config-mux:my_mux)> exit
WARNING: Failed to allocate switch resources for port C3
WARNING: The "optimize" command may help to resolve allocation failures
admin@N3550-F(config)> optimize
This may cause an interruption in network traffic
Are you sure? y
Optimizing switch resources...
Operation completed
admin@N3550-F(config)>
admin@N3550-F(config)>

Fastmux Trade-offs and Limitations

Users should not approach fastmux as if it works as a general purpose switch. Instead, use the general model that fastmux works more like a direct attach cable, connecting your downstream devices to the upstream port (and visa versa). This means that fastmux firmware mode makes some trade-offs in management features, in order to achieve the lowest latency muxing available on the Nexus 3550-F. So features like LLDP, BGP and IGMP are not available in fastmux firmware mode.

Users should also be aware of the following limitations in fastmux firmware.

  • Errors received on a downstream port will be forwarded to the upstream port.
  • MACs connected to downstream ports (e.g. from custom FPGA devices) which do not adhere to the Ethernet spec may cause fastmux to transmit errors.
  • If a downstream port stops receiving a link signal, the upstream port may temporarily stop transmitting a link signal.
  • Chaining fastmux objects on the same Nexus 3550-F via a cable or virtual port is not supported (a separate Nexus 3550-F must be used to chain fastmux objects).