Cisco Catalyst 6500/6800 Supervisor 6T Architecture White Paper

Contents

Introduction. 3

High-Level Description of Supervisor 6T. 3

System Level Requirements. 5

Supervisor Architecture. 9

Baseboard. 9

Supervisor 6T Uplinks. 10

Supervisor 6T Uplinks Operating Modes. 10

Normal/Default Mode. 10

10G Performance Mode. 11

40G Performance Mode. 11

10G Oversubscription Mode. 12

Fabric Interface and Replication Engine (FIRE) ASIC.. 12

Port ASIC.. 13

Multilayer Switch Feature Card (MSFC6) 13

Switch Fabric. 15

Crossbar Switch Fabric. 15

Clock Frequency. 16

Line Encoding. 17

Policy Feature Card (PFC-E) 17

PFC4 Architecture. 19

Layer 2 Forwarding Engine. 20

Layer 3 Forwarding Engine. 21

PFC4 and PFC4-XL Feature Review.. 30

PFC4 Architecture Enhancement Summary. 39

PFC4 Security and QoS Architecture. 39

Switch Fabric and Fabric Connection Daughter Card. 44

Supervisor Engine 6T Switch Fabric. 44

Switch Fabric Architecture. 45

Fabric Connection Daughter Card. 46

Baseboard Temperature Sensors. 47

Supervisor 6T Packet Walks. 48

Ingress Packet Processing. 48

Egress Packet Processing. 49

Glossary. 50

 

Introduction

The Cisco Catalyst^® Supervisor Engine 6T is the latest addition to Cisco^® Catalyst 6500 and 6800 family of Multi-Layer Switching Supervisor Engines. Supervisor 6T is a next-generation supervisor that offers increased scalability for few supported features, much higher per-slot throughput, and supports forwarding performance similar to the previous supervisor.

This white paper will provide an architectural overview of the new Supervisor 6T, physical layout of the Supervisor 6T, and details about its updated hardware components.

High-Level Description of Supervisor 6T

The Supervisor 6T is made up of three main physical components:

●   The Baseboard with Integrated MSFC6

●   The 4^th generation Enhanced Policy Feature Card (PFC4-E)

●   The New 6 Tbps Switch Fabric

The Supervisor baseboard forms the foundation upon which the purpose-built daughter cards and other components are placed. It houses a multitude of application-specific integrated circuits (ASICs), including the ASIC complex that makes up the primary Six Terabit (6.16Tbps) crossbar switch fabric, as well as the port ASICs that control the front-panel 10 GE and 40 GE ports.

The MSFC6 Complex is Integrated onto Baseboard, which serves as the software control plane for the switch. The control plane handles the processing of all software-related features. More details on this new CPU complex will be explored later in this paper.

The PFC4-E is the daughter card that incorporates a special set of ASICs and memory blocks, which provide hardware-accelerated data-plane services for packets traversing the switch. Supervisor 6T PFC4 is a new, improved forwarding ASIC, compared to the one available on the Supervisor 2T, which combines the previous Layer 2 and 3 ASICs with Improved read/write Bandwidth and various other Fixes.

The 6-Tbps Switch Fabric is another Daughter Card that provides 28 Dedicated Fabric Channels capable of operating at different Clock speeds to support 20 Gbps (Cisco Catalyst 6700 and 6800 series Line cards), 40 Gbps (Cisco Catalyst 6900 and 6800 series Line Cards), and 110 Gbps (Next-Generation Line cards) Channels. Each channel further consists of eight parallel serializer/deserializer (SerDes) lanes. These eight parallel SerDes lanes are aggregated together to form a single logical fabric channel. The Supervisor 6T provides up to Four Channels for each Module or Slot when installed on a 6807xl Chassis and up to 2 Fabric Channels for each Module or Slot when installed on a Cisco Catalyst 6500-E Chassis.

A high-level overview of the Supervisor 6T board layout is shown in Figure 1.

A summary of the Supervisor 6T critical features is listed in Table 1.

Table 1.       Important Baseboard Features of Supervisor 6T

The following sections provide more details on each of the major components of the Supervisor 6T.

System Level Requirements

The Supervisor 6T is designed to operate in any Catalyst 6500 E-Series chassis as well as in a Catalyst 6807-XL chassis listed in Table 2. The Supervisor 6T will not be supported in any of the earlier 6500 non-E-Series chassis. Table 2 provides an overview of the supported and non-supported chassis for Supervisor 6T.

Table 2.       Chassis Options for Supervisor 6T

For the E-Series chassis, a corresponding E-Series fan (or high-speed HS fan) for that chassis is required to support Supervisor 6T. While the 2500 W power supply is the minimum-sized power supply that must be used for a 6-, 9-, and 13-slot chassis supporting Supervisor 6T, the current supported minimum shipping power supply is 3000 W.

The Catalyst 6503-E requires a 1400 W power supply and the Catalyst 6504-E requires a 2700 W power supply when a Supervisor 6T is used in each chassis. Either AC or DC power supply options can be used. The 6500 E and 6807-XL series chassis types, as well as corresponding fan and power supply options that can be used with a Supervisor 6T, are detailed in Table 3.

Table 3.       Supported Chassis, Fan, and Power Supply for Supervisor 6T

The installation of the Supervisor 6T into a given chassis is always performed in specific slots. Keyed guide pins are used between the chassis and Supervisor connectors to manage this. Each chassis has two slots reserved for Supervisor use. These are called out Table 4.

Table 4.       Chassis Supervisor Slots

The Supervisor 6T provides backward compatibility with the existing WS-X67xx, WS-X68xx, and WS-X69xx Series Line cards. There is no support for any of the Legacy Line cards such as WS-X61xx, WS-X62xx, WS-X63xx, WS-X64xx, or WS-X65xx, CFC-based Line cards, and Service Modules.

There is no compatibility between the Supervisor 6T and earlier generations of Distributed Forwarding Cards (DFCs), such as DFC, DFC2, or DFC3x. The DFC is used to accelerate forwarding performance for the system as a whole, and uses the same forwarding architecture that is found on the PFC. The PFC4 architecture introduces a number of changes that differentiate it significantly in operation from earlier PFC and DFC models.

These changes require that only the DFC4 can interoperate with the PFC4. Any existing WS-X67xx Line cards that have a DFC3x will have to be upgraded with the new DFC4. WS-X67xx Line cards with a DFC4 installed to function in distributed dCEF720 mode.

Note:    Any newly purchased WS-X6700 Series Line cards that are shipped with a DFC4 or DFC4XL pre-installed have been renamed as WS-X6800 Series Line cards to clearly separate the performance differences.

Note:    Due to compatibility issues, the WS-X6708-10G-3C/3CXL cannot be inserted in a Supervisor 6T system, and must be upgraded to the new WS-X6908-10G-2T/2TXL

Table 5 summarizes the line cards that are supported by Supervisor 6T

Table 5.       Supervisor 6T-Compatible Line Cards

Note:    Any existing WS-X67xx Line cards can be upgraded by removing CFC or DFC3x daughter card and replacing it with a new DFC4 or DFC4XL. They will then be operationally equivalent to the WS-X68xx line cards but will maintain their WS-X67xx identification.

Table 6.       Line Card SKUs and Descriptions

Applicable line cards (from Table 6) utilizing a DFC3x will need to upgrade to a DFC4 in order to operate in a Supervisor 6T chassis. This level of interoperability is summarized in Table 7.

Table 7.       PFC/DFC Interoperability Matrix

Supervisor Architecture

The major processing blocks of the Supervisor 6T include the RP Complex (MSFC6) Port Complex, which houses Enhanced PFC4 and New crossbar Switch Fabric. Each of these block elements and their interconnection can be seen in Figure 2.

The following sections provide more detail for each processing block.

Baseboard

The Baseboard on the Supervisor 6T houses Front Panel ports, Integrated RP Complex, Forwarding ASIC (PFC4) and Switch Fabric daughter Boards. The Supervisor 6T supports two baseboard models that offer different levels of scalability based on the PFC4-E forwarding engine present. The two models are C6800-SUP6T (Lite) and C6800-SUP6T-XL (Heavy).

Table 8 outlines some high level differences between two models.

Table 8.       Comparison of Supervisor 6T Baseboards

Supervisor 6T Uplinks

The New Supervisor 6T includes Eight 10 Gigabit Ethernet Uplinks and two 40 Gigabit Ethernet Uplinks with 80 Gbps (2x20G Per Fire ASIC) on the backplane and 160 Gbps on the Front Panel, making it 2:1 Oversubscribed if all ports are used at line rate. Eight 10 Gigabit Ethernet ports are Native SFP+ ports Capable of operating in 10/100/1000, 1 Gigabit Ethernet, and 10 Gigabit Ethernet.

Similar to the Supervisor 2T, if Active and Standby Supervisor 6T are installed in the Chassis, both 10G and 40G ports are Active on both the supervisors and Forwarding traffic at the same time.

Supervisor 6T Uplinks Operating Modes

The Uplink ports on the Supervisor 6T can operate in one of four modes:

●   Normal/Default Mode – 8x10G and 2x40G Ports Active (2:1 Oversubscribed)

●   10G Performance Mode – 8x10G ports Active

●   40G Performance Mode – 2x40G Ports Active

●   10G Oversubscription Mode – 16x10G Ports Active (2:1 Oversubscribed)

The 10 GE and 40 GE Uplink ports are interleaved among the two EDC PHY and Cross-Connected to the Port ASIC as shown in Figures 3 - 6 to achieve these modes, in which ports can be operated without Oversubscription.

Normal/Default Mode

●   Achieved using the OEDC PHY

●   Port ASIC 0 operates in 40G Mode, hence 40G QSFP ports termination

●   Port ASIC 1 operates in 10G Mode, hence 10G SFP+ ports termination

10G Performance Mode

●   Achieved using the OEDC PHY

●   Port ASIC 0 operates in 10G Mode, hence 4x10 G Local SFP+ ports termination

●   Port ASIC 1 operates in 10G Mode, hence 4x10G Local SFP+ ports termination

●   40G Ports Administratively shutdown

40G Performance Mode

●   Achieved using the OEDC PHY

●   Port ASIC 0 operates in 40G Mode, hence 1x40G (Port 9) Local QSFP ports termination

●   Port ASIC 1 operates in 40G Mode, hence 1x40G (Port 10) Local QSFP ports termination

●   10G Ports Administratively shutdown

10G Oversubscription Mode

●   Achieved using the OEDC PHY

●   Using Cisco QSFP to Four SFP+ Copper Breakout Cable

●   Port ASIC 0 operates in 10G Mode, hence 4x10 G Local SFP+ ports termination

●   Port ASIC 1 operates in 10G Mode, hence 4x10G Local SFP+ ports termination

Fabric Interface and Replication Engine (FIRE) ASIC

This ASIC is used to provide a number of important functions combining multiple standalone ASIC into one Single ASIC for high speed Fabric connection to the Backplane and also enables replacement of RLDRAM’s with DDR3 DRAM’s. First and foremost, it receives packets from the front panel 10GE and 40GE ports, extrapolates valuable information from packet headers, and forwards this information to the PFC4E for packet lookup and associated forwarding services processing (Security, Quality of Service, NetFlow, and others). When packets return from packet lookup processing, this ASIC will perform packet rewrites according to the lookup result.

Another important processing role performed by this ASIC is multicast replication. This includes IGMP snooping for Layer 2 packets, as well as multicast expansion for Layer 3 multicast packets. Additionally, other replication services are supported to provision switched port analyzer (SPAN, ER-SPAN, and more) capabilities.

Port ASIC

There are two port ASICs on the Supervisor used to provision the front panel 8 x 10GE and 2 x 40GE ports. One port ASIC supports the First 4x 10 GE port and a single 40GE port. The other port ASIC supports the next 4x10 GE port and 2^nd 40GE ports with a total of 80G bandwidth towards the Port Side. The following list defines this port ASIC’s capabilities:

●   Per-port VLAN translation

●   VSL support

●   Cisco TrustSec^® support (802.1ae link layer encryption)

●   Jumbo frames (up to 9216 bytes)

●   Flow control

●   1P7Q4T-Default (one strict priority queue, seven normal round robin queues, and four WRED thresholds per normal queue) queue structure for both 10 GE and 40 GE ports – Both Receive and Transmit Queues

●   2P6Q4T-Configurable (Two strict priority queue, six normal round robin queues, and four WRED thresholds per normal queue) queue structure for both 10 GE and 40 GE ports – Both Receive and Transmit Queues

●   8MB/4MB Ingress/Egress buffer per port ASIC

●   Ingress QOS Support for DWRR, WRED, and SRR scheduling schemes

●   WRED and Tail Drop congestion management

●   802.1Q VLAN encapsulation

●   ECC protection

Multilayer Switch Feature Card (MSFC6)

The MSFC6 is a next-generation Intel CPU for the Supervisor 6T (Figure 7). It is no longer a daughter card; all the components are now integrated onto the baseboard. The MSFC6 performs Software control plane services for the switch. Control plane functions typically process all those features and other processes that are not handled directly in hardware by purpose built ASICs. The MSFC6 CPU handles Layer 2 and Layer 3 control plane processes, such as the routing protocols, management protocols like SNMP and SYSLOG, and Layer 2 protocols (such as Spanning Tree, Cisco Discovery Protocol, and others), the switch console, and many more.

Similar to the Supervisor 2T, the Supervisor 6T houses a single CPU that combines the RP and SP into one Complex.

Highlights include:

●   New 2.5-GHz Intel X86 IVB-Gladden dual-core CPU

●   Two DIMM Slots, Single 4GB 1600 Million (s of) Transfers per Second DDR3 ECC SDRAM is populated.

●   DDR3 can transfer data at twice the rate of DDR2.

●   Connectivity Management Processor (CMP) replaced with Direct OOB RJ45 or SFP network interface.

●   Support for USB Type A file system - Supports standard USB thumb drives as a Cisco IOS file system (for example, copy, delete, format, and boot)

●   Support for USB Type B serial console - Supports Standard Type B (mini) serial console cables (in addition to RJ-45 serial)

●   Compact flash replaced with eUSB - Internal Enhanced USB (eUSB) flash will support 4GB of storage for System Boot Image, Configuration Files & On Board Fault Logging.

●   New switched EOBC interface - Supports existing EOBC architecture but uses a point-to-point (P2P) switched design for dedicated bandwidth

●   Secure Bootup (enabled by Default) for securing a Cisco IOS^® Software image.

Figure 8 shows a block diagram of the MSFC6.

The specifications for the MSFC6 CPU complex as compared to the MSFC5 (Supervisor 2T) and MSFC3 (Supervisor 720-10G) are shown in Table 9.

Table 9.       SFC3 Comparison with MSFC5 and MSFC6

Switch Fabric

The Supervisor 6T supports Switch Backplane, which is the Crossbar Switch Fabric as compared to two different switch backplanes on the Supervisor 2T. The Bus Based Backplane is not supported on the Supervisor 6T, which in turn eliminates the supports for CFC based Line cards and Service Modules.

This backplane refers to the 6-Terabit backplane that is contained within the Supervisor’s name. The crossbar switch fabric provides a set of fabric channels (or data paths) that are assigned to the slots in the chassis where line cards are inserted. This is referred to collectively as the crossbar switch backplane. This array of fabric channels provides an any-to-any (full-mesh) connection option for the attached line card to forward data over a dedicated path to any other line card installed in the chassis.

Crossbar Switch Fabric

The crossbar switch fabric on the Supervisor 6T provides 3080 Gbps of switching capacity. This capacity is based on the use of 28 fabric channels that are used to provision data paths to each slot in the chassis. Each fabric channel can operate at 20, 40, and 110 Gbps, depending on the inserted line card. The capacity of the switch fabric is calculated as follows:

●   28 x 110 Gbps = 3080 Gbps

●   3080 Gbps x 2 (full duplex) = 6160 Gbps

The 6160 Gbps number is a marketing number (common among all switch vendors) and is used in all literature to denote that full duplex transmission allows data traffic to be transmitted and received simultaneously, while the switch fabric capacity is documented as a full duplex number.

The Supervisor 6T on the Catalyst 6807x provides up to Four Channels for each Module Slot (Figure 3). Each channel further consists of eight parallel serializer/deserializer (SerDes) lanes. These eight parallel SerDes lanes are aggregated together to form a single logical fabric channel. Figure 9 shows a fabric channel layout in the Catalyst 6807-XL and Figure 10 shows a fabric channel layout in the Catalyst 6509-E.

The 80 Gbps per slot nomenclature on the 6500-E Series represents 2 x 40 Gbps fabric channels that are assigned to each slot providing for 80 Gbps per slot in total. If marketing math were used for this per slot capacity, one could argue that the E-Series chassis provides 160 Gbps per slot.

Clock Frequency

All computer-based data is represented by binary “bits”, often referred to as “1’s” and “0’s”, which are simple representations of either the presence of (1) or absence of (0) an electrical or optical signal. The amount of time between the arrival of each individual “bit” is called its “frequency”, measured in Hertz (Hz), and controlled by a central “clock”.

What we commonly refer to as the “speed” of a piece of hardware is actually a reference to its clock frequency. The faster the frequency, the faster individual bits will arrive. However, these bits must be interpreted correctly in order for the higher functions to make sense of them. This requirement is the purpose of line encoding, discussed in the next section.

Note:    Several lesser factors also influence the maximum clock frequency, such as the characteristics and quality (for example, signal integrity) of the physical medium, the design and operation of the components (such as the SerDes), the accuracy of the central clock, and others.

Note:    These additional factors are also beyond the scope of this document, but it is important to point out that a chassis must also support them in order to support higher frequencies.

The clock frequencies supported by the Supervisor 6T are:

●   3.125 GHz: For up to 20 Gbps per channel

●   6.25 GHz: For up to 40 Gbps per channel

●   15.0 GHz: For up to 11 0Gbps per channel

Line Encoding

Along with clock frequency (which dictates the arrival rate of data “bits”), the transmitting and receiving equipment must both agree about how to interpret “good” data from “bad” (corrupt) data.

Furthermore, if data transmissions (bits) are simply the presence or absence of signal (1 or 0), how can the receiving equipment understand the difference between a “good” transmission of bits, or if it is actually experiencing interference, or if the transmission has completely ceased?

The transmitting equipment sends a fixed number of bits together in a sequence (encoding), and then the receiving equipment uses some portion of the total bits to determine if the transmission was successful (or not).

The line encodings supported by the Supervisor 6T are:

●   08/10b: For up to 20 Gbps per channel

●   24/26b: For up to 40 Gbps per channel

●   64/66b: For up to 110 Gbps per channel

Along with clock frequency and number of channels, this is how a Supervisor 6T is capable of achieving up to 440 Gbps (8 SerDes at 15 GHz at 64/66b = 110 Gbps per channel x 4 channels).

Policy Feature Card (PFC-E)

The PFC present on the Supervisor 6T is an improved Policy Feature Card (PFC4-E) forwarding controller, which is based on the same EARL8 with a few enhancements and fixes, as compared to the PFC present on the Supervisor 2T. The Enhanced PFC4 is still a 60-MPPS Layer 2 and Layer 3 Forwarding Engine that integrates a Layer 2 and Layer 3 ASIC into one Single ASIC with Improved Intra-ASIC Read Write Bandwidth.

Highlights include:

●   PFC4-E merges Layer 2 and Layer3 ASICs

◦     Enhanced Performance with Equal Scale

◦     Improved Intra-ASIC read/write bandwidth

◦     60 Mpps Layer 2 and 3+ Forwarding

◦     256K, 1M, or 2M FIB TCAM Entries

◦     128K MAC Address CAM Entries

●   Integrates external SRAMs

◦     Uses 4 sets of 32K x 96-bit eDRAM

◦     Full ECC with an additional 8 bits on read/write

●   Reduces pin count and block size

◦     Uses 3 RLDRAM3 chips at 600 MHz

◦     Support for 2M FIB entries^* (4 TCAMs)

PFC4 provides hardware accelerated forwarding for packets traversing the switch. This includes forwarding for IPv4 unicast/multicast, IPv6 unicast/multicast, Multi-Protocol Label Switching (MPLS), and Layer 2 packets. Along with forwarding, the PFC4-E is also responsible for processing a number of services that are handled during the forwarding lookup process. This includes, but is not limited to, the processing of security Access Control Lists (ACLs), applying rate limiting policies, quality of service classification and marking, NetFlow flow collection and flow statistics creation, EtherChannel load balancing, packet rewrite lookup, and packet rewrite statistics collection.

The DFC4 is a daughter card that is located on Supervisor 6T supported line cards. The DFC4 contains the same ASIC functional blocks that are found on the PFC4. The main purpose of the DFC4 is to provide local forwarding services for the line card, offloading this function from the PFC4. Using DFC4s helps scale performance of the overall chassis. The maximum performance of PFC4-E on the Supervisor 6T is 60 Mpps (for IPv4 forwarding). Each DFC4 on a line card adds an additional 60 Mpps of forwarding performance to the aggregate forwarding capacity of the chassis. All of the information about the PFC4-E in this white paper equally applies (function, scalability, and performance) to the DFC4.

The PFC4 is located on the right side of the Supervisor 6T baseboard, as shown in Figure 11, and it is not Field Fruable.

The location of the DFC4 on a line card is shown in Figure 12.

PFC4 Architecture

The PFC4 is made up of two main ASIC processing blocks, along with a number of high-speed memory blocks that serve to provide the hardware acceleration of selected features. One ASIC block performs Layer 3 services, while the other ASIC block performs Layer 2 services. A high-level view of the PFC4 is shown in Figure 13.

At the center of the PFC4 complex are the two forwarding engines. These two ASIC complexes are responsible for the forwarding of all Layer 2 and Layer 3 packets in hardware. Attached to each of these ASIC blocks are a series of tables that are used to store information that facilitates the forwarding of packets in hardware.

The following sections provide more details about each of these two forwarding engines and the associated tables with which they interface.

Layer 2 Forwarding Engine

This engine is responsible for Layer 2 packet processing and supports a number of enhancements beyond those found on the Layer 2 forwarding engines in previous PFC3x complexes. Integrated into the forwarding engine ASIC is a MAC address table containing 128 K entries. The MAC address table consists of two banks of 4 K lines with 16 entries per line (2 x 4 K x 16 = 128 K entries). Each entry in the MAC address table is 115 bits wide, and contains forwarding and aging information related to a destination MAC entry and an associated bridge domain pair.

Rather than running a hash operation to find a pointer into the first bank, and then run the hash again to derive a pointer into the second bank, two simultaneous hash functions are performed at the same time to provide a pointer into each of the memory banks. In this manner, the Layer 2 lookup performance is at its best.

Prior to PFC4, every interface in the system was identified by a VLAN ID, including internal uses such as Layer 3 sub-interfaces, VPNs, tunnels, and egress multicast replication-mode. This restricted the total number of unique interfaces to 4096. The new Bridge Domain (BD) concept is one of the more significant enhancements introduced with the PFC4, and is designed to increase scaling of VLANs internally to the switch. When a user creates a VLAN, it maps internally to a unique bridge domain. Support for 16 K bridge domains is built into PFC4 hardware. However, at first customer ship, only 4 K VLANs will be supported by the software running on the Supervisor 2T.

All frames entering the Layer 2 forwarding engine are associated with a Logical Interface (LIF), which is, in essence, a map to a port index and VLAN pair on which the frame entered the switch. A LIF database of 512 K entries (each comprised of BD, LIF, and control bits) resides in the Layer 2 forwarding engine. Each LIF entry is ultimately used to facilitate Layer 3 processing whenever the packet is passed to the Layer 3 forwarding engine. Along with the LIF database is a LIF statistics table. This table maintains diagnostic VLAN counters, along with byte and frame count statistics per ingress and egress LIF, and consists of one million entries.

The Layer 2 forwarding engine maintains a set of Access Control Entry (ACE) counters. When the Layer 3 forwarding engine performs classification processing, it will communicate with the Layer 2 forwarding engine to update ACL counters when a hit is registered against an ACE (such as a line in an ACL list).

Table 10 provides a summary of the main differences between Layer 2 support in the PFC4 and previous PFC3x versions.

Table 10.     PFC4 Layer 2 Forwarding Engine Features

Even for Layer 3 routed flows, the Layer 2 forwarding engine performs a number of critical services prior to handing the packet over to the Layer 3 forwarding engine for Layer 3 processing. Included in its processing list are the following functions:

●   Performs CRC error checking

●   Performs LIF and BD (bridge domain) lookup

●   Maintains LIF statistics

●   Performs Layer 2 MAC table lookup

●   Calculates Result Bundle Hash (RBH) or EtherChannel load-balancing hash

●   Determines 16 static MAC match conditions for system frames (such as CDP, BPDU, and more)

●   Performs IGMP, MLD, and PIM snooping

●   Performs hardware rate limiting

●   Provides ACE counters

Layer 3 Forwarding Engine

The Layer 3 forwarding engine performs Layer 3+ services, including IPv4, IPv6, and MPLS forwarding lookups, as well as Security, QoS, and NetFlow policies on packets traversing the switch. There have been a number of enhancements incorporated into this PFC4 Layer 3 forwarding.

From a capacity standpoint, fundamentally faster packet processing, support for more NetFlow entries, and more ACLs form only part of the wide range of features whose operational limits have been increased. Some new features have also been introduced, such as support for egress NetFlow, egress microflow policing, and distributed policing.

A summary of the major changes for this Layer 3 ASIC, compared to earlier PFC3x found on previous Supervisor engines, is detailed in Table 11.

Table 11.     PFC4 Layer 3 Forwarding Engine Features

Layer 3 Forwarding Engine Processing Paths

One of the Limitations of Previous PFCs is that decisions are made in Multiple Cycles, thereby adding latency to the whole forwarding process. PFC4 makes many of these decisions in a single pass by going through the Layer 2 and Layer 3 components in a step-by-step process. The Component that performs Layer 3, QOS, NetFlow, and ACL functionalities is implemented in a pipeline mode, with each stage in the pipeline performing a specific task. The Two logical pipelines that make up the physical pipeline are the Input Forwarding Engine (IFE) and Output Forwarding Engine (OFE).

●   The IFE pipeline performs ingress functions, including input classification, input QOS, ACLs, RPF checks, ingress NetFlow, and Layer 3 FIB-based forwarding.

●   The OFE pipeline performs egress functions, including adjacency lookup, egress classification, and rewrite instruction generation.

When a packet header enters the Layer 3 ASIC, the IFE pipeline is the first pipeline to process the packet. After completion of IFE processing, the header is then passed onwards to the OFE pipeline, along with the results of the IFE processing. This can be seen in Figure 14.

The processing cycles for IFE and OFE processing are always performed in an IFE/OFE order. At the completion of OFE processing, the Layer 3 forwarding engine will collate the results and hand the packet back to the Layer 2 forwarding engine for onward processing. The processing performed by this ASIC is displayed in Table 12.

Table 12.     PFC4 Layer 3 Forwarding Engine IFE and OFE Functions

PFC4 (and PFC4XL) Functional Elements

The Layer 3 forwarding engine contains a number of functional elements, all of which work together to provide Layer 3 processing for packets. The main functional elements are executed in the order shown and can be seen in Figure 15.

Each of these functional elements is detailed in the following sections.

Interface Processing and LIF Map Table

A LIF is a new concept introduced with the PFC4. The LIF helps enable per-port-per-VLAN interface processing, which separates Layer 2 characteristics from Layer 3 characteristics. The LIF map table (which is separate from, but is mapped to, the LIF table located on the Layer 2 forwarding engine) contains 128 K entries, and helps scale the number of Layer 3 interfaces and sub-interfaces that can be supported by the PFC4. There are two LIF map tables used by the Layer 3 forwarding engine: one for ingress LIFs and one for egress LIFs.

Built within the LIF map table entry are a number of fields that define the operating characteristics associated with each LIF. Some of the information contained within a LIF entry is used as an input for subsequent table lookups by other Layer 3 forwarding engine processes.

Examples of the information contained within a LIF map table entry include:

●   Security Group Tag (SGT) - applicable for CTS processing

●   Tunnel interface information (GRE, EoMPLS, IPv6, and others)

●   RPF lookup requirements

●   MPLS and EoMPLS interface information

●   MPLS VPN ID

●   IPv4/IPv6 VPN ID

●   Trusted or un-trusted state (from a QoS perspective)

●   ACL label

For ingress processing, the LIF table:

●   Helps derive per logical interface configured parameters such as ACL labels, VPN, and more (some of this information forms part of the lookup key in other tables)

●   Holds information that is used for Port ACL (PACL) lookup

●   Supports IP multicast filtering

●   Filters at Layer 3 for MPLS packets

For egress processing, the LIF table:

●   Helps derive per logical interface configured parameters such as ACL labels, VPN, and more (some of this information is used as part of the lookup key in other tables)

●   Performs interface check for routed packets

●   Provides source filtering for multicast packets

●   Performs scope enforcement for IPv6 packets

●   Supports MPLS Fast Reroute for FRR-TE tunnels

RPF Processing and the RPF Map Table

The reverse path forwarding check is used to confirm that the source IP address associated with a frame is received on the interface that the FIB table lists as the correct source or RPF interface. For unicast forwarding, the RPF check is performed to stop IP address spoofing through malformed or forged source IP addresses.

The PFC4 supports a maximum of 16 RPF interfaces for both IPv4 and IPv6. Both PFC3B/XL and PFC3C/XL supported looking up two interfaces during the RPF lookup, and none of the PFC3x forwarding engines supported IPv6 RPF lookups.

Following is an example of how RPF is used. A packet arrives on interface 3/1 and has a source address that is part of the subnet 193.10.1.x. The RPF process performs a reverse lookup on the forwarding table. Rather than looking at the destination address, it uses the source address for the lookup. The lookup determines that packets from network 193.10.1.x should arrive on interface 3/5. In this case, as the packet arrived on interface 3/1, it is deemed to be a spoofed packet, and thus, is dropped in hardware.

The RPF processing block responsible for the RPF check is also responsible for a number of other processing checks as well. Additional processing checks:

●   IP Multicast forwarding requires an RPF check, to build its distribution tree. PIM uses the RPF information to determine which interface to send Join and Prune messages, for a given IP source

●   To support MPLS aggregate VPN support, a lookup using the MPLS label can be performed to determine the MPLS VPN ID associated with it

●   For IPv4 and IPv6 packets, a source MAC to IP binding check can be performed

●   For IPv4 and IPv6 packets, source AS mapping check can be performed, which facilitates a later lookup in the ACL TCAM

●   For IPv4 and IPv6 packets, a source group tag (SGT) lookup can be performed

●   VPN and QoS mapping for MPLS packets are supported

Classification Processing and the Classification Memory (ACLs)

Two TCAM banks providing a total of up to 256 K access control entries are available for classification ACLs. By default, the PFC4 reserves 16 K entries for QoS ACEs and 48 K entries for Security entries. The PFC4XL reserves 64 K entries for QoS ACEs and 192 K entries for Security ACEs. The first ACL TCAM bank is used for standard QoS and Security ACL entries, and the second TCAM bank is used for Security and CTS (RBACL) entries, which require label-based functions. This combined TCAM design can be used to store Security ACLs, QoS ACLs, RBACLs, or accounting results.

The Layer 3 forwarding engine provides for a dual lookup into each bank, allowing for four lookups to be performed simultaneously. This means that for each input packet, up to four classification rules can be matched during IFE (ingress) processing, and up to four classification rules can be matched during OFE (egress) processing.

Each classification TCAM entry is 144 bits wide, and uses a lookup key to initiate a lookup into the TCAM. This lookup key uses input fields such as the ACL label (which is obtained from the LIF table), packet type (IPv4, IPv6, and more) and other fields to generate the key. The result of the TCAM lookup provides a pointer into the classification SRAM that holds the actual ACE entry.

The Layer 3 forwarding engine works in conjunction with the Layer 2 forwarding engine, which holds and maintains the hit counters for each access control entry. Every time a classification ACL is matched during IFE or OFE processing, the ACL counters in the Layer 2 forwarding engine are updated to reflect the hit.

The classification TCAMs have the following capabilities:

●   Ingress DSCP mutation, using one of 14 ingress DSCP mutation maps selected by ingress LIF map table lookup

●   Address compression on IPv6 addresses, prior to lookup into the TCAM

●   Security ACLs (ingress/egress classification) return a permit/deny for the packet, based on configured ACL policies

●   ACL redirects (ingress/egress classification), which define which packets are to be redirected (this mechanism can be used for features such as policy-based routing through input classification)

●   RPF+ (ingress classification only), which provides the ability to ignore an RPF result for certain classified flows

●   RBACL support. The generation of SGT and DGT based on IP classification. Also, classification of flows based on SGT (input and output classification) or DGT (only output classification) of the packet is enabled through these TCAMs

●   Ability to generate ACL-based VPNs (ingress classification only), useful for implementing VRF select or newer routing mechanisms like multi-topology routing (MTR)

●   Service cards virtualization, which provides the ability to generate a virtual ID based on ingress or egress classification

●   Ability to specify the QoS for a classified flow, and a policer index for aggregate policing, based on input or output classification

●   ACL-based accounting (ingress and egress classification), plus the ability to provide drop counters on each ACE to implement ACL-based accounting

●   Generation of fields required for NetFlow lookup (ingress and egress classification)

●   Code logic that interfaces with the classification TCAMs also interfaces with the Layer 2 forwarding engine to maintain ACE statistics

NetFlow Processing and the NetFlow Hash and Data Tables

NetFlow is a hardware-enabled process that collects statistics on packet flows that traverse the switch. A flow is identified by a flow mask, which uses fields from the packet header to determine what constitutes a flow. The default flow mask uses the source and destination IP address, the source and destination port number, and the IP protocol to determine a flow.

Here is an example of how this is used. A user initiates three sessions: one email, one web, and one print. Each packet flow uses the same source IP address, but has a different destination IP address and port number pair. Using the default full flow mask, each session will be viewed as a separate flow, and statistics for each flow would be counted individually (a full flow mask uses the IP protocol field, src/dest ip address and src/dest port number to identify a unique flow).

However, if the administrator were to use, for example, a source-only flow mask (each flow identified by source IP address only), then the statistics count result would be different. A source-only flow mask identifies all packets associated with the same source IP address as part of the same flow. In our example, the same user who initiated three sessions (email, web, and print) would now have all of that session data collected under one flow record, rather than three individual flow records when using the other flow mask. There are a number of flow masks that can be chosen by the user, and these can be configured as needed by the administrator.

At this stage of the processing flow, NetFlow processing is implemented across two banks of Reduced Latency DRAM (RLDRAM). One bank (NetFlow hash table) acts as a hash table holding pointers into the second bank (NetFlow data table), where the actual NetFlow data is held. The NetFlow data table is 288 bits wide, of which 160 bits represent the NetFlow key, and the other 128 bits are used for holding NetFlow data. A total of 1 M NetFlow entries can be stored (in the PFC4XL), and this is split evenly between IFE and OFE. For ingress (IFE) NetFlow, the Layer 3 forwarding engine maintains 512 K entries. Likewise, for egress (OFE) NetFlow, an additional 512 K entries are offered. For the non-XL version of the PFC4, the NetFlow table can hold up to 512 K entries, and those can be shared between both IFE and OFE.

NetFlow flow records can be created for IPv4 flows, IPv6 flows, and VPN flows. The IPv4 and VPN consume one entry, while IPv6 flows consume two entries. NetFlow statistics are maintained for each flow.

When a packet arrives for NetFlow processing, the flow mask is used to determine which fields in the packet header will be used to build a lookup key, which is used to search the NetFlow hash table for an existing entry. If an entry is found, the lookup will return a pointer into the NetFlow table and the NetFlow statistics table. If no entry is found, then a pointer entry is created and a flow record will be created in the NetFlow table.

In parallel with the increase in NetFlow entries is a corresponding increase in the number of flow masks that are supported. The flow mask is an important element in the NetFlow process, and warrants additional attention. As described earlier, the flow mask defines what constitutes a flow, and has an impact on how statistics are collected for different packet streams. With the previous PFC3x forwarding engines, there were six flow masks, of which two were reserved for system use. This left two flow masks that could be used by the user. The main use for the two different flow masks was with user-based rate limiting (UBRL), which allowed the user to configure different policers with different flow masks. With the PFC4, there are now 80 flow masks available for use. Of the 80, there are 32 flow masks for IPv4, 32 flow masks for IPv6, eight flow masks for MPLS, and eight flow masks for Layer 2 packet flows.

Microflow policing can be performed on every NetFlow entry in the NetFlow data table. This means you could potentially have 512 K ingress microflow policers and 512 K egress microflow policers in action at the same time when the system operates in PFC4XL mode.

While sampled NetFlow was available on the PFC3x forwarding engines, it was performed in software by the control plane. Sampled NetFlow on the PFC4 is a new feature now supported by the Layer 3 forwarding engine as a hardware process. It offers a way to reduce the amount of information that is collected about a given flow. Supervisor 2T supports 1:N Sampling, a method which allows one packet to be chosen out of N packets (for example, 1 per 1000). Sampling operates in a random mode, meaning any packet within sample N will be randomly selected. Samplers are global, and a total of 1 K NetFlow samplers are supported.

A third piece of memory (ICAM) is also used for NetFlow. This piece of memory is used to store flows that have a conflict, due to a hash collision or page full condition in the primary NetFlow table. The ICAM can support 16 NetFlow entries that are shared between the IFE and OFE processes. Given the 99 percent efficiency of the NetFlow hashing algorithm and the aging mechanisms to age out flows that have ended, the likelihood of needing this ICAM is greatly decreased.

Layer 3 Processing and the Forwarding Information Base/Adjacency Tables

The Forwarding Information Base (FIB) table contains a hardware representation of the Layer 3 forwarding tables found in the control plane. The Catalyst 6500 uses Cisco Express Forwarding architecture to enable hardware forwarding. The routing protocols collect next-hop information about the routing topology, and maintain this in their respective protocol tables in the control plane. This forwarding information is consolidated into the global routing table called the RIB (Routing Information Base). CEF then takes this global routing table (RIB) and creates a FIB table that is pushed down into the PFC4/DFC4 FIB TCAM. All Layer 3 forwarding in hardware uses this FIB table to perform Layer 3 forwarding lookups.

The FIB in the PFC4 contains 256 K entries, while the FIB in the PFC4XL contains 1 million entries. These are the same as their PFC3x forwarding engine counterparts. The FIB in the PFC4 contains prefix entries for IPv4 and IPv6 global address, IPv4 and IPv6 multicast addresses, and MPLS label entries. There is a level of partitioning that exists to ensure there is always some space available for different types of forwarding entries. There is some flexibility from a user configuration standpoint that allows these partition boundaries to be changed to accommodate more of one type of forwarding entry. For example, in the PFC4XL, the default setting provides for 512 K IPv4 entries, and this can be increased through configuration control to support up to 1 M entries if required.

The PFC4 FIB is actually comprised of two memory blocks: one is TCAM-based and the other is RLDRAM-based. When performing a Layer 3 lookup, the FIB TCAM lookup is performed first. To execute the lookup, a FIB TCAM lookup key is derived, based on incoming packet type and other fields, to perform a FIB TCAM lookup. The result of the FIB lookup returns a pointer into the FIB RLDRAM, which will hold a pointer into the adjacency table for normal forwarded packets. If the adjacency pointer indicates the destination can be reached through multiple paths, it computes a unique adjacency pointer for each path.

With the earlier PFC3x on the Supervisor 720, the adjacency table also contained rewrite information for rewriting the Layer 2 header, when the packet left the switch. With the PFC4, this information has now been moved to the rewrite table (discussed later). The adjacency table contains a pointer to the Local Target Logic (LTL) index. The LTL index is an internal pointer that represents an interface in the switch, which essentially identifies the egress interface out of which the packet is going to be sent. The adjacency table also contains a flood index should the packet need to be sent out multiple interfaces.

The FIB is also now used to perform a RPF check for IP multicast packets. This is a new feature of PFC4 not found in earlier PFC3x forwarding engines.

Note that for OFE processing, the FIB is bypassed and does not participate in packet processing for that pipeline.

Policer Processing and the NetFlow Statistics Table

The policing logic implements the following functionality:

●   Performs the IFE and OFE TTL check

●   Performs an MTU check prior to egress policing

●   Performs distributed and non-distributed three-color aggregate policing

●   Performs two-color shared and non-shared NetFlow (microflow) policing

●   Maintains NetFlow and aggregate policing statistics

●   Supports both packet and byte based policing for aggregate and NetFlow policing

The NetFlow statistics table is maintained by the policing processing logic and consists of 1 M entries. It consists of three banks of memory, and is used to maintain a NetFlow statistics entry for every active flow in the system. A NetFlow statistics entry consists of multiple fields, including the packet timestamp, byte count for NetFlow policed packets, forwarded packet and byte count, last used timestamp, TCP flag, and more.

Rewrite Processing and the Rewrite Info Table

The rewrite process engine on the Layer 3 forwarding engine is designed to perform the tasks associated with generating next-hop rewrite instructions for outbound packets. The rewrite process will generate a new source or destination MAC address, and provide a rewrite pointer for the multicast expansion table (MET). Other important processing elements of the rewrite engine include:

●   The rewrite information for source and destination MAC address

●   The Multicast Expansion Table (MET) pointer that is used to specify all Outgoing Interfaces (or OIFs that a multicast packet needs to be replicated to

●   Can initiate packet recirculation for special processes that require it, such as VPLS, GRE tunneling, and IPv6 tunneling operations

●   MPLS operations for pushing, popping, and swapping labels

◦     Push up to five labels

◦     Pop one or two labels

◦     Swap one label

◦     Swap one label plus push up to four labels

◦     Swap two labels

●   EoMPLS encapsulation for up to five labels

●   EoMPLS de-encapsulation with one non-null label or two labels with top label as null label

●   IPv4 Network Address Translation (NAT)

●   Fast Re-Route (FRR) support

●   Provides TOS, traffic class, and EXP rewrites

●   Provides TTL rewrites

The rewrite information table contains 1 M entries, matching what the LIF table supports. It will generate a rewrite instruction set providing instructions for what the outgoing packet’s Layer 2 frame addressing will use.

Packet Processing and the Adjacency Stats Table

The main objective of this component is to build and pass the result of the processing of all processing blocks above back to the Layer 2 forwarding engine. The packet processor also serves as a conduit between IFE and OFE processing, taking the results of IFE processing and building the input required for OFE processing.

The adjacency statistics table contains 512 K entries and maintains a packet and byte counter for original packets (and not copy packets). In other words, a packet that has SPAN processing applied to it would not be counted by the adjacency statistics.

This processing block also maintains a table that holds accounting statistics (4 K entries).

PFC4 and PFC4-XL Feature Review

The PFC4 offers enhancements and new capabilities over earlier generations of the PFC. These hardware feature capabilities are summarized in Table 13.

Table 13.     New PFC4 Enhancements

The following sections provide a brief overview of each of the enhancements noted in Table 13, divided into Layer 2 and Layer 3 functional groups.

Layer 2 - Increased MAC Address Support

A 128 K MAC address table is standard on both models of the PFC4. The MAC address table has been enhanced to support additional fields in a MAC address table entry, such as the bridge domain (BD) that the MAC address is a part of.

Layer 2 - Bridge Domains

The bridge domain is a new concept that has been introduced with PFC4. A bridge domain is used to help scale traditional VLANs, as well as to scale internal Layer 2 forwarding within the switch. Bridge domains are mapped to each VLAN that a user configures, as well as other resources such as an EoMPLS/VPLS tunnel, Layer 3 sub-interfaces, and multicast Egress replication-mode. In essence, a bridge domain is equal to a VLAN (which is a 12-bit ID, allowing 4096 unique values). Bridge domains use a 14-bit ID (12 bits are VLAN ID), for a total of 16 K bridge domains supported in hardware by the PFC4.

Layer 2 - Increased Logical interfaces

The PFC4 introduces the concept of a Logical Interface (LIF), which is a hardware-independent interface (or port) reference index associated with all frames entering the forwarding engine. A LIF is a 72-bit internal address comprised of the bridge domain (BD), the source port index, and other associated information (for example, protocol type) used by the PFC4 to facilitate forwarding decisions. This allows Layer 2 forwarding characteristics to be logically separated from Layer 3 characteristics. The PFC4 adds hardware support for up to 128 K LIFs.

Layer 2 - Improved EtherChannel Hash

Skew tables have been introduced to overcome issues with distributing traffic over an odd number of links (3, 5, 6, and 7) that form part of an EtherChannel link bundle. Previous PFC3x engines offered balanced distribution across even numbered link (2, 4, and 8) bundles. However, link bundles with 3, 5, 6, or 7 member ports could end up with uneven distribution. The use of skew tables, as part of the processing logic for selecting a link in a bundle, helps alleviate those earlier problems.

Inputs (fields) used for the hash itself have also been extended. Options for including the input and output interfaces into the hash, along with the VLAN ID, provide for more granular link selection. These options also help overcome earlier issues with link selection for multicast traffic over VLAN trunks, where one link in the bundle could be favored over others.

Layer 2 - VSL Support

VSL is a technology that allows two physical Catalyst 6500 switches to operate together as a single logical unit. In this mode, the Supervisor Engine 2T in one chassis is the SSO Active and the Supervisor Engine 2T in the other chassis is the SSO Standby. As with standalone (non-VSS) high availability, the Active Supervisor is responsible for the control plane. While only one of the two control planes is active in a virtual switch setup, both data planes are active. Having both data planes active allows the virtual switch system to optimize hardware forwarding and use both PFC4 engines to perform hardware-enabled tasks for each chassis. Support for VSL is included with the PFC4 and mirrors that found in the previous PFC3C and PFC3C-XL Supervisor Engines 720.

Layer 2 – Per-Port-Per-VLAN

This feature is designed for Metro Ethernet deployments where policies based on both per-port and per- VLAN need to be deployed. Typically, these scenarios define a network deployment model where different VLANs carrying different customer traffic are carried on the same physical interface. Earlier PFC engines only allowed port-based or VLAN-based polices to be applied to a port at any one time. A port-based policy would disregard VLAN information, and likewise a VLAN-based policy would disregard port information. Support for this new interface type in the PFC4 allows per-port-per-VLAN policies to be applied where both the VLAN and port can be considered in the assigned policy.

Layer 3 - Increased Layer 3 Forwarding Performance

The PFC4 forwarding engine now supports forwarding performances of up to 60 Mpps for both Layer 2 and IPv4 Layer 3 forwarding. Technically, it is actually a 120 Mpps forwarding engine, but since each packet passes through the PFC4 twice (once for ingress processing and once for egress processing, which is discussed later), the effective forwarding performance equates to 60 Mpps.

Due to the length of the address, IPv6 requires an additional internal cycle, which makes the effective IPv6 forwarding performance 30 Mpps. Note that the PFC3B and XL forwarding engines supported forwarding performance up to 30 Mpps for IPv4 and 15 Mpps for IPv6, while the PFC3C and XL forwarding engines supported forwarding performance up to 48 Mpps for IPv4 and 24 Mpps for IPv6.

Layer 3 - Unicast Reverse Path Forwarding (uRPF) for IPv6

Unicast Reverse Path Forwarding (uRPF) is a tool that can be used to protect against address spoofing. The uRPF check performs a lookup into the FIB, using the source address as the lookup index (as opposed to the destination address). The lookup is performed to determine if the packet arrived on an interface where the source address of the packet matches the network found through that interface. If the packet had a source IP address of “A”, but it arrived on an interface from where network “A” does not exist, then the packet is deemed to have been spoofed and will be dropped.

The PFC4 extends support for uRPF to include IPv6, which was not available in the PFC3x. More importantly, PFC4 now supports 16 prefixes during both an IPv4 and an IPv6 uRPF lookup, compared to only supporting the lookup of 2 prefixes with the PFC3x.

Layer 3 - Tunnel Source Address Sharing

With the PFC3x family, each tunnel was required to use a unique source IP address as the tunnel source. If two or more tunnels were configured using the same source address, packets for the second and subsequent tunnels configured would be switched in software. With the PFC4, this scenario has been enhanced and now multiple tunnels can be configured to share the same source address, while the tunneled packets are switched in hardware.

Layer 3 - IPv6 Tunneling

The PFC3x did not support IPv6 tunneling options where the outer header was an IPv6 header. This behavior has changed with PFC4. The PFC4 now supports the following new IPv6 tunneling options:

●   ISATAP (Intra-Site Automatic Tunnel Address Protocol): ISATAP tunneling protocol allows IPv6 packets to be globally routed through the IPv6 network, while also being automatically tunneled through IPv4 clouds locally within a site. ISATAP tunneling uses an address format with specially constructed 64-bit EUI-64 interfaces ID. PFC4 fully supports ISATAP tunnels, whereas PFC3x offered only partial support.

●   IPv6 tunnel for GRE packets: PFC4 supports IPv6 GRE tunneling mode, where both IPv4 and IPv6 packets can be encapsulated in an IPv6 GRE header and tunneled across an IPv6 network. The PFC3x had limited support for this tunneling mode. The PFC4 can support up to 256 IPv6 GRE tunnels.

●   IPv6 generic tunneling: IPv6 packets can be encapsulated in an IPv6 or IPv4 header and tunneled across the IPv6 network. The PFC4 offers support for this mode and 256 tunnel source addresses are supported. The PFC3x family did not support this tunneling option.

Layer 3 - VPLS

Virtual Private LAN Service (VPLS) allows for a Layer 3 MPLS network to appear to operate like a Layer 2 Ethernet-based network. It effectively allows native Ethernet frames to be transported across an MPLS network providing any-to-any connectivity, just as if it were on a normal Ethernet LAN. Previous PFC engines required an external WAN card, such as the OSM or SIP-400, to provide support for VPLS. With PFC4, VPLS is supported natively, and no external WAN cards are required for this support.

Layer 3 - MPLS over GRE

The PFC4 now supports the MPLS over GRE tunneling option in hardware. This was not supported in previous PFC3x engines. This capability is important for those customers looking to join MPLS backbones over a common IP network.

Layer 3 - MPLS Tunnel Modes

MPLS provides a number of options for controlling how the experimental bit (EXP) can be set. The EXP value provides a way for a priority value to be assigned to an MPLS packet. The EXP setting is three bits providing 23 values (eight values). The mechanisms used to control the setting of the EXP bit are referred to as tunnel modes, and are defined in RFC3270. The three tunnel modes are called uniform, short pipe, and pipe tunnel modes.

In uniform mode, any changes made to the EXP value of the topmost label on a label stack are propagated both upward, as new labels are added, and downward, as labels are removed.

In short pipe mode, the IP precedence bits in an IP packet are propagated upward into the label stack as labels are added. When labels are swapped, the existing EXP value is kept. If the topmost EXP value is changed, this change is propagated downward only within the label stack, not to the IP packet.

Full pipe mode is just like short pipe mode, except the per-hop behavior (PHB) on the mpls2ip link is selected, based on the removed EXP value rather than the recently exposed type of service value. The underlying type of service in the IP header in the packet is not modified.

Regarding current levels of support for MPLS tunnel modes, previous PFC3x forwarding engines supported short pipe and uniform tunnel modes. The PFC4 now adds support for full pipe tunnel mode.

Layer 3 - Increased Support for Ethernet over MPLS Tunnels

Ethernet over MPLS (EoMPLS) provides a mechanism for two disparate LAN networks to be joined over an MPLS network, thus allowing Ethernet frames to traverse an MPLS cloud. The PFC3x supported a maximum of 4 K EoMPLS tunnels. The number of tunnels now supported with the PFC4 has been increased to 16 K tunnels in hardware.

Layer 3 - MPLS Aggregate Label Support

The number of MPLS aggregate labels supported on the PFC4 has been increased significantly over what was supported on the PFC3x. Aggregate label support has been increased to 16 K, up from 512 on the previous PFC3x family.

Layer 3 - Layer 2 over GRE

The PFC4 family adds support for transporting Layer 2 packets over a generic route encapsulation (GRE) tunnel. This is a new capability added into the PFC4 hardware.

Layer 3 - Increased Multicast Routes

The PFC3x supported a maximum of 32 K multicast routes in hardware. The number of multicast routes that is supported in the PFC4 has been increased to 256 K. Note that the initial release will limit the maximum multicast routes to 128 K for IPv4 and 128 K for IPv6.

Layer 3 - PIM Register Encapsulation/De-Encapsulation for IPv4 and IPv6

Protocol Independent Multicast (PIM) is used to build a forwarding path through a network for multicast packets. As part of the process that PIM uses to build this forwarding path, multicast routers and switches will send PIM register packets to a device called the rendezvous point (RP). When these PIM register packets are sent, they are encapsulated in an IPv4 unicast header. If a Catalyst 6500 is configured as a PIM RP, then the processing of these PIM register packets requires the packets to be de-encapsulated in software.

The PFC4 now supports the ability to both encapsulate and de-encapsulate these PIM register packets in hardware, thus avoiding the performance penalty previously incurred on the supervisor in processing these packets.

Layer 3 - IGMPv3 and MLDv2 Snooping

IGMP is a multicast protocol that allows a host to indicate to the network that they wish to receive a multicast stream. The network uses this information to build a multicast topology over which multicast packets can be efficiently forwarded to hosts. IGMPv3 is the latest version of IGMP that allows hosts to specify a list of sources they would receive traffic from, enabling the network to drop packets from unwanted sources.

Earlier PFC3x engines support hardware-based IGMP Layer 2 snooping for v1 and v2. IGMPv3 source-specific snooping used a combined software and hardware model, where software is used to track specific sources, which is then tied to a non source-specific hardware entry. The PFC4 now adds hardware-based snooping support for IGMP v3.

MLDv2 snooping is Layer 2 snooping for IPv6 multicast packets. The PFC4 supports hardware-based MLDv2 snooping for IPv6 hosts as well as IGMPv3 snooping for IPv4 hosts.

Layer 3 - Increased Support for NetFlow Entries

Up to 512 K NetFlow entries can now be stored in the PFC4, and up to 1 M NetFlow entries (512 K for ingress NetFlow and 512 K for egress NetFlow) can now be stored in PFC4XL. This is quadruple the number of entries offered on the comparable PFC3x forwarding engines.

Layer 3 - Improved NetFlow Hash

The NetFlow implementation on all PFC forwarding engines uses a hash to both store and retrieve entries in the NetFlow table. With each generation of PFC engine, the efficiency of the hash has improved, allowing a greater percentage of the NetFlow table to be utilized. With PFC2 and PFC3a, the hash efficiency was 50 percent. With subsequent PFC3x, it improved to 90 percent. With PFC4, the hash has been improved yet again to provide a hash efficiency of close to 99 percent.

Layer 3 - Egress NetFlow

Previously, NetFlow was only supported for ingress data traffic. Egress NetFlow provides support for collecting flow statistics for packets after they have had ingress processing applied to them, and prior to transmission out the egress interface or interfaces. This can be of value, especially for users wanting to collect flow statistics for data moving into and out of a VPN, for example.

Layer 3 - Sampled NetFlow

Sampled NetFlow is a new feature in the PFC4 that allows customers to opt for NetFlow records to be created based on a sample of the traffic matching the flow. Sample Netflow uses a 1/N based sampling which inspects one packet every N packets. The PFC3x was capable of performing sampling but the operation was performed after the inspection process. The PFC4 performs the sampling during the inspection process, effectively reducing the amount of NetFlow entries. There are 1 K global NetFlow samplers supported in PFC4.

Layer 3 - MPLS NetFlow

The PFC4 provides support for aggregate label at the Provider Edge (PE). This feature allows the inspection of IP traffic belonging to a particular VPN before it is added with a MPLS label (ip2mpls) and after the last label is removed (mpls2ip). The MPLS NetFlow also allows the inspection of the IP header for non-aggregate label at a P device (mpls2mpls).

Layer 3 - Layer 2 Netflow

The layer 2 Netflow feature in the PFC4 allows NetFlow lookups for IPv4-, IPv6-, and MPLS-based packets to be performed using the Layer 2 header.

Layer 3 - Flexible NetFlow

The PFC4 now supports Flexible NetFlow, based on the NetFlow v9 record format. Flexible NetFlow allows users more flexibility in defining which record types they want to use for a v9 record. More importantly, Flexible NetFlow now also includes a number of new field options to allow for collection of MPLS, IPv6, and multicast information in a NetFlow record.

Layer 3 - Distributed Policing

In a PFC3x-based system using DFC3s, an aggregate policer applied on a VLAN that included ports on different DFC3-enabled line cards could not synchronize their token buckets. As such, each DFC3-enabled line card would maintain its own aggregate policed count, resulting in the true aggregate rate being multiplied by the number of DFC3s which apply the policer.

PFC4 solves this problem with the introduction of the distributed policer. This allows the policing state to be synchronized across multiple DFC4-enabled line cards providing for multi-port multi-module policing. A total of 1 K distributed policers are supported with PFC4.

Layer 3 - DSCP Mutation

Quality of Service (QoS) support in PFC4 is enhanced with its support for multiple ingress and egress Differentiated Services Code Point (DSCP) mutation maps. A DSCP mutation map is a table that defines to what an existing DSCP value in a packet can be modified. DSCP mutation maps facilitate the marking (or reprioritizing of packets) process. Up to 14 ingress DSCP mutation maps and up to 16 egress DSCP mutation maps can be defined in the PFC4.

Layer 3 - Aggregate Policers

An aggregate policer is a rate limiting policy that can be applied to a port, group of ports, VLAN, or group of VLANs that limits total traffic through those ports or VLANs to a predetermined bandwidth amount. Traffic in excess of the limit can either be marked down and forwarded, or dropped. The previous PFC3x forwarding engine supported a maximum of 1023 aggregate policers per chassis. PFC4 increases the limit on aggregate policers supported to 6 K.

Layer 3 - Microflow Policers

Like an aggregate policer, a microflow policer is a rate-limiting policy that can be applied to a port, group of ports, VLAN, or group of VLANs that limits total traffic for each flow through those ports or VLANs to a stated bandwidth amount. Traffic in excess of the limit can either be marked down and forwarded, or dropped. The previous PFC3x forwarding engine supported a maximum of 63 microflow policers per chassis. PFC4 increases the limit on microflow policers supported to 127.

Another enhancement to microflow policing is that with PFC4, a microflow policer can now be configured for both ingress and egress. Previously, a microflow policer could only be configured for the ingress direction.

Layer 3 - Policing by Packets or Bytes

The PFC4 introduces support for a policer to enforce its rate, using either a packet or byte count. Previously, only byte count was supported.

Layer 3 - Cisco TrustSec

Cisco TrustSec is an access control architecture where Security policies are enforced based on group membership as opposed to IP or MAC address. Group membership policies are inherently built into each packet by attaching a Security Group Tag (SGT) to each packet that enters a Cisco TrustSec domain. The SGT is assigned by the ingress switch and is used at the egress switch to determine access rights. The SGT is used in conjunction with Roles Based ACL (RBACL).

Layer 3 - Role-Based ACL

RBACL is an integral part of the Cisco TrustSec model that provides for a scalable way to apply access control within a Cisco TrustSec domain. Policies applied using RBACLs are assigned user group policies that encompass multiple end hosts. Data sent by hosts is tagged with an SGT that is carried inside a Cisco TrustSec header. This SGT is assigned by the hardware, and is carried with the packet as it traverses through the network. At intermediate nodes, the SGT can be modified or reassigned using classification ACLs. When the packet arrives at the network edge, the RBACL can be used to enforce Security policies that have been defined by the assigned SGT.

Layer 3 - Layer 2 ACL

PFC4 introduces enhanced support for a Layer 2 ACL that allows inspection of all of the Layer 2 fields. The ACL can inspect source and destination MAC address, Ethertype, VLAN ID, 802.1p user priority (or class of service) bits, and outer and inner tags (for 802.1Q tunnel packets).

Layer 3 - ACL Dry Run

PFC4 introduces the capability to perform a dry run (or pre-commit test) of a Layer 3 ACL. Earlier Ternary Content Addressable Memory (TCAM) implementations attempted to program the ACL without verifying whether enough space was available. The new ACL Dry Run feature allows the user to temporarily use a portion of the ACL TCAM to first test whether the configured ACL will fit. This guarantees Layer 3 ACL hardware programming will complete, prior to commitment.

Layer 3 - ACL Hitless Commit

PFC3x and earlier TCAM programming implementations are executed immediately upon configuration commit. If an ACL is applied to an interface, and then the configuration is changed, TCAM programming must first remove the previous ACL configuration before applying the new. There is a small period of time, while the TCAM programming is still being completed, that the only ACL entry present (default) is implicit deny. During this brief period, packets hitting the interface will be dropped until the updated TCAM programming process is complete.

PFC4 introduces the ability to program the ACL TCAM entries using a special pointer. Using this new ability, the ACL Hitless Commit feature allows the modified ACL to be programmed using new TCAM entries, while the original ACL entry remains intact. Once the new TCAM programming is complete, the old ACL TCAM pointer is removed, and replaced by the new pointer. This eliminates the temporary implicit deny scenario during transition.

Layer 3 - Layer 2 + Layer 3 + Layer 4 ACL

PFC3x provided support for Layer 2 or Layer 3/4 ACLs, but not both at the same time. With this new ACL type, PFC4 allows both Layer 2, 3 and 4 information to be inspected at the same time. Especially useful in wireless networks where mobility can often change the user’s source IP address, the ACL could be built to inspect the source MAC address along with other higher layer information (such as Layer 4 port information) to apply a Security policy.

Layer 3 - Classification Enhancements

The PFC4 engine offers several extensions to classification ACLs. As well as being able to match on the traditional classification options such as IP address, TCP/UDP ports, and others, the PFC4 also offers match on packet length, Time to Live (TTL), IP options, and IPv6 Extended Header. Some worms and other forms of attack sometimes require matching on these fields to make a positive ID.

Layer 3 - Per Protocol Drop (IPv4, IPv6, MPLS)

PFC4 adds support for the ability to only forward protocol traffic if enabled on the interface. The protocols that can be defined at an interface level are IPv4, IPv6, and MPLS. Traffic not matching the defined protocol will be dropped.

Layer 3 - Increase in ACL Label Support

An ACL label is used to group access control entries (ACEs) that are associated with the same access control list. An access control list entry that starts with “access-list 101....” uses the label “101” to indicate which ACL group this entry belongs to. Previously, the PFC3B/XL and PFC3C/XL supported a maximum of 4096 ACL labels. PFC4 increases support for the number of ACL labels to 16 K.

Layer 3 - Increase in ACL TCAM Capacity

The PFC4 forwarding engine implements two banks of TCAMs for classification purposes, providing a total of 256 K access control entries for DFC4XL and 64 K for DFC4. These ACEs can be shared between the security and QoS for both ingress and egress lookups. There is also a corresponding increase in the mask to entry ratio. This capability is discussed in more detail later in this paper. In summary, this allows for more efficient use of the TCAM space when defining security policies.

Layer 3 - Source MAC + IP Binding

A binding of IP address, VLAN, and MAC address can be made to facilitate the decision-making process for forwarding packets. This enforcement is performed by the PFC4 in hardware, and is especially useful in protecting against address spoofing. The IP source guard is an example of one feature that makes use of this capability.

Layer 3 - Drop on Source MAC Miss

This feature is another hardware enhancement that is used to further enhance the port security feature. Port security can be used to bind MAC addresses to a given port, ensuring that only packets with the defined MAC address are forwarded.

Layer 3 - RPF Check Interfaces

The Reverse Path Forwarding (RPF) check is used to help determine if a packet has been spoofed. It uses a reverse lookup, whereby the source address is used to initiate the lookup. If the packet arrives on an interface where its source address is not seen as existing out that interface, it is deemed a spoofed packet and will be dropped. With an RPF check, multiple paths can be incorporated into the lookup. Previous PFC3x engines supported 2 paths in the RPF lookup. PFC4 increases the number of paths included in the lookup to 16.

Layer 3 - RPF Checks for IP Multicast Packets

RPF checks for IP Multicast packets were previously performed in software, and the correct RPF interface was then programmed into the hardware forwarding entries. PFC4 allows full hardware-based RPF checks for IP Multicast. This capability also allows for dual RPF checks to support PIM sparse-mode shortest path tree (SPT) switchover to occur in hardware.

PFC4 Architecture Enhancement Summary

This section provides some details for the PFC4 architecture, its performance metrics, and an explanation of some of the feature enhancements that it introduces.

PFC4 Forwarding Architecture and Performance Metrics

The forwarding architecture of the new Supervisor 6T is based on the Cisco Express Forwarding (CEF) architecture, the same forwarding architecture used on the Supervisor 720 and Supervisor 2T. One of the primary enhancements of the Supervisor 2T and 6T (over the Supervisor 720) is the doubling of the centralized forwarding performance to 60 Mpps.

These next sections offer an overview of the major changes in functionality, scalability, and performance for Layer 2 and Layer 3 forwarding features supported by the PFC4. Table 14 offers a PFC4 Layer 2 and Layer 3 Feature Comparison with Previous Generations of PFCs

Table 14.     PFC4 Layer 2 and Layer 3 Feature Comparison with Previous Generations of PFCs

MAC address learning continues to be performed in hardware, as was done with the Supervisor 720. The MAC (CAM) table in the Supervisor 6T has been increased in size to 128 K, and by using a new architecture the efficiency has been improved to 99 percent.

The aggregate forwarding performance of the chassis will multiply by 60/120 Mpps for each DFC4-based line card that is installed in the chassis. This facilitates an aggregate system performance up to 1440 Mpps, assuming a Catalyst 6513-E and 720 Mpps on the Catalyst 6807xl.

PFC4 Security and QoS Architecture

Security and QoS functionality have been enhanced on the Supervisor 6T. One of the main enhancements has been the move to a consolidated TCAM (memory) bank for holding Security and QoS ACLs that have been defined in Security and QoS Policies in the switch configuration. In previous PFC3x forwarding engines, there were two separate TCAM banks used for this purpose, each 32 K in size. One TCAM bank was used for Security ACLs and the other bank was used for QoS ACLs.

With the Supervisor 6T, up to 256 K entries are available for use (this is in the PFC4-XL). By default, in the PFC4 XL, 64 K entries are reserved for QoS, while 192 K entries are available for Security ACLs and related feature ACLs (such as NAT, WCCP, and others). Based on user-specified software configuration, up to 128 K entries can be made available for QoS ACLs.

Another major improvement is the ACE to mask ratio. An ACE is one ACL permit/deny statement that exists within the framework of an ACL. In the ACE statement, the mask is what is used to identify what portion of the address space should be used to match on the incoming or outgoing address. Let’s look at an example to better understand how the ACE breaks down into the different functional components.

access-list 101 permit ip 10.1.1.0 0.0.0.255 any

In this example, the entire line is one ACE. This ACE typically forms part of a larger ACL, consisting of multiple configuration lines. The mask is the “0.0.0.255” part of the ACE, while the value is the “10.1.1.0” part of this ACE example. In this example, the mask signifies that only the first 24 bits of the IP address should be used to match on classified packets.

With PFC3x forwarding engines, the hardware tables support 32 K ACEs and 4 K masks, which yields an 8:1 ratio of ACEs to masks. Depending on how a customer might build their ACL, there is potential for masks to be consumed before ACEs were consumed, leading to a potential inefficient use of the Security TCAM. With the Supervisor Engine 6T, the mask to value ratio has changed and now supports a 1:1 ratio, providing 1 mask for each ACE (value). This should increase the flexibility for how customers can deploy ACLs and reduce any potential inefficient use of table entries.

Figure 16 shows how the masks and values are represented in the hardware TCAMs with PFC3x on the left and the Supervisor 6T on the right.

QoS on the Supervisor 6T offers support for distributed policing. For a Supervisor 720-based system, a rate-limiting policy applied to a VLAN that had member ports spread across DFC3-enabled line cards would result in each DFC3 maintaining its own token bucket. In other words, a rate-limiting policy of 2 Gbps would result in each DFC3 maintaining its own 2 Gbps rate limiting count. With the Supervisor 6T, synchronization of the rate limiting policy occurs between participating DFC4-enabled line cards, ensuring the aggregate traffic load for traffic, such that VLAN is truly limited to the configured rate.

Table 15 provides a summary of the enhancements incorporated into the Supervisor 6T for Security and QoS.

Table 15.     PFC4 Security and QoS Comparison with Older PFCs

PFC4 NetFlow

The support for NetFlow in the PFC4 has been enhanced on both the scalability and functionality side. Of perhaps the most significance is support for true egress NetFlow. With the change in how a packet is processed and the new pipeline processing method that the PFC4 uses to process a packet, there are two separate processing paths that a packet will take: one for ingress services and one for egress services.

While this is discussed in more detail later in this paper, the PFC4 now allows for both ingress and egress NetFlow services to be performed on all packets. The Two Cycle lookup process allows NetFlow information to be gathered after the Final destination of the flow is known. Even though all forwarding engines in the system can perform NetFlow lookups, it is important to remember that all lookups are performed on the ingress forwarding engine, even for Egress NetFlow.

Support for Flexible NetFlow is now built into hardware. Flexible NetFlow offers a more flexible method to create flow monitors that allow for the collection of data that fits user specified templates. In this manner, an administrator can create a flow monitor to collect IPv6 specific information on one interface, while on another interface create a separate flow monitor to collect IPv4 multicast specific information.

Cisco TrustSec

This architecture uses access control, authentication, and encryption to build a scalable, highly secure network. There are three important elements of the Cisco TrustSec architecture that is part of the hardware capabilities in the Supervisor 6T:

●   Support for SGT and Destination Group Tag (DGT) tagging

●   Role Based ACL (RBACL) ink layer encryption (IEEE 802.1ae)

The Security Group Tag (SGT) and Destination Group Tag (DGT) are tags that are inserted into a packet and are used to define the Security policies that should be applied to this packet as it traverses the Cisco TrustSec cloud. Cisco TrustSec uses an eight-byte header and contains sixteen bits that are used to indicate the SGT or DGT for that packet. RBACL provides a means of classification of packets using the SGT and DGT to apply Security policies.

The PFC4 provides support for SGT, DGT, and RBACL in the following manner:

●   Both SGT and DGT assignment can be performed by the PFC4

●   The SGT can be derived during packet processing on ingress from the input packet or from an ingress ACL

●   The DGT can be derived from the destination IP lookup (in the FIB), from the NetFlow process, or the ingress ACL

●   RBACL is supported on the egress interface

●   Cisco TrustSec tunnel encapsulation

PFC4 (and PFC4XL) Packet Walk

The PFC4 and PFC4XL are capable of processing up to 60 Mpps for both Layer 2 and Layer 3 processed packets. There are four stages to the packet walk. These stages follow, and are shown in Figure 17.

1.     Layer 2 (pre Layer 3) ingress processing

2.     Layer 3 IFE processing

3.     Layer 3 OFE processing

4.     Layer 2 (post Layer 3) egress processing

This section will explore in detail the packet walk through the PFC4 for an IPv4 unicast packet.

Layer 2 (Pre Layer 3) Ingress Processing

The entire process for the packet walk begins with the arrival of a frame over the DBUS destined to the Layer 2 forwarding engine ASIC. The processing steps associated with this stage are:

1.     Packet arrives over the DBUS or the fabric replication ASIC

2.     A CRC check is performed by the Layer 2 forwarding engine

3.     A LIF database lookup is performed which returns the ingress LIF, bridge domain, and lookup index for later Layer 3 forwarding engine lookups

4.     A Result Bundle Hash (RBH) is generated, indicating which link of an EtherChannel bundle is to be used (if applicable)

5.     Performance of a static MAC address match for control packets (CDP, BPDU, and more)

6.     Performance of Layer 2 table MAC address lookup

7.     Merging of the results of the processing (Layer 2 Lookup, RBH, and more) and creation of a frame that is forwarded to the Layer 3 forwarding engine for processing

Layer 3 IFE Processing

Following are the IFE processing steps for the Layer 3 forwarding engine:

1.     The Layer 3 forwarding engine receives a frame from the Layer 2 forwarding engine, which it checks for CRC errors, then forwards to the IFE pipeline.

2.     A LIF map table lookup is performed to collect information regarding the interface on which the packet arrived (for example, the ACL label for input classification, RPF mode, VPN number, and more).

3.     An RPF check is performed on the Source address.

4.     Packet classification is performed on the packet, with a lookup into ACL TCAMs.

5.     The TCAM lookup result is merged into an ACL result, a QoS result, an accounting result, and a Cisco TrustSec result.

6.     Notify the Layer 2 engine to update ACL statistics (using the ACL hit counter, located on Layer 2 engine).

7.     Ingress NetFlow lookup is performed, using results from Step 5.

a.     If it is a hit, NetFlow statistics are updated and Microflow policing is performed (if applicable).

b.     If no hit is found, a new NetFlow entry is created.

8.     The Layer 3 FIB lookup is performed, which returns an adjacency pointer and a link count, which determines how many equal cost paths exist for this prefix.

9.     Ingress aggregate and NetFlow policing is performed.

10.   All the ingress lookups are now complete and a result can be generated.

Layer 3 OFE Processing

Once the IFE pipeline processing is finished, the packet is then passed to the OFE pipeline for further processing. Following are the OFE steps:

1.     The adjacency pointer retrieved by the FIB lookup during IFE processing is used to return an egress LIF index and a rewrite pointer. The rewrite pointer is held for later in the OFE process when rewrite information required for this packet needs to be retrieved from the rewrite table.

2.     The egress LIF lookup is performed to retrieve information about the egress interface out of which the packet is to be sent.

3.     An egress classification lookup is performed into the ACL TCAM for Security and QoS ACLs. If a redirect is found (such as PBR, VACL redirect, and more), then a new rewrite pointer is received for later lookup into the rewrite table.

4.     An egress NetFlow lookup is performed.

5.     Egress aggregate and microflow policing is applied (if applicable).

6.     A lookup into the RIT (rewrite information table) is performed, using the rewrite pointer retrieved earlier.

7.     A final result is generated containing the information retrieved from the OFE lookups.

8.     The result frame is passed back to the Layer 2 engine containing information such as the destination VLAN, egress LIF, and rewrite information.

Layer 2 (Post-Layer 3) Egress Processing

Subsequent to the Layer 3 engine finishing processing, it hands the result of its operation back to the Layer 2 engine for the final processing stage to commence. Following are these final steps. It:

1.     Checks the Layer 3 engine result for CRC errors.

2.     Merges its Layer 2 result with the Layer 3 result from Layer 3 engine.

3.     Inspects results of Layer 3 engine processing for post Layer 2 table lookups, such as LIF.

4.     Performs CPU rate limiting function (if applicable).

5.     Sends a result to the outgoing interface through BUS.

6.     LIF statistics are updated to reflect the forwarding of this packet.

This completes the processing for a single packet passing through the PFC4 complex. Up to 60 million of these packet lookups can be processed in one second. The same processing sequence and performance metrics apply for the DFC4 complex, independent of the PFC4. This allows an aggregate lookup rate of 720 Mpps for a Catalyst 6513-E system.

Switch Fabric and Fabric Connection Daughter Card

This section provides more details about the new switch fabric design and Fabric Connection Daughter Card (FCDC) on the Supervisor 6T.

Supervisor Engine 6T Switch Fabric

LAN switches predominantly use either a shared memory switch fabric or a crossbar switch fabric for their switch backplane. The switch fabric implemented on the Supervisor 6T uses the crossbar architecture, which is the same backplane architecture used on the Supervisor 720 and Supervisor 2T. A crossbar architecture allows multiple simultaneous data transfers to occur at the same time between different line cards.

Each line card slot in a chassis has its own set of dedicated channels over which to send data into the switch fabric. Twenty-eight 110 Gbps fabric channels are provided by the Supervisor 6T Switch Fabric, which distributes two fabric channels to each line card slot on 6500-E chassis and four Fabric channels to each line card slot on 6807XL Chassis.

The fabric ASIC used in the Supervisor 6T represents an upgrade from the fabric ASIC used in the Supervisor 720 and Supervisor 2T. Some of the major enhancements include:

●   Buffered crossbar design

●   28 fabric channels (compared with 20 fabric channels on the Sup 720-10G) carrying both Multicast, Unicast Traffic, and Control Traffic.

●   Each channel can operate at either 110 Gbps for a New Generation of Line cards, 40 Gbps to support the WS-X69xx line cards, or 20 Gbps to provide backward compatibility for WS-X67xx and WS-X68xx line cards

●   Enhanced multi-destination arbitration

●   Hardware Capable of supporting two-priority level data path through two dedicated input queues and two output queues. Current Software utilizes low priority queue only.

●   Packet counters per queue, so packet history is visible for debugging purposes

●   Separate non-shared input and output queuing on a per-port-per-queue basis

●   Multiple modes of backpressure supported

◦     Per-queue flow control from input buffer to downlink line card

◦     Per-queue flow control from output buffer to input buffer (internal to fabric ASIC)

◦     Per-queue flow control from line card to output buffer

●   Support for Jumbo frames up to 9248 bytes in size

Switch Fabric Architecture

The Supervisor 6T Switch Fabric implements two identical crossbar switch fabrics. Each crossbar is used to handle either high priority or low priority traffic. This is shown in Figure 18.

When a line card forwards a packet over its fabric port, it is received by the fabric and stored in the fabric port’s input buffer. The input buffer on the ingress fabric port contains a high priority queue and a low priority queue. Current Implementation supports usage of a lower priority queue only, even though the hardware is capable of two queues. Hence, when a packet arrives on the ingress fabric port, it is de-multiplexed into low Priority.

Within the packet header is a Fabric Port of Exit (FPOE) index field, which indicates the switch fabric port of exit. The switch fabric will arbitrate for output buffers associated with that egress fabric port. Once the arbitration is successful, the packet data will flow from the ingress packet buffer on the ingress fabric port to the egress packet buffers on the egress fabric port. During this transmission, a three-times-over-speed is used to reduce switching latency and reduce Head of Line Blocking (HOLB) contention. Once the entire data packet has been received by the egress fabric interface ASIC and is fully stored in the egress fabric port buffers, it is then transmitted by the egress line card.

Fabric Connection Daughter Card

The Supervisor 720 supported only 20 fabric channels (18 for line cards, two for Supervisor uplinks). When it was installed in the Catalyst 6513 Switch, it limited the top eight slots to a single channel only. This limited line card options for Slots 1 through 8, restricting dual fabric line cards (WS-X67xx) to being inserted in Slots 9 through 13 only. The crossbar switch fabric on the Supervisor 6T and the 6513-E chassis address this limitation.

With this chassis and Supervisor combination, all line card slots (Slots 1 through 6 and Slots 9 through 13) have dual fabric channels built into the chassis, allowing dual fabric line cards (WS-X67xx, WS-X68xx,WS-69xx or New C6800) to operate in all slots. The fabric connection daughter card is a new addition on the Supervisor baseboard. This daughter card provides connection for 6 additional channels across Slots 1 through 6 that were missing in the original Catalyst 6513. This can be seen in Figure 19.

Figure 19 shows the Fabric Connection Daughter Card (FCDC) from the rear of the Supervisor 6T. It provides a second connector that sits just above the original fabric connector found on earlier Supervisor 720 models. For all chassis models (except the 6513-E), this connector does not play a part in provisioning fabric channels to any of the line card slots. In the Catalyst 6513-E, there is an additional connector found for Slots 7 and 8 on the chassis backplane that is used by the FCDC. This additional backplane connector can be seen in Figure 20 of the Catalyst 6513-E.

This additional slot connector allows the 6 additional fabric channels to be relayed to the first six line card slots in the Catalyst 6513-E, thus providing dual fabric channel connections to all line card slots.

Baseboard Temperature Sensors

The Supervisor 6T has four temperature sensors with two Sensors located on each side of the board. The right side sensors are designated the “inlet” sensors, while the sensors on the left of the board are designated the “outlet” sensors. These sensors provide information that is fed into the environmental subsystem, which uses this information to assess temperature thresholds.

Supervisor 6T Packet Walks

All packet processing is performed in a specific sequence through the different ASIC blocks. A high-level packet walk is provided for packets that ingress and egress the local ports on the Supervisor.

Ingress Packet Processing

1.     Packets arriving on either the 10G or 40G ports have preliminary checks (CRCs) performed by the PHY, converting signals and serializing the bits prior to sending the Packet or Frame to the Port ASIC

2.     If Cisco TrustSec is enabled, the Port ASIC performs 802.1ae decryption, parses the packet to derive the VLAN, CoS, etc., and performs ingress QoS. Then it applies an internal header and sends it to the FIRE ASIC

3.     The FIRE ASIC stores the data payload in the local buffer and then sends only the internal header to the forwarding engine (for lookup).

4.     The inband FPGA parses the internal header and then sends it to the forwarding engine.

5.     The forwarding Engine performs Layer 2, Layer 3, ACL, and NetFlow IFE and OFE processing, and determines the Egress Port and Rewrite Information. Then it returns a new, Internal Header to the FIRE ASIC (using Inband FPGA)

6.     The FIRE ASIC uses the new internal header to determine the fabric port mapped to the egress port, and converts the internal header to a fabric header and sends the packet with Fabric Header it to the fabric ASIC.

7.     The Fabric ASIC uses the fabric header to determine the egress fabric port and then sends the packet to the Switch Fabric.

Figure 22 outlines the Supervisor 6T ingress packet walk.

Egress Packet Processing

1.     Packets are received from one of the switch fabric channels and on to the egress FIRE ASIC.

2.     The FIRE ASIC stores the packet, derives a new internal header, and sends the packet header information to the forwarding engine for egress lookup.

3.     The inband FPGA parses the internal header and then sends it to the forwarding engine.

4.     The forwarding engine performs an egress lookup to learn the source MAC address and returns the internal header and results to the FIRE ASIC using Inband FPGA.

5.     The FIRE ASIC uses the new internal header to determine the egress port, performs egress QOS, checks for replication services, and then reassembles the packet and sends it to the Port ASIC.

6.     The Port ASIC removes the internal header and rewrites VLAN, CoS, etc., and forwards the packet to PHY.

7.     If Cisco TrustSec is enabled, the Port ASIC performs 802.1ae encryption prior to forwarding the packet to the PHY.

8.     PHY serializes the bits and converts the signal, then transmits the packet.

Figure 23 visually displays the Supervisor 6T ingress packet walk.

Glossary

This section provides brief explanations for many acronyms used throughout this document.

Table 16.     Definition of Acronyms