Contents

Introduction

Prerequisites

Requirements

Components Used

Background Information

Ethernet OAM

Ethernet OAM Protocols Position

CFM Overview

Key CFM Mechanisms

CFM Concepts

Maintenance Domain

Maintenance Association

Maintenance Point - Maintenance End Point

Maintenance Domain Intermediate Point

UP MEP

UP MEP - Frame Forwarding

DOWN MEP

DOWN MEP - Frame Forwarding

MP Placement in a Bridge Port

MAs and UP/DOWN MEPs

Applicability of UP/DOWN EPs in Switches

Fault Management

CFM Protocols

Continuity Check Protocol

Loopback Protocol

Linktrace Protocol

Implementation Cases

Configuration Management (UP MEP)

Topology

Verify

Show Commands

Verify Continuity Check

Sniffer Results

Configuration Management (DOWN MEP)

Verify

Show Commands

Verify Continuity Check

Debug Commands

Performance Management

Key Performance Indicators (KPIs)

Measuring KPIs

Frame Delay/Delay Variation

Frame Loss

Cisco Performanace Management Solution

Usage Guidelines and Restrictions

Prerequisites

Configuration Management

Verify

Debug Commands

Related Information

Introduction

This document describes Connectivity Fault Management (CFM) technology, configuration, post-checks, and troubleshooting. The basic concepts of CFM, CFM's building blocks, a configuration guide, show commands, and Wireshark analysis of CFM messages is provided. This document does not explain hardware limitations or the supported interface for CFM to work.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

- Ethernet technologies

- Ethernet Virtual Connections (EVCs)

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

Ethernet CFM is an end-to-end per-service-instance Ethernet layer operation, administration, and management (OAM) protocol. It includes proactive connectivity monitoring, fault verification, and fault isolation for large Ethernet metropolitan-area networks (MANs) and WANs.

The advent of Ethernet as a MAN and WAN technology imposes a new set of OAM requirements on Ethernet's traditional operations, which were centered on enterprise networks only. The expansion of Ethernet technology into the domain of service providers, where networks are substantially larger and more complex than enterprise networks and the user base is wider, makes operational management of link uptime crucial. More importantly, the timeliness to isolate and respond to a failure becomes mandatory for normal day-to-day operations, and OAM translates directly to the competitiveness of the service provider.

Ethernet OAM

- Building Block - IEEE 802.1ag

- CFM - IEEE 802.3ah (clause 57)

- Ethernet Link OAM (also referred as 802.3 OAM, Link OAM, or Ethernet in the First Mile (EFM) OAM) - ITU-T Y.1731

- OAM functions and mechanisms for Ethernet-based networks - MEF E-LMI (Ethernet Local Management Interface)

Ethernet OAM Protocols Position

- E-LMI - User to Network Interface (UNI)

- Link OAM - Any point-to-point 802.3 link

- CFM - End-to-End UNI to UNI

- MPLS OAM - within MPLS cloud

CFM Overview

- Family of protocols that provides capabilities to detect, verify, isolate, and report end-to-end Ethernet connectivity faults

- Employs regular Ethernet frames that travel in-band with the customer traffic

- Devices that cannot interpret CFM Messages forward them as normal data frames

- CFM frames are distinguished by Ether-Type (0x8902) and dMAC address (for multicast messages)

- Standardized by IEEE in 2007 IEEE std. 802.1ag-2007

Key CFM Mechanisms

- Nested Maintenance Domains (MDs) that break up the responsibilities for network administration of a given end-to-end service

- Maintenance Associations (MAs) that monitor service instances under a given MD

- Maintenance Points (MPs) that generate and respond to CFM Protocol Data Units (PDUs)

- Protocols (Continuity Check, Loopback, and Linktrace) used for Fault Management activities

CFM Concepts

Maintenance Domain

- Defined by Operational/Contractual Boundaries, such as Customer/Service Provider/Operator

- MD may nest and touch, but never intersect

- Up to eight levels of "nesting": MD Level (0..7) - the higher the level, the broader it's reach

- MD Name Format: null, MAC address, DNS or string-based

Maintenance Association

- Monitors connectivity of a particular service instance in a given MD, such as one service that traverses four MDs = four MAs

- Defined by a set of Maintenance End Points (MEPs) at the edge of a domain

- Identified by MAID - "Short MA" Name + MD Name

- Short MA Name Format - Vlan-ID, VPN-ID, integer or string-based

Maintenance Point - Maintenance End Point

- Maintenance Association End Point

- Define the boundaries of an MD

- Support the detection of connectivity failures between any pair of MEPs in an MA

- Associated per MA and identified by a MEPID (1-8191)

- Can initiate and respond to CFM PDUs

Maintenance Domain Intermediate Point

- Maintenance Domain Intermediate Point (MIP)

- Supports the discovery of paths among MEPs and location of faults along those paths

- Can be associated per MD and VLAN/EVC (manually or automatically created)

- Can add, check, and respond to received CFM PDUs

UP MEP

- CFM PDUs generated by the MEP are sent towards the Bridge's Relay Function and not via the wire connected to the port where the MEP is configured

- CFM PDUs to be responded by the MEP are expected to arrive via the Bridge's Relay Function

- Applicable to switches

UP MEP - Frame Forwarding

DOWN MEP

- CFM PDUs generated by the MEP are sent via the wire connected to the port where the MEP is configured

- CFM PDUs to be responded by the MEP are expected to arrive via the wire connected to the port where the MEP is configured

- Port MEP - special Down MEP at level zero (0) used to detect faults at the link level (rather than service)

- Applicable to routers and switches

DOWN MEP - Frame Forwarding

MP Placement in a Bridge Port

MAs and UP/DOWN MEPs

Applicability of UP/DOWN EPs in Switches

- DOWN MEPs are typically used for MAs that span a single link

- UP MEPs are commonly used for MAs with a wider reach, such as end-to-end and beyond a single link

Fault Management

CFM Protocols

There are three (3) protocols defined by CFM:

1. Continuity Check Protocol

- Fault Detection

- Fault Notification

- Fault Recovery

2. Loopback Protocol

- Fault Verification

3. Linktrace Protocol

- Path Discovery and Fault Isolation

Continuity Check Protocol

- Used for Fault Detection, Notification, and Recovery

- Per-Maintenance Association multicast "heart-beat" messages are transmitted at a configurable periodic interval by MEPs (3.3ms, 10ms, 100ms, 1s, 10s, 1min, 10min) - Uni-directional (no response required)

- Carries status of port on which MEP is configured

- Catalogued by MIPs at the same MD-Level, terminated by remote MEPs in the same MA

Loopback Protocol

- Used for Fault Verification - Ethernet Ping

- MEP can transmit a unicast LBM to a MEP or MIP in the same MA

- MEP can also transmit a multicast LBM (defined by ITU-T Y.1731), where only MEPs in the same MA respond

- Receiving MP responds and transforms the LBM into a unicast LBR sent back to the original MEP

Linktrace Protocol

- Used for Path Discovery and Fault Isolation - Ethernet Traceroute

- MEP can transmit a multicast message (LTM) in order to discover the MPs and path to a MIP or MEP in the same MA

- Each MIP along the path and the terminating MP return a unicast LTR to original MEP

In order to put all three protocols together and implement them in the network, complete these steps:

1. Run a connectivity check in order to proactively detect a soft or hard failure.

2. Upon a failure detection, use loopback, CCM DB, and Error DB in order to verify it.

3. Upon verification, run traceroute in order to isolate it. Multiple segment LBMs can also be used to isolate the fault.

4. If the isolated fault points to a virtual circuit, then the OAM tools for that technology can be used to further fault isolation; as an example for MPLS PW, VCCV and MPLS ping can be used.

Implementation Cases

Configuration Management (UP MEP)

Topology

In order to explore the configuration, a small topology was built for demonstration. The names used for Domain, Service name, and EVC Name are shown here:

Domain: ISPdomain

Domain level: 5

Service Name: XCONN_EVC

EVC Name: EVC_CE1

PE1:

---------------------------Enabling CFM globally-------------------------------------

ethernet cfm ieee

ethernet cfm distribution enable

ethernet cfm global

ethernet cfm traceroute cache

ethernet cfm alarm notification all

ethernet cfm domain ISPdomain level 5

service XCONN_EVC evc EVC_CE1

continuity-check

---------------------------Enabling CFM MEP under EVC----------------------------

int gig4/2

service instance 2100 ethernet EVC_CE1

encapsulation dot1q 2100

xconnect 192.168.3.3 2100 encapsulation mpls

cfm mep domain ISPdomain mpid 102

monitor loss counter

PE3:

---------------------------Enabling CFM globally-------------------------------------

ethernet cfm ieee

ethernet cfm distribution enable

ethernet cfm global

ethernet cfm traceroute cache

ethernet cfm alarm notification all

ethernet cfm domain ISPdomain level 5

service XCONN_EVC evc EVC_CE1

continuity-check

---------------------------Enabling CFM MEP under EVC----------------------------

int gig4/2

service instance 2100 ethernet EVC_CE1

encapsulation dot1q 2100

xconnect 192.168.1.1 2100 encapsulation mpls

cfm mep domain ISPdomain mpid 201

monitor loss counter

Verify

Show Commands

PE1#show ethernet cfm maintenance-points local

Local MEPs:

--------------------------------------------------------------------------------

MPID Domain Name                                 Lvl   MacAddress     Type  CC

Ofld Domain Id                                   Dir   Port           Id

MA Name                                           SrvcInst       Source

EVC name

--------------------------------------------------------------------------------

102  ISPdomain                                   5     ccef.48d0.64b0 XCON  Y

No   ISPdomain                                   Up    Gi4/2          N/A

XCONN_EVC                                         2100           Static

EVC_CE1

Total Local MEPs: 1

PE1#show ethernet cfm maintenance-points remote

--------------------------------------------------------------------------------

MPID  Domain Name                                 MacAddress          IfSt  PtSt

Lvl  Domain ID                                   Ingress

RDI  MA Name                                     Type Id             SrvcInst

EVC Name                                                        Age

Local MEP Info

--------------------------------------------------------------------------------

201   ISPdomain                                   8843.e1df.00b0      Up    Up

5    ISPdomain                                   Gi4/2:(192.168.3.3, 2100)

-    XCONN_EVC                                   XCON N/A            2100

EVC_CE1                                                         5s

MPID: 102 Domain: ISPdomain MA: XCONN_EVC

In this output you can see the remote mpid and remote MAC address. CFM status shows up/up.

Verify Continuity Check

PE1#ping ethernet mpid 201 domain ISPdomain service XCONN_EVC

Type escape sequence to abort.

Sending 5 Ethernet CFM loopback messages to 8843.e1df.00b0, timeout is 5 seconds:!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 4/4/4 ms

PE1#traceroute ethernet mpid 201 domain ISPdomain service XCON$

Type escape sequence to abort. TTL 64. Linktrace Timeout is 5 seconds

Tracing the route to 8843.e1df.00b0 on Domain ISPdomain, Level 5,

service XCONN_EVC, evc EVC_CE1

Traceroute sent via Gi4/2:(192.168.3.3, 2100), path found via MPDB

B = Intermediary Bridge

! = Target Destination

* = Per hop Timeout

--------------------------------------------------------------------------------

MAC      Ingress        Ingr Action  Relay Action

Hops   Host             Forwarded   Egress         Egr Action   Previous Hop

--------------------------------------------------------------------------------

B 1                    ccef.48d0.64b0 Gi4/2          IngOk        RlyMPDB

Forwarded

! 2                    8843.e1df.00b0                             RlyHit:MEP

Not Forwarded                              ccef.48d0.64b0

Sniffer Results

A sniffer device was placed on PE1, which captures all CFM packets that come remotely. An example is shown here:

In the screen shot:

- Sequence Number 2 and 13 shows the general continuity check message (CCM).

- Sequence Number 4, 5, 6, 7, and 8 shows the Loopback Replies (LBRs), which were generated due to a ping test.

- Sequence Number 10 shows the Linetrace Reply (LTR), which was generated due to a traceroute test.

Configuration Management (DOWN MEP)

In the previous example, the EVC can used by CE1 which is located behind the PE1 and PE3. You can enable down MEP on the CE1 device, but with a higher level of MD. MD level 7 is shown in this example.

Domain: CEdomain

Domain level: 7

CE1_A

--------------------Enabling CFM globally---------------------------

ethernet cfm ieee

ethernet cfm global

ethernet cfm domain CEdomain level 7

service CUST vlan 2100 direction down (down Mep)

continuity-check

------------------Enabling CFM MEP under interface--------------------

interface GigabitEthernet1/0/1

switchport access vlan 2100

switchport trunk encapsulation dot1q

switchport mode trunk

ethernet cfm mep domain CEdomain mpid 1002 service CUST

CE1_B

--------------------Enabling CFM globally---------------------------

ethernet cfm ieee

ethernet cfm global

ethernet cfm domain CEdomain level 7

service CUST vlan 2100 direction down

continuity-check

------------------Enabling CFM MEP under interface--------------------

interface GigabitEthernet1/0/1

switchport access vlan 2100

switchport trunk encapsulation dot1q

switchport mode trunk

ethernet cfm mep domain CEdomain mpid 2001 service CUST

Verify

Show Commands

CE1#show ethernet cfm maintenance-points remote

--------------------------------------------------------------------------------

MPID  Domain Name                                 MacAddress          IfSt  PtSt

Lvl  Domain ID                                   Ingress

RDI  MA Name                                     Type Id             SrvcInst

EVC Name                                                        Age

Local MEP Info

--------------------------------------------------------------------------------

2001  CEdomain                                    5835.d970.9381      Up    Up

7    CEdomain                                    Gi1/0/1

-    CUST                                        Vlan 2100           N/A

N/A                                                             3s

MPID: 1002 Domain: CEdomain MA: CUST

Total Remote MEPs: 1

CE1#show ethernet cfm maintenance-points local

Local MEPs:

--------------------------------------------------------------------------------

MPID Domain Name                                 Lvl   MacAddress     Type  CC

Ofld Domain Id                                   Dir   Port           Id

MA Name                                           SrvcInst       Source

EVC name

--------------------------------------------------------------------------------

1002 CEdomain                                    7     0023.eac6.8d01 Vlan  Y

No   CEdomain                                    Down  Gi1/0/1        2100

CUST                                              N/A            Static

N/A

Verify Continuity Check

CE1#ping ethernet mpid 2001 domain CEdomain service CUST

Type escape sequence to abort.

Sending 5 Ethernet CFM loopback messages to 5835.d970.9381, timeout is 5 seconds:!!!!!

Success rate is 100 percent (5/5), round-trip min/avg/max = 1/1/1 ms

Total Local MEPs: 1

Till now MIP is not configured on PE1 and PE3 hence output of show command and

traceroute command will be as per below.

CE1#tracer ethernet mpid 2001 domain CEdomain service CUST

Type escape sequence to abort. TTL 64. Linktrace Timeout is 5 seconds

Tracing the route to 5835.d970.9381 on Domain CEdomain, Level 7, vlan 2100

Traceroute sent via Gi1/0/1

B = Intermediary Bridge

! = Target Destination

* = Per hop Timeout

--------------------------------------------------------------------------------

MAC      Ingress        Ingr Action  Relay Action

Hops   Host             Forwarded   Egress         Egr Action   Previous Hop

--------------------------------------------------------------------------------

! 1                    5835.d970.9381 Gi1/0/1        IngOk        RlyHit:MEP

Not Forwarded                              0023.eac6.8d01

CE1_A can see CE1_B via traceroute.

Now, configure MIP on PE1 and PE2.

PE1:

interface GigabitEthernet 4/2

service instance 2100 ethernet EVC_CE1

cfm mip level 7

PE2:

interface GigabitEthernet 4/2

service instance 2100 ethernet EVC_CE1

cfm mip level 7

Now, check the traceroute results from CE1.

CE1#traceroute ethernet mpid 2001 domain CEdomain service CUST

Type escape sequence to abort. TTL 64. Linktrace Timeout is 5 seconds

Tracing the route to 5835.d970.9381 on Domain CEdomain, Level 7, vlan 2100

Traceroute sent via Gi1/0/1

B = Intermediary Bridge

! = Target Destination

* = Per hop Timeout

--------------------------------------------------------------------------------

MAC      Ingress        Ingr Action  Relay Action

Hops   Host             Forwarded   Egress         Egr Action   Previous Hop

--------------------------------------------------------------------------------

B 1                    ccef.48d0.64b0 Gi4/2          IngOk        RlyMPDB

Forwarded                                  0023.eac6.8d01

B 2                    8843.e1df.00b0                             RlyMPDB

Forwarded      Gi4/2          EgrOK        ccef.48d0.64b0

! 3                    5835.d970.9381 Gi1/0/1        IngOk        RlyHit:MEP

Not Forwarded                              8843.e1df.00b0

You can see the difference in the traceroute output. Iintermediate hops are seen after MIPs on PE1 and PE2 are configured.

Debug Commands

debug ethernet cfm diagnostic packets

debug ethernet cfm packets

Performance Management

Key Performance Indicators (KPIs)

- Frame Loss Ratio - percentage (%) of service frames not delivered/total number of service frames delivered in T time interval

- Frame Delay - round-trip/one-way delay for a service frame

- Frame Delay Variation - variation in frame delay between a pair of service frames

Measuring KPIs

Frame Delay/Delay Variation

- One-way or two-way measurements

- Requires synthetic traffic with timestamps

- Requires time-of-day synchronization for one-way delay

Frame Loss

- One-way Frame Loss

- Source to Destination - Far-End

- Destination to Source - Near-End

- Service Frame Loss (actual loss) - requires counter exchange

- Applicable only to Point-to-Point EVCs

- Statistical Frame Loss - relies on synthetic traffic

- Requires synthetic traffic for multipoint services

- Applicable to Point-to-Point and Multipoint EVCs

Cisco Performanace Management Solution

- Ethernet performance probes based on IEEE 802.1ag and vendor-specific PDUs

- Measure one-way FD/FDV/FL and two-way FD/FDV

- Partial multi-vendor network support

- Configured and scheduled via IP SLA

- Shipped under feature name: IP SLA for Metro Ethernet

- Ethernet performance probes based on Y.1731 PDUs

- Priority to these mechanisms in Cisco IOS®: One-way ETH-DM/Two-way ETH-DM, Single-ended ETH-LM and Cisco-proposed Y.1731 extensions (ETH-SLM) Multi-vendor interoperability

- Software and hardware-assisted implementation configured and scheduled via IP SLA

- Phased-out delivery for selected Cisco IOS and Cisco IOS-XR platforms

Usage Guidelines and Restrictions

- Cisco 7600 Implementation

- Y.1731 PM not supported for these CFM scenarios:

- MEP on switchport

- MEP on VPLS L2VFI

- UP MEP on Service Instance with Bridge-Domain

- DOWN MEP on untagged Service Instance with Bridge-Domain

- DOWN MEP on doubled-tagged routed (sub)interface

- Port MEP

- After a Supervisor switchover, Y.1731 PM stats are cleared

- IPSLA restart required

- Port-Channel considerations

- Member interfaces must reside on ES+ linecards

- For Loss probes (LMM), all members must reside on the same NPU (restriction does not apply to Delay probes)

- When a member link is added/deleted, the session is rendered invalid

- Y.1731 PM not supported on Port-Channel with manual EVC load balancing

- Y.1731 PM not supported on mLACP

Prerequisites

- Configure CFM.

- MD, MA, and MEPs

- Enable distribution of local MEP configuration to ES+ linecards.

- Program hardware to respond to incoming Delay Measurement Message (DMM)/Loss Measurement Message (LMM) PDUs

- Router(config)#ethernet cfm distribution enable

- (Optional) Configure time source protocol (NTP or PTPv2). Required for one-way delay measurement.

- Enable synchronization down to the linecard.

- Router(config)#platform time-source

- (Optional) Enable service frame per-cos/aggregate counter monitoring under CFM MEP. Required for loss probes.

- Router(config-if-srv-ecfm-mep)#monitor loss counter

Configuration Management

The previous commands have already been enabled in Fault Management, therefore just IP SLA is enabled to start with Performance Management.

Ip sla 10

Ethernet y1731 loss LMM domain SPdomain evc EVC_CE1 mpid 201 cos 8 source mpid 102

Frame interval 100

Aggregate interval 180

Ip sla schedule 10 start-time after 00:00:30 life forever.

Verify

PE1#show ip sla stat 10

IPSLAs Latest Operation Statistics

IPSLA operation id: 10

Loss Statistics for Y1731 Operation 10

Type of operation: Y1731 Loss Measurement

Latest operation start time: 09:30:11.332 UTC Fri Dec 20 2013

Latest operation return code: OK

Distribution Statistics:

Interval

Start time:  09:30:11.332 UTC Fri Dec 20 2013

Elapsed time: 56 seconds

Number of measurements initiated: 120

Number of measurements completed: 120

Flag: OK

PE1#show ethernet cfm pm session active

Display of Active Session

----------------------------------------------------------------------------

EPM-ID   SLA-ID    Lvl/Type/ID/Cos/Dir Src-Mac-address Dst-Mac-address

----------------------------------------------------------------------------

0       10        5/XCON/N/A/7/Up     ccef.48d0.64b0  8843.e1df.00b0

Total number of Active Session: 1

--> Src-Mac-address: SRC MAC of MEP,check 'show ethernet cfm maintenance-points local'

--> Dst-Mac-address: MAC of dest MEP,check 'show ethernet cfm maintenance-points remote'

PE1#show ethernet cfm pm session detail 0

Session ID: 0

Sla Session ID: 10

Level: 5

Service Type: XCO

Service Id: N/A

Direction: Up

Source Mac: ccef.48d0.64b0

Destination Mac: 8843.e1df.00b0

Session Status: Active

MPID: 102

Tx active: yes

Rx active: yes

Timeout timer: stopped

Last clearing of counters: 08:54:20.079 UTC Sat Dec 20 2013

DMMs:

Transmitted: 0

DMRs:

Rcvd: 0

1DMs:

Transmitted: 0

Rcvd: 0

LMMs:

Transmitted: 3143161

LMRs

Rcvd: 515720

VSMs:   Transmitted: 0

VSRs:   Rcvd: 0

Debug Commands

debug ip sla trace <oper_id>

debug ip sla error <oper_id

Related Information

- ITU-T Y.1731 Performance Monitoring In a Service Provider Network

- Cisco Carrier Ethernet OAM Overview

- Technical Support & Documentation - Cisco Systems