Contents

Introduction

Prerequisites

Requirements

Components Used

Conventions

Decoding the Active Link Bitmap

Physical Layer Alarm or Incrementing Errors

E1 Troubleshooting Documents

T1 Troubleshooting Documents

Exceeding Differential Delay

IMA MIB Values for Differential Delay

Viewing Configured and Measured Values on NM-IMA

Viewing Configured and Measured Values on PA-A3-IMA

IMA Group Comes Down

Known Issues

Related Information

Introduction

Inverse multiplexing over ATM (IMA) defines a sublayer-1 protocol for building a virtual link that consists of multiple physical T1 or E1 links. The IMA protocol handles link failure and automatic link recovery, and it also adds and deletes links while keeping the IMA group in service.

This document provides troubleshooting steps for you to follow when your router reports up or down link changes to a member interface in an IMA group or when the group interface bounces. Physical T1 interfaces are removed from a group under these conditions:

- Exceeding the differential delay.

- Experiencing a physical layer alarm or error state.

Prerequisites

Requirements

Readers of this document should have knowledge of these topics:

- Inverse Multiplexing for ATM (IMA) Frequently Asked Questions

- IMA (Inverse Multiplexing for ATM) Support Pages

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Decoding the Active Link Bitmap

Before understanding the two reasons for physical T1 lines to be removed from an IMA group, it is important to know how to determine which link was removed.

In this sample output, you can see the IMA group repeatedly reporting a change in the number of active links in the group.

Apr 13 20:45:47.196 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0

now has 5 active links, active link bitmap is 0x37.

APR 13 20:45:47.964 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0

now has 6 active links, active link bitmap is 0x3F.

APR 13 20:45:51.184 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0

now has 5 active links, active link bitmap is 0x37.

APR 13 20:45:51.440 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0

now has 6 active links, active link bitmap is 0x3F.

APR 13 20:45:55.528 CDT: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM2/IMA0

now has 5 active links, active link bitmap is 0x37.

The important thing to note about this output is the active link bitmap. The rest of this section discusses how to decode this bitmap. You can see that the bitmap consists of two hexadecimal characters or eight bits. Read these bits from right to left, with physical T1 port 0 at bit position 0 and physical T1 port 7 at bit position 7.

With the above output as an example and using these tables, you can see that when the active link bitmap changes from 0x37 to 0x3F, physical port 3 drops from the IMA group.

Decode for 0x3F

2x Value

8

4

2

1

8

4

2

1

Binary Value

0

0

1

1

1

1

1

1

T1 Port Number

7

6

5

4

3

2

1

0

Decode for 0x37

2x Value

8

4

2

1

8

4

2

1

Binary Value

0

0

1

1

0

1

1

1

T1 Port Number

7

6

5

4

3

2

1

0

Note: Because repetitious bursts of errors affect the data path if links are brought back in too rapidly, the IMA specification suggests that vendors implement “de-bouncing” techniques to regulate T1 link addition and removal.

Physical Layer Alarm or Incrementing Errors

Issue the show controllers atm command on the physical T1 to determine the line status.

router# show controllers atm0/2

Interface ATM0/2 is administratively down

Hardware is ATM T1

!--- Output suppressed.

Link 2 Framer Info:

framing is ESF, line code is B8ZS, fdl is ANSI

cable-length is long, Rcv gain is 26db and Tx gain is 0db,

clock src is line, payload-scrambling is disabled, no loopback

line status is 0x1064; or TX RAI, Rx LOF, Rx LOS, Rx LCD.

port is active, link is unavailable

0 idle rx, 0 correctable hec rx, 0 uncorrectable hec rx

0 cells rx, 599708004 cells TX, 0 rx fifo overrun.

Link (2):DS1 MIB DATA:

Data in current interval (518 seconds elapsed):

0 Line Code Violations, 0 Path Code Violations

0 Slip Secs, 518 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins

0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 519 Unavail Secs

Total Data (last 24 hours)

0 Line Code Violations, 0 Path Code Violations,

0 Slip Secs, 86400 Fr Loss Secs, 0 Line Err Secs, 0 Degraded Mins,

0 Errored Secs, 0 Bursty Err Secs, 0 Severely Err Secs, 86400 Unavail Secs

The line status bitmap comes from the dsx1LineStatus section of RFC 1406 , the digital signal level 1 (DS-1) management information base (MIB). This contains loopback, failure, received alarm, and transmitted alarm information. The dsx1LineStatus is a bitmap represented as a sum, and thus can simultaneously represent multiple failures (alarms) and a loopback state. These are the various bit positions:

Bit Value

Meaning

Definition

1

dsx1NoAlarm

No alarm present.

2

dsx1RcvFarEndLOF

Far end loss of frame (LOF); also known as a Yellow Alarm.

4

dsx1XmtFarEndLOF

Near end sending LOF indication.

8

dsx1RcvAIS

Far end sending alarm indication signal (AIS).

16

dsx1XmtAIS

Near end sending AIS.

32

dsx1LossOfFrame

Near end LOF; also known as a Red Alarm.

64

dsx1LossOfSignal

Near end loss of signal (LOS).

128

dsx1LoopbackState

Near end is looped.

256

dsx1T16AIS

E1 TS16 AIS.

512

dsx1RcvFarEndLOMF

Far end sending TS16 loss of multiframe (LOMF).

1024

dsx1XmtFarEndLOMF

Near end sending TS16 LOMF.

2048

dsx1RcvTestCode

Near end detects a test code.

4096

dsx1OtherFailure

Any line status that is not defined in this list.

With the IMA network module, Cisco uses the value 4096 to indicate loss of cell delineation.

Note: ATM cell delineation and cell misinsertion problems can affect the IMA frame and lead to problems with an IMA interface. In these cases, the router typically does not report physical layer errors or alarms for the physical T1 interfaces.

Note: The clock source on an IMA link changes from line to internal when an AIS or an LOS is detected.

When troubleshooting IMA link drops, issue the show log command to check the router’s log for link up or down messages, as shown in this sample output:

APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/1, changed state to down

APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/3, changed state to up

APR 2 13:57:18: %LINK-3-UPDOWN: Interface ATM1/2, changed state to up

Cisco offers several T1 and E1 troubleshooting documents to assist you in resolving a line status message other than no alarm present.

E1 Troubleshooting Documents

- E1 Troubleshooting Flowchart

- E1 Alarm Troubleshooting

- E1 Error Events Troubleshooting

- E1 Layer 1 Troubleshooting

- Hard Plug Loopback Tests for E1 Lines

T1 Troubleshooting Documents

- T1 Troubleshooting Flowchart

- T1 Alarm Troubleshooting

- T1 Error Events Troubleshooting

- T1 Layer 1 Troubleshooting

- Loopback Tests for T1/56K Lines

Exceeding Differential Delay

In addition to rebuilding the original ATM cell stream, a receiving IMA interface compensates for delays in timing and in the arrival of adjacent ATM cells. A receiving interface detects and rejects physical links with delay greater than the provisioned maximum differential delay tolerance. This tolerance refers to the differences in the arrival of adjacent cells; that is, cells may be delayed more on one link than on a second link. If a link delay exceeds the specified maximum, the link is dropped. Otherwise, the IMA feature—while multiplexing and demultiplexing—adjusts for differences in delays so that all links in a group are aligned.

A receiving interface uses IMA control protocol (ICP) cells to determine the differential delay between the links in the IMA group. ICP cells are sent on each link once per IMA frame. By default, each frame consists of 128 cells. The ICP cell can be located anywhere inside the frame (cell position 0 or cell position 127 or any cell position in between).

Each frame includes an IMA frame sequence number in the ICP cell. This number increments per link and with each transmitted IMA frame. Over the two or more links in the IMA group, the sequence number in the ICP cell of each link is identical. In other words, link 0 uses IMA sequence 0 at the same time that link 1 uses IMA sequence number 0.

The IMA technical specifications on the ATM Forum define how both the transmitter and the receiver act with respect to differential link delay among the constituent links in the IMA group.

- Transmitter’s requirements—The transmit IMA shall not introduce more than 2.5 cell times at the physical link rate of differential delay among the constituent links.

- Receiver’s requirements—The amount of link differential delay tolerated by an IMA implementation shall be up to at least 25 milliseconds when used over DS-1 or E1 links. The amount of link differential delay tolerance may be configured up to the maximum value supported by the IMA implementation.

Both ends of the IMA virtual link may be configured with different amounts of tolerable differential delay.

Appendix A of the standard provides an example of how the receiving IMA virtual link compensates for differential delay. At the receive end, each physical link has its own circular buffer which should be deep enough to tolerate the maximum differential delay variation.

The next diagram shows the cells of three physical links in the same IMA group. The transmitting IMA process distributes ATM cells over the links in a cyclic, round-robin fashion and on a cell-by-cell basis. Whenever a cell is written into the buffer, the write pointer increments. A read (playback) pointer increments as cells are read out of the buffer. All of the delay compensation is done by adjusting the write pointer. IMA delay compensation changes the virtual alignment of the user cells so that, with each read interval or IMA clock tick, one data cell is removed from the buffer. The receiving IMA process always reads one cell per link in a cyclic, round-robin fashion. One cell time equals 276 microseconds at the DS-1 rate. Thus, assuming a maximum allowable delay of 25 milliseconds, the maximum difference acceptable between the write pointers is 91 cells.

In this sample transmission, the source interface has staggered the ICP cells in various cell positions within an IMA frame. At the destination interface, Link 0 and Link 2 have the same amount of propagation delay, but Link 1 has a delay one cell time longer than Link 0 or Link 2.

IMA MIB Values for Differential Delay

The IMA recommendation defines a MIB for collecting IMA-specific statistics for the member T1s and for the IMA group itself. The IMA MIB provides these counters, which are relevant in troubleshooting problems related to differential delay:

- ImaGroupLeastDelayLink—Interface index (ifIndex) of the link configured in the IMA group which has the smallest link propagation delay. The distinguished value of zero may be used if no link has been configured in the IMA group, or if the link with the smallest link propagation delay has not yet been determined.

- ImaGroupDiffDelayMaxObs—Latest maximum differential delay observed (in milliseconds) between the links with the least and the most link propagation delay, among the receive links that are currently configured in the IMA group.

Viewing Configured and Measured Values on NM-IMA

On the IMA network module for Cisco 2600 and 3600 series routers, issue the show ima interface atm command to view the configured maximum allowed differential delay value for your IMA group.

2600# show ima interface atm2/ima2

Interface atm2/IMA2 is up

Group index is 2

Ne state is operational, failure status is noFailure

active links bitmap 0x30

IMA Group Current Configuration:

TX/Rx configured links bitmap 0x30/0x30

TX/Rx minimum required links 1/1

Maximum allowed diff delay is 25ms, TX frame length 128

NE TX clock mode CTC, configured timing reference link atm2/4

Test pattern procedure is disabled

IMA Group Current Counters (time elapsed 12 seconds):

3 NE Failures, 3 Fe Failures, 4 Unavail Secs

IMA Group Total Counters (last 0 15 minute intervals):

0 NE Failures, 0 Fe Failures, 0 Unavail Secs

IMA link Information:

Physical Status   NearEnd          Rx Status      Test Status

------------ --------------- -----------------    -----------

atm2/4            up                active          disabled

atm2/5            up                active          disabled

Issue the show ima interface atm command on the physical T1 member interface to view the currently measured value. This command also displays IMA link error counters for the physical interface. Refer to Troubleshooting ATM IMA Links on Cisco 2600 and 3600 Routers for an explanation of these error counters.

3640-2.2# show ima interface atm 0/0

Interface ATM0/0 is up

ifIndex 1, Group Index 1, Row Status is active

TX/Rx Lid 0/0, relative delay 0ms

NE TX/Rx state active/active

Fe TX/Rx state active/active

NE Rx failure status is noFailure

Fe Rx failure status is noFailure

Rx test pattern 0x40, test procedure disabled

IMA Link Current Counters (time elapsed 866 seconds):

0 Ima Violations, 0 Oif Anomalies

0 NE Severely Err Secs, 0 Fe    Severely Err Secs

0 NE Unavail Secs, 0 Fe Unavail  Secs

0 NE TX Unusable Secs, 0 NE Rx   Unusable Secs

0 Fe TX Unusable Secs, 0 Fe Rx   Unusable Secs

0 NE TX Failures, 0 NE Rx Failures

0 Fe TX Failures, 0 Fe Rx Failures

!--- Output suppressed.

Finally, issue the show ima interface atm slot/imagroup-number detail command to view the maximum observed differential delay value among the member interfaces and to view the particular member interface that is exhibiting the least amount of delay.

3640-2.2# show ima interface atm0/ima0 detail

Interface ATM0/IMA0 is up

Group index is 1

NE state is operational, failure status is noFailure

Active links bitmap 0x3

IMA Group Current Configuration:

TX/Rx configured links bitmap    0x3/0x3

TX/Rx minimum required links    1/1

Maximum allowed diff delay is    25ms, TX frame length 128

NE TX clock mode CTC, configured    timing reference link ATM0/0

Test pattern procedure is disabled

Detailed group Information:

TX/Rx Ima_id 0x0/0x0, symmetry    symmetricOperation

Number of TX/Rx configured links    2/2

Number of TX/Rx active links    2/2

Fe TX clock mode ctc, Rx frame    length 128

TX/Rx timing reference link 0/0

Maximum observed diff delay 0ms, least delayed link 0

Running seconds 101257

GTSM last changed 23:35:52 UTC    Sat Mar 6 1993

IMA Group Current Counters (time elapsed 896 seconds):

0 NE Failures, 0 Fe Failures,    0 Unavail Secs

IMA Group Interval(1) Counters:

0 NE Failures, 0 Fe Failures,    0 Unavail Secs

IMA Group Interval(2) Counters:

0 NE Failures, 0 Fe Failures,    0 Unavail Secs

Viewing Configured and Measured Values on PA-A3-IMA

On a PA-A3-8T1 or E1 IMA port adapter, issue the show controllers atm command to see the configured value.

atm# show controllers atm1/ima0

Interface ATM1/IMA0 is down

Hardware is ATM IMA

!--- Output suppressed.

ATM channel number is 2

link members are 0x4, active links are 0x0

Group status is blockedNe, 1 links configured,

Group Info: Configured links bitmap 0x4, Active links bitmap 0x0,

TX/Rx IMA_id 0x11/0x63,

NE Group status is startUp,

frame length 0x80, Max Diff Delay 50,

1 min links, clock mode ctc, symmetry symmetricOperation, trl 2,

Group Failure status is startUpNe.

Test pattern procedure is disabled

On a PA-A3-8T1 or E1 IMA port adapter, issue the show ima interface atm slot/imagroup-number detail command to view the currently measured values for differential delay.

7200# show ima interface atm 1/ima0 detail

ATM1/ima0 is up

ImaGroupState:NearEnd = operational, FarEnd = operational

ImaGroupFailureStatus = noFailure

IMA Group Current Configuration:

ImaGroupMinNumTxLinks = 2 ImaGroupMinNumRxLinks = 2

ImaGroupDiffDelayMax = 25 ImaGroupNeTxClkMode = common(ctc)

ImaGroupFrameLength = 128 ImaTestProcStatus = disabled

ImaGroupTestLink = 0 ImaGroupTestPattern = 0xFF

IMA MIB Information:

ImaGroupSymmetry = symmetricOperation

ImaGroupFeTxClkMode = common(ctc)

ImaGroupRxFrameLength = 128

ImaGroupTxTimingRefLink = 0 ImaGroupRxTimingRefLink = 0

ImaGroupTxImaId = 0 ImaGroupRxImaId = 0

ImaGroupNumTxCfgLinks = 2 ImaGroupNumRxCfgLinks = 2

ImaGroupNumTxActLinks = 2 ImaGroupNumRxActLinks = 2

ImaGroupLeastDelayLink = 1 ImaGroupDiffDelayMaxObs = 0

IMA group counters:

ImaGroupNeNumFailures = 78 ImaGroupFeNumFailures = 68

ImaGroupUnAvailSecs = 441453 ImaGroupRunningSecs = 445036

IMA Group Comes Down

Every IMA group must have a minimum number of links to be active. Issue the ima active-links-minimum number command to change this value. If the number of active links falls below the configured value, your router legitimately brings down the IMA virtual interface.

If your router’s IMA group interface goes down unexpectedly, determine what equipment terminates the IMA protocol at the remote end of the link before contacting Cisco Technical Support. Cisco Bug ID CSCdr93036 (duplicated to CSCdr19633) documents a problem with the Processor Switching Module (PXM) and the AUSMB-8T1, which provides IMA services for the Cisco MGX WAN switch. In rare conditions, one physical T1 link coming down brings down the entire IMA group.

If the IMA virtual interface is assigned a number as a group name and a physical link with the same number goes down, the IMA group comes down. In this sample output, the AUSMB-8 card is configured with an IMA group identified with the value of 1. The group consists of six links identified by link numbers 1 through 6.

MGXISH03.1.29.AUSMB8.a > dspimainfo

Link Group NeTx   NeRx   FeTx   FeRx   TxLID RxID

State  State  State  State

-------------------------------------------------

1    1     Active Active Active Active 0     0

2    1     Active Active Active Active 1     1

3    1     Active Active Active Active 2     2

4    1     Active Active Active Active 3     3

5    1     Active Active Active Active 4     4

6    1     Active Active Active Active 5     5

The root cause of the problem was that the line number going to alarm generated an interface trap (note that the other lines going into alarm did not generate such a trap).

An ATM user service module (AUSM) generates a port trap to the PXM when a line goes down and there is a port within the line. The fix changes the behavior of the AUSM and ensures that the AUSM sends a port trap only when the number of active links falls below the required minimum.

Known Issues

In rare circumstances, an IMA group may experience the removal of multiple T1 physical links, as reported in the router’s log.

!--- Each of these timestamped lines of output appear on one line.

APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now

has 4 active links, active link bitmap is 0xD.

APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now

has 1 active links, active link bitmap is 0xD.

APR 2 13:57:17: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now

has 4 active links, active link bitmap is 0xD.

APR 2 13:57:18: %IMA-5-ACTIVE_LINK_CHANGE: IMA Group ATM1/IMA1 now

has 3 active links, active link bitmap is 0xD.

These symptoms match Cisco Bug ID CSCdr39332, which documents a rare condition in which exceeding the configured maximum differential delay value causes the IMA process to remove the good links or to bring down the entire IMA group. The expected action is that the delay link comes down. The problem is related to how the IMA firmware uses the delay compensation buffer and adjusts the write pointers, as described in this document. As a workaround, try increasing the configured maximum value with the ima differential-delay-maximum milliseconds command. Specify a value from 25 to 250 milliseconds.

If you experience this issue, collect output from the show log and show tech commands and open a service request with Cisco Technical Support.

Related Information

- IMA (Inverse Multiplexing for ATM) Support Pages