This list describes the properties of ethernet link bundles:

• The router supports mixed speed bundles. Mixed speed bundles allow member links of different bandwidth to be configured as active members in a single bundle. The ratio of the bandwidth for bundle members must not exceed 10. Also, the total weight of the bundle must not exceed 64. For example, 100Gbps link and 10Gbps links can be active members in a bundle and load-balancing on member links is based on bandwidth weightage.

• The weight of each bundle member is the ratio of its bandwidth to the lowest bandwidth member. Total weight of the bundle is the sum of weights or relative bandwidth of each bundle member. Since the weight for a bundle member is greater than or equal to 1 and less than or equal to 10, the total member of links in a bundle is less than 64 in mixed bundle case.

• Any type of Ethernet interfaces can be bundled, with or without the use of LACP (Link Aggregation Control Protocol).

• A single router can support a maximum of 256 bundle interfaces. Link bundles of only physical interfaces are supported.

• Physical layer and link layer configuration are performed on individual member links of a bundle.

• Configuration of network layer protocols and higher layer applications is performed on the bundle itself.

• IPv4 and IPv6 addressing is supported on ethernet link bundles.

• A bundle can be administratively enabled or disabled.

• Each individual link within a bundle can be administratively enabled or disabled.

• Ethernet link bundles are created in the same way as Ethernet channels, where the user enters the same configuration on both end systems.

• Prior to Cisco IOS XR Release 7.3.1, the router supported 8 bundle members. The maximum number of supported sub-interfaces for Link Aggregation Group (LAG) or LACP are:

  • Layer 3 sub-interfaces is 1000.

  • Layer 2 sub-interfaces is 4000 per bundle.

  • Member links per bundle is 8.

  Starting with Cisco IOS XR Release 7.3.1, the router supports 16 bundle members, and the database entries are optimized to use one entry per bundle sub-interface, regardless of the number of members in a bundle. The maximum number of supported sub-interfaces for LAG or LACP are:

  • Layer 3 sub-interfaces is 1000.

  • Layer 2 sub-interfaces is 16000 per router.

  • Layer 2 sub-interfaces is 4000 per bundle.

  • Member links per bundle is 16.

• QoS is supported and is applied proportionally on each bundle member.

• If the HQoS profile is enabled, the maximum available trunks by default (physical+sub-interfaces) are 256. If more trunks are required, you can configure the hw-module profile bundle-scale <256/512/1024> command. With HQoS enabled on bundle interfaces, a maximum of 4 priority levels are supported.

  The maximum number of supported bundle members with HQoS profile on Layer2 and Layer3 interfaces:

  • Maximum of 1024 trunks (128 physical interfaces + 896 sub-interfaces) and 16 bundle members.

  • Maximum of 256 trunks (128 physical interfaces + 128 sub-interfaces) and 64 bundle members.

  • Maximum of 512 trunks (128 physical interfaces + 384 sub-interfaces) and 32 bundle members.

• In case static MAC address is configured on a bundle-ether interface, the following limitations are applied:

  • Locally generated packets, such as ICMP, BGP, and so on, going out from the interface have the source MAC address as the statically configured MAC address.

  • Transit (forwarded) packets going out of the interface do not have the configured static MAC as source MAC address. In such a scenario, the upper 36-bits come from the system MAC address (or the original/dynamic MAC address) and the lower 12-bits come from the MAC address configured on the bundle. To check the dynamic pool of MAC addresses included, use the show ethernet mac-allocation detail command.

    For example, if the dynamic MAC address was 008A.9624.48D8 and the configured static MAC address is 0011.2222.ABCD. Then, the source MAC for transit (forwarded) traffic will be 008A.9624.4BCD.

    Note	This limitation can cause traffic blackholing for the transit traffic, in case there is L2 ACL applied for security purpose. In such case, it is necessary to add permit statement for both MAC addresses in the L2 ACL.

• Load balancing (the distribution of data between member links) is done by flow instead of by packet. Data is distributed to a link in proportion to the bandwidth of the link in relation to its bundle.

• All links within a single bundle must terminate on the same two systems.

• Bundled interfaces are point-to-point.

• A link must be in the up state before it can be in distributing state in a bundle.

• Only physical links can be bundle members.

• Multicast traffic is load balanced over the members of a bundle. For a given flow, the internal processes selects the member link, and the traffic for the flow is sent over that member.

• Ensure the load interval of the bundle interface is configured to 300 seconds (default value) on router to prevent fluctuation in the outgoing interface packet and packet mismatch in the interface statistics.

To configure link bundling, you must understand the following concepts:

802.1Q VLAN subinterfaces can be configured on 802.3ad Ethernet link bundles. Keep the following information in mind when adding VLANs on an Ethernet link bundle:

Note	The memory requirement for bundle VLANs is slightly higher than standard physical interfaces.

To create a VLAN subinterface on a bundle, include the VLAN subinterface instance with the interface Bundle-Ether command, as follows:

interface Bundle-Ether interface-bundle-id.subinterface

After you create a VLAN on an Ethernet link bundle, all VLAN subinterface configuration is supported on that link bundle.

VLAN subinterfaces can support multiple Layer 2 frame types and services, such as Ethernet Flow Points - EFPs) and Layer 3 services.

Layer 2 EFPs are configured as follows:

interface bundle-ether instance.subinterface l2transport. encapsulation dot1q xxxxx 

Layer 3 VLAN subinterfaces are configured as follows:

interface bundle-ether instance.subinterface, encapsulation dot1q xxxxx 

Note	The difference between the Layer 2 and Layer 3 interfaces is the l2transport keyword. Both types of interfaces use dot1q encapsulation .

As packets are exchanged across member links of a bundled interface, some member links may slow down or time-out and fail. LACP packets are exchanged periodically across these links to verify the stability and reliability of the links over which they pass. The configuration of short period time intervals, in which LACP packets are sent, enables faster detection and recovery from link failures.

Short period time intervals are configured as follows:

• In milliseconds

• In increments of 100 milliseconds

• In the range 100 to 1000 milliseconds

• The default is 1000 milliseconds (1 second)

• Up to 8 member links

• Up to 1280 packets per second (pps)

After 3missed packets, the link is detached from the bundle.

When the short period time interval is not configured, LACP packets are transmitted over a member link every 30 seconds by default.

When the short period time interval is configured, LACP packets are transmitted over a member link once every 1000 milliseconds (1 second) by default. Optionally, both the transmit and receive intervals can be configured to less than 1000 milliseconds, independently or together, in increments of 100 milliseconds (100, 200, 300, and so on).

When you configure a custom LACP short period transmit interval at one end of a link, you must configure the same time period for the receive interval at the other end of the link.

Note

You must always configure the transmit interval at both ends of the connection before you configure the receive interval at either end of the connection. Failure to configure the transmit interval at both ends first results in route flapping (a route going up and down continuously). When you remove a custom LACP short period, you must do it in reverse order. You must remove the receive intervals first and then the transmit intervals.

In a scaled setup, a bundle programming check is difficult to perform and time consuming. Moreover, an issue is reported only when the user detects it, and not automatically. During multiple test executions, it isn’t possible to detect the initial failure, which causes other subsequent failures. Bundle Consistency Checker (BCC) implements bundle programming and consistency check by using the following steps:

1. BCC uses the running configuration to detect discrepancies.

2. BCC forms a Bundle Consistency Checker Data Base (BCCDB) with the bundle, sub-bundle, or member information fetched from the running configuration.

3. BCC dumps the required data from all available nodes. It then uses BCCDB as a source to verify bundle programming and consistency in all other layer dumps.

4. BCC reports inconsistencies, programming errors, stale entries, and deletes any pending objects.

Supporting Interfaces

The following interfaces support BCC:

• Bundle

• Bundle sub-interface

The following table lists BCC behaviour during inconsistencies in bundle configuration or programming errors.

Router# show bundle consistency 
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Not Done

BCC Stopped: Found 3 info/exceptions/errors

Logs Preview:

2020-07-13 10:34:22,774: INFO: Bundlemgr PD dont have any bundle data

2020-07-13 10:34:22,832: INFO: BMPI dont have any bundle data

2020-07-13 10:34:23,728: INFO: No Bundle is configured/No member is added to Bundle

Logs: 
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Not Done

BCC Stopped: Found 4 info/exceptions/errors

Logs Preview:

2020-07-13 10:36:32,513: INFO: Bundlemgr PD dont have any bundle data

2020-07-13 10:36:32,566: INFO: No member is added to bundle BE1(0x3c00400c)

2020-07-13 10:36:32,566: INFO: BMPI dont have any bundle data

2020-07-13 10:36:33,453: INFO: No Bundle is configured/No member is added to Bundle

Logs: 
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 0        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 0        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Router# show bundle consistency
Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 0        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 3        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Logs Preview:


2020-07-12 17:38:26,568: INFO: No member is added to bundle BE2(0x80042bc) ==> l2 bundle main

2020-07-12 17:38:32,573: interface Bundle-Ether1.1: Dont have any encapsulation config. ==> l3 sub

2020-07-12 17:38:32,574: interface Bundle-Ether1.130: Dont have any encapsulation config.  ==> l2sub

Logs:

/var/log/bcc_inconsistencies.log
/var/log/bcc_programming_error.log
/var/log/bcc_stale_entries.log
/var/log/bcc_delay_delete.log
/var/log/bcc_bundle_health.log
/var/log/bcc_exception.log
/var/log/bcc_debug.log

Router# show bundle consistency

Building configuration...
Dumping Data............................ Done
Parsing Data............................ Done
Bundle Consistency Check................ Done
Bundle Programming Check................ Done
Stale Entry Check....................... Done
Bundle Health Check..................... Done

Overall Results:

Inconsistencies             : 0        Stale Entries                : 0
BCM Programming Error       : 1        Delete Pending DPA Objects   : 0
Info/Error/Python Exception : 0        Overall Bundle Health Status : WARNING

Execute 'show bundle status' to see detailed reason for 'WARNING' in bundle health check

Logs Preview:

2020-07-12 18:48:22,658: Programming Error 1: BE1(0x80041ec) NPU 0,0/RP0/CPU0 Vlan Domain 0x33 != GigabitEthernet0_0_0_2 Vlan Domain 0xa

Logs:

/var/log/bcc_inconsistencies.log
/var/log/bcc_programming_error.log
/var/log/bcc_stale_entries.log
/var/log/bcc_delay_delete.log
/var/log/bcc_bundle_health.log
/var/log/bcc_exception.log
/var/log/bcc_debug.log