Troubleshooting High CPU Utilization Due to Interrupts

Introduction

This document explains the reasons for high CPU utilization due to interrupts, and provides troubleshooting tips and guidelines.

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Conventions

For more information on document conventions, refer to the Cisco Technical Tips Conventions.

Possible Causes of High CPU Utilization Due to Interrupts

High CPU utilization on an interrupt level is primarily caused by packets handled on interrupt level. Interrupts are generated any time a character is output from the console or auxiliary ports of a router.

Universal Asynchronous Receiver/Transmitters (UARTs) are slow compared to the processing speed of the router, so it is unlikely, though possible, that console or auxiliary interrupts can cause a high CPU utilization on the router (unless the router has a large number of tty lines in use).

There are several reasons for high CPU utilization due to interrupts:

• An inappropriate switching path is configured on the router

• The CPU is performing alignment corrections

• The router is overloaded with traffic

• There's a bug in the Cisco IOS® software running on the router

• Voice ports are configured on the router

• There are active Asynchronous Transfer Mode (ATM) interfaces on the router

• Too many packets are being punted from PXF to the Route Processor (RP)

Inappropriate Switching Path

To troubleshoot this potential issue, verify the following:

• Check whether or not the router is running Cisco Express Forwarding:

  • Verify the configuration for the ip cef global configuration command.

  • Verify that Cisco Express Forwarding is enabled and working by issuing the show ip cef summary command.

  • Verify that Cisco Express Forwarding is enabled as the switching path on all interfaces. You can see this in show cef interface and show ip interface output. If Cisco Express Forwarding is configured, but not enabled on an interface, this means that the interface encapsulation is not supported in Cisco Express Forwarding. Verify that Cisco Express Forwarding is operational, that is, check whether packets are really switched through the router using Cisco Express Forwarding by looking at the show cef not-cef-switched .

  • Using the show cef drop command and the show interfaces switching command (this is a hidden command you can use to look for cache misses), verify that Cisco Express Forwarding isn't dropping packets. If this is the case, see the CEF troubleshooting page.

• Verify if any of the interfaces have long access lists configured.

  • As a general rule of thumb, any access list with over ten lines is considered long.

  • Repeatedly going over long access lists is very CPU-intensive. With NetFlow switching, if the flow is already in the cache, you no longer need to check the access list. So in this case, NetFlow switching would be useful. You can enable NetFlow switching by issuing the ip route-cache flow command.

  • Note that if Cisco Express Forwarding and NetFlow are both configured on an interface, Cisco Express Forwarding will be used to make a switching decision.

• Verify that NetFlow switching is configured on the router:

  • Check the statistics by issuing the show ip cache flow command. Look at the number of new flows per second.

  • If Cisco Express Forwarding is not enabled, enable Cisco Express Forwarding to speed up the switching decision.

  • If there are no long access lists, try disabling NetFlow switching.

CPU Performing Alignment Corrections

Alignment errors are caused by misaligned reads and writes. For example, a two-byte read where the memory address is not an even multiple of two bytes is an alignment error.

Alignment errors are usually caused by a software bug. The CPU corrects this error, but if there are many corrections to do, this becomes CPU- intensive. For troubleshooting this type of error, see Troubleshooting Spurious Accesses, Alignment Errors, and Spurious Interrupts.

Router Overloaded with Traffic

The output of the show interfaces and show interfaces switching (hidden) commands provide information about which interfaces are overloaded. To capture the output of these commands in a log file for later analysis, follow the steps below.

1. Issue the terminal length 0 command.

2. Check the output of show interfaces . Examine the load and the number of throttles on interfaces. The load is an average value computed, by default, over five minutes. To change this interval, issue the load-interval seconds command, where the seconds represent the length of time for which data is used to compute load statistics. Use a value that is a multiple of 30.

   Throttles are a good indication of an overloaded router. They show the number of times the receiver on the port has been disabled, possibly due to buffer or processor overload. Together with high CPU utilization on an interrupt level, throttles indicate that the router is overloaded with traffic.

3. Check the output of the show interfaces switching (hidden) command to see what kind of traffic (protocol and switching path) is going through the overloaded interface. If some interfaces are too overloaded with traffic, consider redesigning the traffic flow in the network or upgrading the hardware.

4. The network loop can also be a reason for the traffic overload. Verify your network topology.

If there is a possibility that a single device is generating packets at an extremely high rate and thus overloading the router, you can determine the MAC address of that device by adding the ip accounting mac-address {input|output} interface configuration command to the configuration of the overloaded interface.

The show interfaces [ ] mac-accounting command displays the collected information. Once the source device's MAC address is found, the corresponding IP address can be found by checking the output of the show ip arp command.

Software Bug

If you suspect a bug in the Cisco IOS software version running on the router, you can check the Bug Toolkit (registered customers only) for a bug that reports similar symptoms in a similar environment.

Voice Ports Configured on the Router

Even if there is no traffic, software continues to monitor channel-associated signaling (CAS), which uses CPU resources.

Active Asynchronous Transfer Mode (ATM) Interfaces on the Router

Even if there is no traffic, the ATM interfaces send out null cell (per ATM standards) and continue to use CPU resources.

Many Parallel Express Forwarding (PXF) Punts to the RP

When PXF punts too many packets to the RP, the RP may get overloaded. You can compare the amount of punted packets to the total amount of incoming packets by issuing the show pxf accounting summary command. Use the same command to find out why the packets are punted to the RP. This could be either a software bug, or traffic is not supported by PXF.

CPU Profiling

CPU profiling is a low-overhead way of determining where the CPU spends its time. The system works by sampling the processor location every four milliseconds. The count for that location in memory is incremented. The root cause of this CPU utilization will be determined by CPU Profiling.

Complete these steps in order to perform CPU profiling. CPU utilization has be done when you are experiencing high CPU utilization.

Note: All these commands must be typed when in enable mode

1. Capture the output of show region and take the starting address, the ending address and the size of main:text region

2. Capture the output of show memory statistics and check the size of the largest block in processor memory.

3. Do profile task interrupt to configure profiling only for interrupts.

4. Compare the size of main:text region with the size of the largest block of free processor memory. Ideally the largest block should be larger than the main:text.

   If the largest block is smaller than main:text size, then adjust the granularity to make sure that the profiling will be able to get a block of processor memory.

   If the largest block is larger than main:text region, use a granularity of 4.

   If the largest block is larger than the half of the main:text region, use a granularity of 8.

   If the largest block is larger than a quarter of the main:text region, use a granularity of 10 ( 16 in hexadecimal).

   Note: Granularity must be a power of 2 and should be as small as possible (but not smaller than 4)

5. Start profiling by doing profile

   Profile <starting address> <ending address> <granularity value>

   The starting address and ending address is determined from the Step1.

6. Wait 5 to 10 minutes

7. Stop profiling by doing profile stop

8. Capture the output of show profile terse.

9. Make sure the memory is freed by doing unprofile all

show interfaces switching Command

This command is used for determining active switching paths on interfaces. For more information about switching paths in Cisco IOS software, refer to Configuring Switching Paths .

The following is a sample output of the show interfaces switching command for one interface:

RouterA#show interfaces switching  
 Ethernet0            
           Throttle count          0          
         Drops         RP          0         SP          0    
   SPD Flushes       Fast          0        SSE          0    
   SPD Aggress       Fast          0   
  SPD Priority     Inputs          0      Drops          0       
      Protocol       Path    Pkts In   Chars In   Pkts Out  Chars Out          
         Other    Process          0          0        595      35700              
             Cache misses          0                      
                     Fast          0          0          0          0                 
                Auton/SSE          0          0          0          0             
               IP Process          4        456          4        456              
             Cache misses          0                      
                     Fast          0          0          0          0                 
                Auton/SSE          0          0          0          0            
              IPX Process          0          0          2        120              
             Cache misses          0                      
                     Fast          0          0          0          0                 
                Auton/SSE          0          0          0          0  
    Trans. Bridge Process          0          0          0          0              
             Cache misses          0                      
                     Fast         11        660          0          0                 
                Auton/SSE          0          0          0          0        
          DEC MOP Process          0          0         10        770              
             Cache misses          0                      
                     Fast          0          0          0          0                 
                Auton/SSE          0          0          0          0             
              ARP Process          1         60          2        120              
             Cache misses          0                      
                     Fast          0          0          0          0                 
                Auton/SSE          0          0          0          0            
              CDP Process        200      63700        100      31183              
             Cache misses          0                      
                     Fast          0          0          0         0                 
                Auton/SSE          0          0          0          0 

The output lists the switching paths for all protocols configured on the interface, so you can easily see what kind and the amount of traffic is going through the router. The following table explains the output fields:

Field	Definition
Process	Processed packets. These can be packets destined for the router, or packets for which there was no entry in the fast switching cache.
Cache misses	Packets for which there was no entry in fast switching cache. The first packet for this destination (or flow - depending on the type of fast switching configured) will be processed. All subsequent packets will be fast switched, unless fast switching is explicitly disabled on the outgoing interface.
Fast	Fast switched packets. Fast switching is enabled by default.
Auton/SSE	Autonomous switched, silicon switched, or distributed switched packets. Available only on Cisco 7000 Series Routers with a Switch Processor or Silicon Switch Processor (for autonomous switching or silicon switching, respectively), or on Cisco 7500 Series Routers with a VIP (for distributed switching).

Sample Script in order to Gain CPU Profiling At High CPU

This script saves the outputs on flash:CPU_Profile when the CPU utilization is more than 75%:

service internal

event manager applet High_CPU

event snmp oid 1.3.6.1.4.1.9.9.109.1.1.1.1.6 get-type
	 next entry-opge entry-val 75

exit-time 10 poll-interval 5

action 0.1 syslog msg "CPU Utilization is high"

action 0.2 cli command "enable"

action 0.4 cli command "show log | append flash:CPU_Profile.txt"

action 0.5 cli command "show process cpu sorted | append
	 flash:CPU_Profile.txt"

action 0.6 cli command "show interfaces | append
	 flash:CPU_Profile.txt"

action 1.1 cli command "configure terminal"

action 1.2 cli command "profile xxxxxxx yyyyyyyyZ"

action 1.3 cli command "profile start"

action 2.3 syslog msg "Entering TCLSH"

action 2.4 cli command "tclsh"

action 2.5 cli command "after 240000"

action 2.6 cli command "exit"

action 2.9 syslog msg "Exiting TCLSH"

action 3.0 cli command "profile stop"

action 3.1 cli command "show profile terse | append flash:CPU_Profile.txt"

action 3.2 cli command "clear profile"

action 3.3 cli command "unprofile all"

action 4.1 syslog msg "Finished logging information to flash:CPU_Profile.txt..."

action 4.2 cli command "end"

Related Information

• Troubleshooting High CPU Utilization on Cisco Routers
• Technical Support & Documentation - Cisco Systems