Contents

Introduction

Prerequisites

Topology

Generic outputs to collect from WLC

Specific outputs from WLC for concrete AP

Advanced logs from WLC and AP for concrete AP

Introduction

This document describes a systematic approach and list of commands to collect for troubleshooting 9800 AP join/disconnection issues.

Prerequisites

Cisco recommends that you have knowledge of these topics:

Basic knowledge of Cisco WLC 9800

Basic knowledge of Cisco Wave2 and/or 11AX APs

Topology

This troubleshooting flow is applicable for APs connected in local mode or APs connected in flexconnect mode in a branch site.

Generic outputs to collect from WLC

1.- We can start to verify that the number of APs matches the expected number of APs connnected to WLC. And reviewing WLC logs for AP disconnections.

Identify if we are missing any APs and if in the logs we observe APs disconnecting at the same time or always same APs connecting/disconnecting

show ap summary | i Number of APs !!Check if number of APs matches with number we expect

sh log | i AP Event: !!Check if APs are disconnecting at same time, or exist any pattern for APs connecting/disconnecting.

2.- We can get list of all the APs and check for missing APs. Identify APs with lower Up Time and lower Assoc Uptime, that helps to identify if there are APs reloading or reconnecting to capwap.

If APs are reloading we can check for APs with similar uptime and check if those APs were in same switch. If we are seeing APs with high Up time and lower Association time we need to check if

there were changes done to those APs that could cause capwap restart or if there are capwap flaps due to retransmissions. We can also check if there are any AP crashes.

show ap uptime !! Check Up Time vs Assoc Up Time. Check for patterns

show ap cdp nei !! Identify if all APs with similar Up Time were in same switch.

show ap crash !! Check if there are any AP crashes.

dir all | i crash !! Find AP crashes stored in WLC.

3.- We can check for all AP connect/disconnect historical events, and disconnect reasons. We can find out if disconnect reason is similar to all the APs and in which phase of the connection did disconnection happen.

Identify top disconnection reasons and if there is any time pattern for those disconnections.

show wireless stats ap history !! Find connect/disconnect events, time for those events, disconnect reason and count.

show wireless stats ap discovery !! Find discovery requests been received by WLC and time for those discovery requests

show wireless stats ap join summary !! Find status of the AP, last disconnection reason and in which phase disconnection occurred.

4.- In case the errors are seen in the DTLS phase we can check which type of certificate and ciphers are used for AP DTLS handshake.

show wireless certification config !! Check DTLS version and cipher suite

show wireless management trustpoint !! Type of certificate used

show wireless dtls connections !! Show if DTLS is established for capwap control/data ports used

Specific outputs from WLC for concrete AP

5.- Now we can focus on some concrete AP that is having issues. First we need to find the ethernet mac and radio mac for that AP. Check history for that AP and always-on-tracing

Use the show commands to have a summary of events with time reference, different phases of AP association and in which we could observe failures, reason for reboot or disconnect.

We can find out if WLC rebooted the AP due to image upgrade. Or if AP disconnected due to keepalive failure.

Use then always-on-tracing to have more details about what occurred to the AP showing sequence of events. With show command time reference we can focus on the events occurring around that time.

Collecting show tech wireless for concrete AP provides us config details, tag assigment, info about model, radios channels, ...

show wireless stats ap history mac-address Ethernet_MAC@ !!Check type of event and time for the event and disconnect reason and count for specific AP.

show wireless stats ap mac Radio_MAC@ discovery detailed !!Check number of discovery request/responses, discovery failures and type for last working discovery and non working discovery.

show wireless stats ap mac Radio_MAC@ join detailed !!Counters for different phases discovery, dtls, join, config, data dtls. Also shows last reboot type and reason. Disconnect type and reason.

show logging profile wireless start last X days filter mac <radio-or-ethernet-AP-mac> !!Always-on-tracing for this AP shows more detailed events errors stored in the WLC trace database. Config changes, radio events, association/disassociation events.

show tech wireless ap name <ap-name> !! Config details, tag, radio info channels/txpower, SSIDs, ...

6.- If we are observing that multiple APs that are not in same switch are disconnecting around the same time then we can confirm if all disconnecting APs are in the same wncd.

If that is the case then we can check wncd CPU utilization to see if disconnections could be due to high wncd CPU utilization and WLC not been able to process packets received from APs.

show wireless loadbalance ap affinity mac Ethernet_MAC@ !!Check wncd assigned to concrete AP mac address, we can also get wncd for concrete site-tag

show wireless loadbalance ap affinity wncd <0-7> !!Other option is to check all APs assigned to a concrete wncd.

sh proc cpu platform | i wncd !! Check CPU utilization per wncd

Advanced logs from WLC and AP for concrete AP

7.- If with previous information we are not able to identify reason for AP joins then we need to capture ra-traces and packet captures and AP debugs in case we can access the AP for next event.

This provides packet captures from AP and verbose level traces to identify reasons for the AP disconnections. Need to enable traces and captures before the next event to capture the data.

In case AP is accessible through SSH we can enable debugs in the AP that provides point of view of the AP about disconnections. Collecting packet capture in WLC and AP switchport could be

helpful to identify if disconnections are due to some packet drops in the network.

Logs from WLC:

!! Enable ra-trace for AP using default monitor-time is 1800s increse it to max in case you do not know when AP disconnection occurs.

debug wireless mac <AP_Radio_MAC> internal monitor-time 2085978494 !!Using AP radio mac to capture traces with verbose level from WLC. Setting time allows us to enable traces for up to 24 days

!!Or

debug wireless ip <AP_IP> internal monitor-time 2085978494 !!Using AP ip address to capture traces with verbose level from WLC. Setting time allows us to enable traces for up to 24 days

!!Reproduce

no debug wireless mac <AP_Radio_MAC|AP_IP> internal monitor-time 2085978494

!!WLC generates an ra_trace file with AP_info, command to check for ra_trace file generated.

dir bootflash: | i ra_trace

!!Embedded Captures filtered by AP IP address ACL. Filter packet captures for AP ip address in both directions and have a circular buffer to ensure that we get latest captures in case buffer exceeds 100M

!!Create ACL

ip access-list extended CAP-FILTER

permit ip host <AP_IP> any

permit ip any host <AP_IP>

!!Create packet capture

monitor capture MYCAP clear

monitor capture MYCAP interface Po1 both

monitor capture MYCAP buffer circular size 100

monitor capture MYCAP match any

monitor capture MYCAP access-list CAP-FILTER

monitor capture MYCAP start

!!Reproduce

monitor capture MYCAP stop

monitor capture export flash:|tftp:|http:.../filename.pcap

Logs From AP

show tech !! Collect show tech to have all config details and radio stats for the AP.

show dtls connection !! Check certificates, ports and ciphers, versions for DTLS

term mon

!!Basic

debug capwap client events

debug capwap client error

!! Advanced

debug capwap client pmtu

debug capwap client keepalive

debug capwap client payload

debug capwap client details

debug capwap client info

List of all commands

List of all commands from WLC

show ap summary | i Number of APs

sh log | i AP Event:

show ap uptime

show ap cdp nei

show ap crash

dir all | i crash

show wireless stats ap history

show wireless stats ap discovery

show wireless stats ap join summary

show wireless certification config

show wireless management trustpoint

show wireless dtls connections

show wireless stats ap history mac-address Ethernet_MAC@

show wireless stats ap mac Radio_MAC@ discovery detailed

show wireless stats ap mac Radio_MAC@ join detailed

show logging profile wireless start last X days filter mac <radio-or-ethernet-AP-mac>

show tech wireless ap name <ap-name>

show wireless loadbalance ap affinity mac Ethernet_MAC@

show wireless loadbalance ap affinity wncd <0-7>

sh proc cpu platform | i wncd

debug wireless mac <AP_Radio_MAC> internal monitor-time 2085978494

List of all commands from AP

show tech

show dtls connection

term mon

debug capwap client events

debug capwap client error

debug capwap client pmtu

debug capwap client keepalive

debug capwap client payload

debug capwap client details

debug capwap client info