Troubleshoot Catalyst 9800 Wireless Controllers

Nicolas Darchis

Technical Leader, CX

BRKEWN-3013

Troubleshoot

Catalyst 9800 Wireless Controllers

Questions?

Use Cisco Webex Teams to chat

with the speaker after the session

Find this session in the Cisco Events Mobile App

Click “Join the Discussion”

Install Webex Teams or go directly to the team space

Enter messages/questions in the team space

How

1

2

3

4

Cisco Webex Teams

BRKEWN-3013

3

Every day

LABEWN-1505

Cisco 9800 Controllers

- Understanding,

deploying and

troubleshooting

09:00

Opening Keynote

17:00

Guest Keynote

18:30

Cisco Live

Celebration

09:00

BRKEWN-3013

Advanced

Troubleshooting of

Cisco Catalyst 9800

Wireless Controller

11:00

BRKEWN-3011

Advanced

Troubleshooting of

Wireless LANs

16:45

BRKEWN-2480

Plan, design and

troubleshoot your Cisco

DNA driven 9800 WLC

wireless network: Best

Practices and lessons

learnt from the field

14:45

BRKEWN-2809

The Final Fails. 6 for

(WiFi) 6

Troubleshooting

MOB

Mobility Track

FRI

THU

WED

TUE

BRKEWN-3013

9

• C9800 Quick Intro

• C9800 Software Architecture

• Life of a Packet

• New Config Model

• IOS-XE

Tracing and Debugging

• Embedded Packet Capture

• Design considerations and gotchas(HA,LAG,VMs)

• Health Monitoring

• GUI Troubleshooting Dashboard

10

Agenda

BRKEWN-3013

Session objectives

11

• Understand Catalyst 9800 WLC architecture

• Understand basic packet flow through IOS-XE

• Understand new platform debug options

• serviceability enhancements

• spare memorizing – focus on understanding

• Centered around 16.12/17.1

BRKEWN-3013

Introduction

• Debugging process is different…

• Simplified Object model

• “Store and Process”

• Always On

• Trace on Failures

• Improvements in Serviceability

• Traceability

• Large collaboration between TAC/BU/Customers

12

BRKEWN-3013

C9800 Quick Intro

C

2,000+ unique customers

ENCS

7000+ units sold

Catalyst 9800 - Fastest Ramping Wireless Controller

BRKEWN-3013

15

Cat 9800 Platforms

9800-40/80

• 2k/6K APs

• 32/64K

clients

• 40/80Gbps

throughput

• All AP

modes

17

BRKEWN-3013

9800-L (C/F)

• 250/500 APs

• 5K/10K clients

• 5/10Gbps

throughput

• All AP modes

• Performance

license avail.

9800-AP

EWC-AP

• 50/100 APs

• 1K/2K clients

• Flexconnect

only

9800-CL

• Private/Public

Cloud

• 1K/3K/6K APs

• 10K/32k/64k

clients

• Flexconnect

recommended

9800-SW

• 200 APs

• 4K clients

• SD-Access

Only

Cat 9800 Platforms

Catalyst 9800-CL

Catalyst 9800-SW

embedded on switch Catalyst 9800-40 Catalyst 9800-80

Key Features

Ideal for

Multiple scale options with

a single deployment

package

Small campuses or

distributed branches

Midsize to large enterprise

Large enterprise and

service provider

Form factor

Virtual machine software

Installed on Cisco Catalyst

9300 switches 1 RU appliance 2 RU appliance

Deployment Mode

FlexConnect Yes No Yes Yes

Central mode (formerly

local mode) Only private cloud No Yes Yes

Mesh Yes No Yes Yes

FlexConnect + Mesh Yes No Yes Yes

OfficeExtend Yes No Yes Yes

Fabric / SDA Yes Yes Yes Yes

Scalability

Minimum access points 1 1 1 1

Maximum access points

6000 (local switching),

3000 (central switching) 200 2000 6000

Maximum client support

64,000 (local switching);

32,000 (central switching)

4000 32000 64000

18

BRKEWN-3013

C9800-SW – Forwarding Engine Driver

(FED)/Doppler Data Plane

19

• 9800-SW on 9300/9500 switches uses Doppler chipset

• Doppler (UADP) hardware chipset. FED (Forwarding Engine Driver) architecture for

packet forwarding

• Software & control plane stays the same but data plane is completely different

• We will not cover this data plane type. More details in BRKARC-2035

• Controller and switch act as one (same CLI, same IP, same Web UI)

BRKEWN-3013

C9800: Cisco Packet Processor (CPP) Data Plane

• Cloud and appliances : same CPP architecture

• Cloud just virtualizes the data plane

• Large appliances have the QFP (Quantum Flow

Processor) hardware for the CPP.

• No differences apart from the forwarding

performance (and Manufacture Installed Certificate)

• More details in BRKARC-3147 (ASR1k architecture

and troubleshooting)

20

BRKEWN-3013

C9800-AP : Embedded Wireless Controller (on AP)

• AP still run AP COS operating system like Wave2 APs

• AP will place IOS-XE controller code in one partition and

AP code in the backup partition

• Controller code is the same as 9800, but with less

processes, single database, and without dataplane (AP

provides dataplane)

• Flexconnect only

• Any other supported AP (wave2, wifi6…) can join

• Only one AP runs controller code

21

BRKEWN-3013

C9800 Software

Architecture

IOS-XE

• Based on BinOS (linux + Cisco patches)

• IOS is now IOSd

• IOS-XE 16.x/17.x same in EN products

• Previous IOS-XE releases

• Denali 16.1.1, 16.2.1, 16.3.x

• Everest 16.4.1, 16.5.1, 16.6.x

• Fuji 16.7.1, 16.8.1, 16.9.x

• First IOS-XE for 9800 : Gibraltar 16.10.1,16.11.1,16.12.x

• Amsterdam 17.1.1,17.2.1,17.3.x

• B…. : 17.4, …

25

BRKEWN-3013

IOSd

BinOS

IOSd

Blob

IOSd

subsystem

IOSd

subsystem

IOSd

subsystem

HA

Management interface

Module drivers

Kernel

Hosted

Apps

WNCd

Wireshark

WNCd

RRMd

Rogued

Compatibility

• AP Models Supported

• 11ac wave 1 = 1700, 2700, 3700

• 11ac wave 2 = 1800, 2800, 3800, 4800, 1540, 1560, 1570

• 11ax APs = 9115, 9117,

• IOT APs = IW3700 (17.1), IW6300 (17.1)

• Small catch : 9120AXI on 16.12.1. 9120AXE and 9130AXI on 16.12.2. 9130 AXE on 17.1.1

• AP Modes Supported

• Local, FlexConnect, Monitor, Mesh*, Flex+Mesh*, Sensor, Sniffer

*Only on wave 1 and outdoor wave 2 APs on 16.x. On all APs starting 17.1

26

BRKEWN-3013

Compatibility

27

IOS-XE DNACenter Identity

Services

Engine

Prime

Infrastructure

CMX, DNA

Spaces

16.10.1 - 2.2, 2.3, 2.4 3.5 10.5.1

16.10.1e 1.2.10 2.2, 2.3, 2.4 3.5 10.5.1

16.11.1b 1.3 (non

SDA)

2.4, 2.6 3.6 10.5.1>10.6.1

16.11.1c 1.3.0 2.4, 2.6 3.6 10.5.1>10.6.1

16.12.1 1.3.0 2.4, 2.6 3.7 10.5.1>10.6.2

16.12.1s 1.3.1.2 /

1.3.1.3

2.4, 2.6 3.7 10.5.1>10.6.2

16.12.2s 1.3.3 2.4,2.6 3.7 10.5.1>10.6.2

17.1.1s 1.3.3 2.4,2.6 3.7.1 10.5.1>10.6.2

BRKEWN-3013

IRCM Compatibility

28

BRKEWN-3013

• In a nutshell : 9800 can work in mobility with any 8.8.111 or later AireOS or any

version of 9800.

• A special IRCM version of 8.5 is also available with TAC with the same compatibility

Process Architecture

31

BRKEWN-3013

Dataplane (CPP/CPP SW/Doppler)

IPC

WNCd

Wireless processes

(mobilityd, rrmd,

rogued, nmspd)

Wireless

process

Punt Path

Legend

Linux

kernel

LSMPI

LFTS

Linux

TCP/IP

Plumbing Path

IOSd

FMAN RP

FMAN FP

ASIC

Driver

IOS-XE

infra

Db

Cisco

DNA-C

DMI procs

REPM

infra

3

rd

party

Prime

Infra

webUI

DBM

ODM

CLI Agent

CLI

Management

access

DB Access

Programmable Interface

Horizontal Scaling WNCd

32

BRKEWN-3013

Horizontally scaled

WNCmgr

RRM

rogued

NMSPd

mobilityd

WNCd

Wireless Network Control Daemon (WNCd)

33

BRKEWN-3013

Policy

Manager

WNCd : controller process managing AP and client

session

• Capwap : AP discovery

• Dot11 : Client dot11

• SANET/AAA: Client authentication

• EPM : Client policies

• SISF : client IP learning

• Client Orchestrator : Client State Transitions

• LISP-agent : L2 Lisp handling for Fabric deployment

IP

Learn

CAPWAP

AAA

Dot11

LISP

EWC architecture

35

BRKEWN-3013

9800 controller on 9100 series APs

Linux kernel

COS dataplane

Linux sockets

AP process (Capwapd)

Controller (BinOS)

Wireless processes

IOSd punt/inject

EWC triage

Life of a Packet

QFP

ASIC

X86 CPU

complex

Ingress

buffering

Buffering

10G PHY

SFP+ x4

or x8

Astro

Crypto chip

Life of a Packet : Control plane

BRKEWN-3013

37

Punt/Inject/Plumb Path

38

BRKEWN-3013

Dataplane (CPP/ CPP SW/Doppler)

WNCd

Legend

Linux kernel

IOSd

FMAN RP

FMAN FP

CPP-Client/FED

IOS-XE infra

IPC

Punt Path

Plumbing Path

Inject Path

QFP

ASIC

X86 CPU

complex

Ingress

buffering

Buffering

10G PHY

SFP+ x4

or x8

Astro

Crypto chip

Life of a Packet : data plane

wireless client traffic

BRKEWN-3013

39

New Config Model

AireOS vs. Catalyst 9800 Config Model

41

BRKEWN-3013

Granular & simplified

What Policies on which Sites

with what RF characteristics

Going towards a more Modularized and Reusable model with Logical decoupling of configuration entities

Basic

Wireless

Advanced

Wireless

Wireless Security

Switching Policy

Network Policy

WLAN

AP Group

Flex Group

Network Policies

Wireless site

settings

RF Parameters

Site Specific

Policies

RF Profiles

Network Policies

Wireless security

Remote Site

Config

Remote site

parameters

Switching Policies

RF Profile

High Density HDX

Data Rates

DCA, TPC, CHDM

Profile threshold

for traps

Client Distribution

Decouple

Modularize

AireOS Config Model

High Density HDX

Data Rates

DCA, TPC, CHDM

Profile threshold

for traps

Client Distribution

WLAN

Policy

Profile

Flex

Profile

AP Join

Profile

Basic

Wireless

Advanced

Wireless

Wireless Security

Switching Policy

Network Policy

Site

Tag

RF

Tag

Wireless site

settings

Site Specific

Policies

Remote Site

Config

Remote site

parameters

High Density HDX

Data Rates

DCA, TPC, CHDM

Profile threshold

for traps

Client Distribution

RF Profile

Policy

Tag

Site

Tag

RF Tag

Policy

Tag

b/g

a/n/ac

Join Profile

RF Profile

WLAN

Profile

Flex Profile

Site Tag

Policy

Profile

Policy

Tag

RF Tag

AP

Security

11k/v

LB/BS

Client limits

P2P

Profiling

VLAN

QoS

Anchoring

Timers

DHCP

Flex

Local Vlans

Flex

Local Auth

Policy ACL

Led

Stat Timers

TCP MSS

CAPWAP Timers

CAPWAP Features

AP management

Rogue

Band

Rates

RRM

HD

LB

HSR

BRKEWN-3013

42

AireOS to 9800 Configuration Translator

43

BRKEWN-3013

Cloud Tool https://cway.cisco.com/tools/WirelessConfigConverter/

New Config Model Tips

44

BRKEWN-3013

New Config Model Tips

45

BRKEWN-3013

• Configured Tags and Profiles Active Tags and Profiles

• Remember : Tag APs with tags

• How to monitor tag configuration:

# show ap tag summary

Number of APs: 3

AP Name AP Mac Site Tag Name Policy Tag Name RF Tag Name Misconfigured Tag Source

---------------------------------------------------------------------------------------------------------------------------------------------------------------------------

ap3800i-r2-sw1-te0-1 0042.68a0.ee78 default-site-tag floor2-tag default-rf-tag No Static

ap2800-r2-sw1-2-0-4 002c.c899.b9ac default-site-tag default-policy-tag default-rf-tag No Default

# wireless config validate

Or use WCAE tool

New Config Model Tips

46

BRKEWN-3013

• We do not support at the moment (until a couple of releases) the roaming of a client

between 2 APs who have different policy profile attached

• Therefore the “AP group with one subnet per floor” is a risky game with 9800

New Config Model:

Troubleshoot

New config model

9800#show ap name APa80c.0dd2.1fa8 tag detail

AP Name : APa80c.0dd2.1fa8

AP Mac : a80c.0dd2.1fa8

Tag Type Tag Name

-----------------------------

Policy Tag default-policy-tag

RF Tag default-rf-tag

Site Tag default-site-tag

Policy tag mapping

------------------

WLAN Profile Name Policy Name VLAN Central Switching IPv4 ACL IPv6 ACL

----------------------------------------------------------------------------------------------------------------------

dot1x-test default-policy-profile VLAN0711 ENABLED Not Configured Not Configured

Site tag mapping

----------------

Flex Profile : default-flex-profile

AP Profile : default-ap-profile

Local-site : Yes

Verifying applied configuration – WLC CLI

48

BRKEWN-3013

New config model

Verify the mappings on the AP as well !

LAB_3802#show capwap client config

AdminState : ADMIN_ENABLED(1)

Name : Lab_3802

<snip>

AP Policy Tag : ewlc_policy_tag

AP RF Tag : default-rf-tag

AP Site Tag : elwc_site_tag

Verifying applied configuration – AP CLI

49

BRKEWN-3013

New config model

Verifying applied configuration – Web UI

50

BRKEWN-3013

IOS-XE

Tracing and

Debugging

IOS-XE Tracing/Debugging

• IOSd Logging

• Binary Tracing

• Always On Tracing

• Conditional Debugging/Radioactive Tracing

• Non-Conditional Debugging/Per Process Tracing

52

BRKEWN-3013

What is what?

IOSd Logging

• Your Traditional Syslog

Binary Tracing

• Fast infrastructure for real-time

logging

Always On Tracing

• Real time data collection for all

relevant events

53

Conditional Debugging/Radioactive

Tracing

• Per IP/MAC address debugging

Non-Conditional Debugging/Per Process

Tracing

• Your traditional debug

BRKEWN-3013

IOS-XE Logging architecture

54

IOSd logging Vs btrace

L2 Authentication Key Exchange Start. EAP type: PEAP, Resolved VLAN: 185, Audit Session id:ABCD

EAP Key management successful. AKM:FT-DOT1X Cipher:CCMP WPA2

Mobility discovery triggered. Client mode: Local

ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: abcd.abcd.cdef capwap IFID: 0x1234

Client IP learn successful. Method: IP Snooping IP: 10.0.0.1

Client state transition: S_CO_IP_LEARN_IN_PROGRESS -> S_CO_RUN

Oct 9 09:12:15.363 UTC: %CLIENT_ORCH_LOG-6-CLIENT_ADDED_TO_RUN_STATE: Chassis 2 R0/0:

wncd: Username entry (bob) joined with ssid (foo) for device with MAC: 1234.1234.5678

WNCd-0

IOSd

Btrace Library

IOS Logger

WNCd-0 tracelog (wncd_x_R0-0.2280_41.20181009080530.bin)

IOSd traceIog (IOSRP_R0-0.14671_21.20181009041228.bin)

Btrace Library

Client join

messages

Final RUN state

message

Oct 9 09:12:15.363 UTC: %CLIENT_ORCH_LOG-6-CLIENT_ADDED_TO_RUN_STATE: Chassis 2 R0/0:

wncd: Username entry (bob) joined with ssid (foo) for device with MAC: 1234.1234.5678

Syslog, VTY (term mon), console, …

BRKEWN-3013

IOS-XE Tracing

BinOS Trace Levels

55

• ERROR level represent abnormal situations. We want to raise the user attention to these

• WARNING represent an incident that could potentially lead to an error (or not…)

• NOTICE is the default logging level for binos daemons. It captures significant events if they are

normal working conditions. (client connect, failover)

• INFO contains details about state machines and the communication flow

• DEBUG contains traces needed to root cause failure conditions

• VERBOSE :

• INTERNAL is not a level but a flag on any log line when it is not meant to be

understood by mere mortals but only by developers

2-Critical

3-Error

4-Warning

5-Notice

6-Info

7-Debug

8-Verbose

BRKEWN-3013

Introducing Always On tracing

Contextual Logs WITHOUT enabling debugs

56

• Each process writes relevant events at Notice level

• No debug required

• Problem isolation assistance

• Is client facing authentication issues or DHCP issue or something else

• Helps establish trends

• Isolate if reported client connectivity problem is specific to certain APs or certain client mac addresses

• Box can store 48h approx. at max HW capacity, weeks typically

BRKEWN-3013

Always On tracing CLI

• Pre Process (in memory):

# show logging process <process daemon>

• Export to file:

# show logging process <process daemon> to-file <alwayson-processname.txt>

• Display in console:

# more bootflash:alwayson-processname.txt

• Export:

# copy bootflash:alwayson-processname.txt tftp://<serverip>/path OR

ftp://user:pass@serverip/path

57

BRKEWN-3013

Always On tracing – How to view

• Aggregated view across processes:

# show logging profile wireless filter {mac | ip} {client-mac | mobility-peer-ip}

to-file <alwayson-clientmac>.txt

• Focus on time window, export to file

# show logging profile wireless start timestamp “MM/DD/YYYY HH:MM:SS” filter mac

Default time in 16.12: since last boot

Default time in 17.1 : last 10 minutes

• Focus last 5 minutes:

# show logging profile wireless start last 5 minutes

58

BRKEWN-3013

Always On: successful client connection

59

BRKEWN-3013

# show log profile wireless filter mac 0040.96b9.b5c4 to-file output.txt

[client-orch-sm] [21109]: (note): MAC: f0c1.f10b.8ac1 Association received. BSSID 7069.5a51.4ec0, old BSSID 0000.0000.0000, WLAN RomanTest, Slot

0 AP 7069.5a51.4ec0, AP4C77.6D9E.6162

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_INIT -> S_CO_ASSOCIATING

[dot11] [21109]: (note): MAC: f0c1.f10b.8ac1 Association success. AID 1, Roaming = False, WGB = False, 11r = False, 11w = False AID list: 0x1|

0x0| 0x0| 0x0

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_ASSOCIATING -> S_CO_L2_AUTH_IN_PROGRESS

[client-auth] [21109]: (note): MAC: f0c1.f10b.8ac1 ADD MOBILE sent. Client state flags: 0x71 BSSID: MAC: 7069.5a51.4ec0 capwap IFID:

0x90000004

[client-auth] [21109]: (note): MAC: f0c1.f10b.8ac1 L2 Authentication initiated. method DOT1X, Policy VLAN 1477,AAA override = 0 , NAC = 0

[ewlc-infra-evq] [21109]: (note): Authentication Success. Resolved Policy bitmap:11 for client f0c1.f10b.8ac1

[client-auth] [21109]: (note): MAC: f0c1.f10b.8ac1 L2 Authentication Key Exchange Start. Resolved VLAN: 1477, Audit Session id:

1E27300A0000000E127592C3

[client-keymgmt] [21109]: (note): MAC: f0c1.f10b.8ac1 EAP Key management successful. AKM:DOT1X Cipher:CCMP WPA2

[client-orch-sm] [21109]: (note): MAC: f0c1.f10b.8ac1 Mobility discovery triggered. Client mode: Local

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_L2_AUTH_IN_PROGRESS ->

S_CO_MOBILITY_DISCOVERY_IN_PROGRESS

[mm-client] [21109]: (note): MAC: f0c1.f10b.8ac1 Mobility Successful. Roam Type None, Sub Roam Type MM_SUB_ROAM_TYPE_NONE, Previous BSSID MAC:

0000.0000.0000 Client IFID: 0xa0000001, Client Role: Local PoA: 0x90000004 PoP: 0x0

[client-auth] [21109]: (note): MAC: f0c1.f10b.8ac1 ADD MOBILE sent. Client state flags: 0x72 BSSID: MAC: 7069.5a51.4ec0 capwap IFID:

0x90000004

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_MOBILITY_DISCOVERY_IN_PROGRESS ->

S_CO_DPATH_PLUMB_IN_PROGRESS

[dot11] [21109]: (note): MAC: f0c1.f10b.8ac1 Client datapath entry params - ssid:RomanTest,slot_id:0 bssid ifid: 0x0, radio_ifid: 0x90000003,

wlan_ifid: 0xf0400002

[dpath_svc] [21109]: (note): MAC: f0c1.f10b.8ac1 Client datapath entry created for ifid 0xa0000001

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_DPATH_PLUMB_IN_PROGRESS -> S_CO_IP_LEARN_IN_PROGRESS

[client-iplearn] [21109]: (note): MAC: f0c1.f10b.8ac1 Client IP learn successful. Method: DHCP IP: 192.168.77.200

[client-orch-state] [21109]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_IP_LEARN_IN_PROGRESS -> S_CO_RUN

Always On: client connection failure

60

BRKEWN-3013

#sh logging profile wireless filter mac f0c1.f10b.8ac to-file dot1x-failure.txt

2019/10/29 09:35:34.048 {wncd_x_R0-0}{1}: [client-orch-sm] [19470]: (note): MAC: f0c1.f10b.8ac1 Association received. BSSID 7069.5a51.4ec0, old BSSID

7069.5a51.4ec0, WLAN RomanTest, Slot 0 AP 7069.5a51.4ec0, AP4C77.6D9E.6162

2019/10/29 09:35:34.048 {wncd_x_R0-0}{1}: [client-orch-state] [19470]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_L2_AUTH_IN_PROGRESS

-> S_CO_L2_AUTH_IN_PROGRESS

2019/10/29 09:35:34.048 {wncd_x_R0-0}{1}: [dot11] [19470]: (note): MAC: f0c1.f10b.8ac1 Association success. AID 1, Roaming = False, WGB = False, 11r

= False, 11w = False AID list: 0x1| 0x0| 0x0| 0x0

2019/10/29 09:35:34.048 {wncd_x_R0-0}{1}: [client-orch-state] [19470]: (note): MAC: f0c1.f10b.8ac1 Client state transition: S_CO_L2_AUTH_IN_PROGRESS

-> S_CO_L2_AUTH_IN_PROGRESS

2019/10/29 09:35:34.048 {wncd_x_R0-0}{1}: [client-auth] [19470]: (note): MAC: f0c1.f10b.8ac1 ADD MOBILE sent. Client state flags: 0x71 BSSID: MAC:

7069.5a51.4ec0 capwap IFID: 0x90000004

2019/10/29 09:35:34.051 {wncd_x_R0-0}{1}: [client-auth] [19470]: (note): MAC: f0c1.f10b.8ac1 L2 Authentication initiated. method DOT1X, Policy VLAN

1477,AAA override = 0 , NAC = 0

2019/10/29 09:35:34.330 {wncd_x_R0-0}{1}: [errmsg] [19470]: (note): %DOT1X-5-FAIL: Authentication failed for client (f0c1.f10b.8ac1) with reason (Cred

Fail) on Interface capwap_90000004 AuditSessionID 000000000000000B16D9A13D Username: drghgdf

Always On : AP join failures

61

BRKEWN-3013

# show log profile wir filter mac <ap radio mac> to-file output.txt

• Unsupported AP

[apmgr-capwap-join] [1263]: UUID: 0, ra: 0, TID: 0 (ERR): d824.bde8.3690 Join request not accepted: Unsupported AP

Model AIR-LAP1142N-A-K9

• Reg Domain failure

[apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to verify reg domain slot.

validation of country code(UX) to regulatory domain(-A) error:1

[apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to get ap default country code. Get

default country code for AP error.

[apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR): f44e.0597.fb50 Failed to set reg domain check status.

country code US is not configured on WLC

• Cert Failure

[apmgr-capwap-config] [1394]: UUID: 10000000002ed, (ERR), %PKI-3-CERTIFICATE_INVALID_NOT_YET_VALID: Certificate

chain validation has failed. The certificate (SN: 6B4F09560000001763DF) is not yet valid Validity period starts

on 22:48:43 IST Sep 9 2014

• Discovery to non wireless mgmt interface

{wncmgrd_R0-0}{2}: [capwapac-srvr] [16320]: UUID: 0, ra::0, TID: 0 (ERR): IP:3.3.3.1[5246], Discovery to non

wireless mgmt interface

Trace-on-Failure (TOF)

(Not fully supported until 17.3)

62

BRKEWN-3013

# show wireless stats trace-on-failure

001. AP radio reset......................................: 0

002. AP reset............................................: 0

003. Client disjoin due to AP radio reset................: 0

004. Client disjoin due to AP reset......................: 0

005. Export client MMIF..................................: 0

006. Export client MM....................................: 0

007. Export client generic...............................: 0

011. AP join failure.....................................: 0

012. AP initial configuration failure....................:

44335

• 55 Predefined failure codes tracked

• Available stats

Radioactive tracing

Building on existing conditional debugging CLI

65

BRKEWN-3013

• Collect additional data about a particular IP or mac

• Roughly similar to the old “deb client mac”

• Formally: it is “store and display” process

• Filter needs to match available info

• A lot more detailed

• Always On: 18 lines

• Radioactive: 180 lines

• Radioactive + Internal: 1800 lines

Radioactive tracing

66

BRKEWN-3013

# debug platform condition feature wireless mac <client mac>

# debug platform condition start

(reproduce issue)

# debug platform condition stop

# show logging profile wireless [(start timestamp “Date&time”) level

debug filter mac <client mac> to-file <filename>]

# more flash:<filename>

Radioactive Tracing: Easy way…

• Macro to collect and export in one go:

# debug wireless mac <mac-of-client> ftp-server ser.ver.ip.add /directory

• Runs for 30 min, or set a timer

• Stop with

# no debug wireless mac <mac-of-client>

• Destination can be FTP or File (flash)

• File is more reliable

• FTP needs write access, previous config

67

BRKEWN-3013

Radioactive Tracing: Even Easier…

68

BRKEWN-3013

Radioactive Tracing Filtering and Cleanup

69

• AP debugging by mac works for all radio/rrm/etc processes. DTLS will not work

• AP debugging by its IP address works for DTLS, but misses all later processes

Remember: set filter for the desired context

• Always remove conditions

# clear platform condition all

# undebug all

BRKEWN-3013

Process Daemon Specific Debugging

• Single process focused troubleshooting

• Examples: RRM, nginx web server

• To view current log level set for a process trace

# show platform software trace level <rrm-mgrd> chassis active R0

# show platform software trace level <wireless> chassis active R0

# show platform software trace level <wncd> chassis active R0

# show platform software trace level <mobility> chassis active R0

…

Unconditional Debugging

70

BRKEWN-3013

Process Daemon Specific Debugging

• Enable:

# set platform software trace <rrm-mgrd | nginx | nmspd> chassis active R0 all debug

(reproduce issue)

• Collect traces:

# show logging process <rrm-mgrd | nginx | nmspd> to-file <debugtrace-rrmd.txt>

• View:

# more bootflash:debugtrace-rrmd.txt

• Export:

# copy bootflash:debugtrace-rrmd.txt { tftp:, ftp:, http:, https:, scp: }

• Disable:

# undebug all OR # set platform software trace <> chassis active R0 all notice

71

BRKEWN-3013

Process Daemon Specific Debugging for mDNS

• Enable:

# set platform software trace wncd 0 chassis active r0 mdns verbose

# sh platform software trace level wncd 0 chassis active R0 | in Verbose

(reproduce issue)

• Collect traces:

# sh platform software trace message wncd 0 chassis active R0

• Disable:

# undebug all OR # set platform software trace <> chassis active R0 all notice

72

BRKEWN-3013

Process Daemon Specific Debugging for mDNS

• Successful service learning:

2019/11/08 07:37:06.976 {wncd_x_R0-0}{1}: [mdns] [28837]: (verbose): Received READ Callback for IPV4 mDNS

packet

…

2019/11/08 07:37:06.975 {wncd_x_R0-0}{1}: [mdns] [28837]: (debug): MDNS_ADVT:[MAC:88e9.fe7a.04c8]TXT record

added/updated sucessfully : Nxxxxx-M-X2HX._airplay._tcp.local

…

2019/11/08 07:37:06.975 {wncd_x_R0-0}{1}: [mdns] [28837]: (debug): MDNS_ADVT:[MAC:88e9.fe7a.04c8]TXT record

added/updated sucessfully : Nxxxxx-M-X2HX._airserver._tcp.local

• Failed mDNS processing:

2019/11/08 07:37:36:50.711 {wncd_x_R0-0}{1}: [mdns] [26786]: (debug): Received READ Callback for IPV4 mDNS

packet

2019/11/08 07:37:36:50.711 {wncd_x_R0-0}{1}: [mdns] [26786]: (verbose): In ret_buffer pak: 0x55bd04ee9ff8

bpak->buffer_start 0x55bd04ee1098 bpak->subblock 0x0

2019/11/08 07:37:36:50.711 {wncd_x_R0-0}{1}: [mdns] [26786]: (verbose): MDNS record Search: record with

wlan_id: 2 found

2019/11/08 07:37:36:50.711 {wncd_x_R0-0}{1}: [mdns] [26786]: (verbose): Dropping mDNS packet, SVI interface

(VLAN : 1477) not present/UP

73

BRKEWN-3013

In summary

Basic client/AP data collection:

• Data is there, just pull it…

• Collect data with “show logging profile wireless filter {mac | ip}”..

Advanced client/AP:

• Use Radioactive Tracing

• Collect data with “debug wireless mac <mac-of-client> ftp-server ser.ver.ip.add

/directory”

Basic Box logs

• Traditional show logs/syslog

75

BRKEWN-3013

Packet Captures

and Tracing

Embedded Packet Capture

• Get packets sent from or to and through the controller

• Export to Wireshark

• No need for switch capture

• Accessible either from GUI or CLI

77

BRKEWN-3013

Embedded Packet Capture (EPC) web interface

• Web interface to the existing

EPC CLI “monitor capture …”

• One click start/stop/download

• Physical and VLAN interfaces

can be selected

78

BRKEWN-3013

Embedded Packet Capture web interface

One click start/stop/download

79

BRKEWN-3013

• monitor capture test interface GigabitEthernet2 both

• monitor capture test control-plane both

• monitor capture test match any

• monitor capture test buffer size 100 circular

• monitor capture test limit pps 1000000

• monitor capture test start

• monitor capture test stop

• monitor capture test export bootflash:test.pcap

Embedded Packet Capture CLI

80

BRKEWN-3013

• Although EPC can be collected easily from Web UI and most of the time options provided there

are sufficient for certain troubleshooting tasks, CLI provides more granular settings for EPC

configuration. It can be configured to match inner identity (currently mac-address only) which

allows to focus on traffic related to specific client event when CAPWAP encapsulated.

• monitor capture client_inner_mac inner mac f0c1.f10b.8ac1 interface vlan39 both control-plane

both

• monitor capture client_inner_mac match any

• monitor capture client_inner_mac start

• monitor capture client_inner_mac stop

• monitor capture client_inner_mac export bootflash:inner-mac.pcap

Embedded Packet Capture CLI

81

BRKEWN-3013

Collected captures can be either uploaded to some file server in the network or

downloaded from WLC web interface directly.

Embedded Packet Capture CLI

82

BRKEWN-3013

Packet Tracing

• Data plane “view” of specified traffic

• Collect X packets, and dump information

• Verify which features are processing each frame

• It is not a packet capture -> EPC

• Mostly IP related traffic (no wireless info)

83

BRKEWN-3013

Packet tracing

84

BRKEWN-3013

Capwap data packet PPE Processing: Wireless -> Wired

L2 ( src mac

lookup and

learn)

CAPWAP

Encapsulated

Capwap

tunnel

lookup (5

tuple hash)

SVI (L3 IP

lookup to

terminal

tunnel)

Wireless

Client

lookup

Feat

ures

ACL, QoS …

Re-Write header

(strip outer l2, ip,

capwap, 801.11 ->

802.3)

Dst Mac

lookup

Flood

Unicast L2 switch

OR

802.3 pkt

DTLS, Re-

assembly

optional

Pkt Entry Point

B1.1

B1.2

B1.3

B1.4

Sub

Block

B1

Packet tracing

FIA Tracing: Wireless -> Wired

85

Feature: SWPORT_PM_INPUT_CHECK

Entry : Input - 0x7001247c

Input : Port-channel1

Output : <unknown>

Lapsed time : 629 ns

Feature: IPV4_INPUT_ARL

Entry : Input - 0x70012964

Input : Port-channel1

Output : <unknown>

Lapsed time : 394 ns

Feature: SWPORT_PM_FWD_VLAN

Entry : Input - 0x700124a0

Input : VLAN-CPPIF-0010

Output : <unknown>

Lapsed time : 960 ns

Feature: SWPORT_VLAN_LEARN

Entry : Input - 0x700124e4

Input : VLAN-CPPIF-0010

Output : <unknown>

Lapsed time : 2304 ns

Feature: SWPORT_VLAN_BRIDGING

Entry : Input - 0x700123b8

Input : Vlan10

Output : <unknown>

Lapsed time : 2624 ns

L2 ( src mac lookup and learn)

SVI (L3 IP lookup to

terminal tunnel)

Feature: IPV4_INPUT_DST_LOOKUP_ISSUE

Entry : Input - 0x70012220

Input : Vlan10

Output : <unknown>

Lapsed time : 149 ns

Feature: IPV4_INPUT_ARL_SANITY

Entry : Input - 0x70012610

Input : Vlan10

Output : <unknown>

Lapsed time : 608 ns

Feature: IPV4_INPUT_DST_LOOKUP_CONSUME

Entry : Input - 0x70012280

Input : Vlan10

Output : <unknown>

Lapsed time : 192 ns

Feature: IPV4_INPUT_FOR_US_MARTIAN

Entry : Input - 0x70012284

Input : Vlan10

Output : <unknown>

Lapsed time : 4917 ns

Feature: IPV4_INPUT_LOOKUP_PROCESS

Entry : Input - 0x70012228

Input : Vlan10

Output : <unknown>

Lapsed time : 2016 ns

Feature: IPV4_INPUT_IPOPTIONS_PROCESS

Entry : Input - 0x700127a8

Input : Vlan10

Output : <unknown>

Lapsed time : 85 ns

Feature: IPV4_INPUT_GOTO_OUTPUT_FEATURE

Entry : Input - 0x700127c8

Input : CAPWAP-IF-0x0091000004

Output : <unknown>

Lapsed time : 938 ns

To CAPWAP features (next slide)

BRKEWN-3013

Feature: SWPORT_VLAN_BRIDGING

Entry : Input - 0x700123b8

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 3189 ns

Feature: MPASS_SET_IN_VECTOR

Entry : Input - 0x700129fc

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 53 ns

Feature:

L2_MC_INPUT_REPLICATION_MODULE

Entry : Input - 0x700be7f8

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 20864 ns

Feature: LAYER2_IPV4_INPUT_ARL_SANITY

Entry : Input - 0x70012968

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 458 ns

Feature: WLCLIENT_INGRESS_IPV4_FWD

Entry : Input - 0x7001299c

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 490 ns

Feature: CAPWAP_INPUT_REASS_FEATURE

Entry : Input - 0x70012988

Input : CAPWAP-IF-0x0091000004

Output : <unknown>

Lapsed time : 502016 ns

Feature:

CAPWAP_TUNNEL_INGRESS_FEATURE

Entry : Input - 0x7001298c

Input : WLCLIENT-IF-0x00a0000002

Output : <unknown>

Lapsed time : 2965 ns

Packet tracing

FIA Tracing: Wireless -> Wired (contd..)

86

Capwap

tunnel lkp

(5 tuple

hash)

Wireless Client

lookup (src mac &

vlan lookup)

L2/SWPORT Dst

Mac lookup

From SVI

BRKEWN-3013

Packet Tracing

• Set condition

# debug platform condition mac 001e.e5e2.35cf both

• Set number of packets to collect

# debug platform packet-trace packet 128 fia-trace

• Check stats

# show platform packet-trace statistics

Packets Summary

Matched 384

Traced 129

Packets Received

Ingress 264

87

BRKEWN-3013

Packet Tracing

#show platform packet-trace packet 47

Feature: IPV4_INPUT_GOTO_OUTPUT_FEATUREEntry : Input - 0x8173e358

Input : Vlan1104

Output : <unknown>

Lapsed time : 4000 ns

Feature: CAPWAP_DTLS_CTRL_DECRYPT_PRE_EXT

Entry : Input - 0x8178ff90

Input : Vlan1104

Output : <unknown>

Lapsed time : 933 ns

Feature: CAPWAP_CTRL_PUNT_EXT

Entry : Output - 0x8178f660

Input : Vlan1104

Output : internal0/0/rp:0

Lapsed time : 4913 ns

Obtain Packet information

88

BRKEWN-3013

Design

considerations

VM and 9800-CL deployments

90

BRKEWN-3013

So you want to deploy c9800-CL ?

• Consider the throughput limitation for local mode APs or centrally switched SSIDs

• If deploying on AWS, don’t hope for too much RCA when going unreachable

• 9800-CL shows up with “GigabitEthernet” interfaces but you can set speed to

10000 if the VM NIC supports it

Things to keep in mind

91

BRKEWN-3013

So you want to deploy c9800-CL ?

• “Forged transmits” and “Promiscuous mode” have to be enabled on Vmware

• Consequence : all VMs in the same port group and

handling the same VLANs will receive each other’s

traffic. Try to assign WLC VMs to different physical

port or different VLANs

• VMware ESXi 6.7 and later should have the

capability to learn MAC addresses.

• HyperV (IOS-XE 17.1) does not require promiscuous mode

Things to keep in mind

92

BRKEWN-3013

Client A Client B Client C

WLC VM - MAC D

vSwitch

Forged transmit disabled means

clients won’t be allowed to send

So you want to deploy c9800-CL ?

• Example of 9800CL on high CPU due to promiscuous mode

Things to keep in mind

93

BRKEWN-3013

9800-17#show proc cpu platform sorted

CPU utilization for five seconds: 15%, one minute: 15%, five minutes: 16%

Core 0: CPU utilization for five seconds: 3%, one minute: 3%, five minutes: 3%

Core 1: CPU utilization for five seconds: 4%, one minute: 3%, five minutes: 3%

Core 2: CPU utilization for five seconds: 25%, one minute: 18%, five minutes: 19%

Core 3: CPU utilization for five seconds: 29%, one minute: 39%, five minutes: 38%

Pid PPid 5Sec 1Min 5Min Status Size Name

--------------------------------------------------------------------------------

27973 27436 61% 61% 63% S 222236 ucode_pkt_PPE0

1030 15026 3% 3% 2% R 1069784 linux_iosd-imag

321 2 3% 3% 3% S 0 ksmd

30585 30281 0% 0% 0% S 166852 cli_agent

30429 1 0% 0% 0% S 2712 rotee

30345 29795 0% 0% 0% S 241800 dbm

30281 14672 0% 0% 0% S 4040 pman.sh

30029 1 0% 0% 0% S 2640 rotee

So you want to deploy c9800-CL ?

• Output of installer when it redirects to serial port and you are monitoring KVM

Things to keep in mind

94

BRKEWN-3013

Certificates on VM and

appliances

95

BRKEWN-3013

C9800 WLC certificates

Tips about the self-signed certificate on the C9800-CL

96

• It has a self-signed certificate automatically generated that you will have to renew if

you change hostname or management interface

• That password will most likely never be used again

• While appliances have a Manufacture-Installed Certificate, virtual controllers can only

rely on Self-Signed Certificate.

• It is generated by the day-0 wizard once you enabled the wireless management

interface and configured the country.

• In case of issues, it can be re-generated with this CLI : #wireless config vwlc-ssc

key-size 2048 signature-algo sha256 password <yourpassword>

• As of 16.10, all SSC had the same Serial Number and used static CA information which was causing browser issues

when you have multiple C9800-CL. You can install another certificate for UI management to work around the problem.

• Issue is no longer applicable to 17.x code since now CA is same as hostname

BRKEWN-3013

C9800 WLC certificates

Tips about the MIC certificates on appliances

97

• Appliance have a MIC (SUDI) certificate. It is automatically used when no trustpoint

are configured

• Therefore :

• Do not use the vwlc-ssc command to generate a self-signed certificate on

an appliance

• Do not assign any trustpoint to the wireless management

BRKEWN-3013

High Availability

98

BRKEWN-3013

High Availability

• Platform details must match

• Same HW model

• For 9800-CL: Number of cores, memory, storage size

• Image Version

• Installation Mode (bundle vs install)

You have HA failovers regularly at the same time ? Are you taking VM snapshots ?

Is your HA pair crashing at regular times ? Are you really not taking VM snapshots ?

Tricks

99

BRKEWN-3013

High Availability

• Pairing between boxes in Install Mode and Bundle mode returns Version Mismatch

%BOOT-3-BOOTTIME_INCOMPATIBLE_SW_DETECTED: R0/0: issu_stack: Incompatible software

detected. Details: Active's super boot mode does not match with member's

subpackage boot mode. Please boot switch 1 in super mode.

C9800-2#sh chassis

Chassis/Stack Mac Address : 00a3.8e23.a0e0 - Local Mac Address

Mac persistency wait time: Indefinite

Local Redundancy Port Type: Twisted Pair

H/W Current

Chassis# Role Mac Address Priority Version State IP

-------------------------------------------------------------------------------------

1 Member 00a3.8e23.a320 1 V02 V-Mismatch 192.168.1.171

*2 Active 00a3.8e23.a0e0 1 V02 Ready 192.168.1.172

Pair boxes in same Installation Mode

100

BRKEWN-3013

High Availability

• Until 16.12 included, there is no gateway reachability check

Tricks

101

BRKEWN-3013

RP

High Availability

• If HA sync fails/Split Brain. How to recover:

• Re-ip the boxes to avoid duplicate

• Regenerate certificates and keys post HA breakup

• Bounce the http service to get GUI Access

Tricks

102

BRKEWN-3013

High Availability

• Starting from 17.x gateway connectivity check through RMI was implemented. Both

active and standby WLC are going to monitor GW connectivity with ICMP and ARP.

“ip default-gateway <ip address>” - needs to be configured in 17.1; requirements will be removed

in later versions.

After 17.1

103

BRKEWN-3013

High Availability

• In case GW connectivity loss is detected on standby unit, it will be moved

to “recovery-mode”.

Log events on Active:

%RIF_MGR_FSM-6-GW_UNREACHABLE_STANDBY: Chassis 2 R0/0: rif_mgr: Gateway not reachable from Standby

%RIF_MGR_FSM-6-RMI_STBY_TO_STDBY_REC: Chassis 2 R0/0: rif_mgr: Going from Standby to Standby(Recovery)

state

Log events on Standby:

Ewlc: triggered dual-active recovery,setting hostname to Katar(recovery-mode)

% Hostname "KATAR(RECOVERY-M " is not a legal LAT node name, Using "CISCO_3C56AC"

Note: access to standby console is disabled by default, can be enabled with:

redundancy

mode sso

main-cpu

standby console enable

After 17.1

104

BRKEWN-3013

High Availability

• In case GW connectivity loss is detected on active, reload will be triggered

on active unit to initiate RP communication failure and to trigger SSO

failover.

Platform SSO debug commands:

debug redundancy …

IOS XE SSO component debugs:

debug ewlc-gateway {error|info|packet}

debug ewlc-hainfra {error|info|packet}

After 17.1

105

BRKEWN-3013

High Availability

show platform software stack-mgr chassis active R0 sdp-counters

show platform software stack-mgr chassis active R0 peer-timeout

show platform software stack-mgr chassis standby R0 sdp-counters

show platform software stack-mgr chassis standby R0 peer-timeout

show chassis

show chassis ha-status local

show chassis ha-status active

show chassis ha-status standby

show redundancy

show redundancy history

show redundancy switchover history

show redundancy states

show logging process stack_mgr internal to-file bootflash:<FILENAME.txt>

Troubleshooting

106

BRKEWN-3013

Low level

High level

Interfaces and IP learn

107

BRKEWN-3013

Interfaces and IP learn

• No particular need for L3 SVI for client subnets in

case you have DHCP server (or IP helper) in the same

VLAN as clients (bridging mode)

• VRFs are not supported !

• Using the WLC as a router/gateway is not supported

• You may also use L3 SVI interfaces to have relay functionality with “ip helper-

address …”

• DHCP proxy (in the wlan profile) has the same effect as ip helper.

• No DHCP snooping

108

BRKEWN-3013

Interfaces and IP learn

• You may face issues if your next hop has “ip verify unicast source reachable-via rx”

enabled. Don’t use IP helpers on 9800 if that is the case (CSCvr86358)

• IP learning done by the wireless device-tracking code (SISF, not IPDT)

• 9800 does not do proxy ARP like AireOS did

• 9800 transforms ARP (meant for wireless clients) broadcasts to unicast. Still no

problem to have large wireless subnets

• Possible battery life reduced if network devices ARP-probe your wireless clients

regularly.

• Proxy ARP coming in 17.3

• Roaming between APs having policies assigning different subnet is going to

disconnect the client. Change coming in 17.3

109

BRKEWN-3013

Interfaces and IP learn : troubleshoot

• # show wireless device-tracking database ip <ip address under theft>

• # show wireless device-tracking database mac <whatever mac was thieving the ip>

• IP theft exclusion policy can be enabled or disabled, but several clients having the

same IP address is not tolerated

• Radioactive tracing with a packet capture will best troubleshoot a client stuck in IP

learn

110

BRKEWN-3013

Flexconnect local switching / local assoc

• On AireOS, “Flex central association” was always a corner case feature

• In 9800, moving the knob to “central switching disabled” leaves central association

enabled by default. This brings a lot of possible issues and added delay

111

BRKEWN-3013

CPU ACLs ?

Since we are in the interface topic, how does one restrict traffic to the WLC itself ?

112

BRKEWN-3013

Mobility

BRKEWN-3013

113

Mobility

• 9800 supports Secure DTLS Mobility

• Mobility Ports: UDP 16666, 16667

• For mixed deployment (AireOS and 9800), secure mobility needs to be enabled on

AireOS side explicitly

• For C9800-CL, SSC hash key needs to be provided to AireOS WLC

• Data DTLS encryption, needs to be enabled or disabled on both ends

• 9800 still has a max of 24 WLCs per mobility group, 72 in total (possibly more in

future)

Basics

114

BRKEWN-3013

Mobility

• Seamless roaming requires

• WLAN Profile Name and SSID need to match

• WLAN security settings

• DHCP Required

• Peer to Peer Blocking

• 802.11i

• Various L2/L3 security schemes

{wncd_x_R0-0}{1}: [client-orch-sm] [30764]: (ERR): Security Policy Mismatch,

Local: [ ], Remote: [ DHCP ]

{wncd_x_R0-0}{1}: [client-orch-sm] [30764]: (ERR): MAC: aaaa.bbbb.cccc Handoff

Deny: Security Policy Mismatch

Client Roaming

115

BRKEWN-3013

Mobility

• Show tech wireless mobility

• Radio Active tracing using a WLC IP address

• Set platform software trace mobility (...) all-modules debug

Troubleshooting

116

BRKEWN-3013

Health Monitoring

Useful Commands

HW monitoring

HW sensors and status

118

# show environment all

Sensor List: Environmental Monitoring

Sensor Location State Reading

Vin P0 Normal 119 V AC

Iin P0 Normal 2 A

Vout P0 Normal 12 V DC

Iout P0 Normal 20 A

Temp1 P0 Normal 33 Celsius

Temp2 P0 Normal 29 Celsius

Temp3 P0 Normal 37 Celsius

VRRX1: VX1 R0 Normal 751 mV

VRRX1: VX2 R0 Normal 6909 mV

VRRX1: VX3 R0 Normal 1216 mV

BRKEWN-3013

Virtual “HW” monitoring

Box specifications and environment

119

#sh platform software system all

Processor Details

=================

Number of Processors : 4

Processor : 1 - 4

vendor_id : GenuineIntel

cpu MHz : 2266.747

cache size : 8192 KB

Crypto Supported : No

model name : Int

Hypervisor Details

===================

Hypervisor: VMWARE

Manufacturer: VMware, Inc.

Product Name: VMware Virtual Platform

BRKEWN-3013

AP Health

Verifying AP discovery

120

# show wireless stats ap discovery

Discovery requests received from total number of APs : 3

AP Radio MAC AP Ethernet MAC IP Address Last Success time Last failure type Last failure time

------------------------------------------------------------------------------------------------------------------------------------------------

0062.ecaa.de80 0042.68a0.ee78 192.168.26.101 05/28/19 10:00:02 None NA

00a3.8ec2.da00 002c.c899.b9ac 192.168.25.102 05/28/19 10:00:02 None NA

cc16.7e30.3980 58ac.78de.891e 192.168.26.102 05/28/19 10:00:09 Non-wireless Mgmt interface NA

• Single view for all Aps that tried to find the controller

BRKEWN-3013

AP Health

AP reliability

121

# show ap uptime

Number of APs: 3

AP Name Ethernet MAC Radio MAC AP Up Time Association Up Time

---------------------------------------------------------------------------------------------------------------------------

ap3800i-r2-sw1-te0-1 0042.68a0.ee78 0062.ecaa.de80 1 day 0 hour 37 minutes 1 day 0 hour 21

ap2800-r2-sw1-2-0-4 002c.c899.b9ac 00a3.8ec2.da00 1 day 0 hour 38 minutes 1 day 0 hour 21

ap3800i-r2-sw1-te0-2 58ac.78de.891e cc16.7e30.3980 1 day 0 hour 36 minutes 1 day 0 hour 21

• Single view:

• AP crashes

• CAPWAP bounces

BRKEWN-3013

AP Health

Verifying AP join

122

# show wireless stats ap join summary

Number of APs: 2

Base MAC Ethernet MAC AP Name IP Address Status Last Failure Type Last Disconnect Reason

-----------------------------------------------------------------------------------------------------------------------------------------------------

0062.ec06.8d10 0000.0000.0000 NA NA Not Joined Dtls NA

00be.75ba.1220 0000.0000.0000 NA NA Not Joined Dtls NA

7c0e.cea0.7680 58f3.9cc4.4864 AP58f3.9cc4.4864 192.168.16.92 Not Joined NA Heart beat timer expiry

84b8.021d.1c70 64f6.9d58.5d3c 2702I-sniffer 192.168.16.198 Joined Join Wtp reset config cmd sent

a80c.0ddb.c720 a80c.0dd2.1fa8 APa80c.0dd2.1fa8 192.168.18.52 Joined NA DTLS alert from AP

• Single view:

• AP Join failures

• Reason codes

• AP mac/IP for debugging

BRKEWN-3013

AP Health

Verifying DTLS

123

# show wireless dtls connections

AP Name Local Port Peer IP Peer Port Version Ciphersuite

-----------------------------------------------------------------------------------------

APD4E8.8019.49E0 Capwap_Ctrl 170.85.125.43 5250 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

EDU_BR_01_00_01_1852 Capwap_Ctrl 170.85.142.18 5264 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

EDU_BR_01_00_02_3702 Capwap_Ctrl 170.85.125.14 56998 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

EDU_BR_01_00_03_1832 Capwap_Ctrl 170.85.145.85 5264 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

EDU_BR_01_00_10_1832 Capwap_Ctrl 170.85.151.11 5272 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

EDU_BR_01_00_13_3702 Capwap_Ctrl 170.85.125.20 62903 DTLSv1.0 TLS_NUM_RSA_WITH_AES_128_CBC_SHA

• Single view:

• Connections per AP

• Ciphers in use

• Source ports for NAT/PAT problems

• Mobility will show here

BRKEWN-3013

AP Health

What happened

124

# show wireless stats ap history

AP Name Ethernet MAC Event Time Recent Disconnect Time Disconnect Reason

Disconnect Count

----------------------------------------------------------------------------------------------------------------------------- -----------

-----------

ap2800-r2-sw1-2-0-4 002c.c899.b9ac Joined 05/29/19 10:49:35 NA

ap2800-r2-sw1-2-0-4 002c.c899.b9ac Disjoined 05/29/19 10:48:18 NA Heart beat timer expiry

ap2800-r2-sw1-2-0-4 002c.c899.b9ac Joined 05/28/19 10:00:12 NA

ap3800i-r2-sw1-te0-1 0042.68a0.ee78 Joined 05/28/19 10:00:13 NA

ap3800i-r2-sw1-te0-2 58ac.78de.891e Joined 05/28/19 10:00:19 NA

• Single view:

• Recent events per AP

• What happened and when

• No debug or data collection needed

BRKEWN-3013

Verifying AP Plumbed path

# show ap summary

Number of APs: 1

AP Name Slots AP Model Ethernet MAC Radio MAC Location Country IP Address State

--------------------------------------------------------------------------------------------------------------------------------------------------------

AP4C77.6D9E.6162 3 4800 4c77.6d9e.6162 7069.5a51.4ec0 default location BE 192.168.79.249 Registered

# show platform software capwap chassis active R0

sh platform software capwap chassis active R0

Tunnel ID AP MAC Type IP Port

------------------------------------------------------------------

0x90000004 7069.5a51.4ec0 Data 192.168.79.249 5272

0xa0000001 0000.0000.0000 Mobility Data 10.48.71.113 16667

125

BRKEWN-3013

Verifying AP Plumbed path

# show platform software capwap chassis active F0

Tunnel ID AP MAC Type IP Port AOM ID Status

--------------------------------------------------------------------------------------------

0x90000004 7069.5a51.4ec0 Data 192.168.79.249 5272 567 Done

0xa0000001 0000.0000.0000 Mobility Data 10.48.71.113 16667 519 Done

# show platform hardware chassis active qfp feature wireless capwap cpp-client

summary

cpp_if_hdl pal_if_hdl AP MAC Src IP Dst IP Dst Port Tun Type

-----------------------------------------------------------------------------------------------------

0X33 0XA0000001 0000.0000.0000 10.48.39.30 10.48.71.113 16667 MOBILITY

0X34 0X90000004 7069.5a51.4ec0 10.48.39.30 192.168.79.249 5272 DATA

126

BRKEWN-3013

Verifying AP Plumbed path

# show platform hardware chassis active qfp feature wireless capwap datapath

summary

Vrf Src Port Dst IP Dst Port Input Uidb Output Uidb Instance Id

--- -------- ------ -------- ---------- ----------- -----------

0 5247 192.168.79.249 5272 65490 65484 3

0 16667 10.48.71.113 16667 65491 65485 0

127

BRKEWN-3013

Troubleshooting APs the easy way

128

BRKEWN-3013

Client Health Monitoring

The SUPER command

129

# show wireless stats client detail

Total Number of Clients : 4

Protocol Statistics

-----------------------------------------------------------------------------

Protocol Client Count

802.11b 0

802.11g 0

802.11a 0

802.11n-2.4 GHz 0

802.11n-5 GHz 0

802.11ac 4

802.11ax-5 GHz 0

802.11ax-2.4 GHz 0

Client Summary

-----------------------------

Current Clients : 4

Excluded Clients : 1

Disabled Clients : 0

Foreign Clients : 0

Anchor Clients : 0

Local Clients : 4

BRKEWN-3013

•Single view:

•Total clients connected

•Per Protocol distribution

•State Distribution : easy to spot network wide

problems

Client Health Monitoring

The SUPER command (part 2)

130

client global statistics:

-----------------------------------------------------------------------------

Total association requests received : 449775

Total association attempts : 376239

Total FT/LocalAuth requests : 0

Total association failures : 382

Total association response accepts : 375857

Total association response rejects : 382

Total association response errors : 382

Total association failures due to blacklist : 2093

Total association drops due to multicast mac : 0

Total association drops due to throttling : 0

Total association drops due to unknown bssid : 71375

Total association drops due to parse failure : 0

Total association drops due to other reasons : 0

Total 11r ft authentication requests received : 204

Total 11r ft authentication response success : 200

Total 11r ft authentication response failure : 4

Total 11r ft action requests received : 234

Total 11r ft action response success : 234

Total 11r ft action response failure : 0

BRKEWN-3013

•Single view:

•98 different stats counters

•Easy to spot:

•Frequent Bcast rotation issues

•Frequent L2/L3 auth failures

•Frequent IP address learning failures

•Roaming types

Client Health Monitoring

The SUPER command (part 3)

131

client state statistics:

-----------------------------------------------------------------------------

Average Time in Each State (ms)

Associated State : 0

L2 State : 85

Mobility State : 2

IP Learn State : 2117

L3 Auth State : 0

Average Run State Latency (ms) : 1102

Average Run State Latency without user delay (ms) : 1061

Latency Distribution (ms)

1 - 100 : 278025

100 - 200 : 11511

200 - 300 : 5590

300 - 600 : 3519

600 - 1000 : 6546

1000+ : 41184

BRKEWN-3013

•Single view:

•Average time per state

•Spotting performance problems

•Variations over time

Client Health Monitoring

The SUPER command (part 4)

132

Deauthentication or disassociation request :0

Session Manager :0

L3 authentication failure :0

Delete received from AP :0

WLAN down :282

AP down/disjoin :79

Connection timeout :0

MAC authentication failure :0

Datapath plumb :0

Due to SSID change :68

Due to VLAN change :0

Admin deauthentication :21

QoS failure :0

WPA key exchange timeout :2

WPA group key update timeout :1

802.11w MAX SA queries reached :0

Client deleted during HA recovery :0

Client blacklist :0

Inter instance roam success :0

BRKEWN-3013

Single view:

•Both good and bad

•177 tracking points

•Network wide problem isolation

Client Health Monitoring

Client not connected anymore ?

133

Nico9800#show wir cli history disconnected summ

Client MAC BSSID #WNCD

----------------------------------------------

e836.171f.a162 0027.e38f.33a0 0

BRKEWN-3013

GUI

Troubleshooting

Dashboard

135

GUI Troubleshooting Dashboard

Debug Bundle Page

136

BRKEWN-3013

GUI Troubleshooting Dashboard

Debug Bundle Page

137

BRKEWN-3013

# show tech wireless # show tech wireless client #show tech wireless qos

# show tech memory. # show tech wireless multicast #show tech wireless datapath

Click here

“show tech-

support”

“show tech-

support wireless”

GUI Troubleshooting Dashboard

Syslog page

138

BRKEWN-3013

GUI Troubleshooting Dashboard

Core Dump and System Report page

139

BRKEWN-3013

Useful commands and tools

Ping and Traceroute page

140

BRKEWN-3013

GUI based CLI Editor

Administration -> Command line interface page

141

BRKEWN-3013

Other Tools

Wireless Troubleshooting Tools

142

BRKEWN-3013

https://developer.cisco.com/docs/wireless-troubleshooting-tools/

Quick Recap

Conclusion : troubleshooting recap

Step 1 : Health Monitoring

144

# show wireless stats trace-on-failure

# show logging trace-on-failure summary / show logging profile wireless

trace-on-failure

# show wireless stats ap join summary

# show wireless stats ap history

# show wireless stats client detail

BRKEWN-3013

Conclusion : troubleshooting recap

Step 2 : Basic logging tracking

145

# show log

Dec 18 13:38:18.228: %LINEPROTO-5-UPDOWN: Line protocol on Interface Capwap1, changed state to down

Dec 18 13:38:18.205: %CAPWAPAC_SMGR_TRACE_MESSAGE-3-EWLC_GEN_ERR: Chassis 1 R0/0: wncd: Error in Session-

IP: 192.168.16.134[5264] Mac: 7069.5a51.46e0 Heartbeat timer expiry for AP. Close CAPWAP DTLS session

Dec 18 13:38:18.231: %CAPWAPAC_SMGR_TRACE_MESSAGE-5-AP_JOIN_DISJOIN: Chassis 1 R0/0: wncd: AP Event: AP

Name: 4802paolo, MAC: 4c77.6d9e.60e4 Disjoined

Dec 21 06:19:45.425: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the

maximum limit(500)as specified by the platform.

..Dec 21 06:20:00.748: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the

maximum limit(500)as specified by the platform.

.Dec 21 06:20:00.785: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the

maximum limit(500)as specified by the platform.

.Dec 21 06:20:15.616: %HTTP-4-SERVER_CONN_RATE_EXCEED: Number of connections per minute has exceeded the

maximum limit(500)as specified by the platform.

BRKEWN-3013

Conclusion : troubleshooting recap

Step 3 : Pull always on data for a client/AP

146

# show logging profile wireless filter-mac <mac> to-file <filename>

start last <minutes>

• Notice level data

• Logs will be rotated every 24/48h or more depending on platform and load

BRKEWN-3013

Conclusion : troubleshooting recap

Step 4 : More information needed? RA Traces

147

# debug wireless mac aaaa.bbbb.cccc monitor-time 10

Use the Web UI for it !

BRKEWN-3013

Conclusion : troubleshooting recap

Step 5 : TAC case

148

• RA-trace output (internal level, while we’re at it) or show logging profile

wireless of always-on output filtered for the problematic mac or timestamp

• Relevant show techs (at least show tech + show tech wireless)

• Your observations from “show logging” or “show logging trace-on-failure

summary” (timestamps, affected macs)

• Core dump files from the web UI troubleshooting page (if the problem is a crash)

BRKEWN-3013

Complete your

online session

survey

• Please complete your session survey

after each session. Your feedback

is very important.

• Complete a minimum of 4 session

surveys and the Overall Conference

survey (starting on Thursday) to

receive your Cisco Live t-shirt.

• All surveys can be taken in the Cisco Events

Mobile App or by logging in to the Content

Catalog on ciscolive.com/emea.

Cisco Live sessions will be available for viewing on

demand after the event at ciscolive.com.

160

Related sessions

Walk-In Labs

Demos in the

Cisco Showcase

Meet the Engineer

1:1 meetings

Continue your education

161

BRKEWN-3013

Thank youThank you