Introduction
This document describes how to increase Data Plane (DP) Quantum Flow Processor (QFP) External Memory Manger (exmem) memory on Cisco Cloud Services Router 1000V (CSR1000V).
Prerequisites
Requirements
Cisco recommends that you have knowledge of these topics:
- CSR1000V routers.
- Cisco IOS®-XE.
- ESXi vCenter.
Components Used
The information in this document is based on these software and hardware versions:
- CSR1000V router deployed in VMware ESXi, 6.7.0
- Cisco IOS®-XE 16.12.6
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
Background
It is important to understand how Cisco IOS®-XE is built. With Cisco IOS®-XE, Cisco has moved to a Linux kernel and all the subsystems have been broken down into processes. All the subsystems that were inside Cisco IOS® before, now run as software processes within the Linux Operating System (OS). Cisco IOS® itself runs as a daemon (Cisco IOSd) within the Linux OS. Cisco IOS®-XE retains not only the same look and feel of the classic Cisco IOS®, but also its operation, support, and management.
Memory assigned to Virtual Machine (VM) is allocated to Cisco IOS® daemon (Cisco IOSd) and to Data Plane memory.
By default, memory allocation on Cisco IOSd is 2.5 GB and for Data Plane is 1.5 GB, the amount of memory available to Cisco IOSd is controlled by memory add-on licenses and certain feature licenses (broadband feature license) and it does not change when more memory is allocated to VM.
In Data Plane memory, exmem memory size on CSR1000V by default is 256MB (268435456 Bytes), which can very small amount for certain specific features configurations, like QoS Polices, NAT sessions, IPSec Security Associations, and so on. and when exmem high utilization is observed -
CSR_ESXi#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics
Type: Name: DRAM, QFP: 0
Total: 268435456
InUse: 263066746
Free: 5368710
Lowest free water mark: 206885888
Type: Name: IRAM, QFP: 0
Total: 2097152
InUse: 218112
Free: 1879040
Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
Total: 0
InUse: 0
Free: 0
Lowest free water mark: 0
Jul 5 08:016:32.486: %QFPOOR-2-LOWRSRC_PERCENT_CRIT: R0/0: cpp_ha: QFP 0 DRAM(EXMEM) at 99 percent, exceeds critical level 95
Exmem is the dynamic memory that allows the actual packet handling, used for features to scale and manages static and dynamic memory used by Cisco Packet Processor (CPP).
Note: Refer to Cisco CSR 1000v Memory Allocation for detailed Cisco IOSd memory add-on information.
Configure CSR1000V exmem allocation
This section describes how to configure Data Plane templates to increase QFP exmem size.
Template options available are -
control-plane-extra-heavy
control-plane-heavy
data-plane-heavy
data-plane-normal
service-plane-heavy
service-plane-medium
Note: This document does not cover control plane templates and service plane templates, for more details about these templates please refer to - Cisco CSR 1000v and Cisco ISRv Software Configuration Guide
Template: Data Plane Heavy
The next table shows the QFP exmem memory allocation for the Data Plane Heavy template.
VM Memory |
QFP exmem Size |
4GB |
256MB (default value) |
8GB |
512MB |
16GB |
1024MB |
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#platform resource data-plane-heavy
Please reboot to activate this template
Router(config)#end
Router#write memory
Building configuration...
[OK]
CSR_ESXi#reload
Summary Steps
- enable
- configure terminal
- platform resource data-plane-heavy
- end
- write memory
- reload
Verify
1. 8GB assigned to VM .
cisco CSR1000V (VXE) processor (revision VXE) with 2295972K/3075K bytes of memory.
Processor board ID 949XJJNMSHM
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
8105924K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.
2. No changes observed in terms of memory allocated to Cisco IOSd and Data Plane after template change.
Router#show platform software vmemory info
Memory Upgrade Limits:
Total System Memory: 7915 MB
Memory From Upgrade Licenses: N/A(Smart License Enabled)
Memory From Feature Licenses: N/A(Smart License Enabled)
Memory Available For Upgrade:
Available System Memory: 3819 MB
Available Upgrade Licensed Memory: N/A(Smart License Enabled)
Available Feature Licensed Memory: N/A(Smart License Enabled)
Current Memory Allocation:
Cisco IOSD: 2560 MB (default) + 0 MB upgrade
Data Plane: 1536 MB (default) + 0 MB upgrade
3. QFP exmem total available is now 512MB.
Router#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics
Type: Name: DRAM, QFP: 0
Total: 536870912
InUse: 61156352
Free: 475714560
Lowest free water mark: 475321344
Type: Name: IRAM, QFP: 0
Total: 2097152
InUse: 218112
Free: 1879040
Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
Total: 0
InUse: 0
Free: 0
Lowest free water mark: 0
Router#show platform software cpu alloc
CPU alloc information:
Control plane cpu alloc: 0
Data plane cpu alloc: 1-7
Service plane cpu alloc: 0
Template used: CLI-data_plane_heavy
Template: Data Plane Normal
Router#configure terminal
Enter configuration commands, one per line. End with CNTL/Z.
Router(config)#platform resource data-plane-normal
Please reboot to activate this template
Router(config)#end
Router#write memory
Building configuration...
[OK]
CSR_ESXi#reload
Summary Steps
- enable
- configure terminal
- platform resource data-plane-normal
- end
- write memory
- reload
Data Plane Normal, uses 256MB in QFP exmem regardless of amount of VM Memory assigned.
cisco CSR1000V (VXE) processor (revision VXE) with 2295972K/3075K bytes of memory.
Processor board ID 949XJJNMSHM
3 Gigabit Ethernet interfaces
32768K bytes of non-volatile configuration memory.
16363460K bytes of physical memory.
7774207K bytes of virtual hard disk at bootflash:.
0K bytes of WebUI ODM Files at webui:.
Router#show platform software vmemory info
Memory Upgrade Limits:
Total System Memory: 15979 MB
Memory From Upgrade Licenses: N/A(Smart License Enabled)
Memory From Feature Licenses: N/A(Smart License Enabled)
Memory Available For Upgrade:
Available System Memory: 11883 MB
Available Upgrade Licensed Memory: N/A(Smart License Enabled)
Available Feature Licensed Memory: N/A(Smart License Enabled)
Current Memory Allocation:
Cisco IOSD: 2560 MB (default) + 0 MB upgrade
Data Plane: 1536 MB (default) + 0 MB upgrade
Router#show platform hardware qfp active infrastructure exmem statistics
QFP exmem statistics
Type: Name: DRAM, QFP: 0
Total: 268435456
InUse: 61156352
Free: 207279104
Lowest free water mark: 206885888
Type: Name: IRAM, QFP: 0
Total: 2097152
InUse: 218112
Free: 1879040
Lowest free water mark: 1879040
Type: Name: SRAM, QFP: 0
Total: 0
InUse: 0
Free: 0
Lowest free water mark: 0
Note: QFP exmem memory allocation depends solely on the amount of memory assigned to VM and the data plane template applied. It does not depend on memory add-on licenses installed.
Troubleshoot
There is currently no specific information available on how to troubleshoot this configuration.