Run Backhaul Manager Script on CGOS CGR 1000

Introduction

This document describes how to run the bhmgr.tcl script on the Cisco 1000 Series Connected Grid Router (CGR1000) with Connected Grid Operating System (CGOS). The backhaul refers to the ipsec tunnel that connects back into Head End Router (HER).

Prerequisites

Requirements

There are no specific requirements for this document.

Components Used

This document is restricted for CGOS version on CGR1000 series routers; CGR1120, CGR1240.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, make sure that you understand the potential impact of any command.

Background Information

Normally CGR is located in a remote area with 3G connection as the primary connection. In order to secure the traffic to the datacenter, the CGR uses an ipsec tunnel. As a result, consider the ipsec tunnel as the backhaul. The bhmgr.tcl script when setup correctly, attempts to restart the connection of the backhaul if there is a link problem. For example, 3G card connectivity can be spotty. Since the interface can get stuck, the backhaul script can reload the 3G module on the CGR to recovery the link. If this procedure is unsuccessful, the script attempts to reboot the CGR as a last resort.

Steps to Run Script

1. Determine what interface(s) need to be monitor. Normally, backhual should monitor the ipsec tunnel because the tunnel forms the connection back to the HER. Also, you need to determine what interface the ipsec tunnel use to form the tunnel. For example; Ethernet2/1 is the main connection and Tunnel 0 is the ipsec tunnel.

2. Setup the environment variable. Only certain environment variable is required, the other ones will be default if not set. More information can be found from reading the script itself. By default, the interface reset is done every 360 minutes and reload is done at 720 minutes when the timer expires.

For example;

event manager environment bh_iflist "eth2/1 tunnel0"
event manager environment bhmgr_track_obj_instance "1"
event manager environment eem_dbg_level "2"

3. Configure the track object to point to the ip sec tunnel. Use track 1 as this will be configure for step 4 during the EEM applet configuration portion.

For example;

track 1 interface tunnel0 line-protocol
delay up 120 down 120

4. Use this configuration to track object 1 in the EEM applet.

For example;

event manager applet bhmgrbhdown
event track 1 state down
action 1.0 syslog priority critical msg Backhaul is down
action 2.0 cli tclsh volatile:bhmgr.tcl bhmgr_process_bh_down
action 3.0 cli command maximum-timeout
event manager applet bhmgrbhup
event track 1 state up
action 1.0 syslog priority errors msg Backhaul is up
action 2.0 cli tclsh volatile:bhmgr.tcl bhmgr_process_bh_up
action 3.0 cli command maximum-timeout

5. The backhaul manager script is integrated on the CGOS and is located on volatile partition. If the backhaul script is missing, put the bhmgrl.tcl on volatile partition.

6. Schedule the bhmgr_monitor with the configuration. Once this runs, the script will setup the environment variable setup in Step 2. If enviornment variable shows not found, the script uses default setting setup within the script.

For example;

scheduler job name bhmgr_monitor
tclsh volatile:/bhmgr.tcl bhmgr_monitor

Output:

                 eem_dbg_level : 2

                 eem_dbg_level : 2

      bhmgr_track_obj_instance : 1

      bhmgr_track_obj_instance : 1

Environment variable not found

Environment variable not found

7. Schedule the script to run periodically.

For example;

    scheduler schedule name bhmgr_monitor_schedule
    job name bhmgr_monitor
    time start now repeat 0:0:10

Log Ouput

When ipsec tunnel is up, logging output shows that backhaul is up. When ipsec tunnel is down, logging output shows backhaul is down. At the time of the down phase, the counter starts to decrease to 0 and execute the reset of the interface or reload of the router. If the backhaul goes back online, the counter resets to the default value.

When ipsec tunnel is online, logging output shows Backhaul is up.  When logging output shows Backhaul is down, ipsec tunnel is down.  Counter will start to decrease:

2017 Mar 20 18:36:35 AST05-CGR %EEM_ACTION-3-ERROR: Backhaul is up
2017 Mar 20 18:36:36 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: Backhaul is up

2017 Mar 20 20:29:02 AST05-CGR %EEM_ACTION-2-CRIT: Backhaul is down

2017 Mar 20 20:29:40 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 29 mins and 23 sec to BH Reset
2017 Mar 20 20:29:40 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 59 mins and 23 sec to RELOAD

Once the counter for BH Reset goes down to 0,  the backhaul manager script attempts to reset the interface when log output shows BH Reset policy hit:

2017 Mar 20 17:43:33 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 7 mins and 57 sec to BH Reset 
2017 Mar 20 17:43:33 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 37 mins and 57 sec to RELOAD 
2017 Mar 20 17:43:34 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 17:43:35 AST05-CGR last message repeated 2 times
2017 Mar 20 17:43:35 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 7 mins and 55 sec to BH Reset 
2017 Mar 20 17:43:35 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 37 mins and 55 sec to RELOAD 
2017 Mar 20 17:53:36 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 17:53:37 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: BH Reset policy hit 
2017 Mar 20 17:53:37 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 17:53:38 AST05-CGR last message repeated 1 time
2017 Mar 20 17:53:38 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: Performing module 2 reload in 30 seconds. 

Once the counter for RELOAD goes down to 0, the backhaul manager script attempts to reload the router when log output shows BH RELOAD policy hit:

2017 Mar 20 18:04:18 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 49 mins and 55 sec to BH Reset 
2017 Mar 20 18:04:18 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 17 mins and 12 sec to RELOAD 
2017 Mar 20 18:04:19 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 18:04:20 AST05-CGR last message repeated 2 times
2017 Mar 20 18:04:20 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 49 mins and 53 sec to BH Reset 
2017 Mar 20 18:04:20 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 17 mins and 10 sec to RELOAD 
2017 Mar 20 18:14:21 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 18:14:22 AST05-CGR last message repeated 2 times
2017 Mar 20 18:14:22 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 39 mins and 51 sec to BH Reset 
2017 Mar 20 18:14:23 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 7 mins and 8 sec to RELOAD 
2017 Mar 20 18:14:24 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 18:14:24 AST05-CGR last message repeated 1 time
2017 Mar 20 18:14:24 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 39 mins and 49 sec to BH Reset 
2017 Mar 20 18:14:24 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 18:14:25 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 7 mins and 6 sec to RELOAD 
2017 Mar 20 18:24:26 AST05-CGR %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configured from vty by admin on scheduler
2017 Mar 20 18:24:27 AST05-CGR last message repeated 2 times
2017 Mar 20 18:24:27 AST05-CGR %EEM_ACTION-2-CRIT: bhmgr: 29 mins and 46 sec to BH Reset 
2017 Mar 20 18:24:27 AST05-CGR %EEM_ACTION-0-EMERG: BH RELOAD policy hit. Performing reload in 30 seconds