Introduction
This document describes the troubleshooting tips for first-time deployments of Zero Touch Provisioning (ZTP).
ZTP was introduced to reduce human interaction in the provision of XR devices. ZTP uses an implementation of the Preboot eXecution Environment (iPXE).
Background Information
ZTP can perform:
- Auto-Configuration update: Download and apply the XR configuration after image installation
- Script execution: Download and execute the user-defined script files. Various methods/functions can be implemented as a part of script execution:
- Multiple packages update
- SMU installation
- Verifications
- eXR decides which of the actions to perform based on the contents of the first line in the file
- If the file starts with !! IOS XR denotes a configuration file and performs apply_config.
- Similarly, the files that start with #! /bin/bash or #! /bin/sh or #!/usr/bin/python denotes a script file, and perform script execution.
- Support for python is introduced from eXR 6.2.2
- eXR iPXE supports Trivial File Transfer Protocol (TFTP), File Transfer Protocol (FTP), and Hypertext Transfer Protocol (HTTP).
- Hypertext Transfer Protocol Secure (HTTPS) is not supported as it cannot predict which signature to verify.
Linux Server Configuration
iPXE is an enhancement over PXE that requires TFTP/FTP/HTTP for image/configuration download and uses Dynamic Host Configuration Protocol (DHCP) to get/provide information for the image and configuration.
DHCP Configuration
In a later example, there is a review of a packet capture to confirm the DHCP operation.
HTTP Server Requirements
HTTP server must be reachable from the Mgmt Ethernet interface.
Troubleshooting Tips
When the Linux server is set up, Perform DHCP/HTTP server reachability and functionality Check.
In this setup, a single Linux server is used as DHCP/HTTP server. If there are separate servers for these functions, verify these steps on all the servers, if needed.
[root@xxxxxxxxxx]# service dhcpd status
Redirecting to /bin/systemctl status dhcpd.service
dhcpd.service - DHCPv4 Server Daemon
Loaded: loaded (/usr/lib/systemd/system/dhcpd.service; disabled; vendor preset: disabled)
Active: active (running) since Mon 2017-05-29 10:30:59 PDT; 15h ago
Docs: man:dhcpd(8)
man:dhcpd.conf(5)
Main PID: 26913 (dhcpd)
Status: "Dispatching packets..."
CGroup: /system.slice/dhcpd.service
└─26913 /usr/sbin/dhcpd -f -cf /etc/dhcp/dhcpd.conf -user dhcpd -group dhcpd --no-pid
<SNIP>
Hint: Some lines were ellipsized, use -l to show in full.
[root@xxxxxxxx]# service httpd status
Redirecting to /bin/systemctl status httpd.service
httpd.service - The Apache HTTP Server
Loaded: loaded (/usr/lib/systemd/system/httpd.service; disabled; vendor preset: disabled)
Active: active (running) since Fri 2017-05-26 05:50:30 PDT; 3 days ago
Docs: man:httpd(8)
man:apachectl(8)
Process: 28088 ExecStop=/bin/kill -WINCH ${MAINPID} (code=exited, status=0/SUCCESS)
Process: 11036 ExecReload=/usr/sbin/httpd $OPTIONS -k graceful (code=exited, status=0/SUCCESS)
Main PID: 28095 (httpd)
Status: "Total requests: 0; Current requests/sec: 0; Current traffic: 0 B/sec"
CGroup: /system.slice/httpd.service
├─11037 /usr/sbin/httpd -DFOREGROUND
├─11038 /usr/sbin/httpd -DFOREGROUND
├─11039 /usr/sbin/httpd -DFOREGROUND
├─11040 /usr/sbin/httpd -DFOREGROUND
├─11041 /usr/sbin/httpd -DFOREGROUND
├─26998 /usr/sbin/httpd -DFOREGROUND
├─27426 /usr/sbin/httpd -DFOREGROUND
├─27427 /usr/sbin/httpd -DFOREGROUND
├─27428 /usr/sbin/httpd -DFOREGROUND
├─27889 /usr/sbin/httpd -DFOREGROUND
└─28095 /usr/sbin/httpd -DFOREGROUND
May 26 05:50:30 xxxxx systemd[1]: Starting The Apache HTTP Server...
May 26 05:50:30 xxxxx systemd[1]: Started The Apache HTTP Server.
May 27 03:16:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.
May 28 03:37:01 xxxxx systemd[1]: Reloaded The Apache HTTP Server.
Enable Router To Get IP Address With DHCP
Interface MgmtEth 0/RP0/CPU0/0
Ipv4 address dhcp
Shut/no shut
Verify HTTP Server Is Functional
- Test that http://<ip-servername> is works.
- Open a browser to the IP address/server name.
If DHCP or HTTP fails to work, there can be firewall issues or reachability problems.
To check the firewall properties on the server perform these commands or add the specific protocols:
Use IP tables command to verify firewall rules:
Iptables –L –n
Chain IN_public_allow (1 references)
target prot opt source destination
ACCEPT udp -- 0.0.0.0/0 0.0.0.0/0 udp dpt:67 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 ctstate NEW
ACCEPT tcp -- 0.0.0.0/0 0.0.0.0/0 tcp dpt:22 ctstate NEW
Use this firewall command to list the entries allowed:
[root@xxxxxxxxx ~]# firewall-cmd --list-all
public (active)
target: default
icmp-block-inversion: no
interfaces: enp2s0f0
sources:
services: dhcp dhcpv6-client http ssh
ports:
protocols:
masquerade: no
forward-ports:
sourceports:
icmp-blocks:
rich rules:
Use the commands to permanently allow ports:
firewall-cmd --permanent --add-service=http
firewall-cmd --permanent --add-service=dhcp
Enable Packet capture on DHCP server:
tcpdump –i <interface id> port (bootpc & bootps & port http) –s 0(buffer size) –w <dest. File name>
ex: tcpdump -i enp2s0f0 port 67 or port 68 or port 80 -s 0 -w iPXEboot.pcap
How To Start ZTP?
Manually Initiate ZTP
Use ZTP initiate exec CLI command to invoke ZTP. By default, the interface used for ZTP is MGMTEth. To initiate this process on other interfaces use this option:
ZTP initiate interface <type> <number> <cr>
OR
Ztp initiate <cr>
Boot The Router With iPXE
1. From Calvados, initiate the reload CLI command:
This command causes a router to reload, which initiates the iPXE boot.
Note: User-class information is interpreted as a Malformed option due to a bug in Wireshark.
2. The router initiates DHCP Discover, notice that user-class is populated with iPXE option:
3. The DHCP Offer from the server includes the boot file name in Option 67:
4. The router initiates the image download:
5. Once the image is successfully downloaded, the installation of the images on the device begins.
6. Once the router successfully boots with the downloaded image, it initiates another DHCP request:
In this Discover, notice that user-class info includes *.exr-config. Since the DHCP is configured to either return Config file or Script (ie. Else statement in etc/dhcp/dhcpd.conf).
7. DHCP server returns the required file information in option 67:
8. The router downloads the config:
The script can be included as a response from the DHCP server and have them copy the image/configuration. This can also be used as a post-install script like in the below example.
Post-Install Automation
Various automation can be performed post-installation on the eXR software.
In this example, this script installs all required packages and applies the Config. This is a slightly modified version of ztp_helper.sh. This script is invoked from xr-linux-shell:
RP/0/RP0/CPU0:NCS-5502-A#more disk0:/ztp/ztp_helper_file.sh
Wed May 31 00:55:54.529 UTC
#!/bin/bash
################################################################################
# Install config and additional packages
################################################################################
source /disk0:/ztp/ztp_helper.sh
export HTTP_SERVER=http://10.10.10.10
export RPM_PATH=images
export CONFIG_PATH=images
#Config
export INITIAL_CONFIG=NCS-5502-A.cfg
export FINAL_CONFIG=NCS-5502-A.cfg
#Packages
K9SEC_RPM=ncs5500-k9sec-2.2.0.0-r612.x86_64.rpm
MCAST_RPM=ncs5500-mcast-2.0.0.0-r612.x86_64.rpm
ISIS_RPM=ncs5500-isis-1.1.0.0-r612.x86_64.rpm
OSPF_RPM=ncs5500-ospf-1.1.0.0-r612.x86_64.rpm
MGBL_RPM=ncs5500-mgbl-3.0.0.0-r612.x86_64.rpm
MPLS_RPM=ncs5500-mpls-2.1.0.0-r612.x86_64.rpm
MPLSTE_RPM=ncs5500-mpls-te-rsvp-2.2.0.0-r612.x86_64.rpm
function download_config(){
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${CONFIG_PATH}/${FINAL_CONFIG} -O /harddisk:/new-config 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading system configuration ###"
else
printf "### Downloading system configuration complete ###";
fi
}
function apply_config(){
# Applies initial configuration
printf "### Applying initial system configuration ###";
xrapply_with_reason "Initial ZTP configuration" /harddisk:/new-config 2>&1;
printf "### Checking for errors ###";
local config_status=$(xrcmd "show configuration failed");
if [[ $config_status ]]; then
echo $config_status
printf "!!! Error encounter applying configuration file, review the log !!!!";
fi
printf "### Applying system configuration complete ###";
}
function install_pkg(){
#Download packages
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${K9SEC_RPM} -O /harddisk:/$K9SEC_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $K9SEC_RPM ###"
else
printf "### Downloading $K9SEC_PKG complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MCAST_RPM} -O /harddisk:/$MCAST_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $MCAST_RPM ###"
else
printf "### Downloading $MCAST_RPM complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${ISIS_RPM} -O /harddisk:/$ISIS_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $ISIS_RPM ###"
else
printf "### Downloading $ISIS_RPM complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${OSPF_RPM} -O /harddisk:/$OSPF_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $OSPF_RPM ###"
else
printf "### Downloading $OSPF_RPM complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MGBL_RPM} -O /harddisk:/$MGBL_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $MGBL_RPM ###"
else
printf "### Downloading $MGBL_RPM complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLS_RPM} -O /harddisk:/$MPLS_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $MPLS_RPM ###"
else
printf "### Downloading $MPLS_RPM complete ###";
fi
printf "Downloading Packages"
ip netns exec tpnns /usr/bin/wget ${HTTP_SERVER}/${RPM_PATH}/${MPLSTE_RPM} -O /harddisk:/$MPLSTE_RPM 2>&1
if [[ "$?" != 0 ]]; then
printf "### Error downloading $MPLSTE_RPM ###"
else
printf "### Downloading $MPLSTE_RPM complete ###";
fi
xrcmd "install update source /harddisk:/ $K9SEC_RPM $MCAST_RPM $ISIS_RPM $OSPF_RPM $MGBL_RPM $MPLS_RPM $MPLSTE_RPM" 2>&1
local complete=0
while [ "$complete" = 0 ]; do
complete=`xrcmd "show install active" | grep k9sec | head -n1 | wc -l`
printf "Waiting for k9sec package to be activated"
sleep 5
done
rm -f /harddisk:/$K9SEC_RPM /harddisk:/$MCAST_RPM /harddisk:/$MCAST_RPM /harddisk:/$ISIS_RPM /harddisk:/$OSPF_RPM /harddisk:/$MGBL_RPM /harddisk:/$MPLSTE_RPM /harddisk:/$MPLS_RPM
printf "### XR PACKAGE INSTALL COMPLETE ###"
}
printf "Start Auto provision"
install_pkg;
download_config;
apply_config;
Related Information