Automate Network Changes

This section contains the following topics:

Change Automation Overview

The Change Automation application automates the process of deploying changes to the network. You can define automation tasks to achieve the intended network states in Change Automation using Playbooks that consists of plays written using YAML. You can then push configuration changes to Cisco Network Service Orchestrator (NSO), which deploys these changes to the network devices.

The difference between Change Automation and other existing scripted automation frameworks is that Change Automation is a closed-loop framework. Changes are deployed to the router or other device using programmable APIs, and the intent of the change is verified using telemetry that comes back from the router. Change Automation relies on telemetry to verify the intent of the change, avoiding the need to frequently poll the device for updates.

The following is a high-level Change Automation workflow:

  1. Review the existing plays and Playbooks to see if any of them meet your needs fully or partially.


    Note
    Change Automation comes with a robust library of Playbooks, each with its own collection of atomic configuration and check plays. (A Playbook consists of multiple plays.)

  2. Build Playbook as required:

    • If the required Playbook is available, use it.

    • If the required Playbook is not available, build a new Playbook using the exisitng applicable plays.

    • If some of the required plays are not available, create new plays and build a new Playbook using the new and existing plays.

    • If none of the required plays are available, create new plays and build a new Playbook using the new plays.

  3. Dry run the Playbook to test if it accomplishes the desired changes in the network.

  4. Deploy the Playbook.

Change Automation allows you to customize and generate plays and Playbooks using its API interface. For more information, see About Custom Plays and About Customizing Playbooks.

Configure Change Automation Settings

Configuring Change Automation settings is a post-installation activity. This section explains the initial settings that need to be configured before you can start using Change Automation.


Note
The Change Automation settings can only be configured once. If you want to modify the settings, Change Automation must be re-installed. Before re-installing, make sure to export the Playbooks, and after re-installing, import the Playbooks. For more information, see Export Playbooks and Import Playbooks.

System settings

After Change Automation is installed in your Crosswork platform, you can access Change Automation from the main menu (go to Network Automation > Dashboard). The Change Automation window is displayed prompting you to complete the configuration. Click Start Configuration to review the change Automation settings. Alternatively, you can navigate to Administration > Settings > Device Override Credentials to view the settings.

Figure 1. Change Automation settings

Click Save after you configure the following settings:

  • Playbook Job Scheduling: Enable or disable the ability the schedule the Playbook jobs.

  • Credential Prompt: Device override credentials are an additional level of authentication that can improve change auditing. If enabled, users will be prompted to enter the credentials (device override credentials) before each Playbook execution. You need to create the relevant credential profile and provider settings for the override credentials to work. Follow the prompts on the window to meet each requirement.


Note
If Playbook Job Scheduling is disabled, then the Credential Prompt is enabled, by default. You cannot disable the credential prompt in this case.

Note
While executing Device Config plays, entering incorrect device override credentials will cause the playbook execution to fail. However, for a Check play or Data Collection play, the device override credentials are not validated and the Playbook will execute successfully irrespective of the accuracy of the override credentials. Device override credentials are only validated while pushing configuration changes.

Note
Also, users with write permissions for the Administration APIs under Change Automation (Go to Administration > Users and Roles > Roles) can only perform this task.

Assign proper level of access to other users

Once the system settings are completed, the admin user must review the user roles to make sure that all the users they have are configured with the proper level of access to run, import, and create Plays and Playbooks. To provide the access, the admin user must:

  1. Go to Administration > Users and Roles > Roles.

  2. Under Roles pane, select the role for which you want to grant the access.

  3. Under the Role Permissions > Global, enable Read and Write checkboxes (as necessary) for Play APIs and Playbook APIs under Change Automation.


    Note
    Also, the user with write permissions for Administration APIs can only disable or enable run Playbook access and assign labels.

Use the Change Automation Dashboard

The Change Automation application's Dashboard window (shown in the following figure) lets you view all Playbook-related activity and initiate Playbook runs. It displays the total number of Playbooks, the Playbook Jobs Calendar, the most recently run Playbook jobs, and the same network topology map you see when you select Topology from the main menu.

To view the Change Automation Dashboard window, select Network Automation > Dashboard.

Figure 2. Change Automation Dashboard Window

The Playbooks tile displays the total number of Playbooks (pre-defined and custom). Clicking on a specific number displays all the Playbooks that correspond to the selected category.


Note

Total Playbooks indicate the total number of pre-defined and user created Playbooks (My Playbooks) that exist in the system.

Predefined Playbooks indicate the number of pre-defined Playbooks that exist in the system.

My Playbooks indicate the number of custom Playbooks created by the current user.

Creating Playbooks does not use a license until the Playbook is run either successfully or unsuccessfully for the first time.

The Jobs Calendar tile displays a calendar (month, week, day) with the number of job sets executed on a given day marked in a circle against the corresponding date. Clicking on the number displays a dialog box with the names of the Playbook job sets and their execution time. Click on the desired job set to view the execution details.


Note

The color of the circle indicates the overall status of the job sets.

  • A red circle indicates at least one job set with Failed status among the day's overall job sets.

  • A grey circle indicates that all job sets are either in Scheduled or Running status.

  • A blue circle indicates at least one critical job set in Recovered status among the day's overall job sets

  • A green circle indicates most of the Playbooks are in success state. Clicking on it displays all the jobs that are Recovered, Scheduled, or Running.

The View All Jobs link on the Jobs tile give you direct access to the Change Automation Automation Job History window.

View the Play list

The Play List window of the Change Automation application gives you a consolidated list of all the Plays in the system.

From the main menu, select Network Automation > Play List to view the Play List window.

Item Description

1

Click Add icon to create a custom Play. See Create a Custom Play using Templates.

Click Delete icon to delete a custom Play. See Delete Custom Plays.

Click Import icon to import custom Play from a gzipped TAR archive file. See Import Plays.

Click Export icon to export a custom Play as a gzipped TAR archive file. See Export Plays.

2

Click Details iconto see a popup Play Details window showing the Play's description and schema. When you are finished viewing these details, click Close icon to close the popup window.

3

The Type column indicates the type of the Play. You can click on the column headings (Name, Description, Type, Labels, and Modified by) to sort the table by that column's data.

4

Click Refresh icon to refresh the Plays list.

5

Click Set Filter icon to set filter criteria on one or more columns in the table.

Click the Clear Filter link to clear any filter criteria you may have set.

View the Playbook List

The Change Automation application's Playbook List window (in the following figure) gives you a consolidated list of all the Playbooks in the system. To view the Playbook List window, select Network Automation > Playbook List.

Item Description

1

Click Add icon to create a custom Playbook. See Create a Custom Playbook.

Click Delete icon to delete the currently selected custom Playbook. See Delete Custom Playbooks.

Click Import icon to import Playbooks from a gzipped TAR archive file. See Import Playbooks.

Click Export icon to export the currently selected Playbook(s) as a gzipped TAR archive file. See Export Playbooks.

2

Click Assign Labels to assign lable(s) to the Playbook. Assigning label(s) to the Playbooks gives the system administrator the ability to control which Playbooks each user role is allowed to run.

3

Click Details iconto see a popup Playbook Details window showing the Playbook's description, software compatibility, version number, and its plays. When you are finished viewing these details, click Close icon to close the popup window.

3

Click on the Name, Description, Version, Software Platform, and Last Modified column headings in the table to sort the table by that column's data. You can also choose which columns are shown, and set quick or advanced filters on any column.

4

Click Refresh icon to refresh the Playbooks list.

5

Click Set Filter icon to set filter criteria on one or more columns in the table.

Click the Clear Filter link to clear any filter criteria you may have set.

About Custom Plays

Change Automation allows users to create their own custom Plays, either based on Cisco models or from scratch. Users can also import, export and delete their custom Plays.

You can create custom Plays in any of the following types:

  • Check Plays: Verifies the data from your devices using a logical expression.

  • Data Collection Plays: Collects data from your devices.

  • Device Config Plays: Performs configuration changes on your device

  • Service Plays: Provisions and manages a service deployed


Note

Cisco-supplied Plays cannot be edited, exported or deleted by the user.


Note

Check Play and Data Collection Play support MDT and SNMP collection.

Create a Custom Play using Templates

This section explains the procedure to create a custom Play. The stages of Play creation will vary depending on the Play type you choose:

  • Check Play: Select Play Type > Select Sensor Path > Build Check Expression > Review Play

  • Data Collection Play: Select Play Type > Select Sensor Path > Build Filter Expression > Review Play

  • Device Config Play or Service Play: Select Play Type > Configure Play (using sample payload in JSON format) > Review Play


Note

Cisco provides YANG modules for Cisco products. The process described in this section assumes that the sensor you want to use or the field that you want to modify is included in the modules provided by Cisco. If the devices are not listed in the default YANG modules, Cisco allows you to expand the device coverage. For information on loading a new or modified modules, see Section: Manage Custom Software Packages in the Cisco Crosswork Infrastructure 4.1 and Applications Administration Guide.


Note

When creating a Service Play, you are not creating a new service for NSO, but creating a Play to manage and provision an existing service in one or more NSO instances. For more information, see https://developer.cisco.com/docs/nso/.

Before you begin

Ensure that the configuration that you are trying to create is available in the NSO, or else it will show an error.

Procedure

Step 1

From the main menu, choose Network Automation > Play List. The Play List window is displayed.

Step 2

Click Add icon to create a custom Play. The Select Play Type window opens displaying the types of Plays supported and a description for each. The stages of creation is also displayed, and it will vary depending on the Play type you select.

Select the Play type you want to create and click Next.

Step 3

If you are creating a Check Play or Data Collection Play, perform the following:

  1. In the Select Sensor Paths window, select the required YANG module, Gather Path, and Sensor Paths. Click Next to continue.

  2. Depending on the Play type you have selected, you need to Build Check (for Check Play) or Build Filter (for Data Collection Play) to apply in your Play. Click Add Rule to add a logic expression using the keys and fields of the selected sensor path(s). Click Add Group to add a new logic group. Select the sensor field, operator and value from the dropdown lists. Select the desired logic operation (AND/OR) between each rule or group.

    Click on Runtime checkbox if you prefer to enter the value of the sensor field dynamically during runtime. If you select this checkbox, the value field is disabled, and you will be prompted to enter the input parameter when this Play is executed (as part of a Playbook) during runtime.

    Click Next to continue.

Step 4

If you are creating a Device Config Play or Service Play, perform the following:

  1. In the Configure Play window, click Import icon or the Import link to import your device config (.JSON) file. You can download and use the sample configuration template. Browse and select your .JSON file, and click Import.

  2. In the acknowledgement prompt, click Continue to select the NSO instance for the config you have imported.

  3. Select the NSO provider instance from the dialog box, and click Process Payload.

    Note

     

    The creation workflow of a Service Play is similar to the Device Config Play, except in the template of the payload file used.

  4. The Configure Play window opens displaying information from the payload file. You can edit the value or description columns with the values that you want to see during a Playbook execution.

    Click Next to continue.

Step 5

In the Review Play window, review the parameters of your Play. Click Dry Run to validate your parameters.

Label your Play with a unique Name and Description.

Note

 

Cisco also formats the play names with indicators such as cfg for configuration, chk for check, and so on, in the name to help you organize the plays properly. You can also use a similar tagging for they plays you create.

You can also add labels to your Play to group it in the future (optional).

Note

 

The labels determine the type of devices you can use the play with. For example, IOS XR Play cannot run on an IOS XE devices. Ensure to review the labels (IOS XR, IOS XE, and so on) that already exist when you add them.

Step 6

If you are satisfied with your changes, click Create.

The Play List window opens displaying your new custom Play in the Play list.

Export Plays

You can export any custom Play authored or imported by you or another user, or have imported into Cisco Crosswork Change Automation.

The exported archive will contain only the user-customizable files listed in Playbook Components and Files. Once you extract them from the archive, you can identify the Play components by their file names and filename extensions.

Your user ID must have Change Automation read permission to export Plays.

Procedure

Step 1

From the main menu, choose Network Automation > Play List.

Step 2

Check the check boxes for the custom Plays you want to export.

Step 3

Click Export icon. Your browser will prompt you to select a path and the file name to use when saving the gzipped tar archive. Follow the prompts to save the file.

Import Plays

You can import any custom Play that meets the following requirements:

  • The Play files must be packaged as a gzipped tar archive.

  • The archive must contain a .play file (a data spec file for the play), at minimum.

  • The archive file must have a unique name.


Note

For more details about editing and importing, see Cisco Crosswork Change Automation Developer Guide.

Note that you can overwrite a custom Play. The system will warn you when you are about to overwrite a custom Play, but will not prevent you from doing so. Take precautions to ensure that you do not overwrite your custom Plays accidentally.

Before you begin

A user must have the write access to import Plays. For more information about granting read-writer role access to a user, see Assign proper level of access to other users in Configure Change Automation Settings.

Procedure

Step 1

From the main menu, choose Network Automation > Play List.

Step 2

Click Import icon. Your browser will prompt you to browse to and select the gzipped archive file containing the Plays you want to import.

Make sure there is no existing Plays with the same name as the Play you intent to import. If you import a Play with the same name, it fails.

Step 3

Follow the prompts to import the archive file.

Delete Custom Plays

You can delete user-defined Plays only. You cannot delete a Cisco-supplied Play.

Your user ID must have Change Automation delete permission to delete Plays.

Procedure

Step 1

From the main menu, choose Network Automation > Play List.

Step 2

In the Play List window, select the custom Plays that you want to delete.

Step 3

Click Delete icon icon.

Step 4

When prompted, click Delete again to confirm.

About Customizing Playbooks

Users can create their own Playsbooks from scratch based on the details from Cisco-supplied Playbooks. They can also create custom Playbooks using the available Plays.

Creating and modifying Cisco-supplied Playbooks are engineering tasks that take place outside of the user interface for Cisco Crosswork Change Automation. As such, they are outside the scope of this User Guide.

Cisco supplies developer-level documentation for Cisco-supplied Playbooks. For more information on how to create custom plays and Playbooks, see:

Playbook Components and Files

Change Automation Playbooks contain a variety of components, referred to using specialized names. The components are implemented in the Playbook as files. Some of these components' names are borrowed from the Ansible specification, but all have their own definitions, and not all of the corresponding files can be customized by users. Some components are Cisco-proprietary intellectual property; while you can use them in custom Plays and Playbooks, you cannot customize them directly. For more information, see Section: Writing Custom Playbooks at Cisco Crosswork Change Automation Developer Guide.

Create a Custom Playbook

Change Automation allow users with admin and read-write roles to create custom Playbooks using the available Plays. For more information about granting read-writer role access to a user, see Assign proper level of access to other users in Configure Change Automation Settings.


Note
You cannot edit a custom Playbook after it is created. It is recommended to Dry Run the Playbook before completing the creation to make sure that the purpose of the Playbook is met. Once created, if you need to make changes to the custom Playbook, you have to recreate your Playbook with the relevant changes.

Procedure

Step 1

From the main menu, choose Network Automation > Playbook List. The Playbook List window is displayed.

Step 2

Click Add icon to create a custom Playbook. The Select Plays window opens displaying the available Plays. The stages of creation are also displayed.

Select all the Plays you want in your Playbook, and click Next.

Note

 
It is recommended to include Sync NSO from device Play. This Play performs the NSO sync from device to get the configuration in NSO to match the device. This will not fail due to the device and NSO being out of sync.

Step 3

In the Order Playbook window, arrange the order of the Plays in the Playbook as per the execution phase (Continuous, Pre-maintenance, Maintenance, Post Maintenance). You can click and drag the play to rearrange them in the sequence you prefer. You can also duplicate or delete a play by clicking on the icons provided. For more information on execution order, see Playbook Execution Order.

Note

 
By default, all the selected plays are displayed within the Maintenance phase. You can move the check plays to the appropriate phase (Continuous, Pre-maintenance, Maintenance, Post Maintenance). Also, depending on the type of play you have selected, it may be restricted from being used in certain phases. For example, a configuration Play cannot be used outside of the maintenance phase. Crosswork Change Automation also formats the Play names with indicators such as cfg for configuration, chk for check, and so on, in the name to help you organize the plays properly. You can use similar tagging for the Plays you create.

Click Next.

Step 4

The Configure Plays window opens displaying the Plays in each execution phase, and the Play schemas.

You can perform the following:

  • Click to specify policy for a Play. In the Specify Policy dialog box, specify relevant values for the fields provided. Click Field Help icon for more information about each field. Click Save to save your policy values.

    Note

     

    Policies are mainly applicable to Check Plays.

    Figure 3. Specify Policy for Check Play
  • Click to apply a condition to a Play. Execution of the play will proceed only if the condition is met. In the Specify Conditionals dialog box, click Add Condition to add a condition. Click Save to save your conditional values.
  • Click to specify a register for a Play. Specifying registers allows you to use the output of a previous Play as the input for another Play. Click Save to save your registers.
  • (Optional) Rename the Plays if you want them to be displayed with different names during the Playbook execution.

Click Next to continue.

Step 5

In the Review Playbook window, review the Plays in your Playbook. Enter relevant values for the Playbook details fields. You can click Field Help icon for more information about each field.

Note

 
For the Software Platform(s) field, make sure to use the exact software type name as it is mentioned in Device Management > Network Devices > Software Type column.

Step 6

(Optional) To set the Labels, click Select and perform one of the following, as applicable:

  • Select the applicable label and click Done.
  • Click Create, enter relevant values for Label and Roles, and click Save. Select the new label and click Done.

Note

 

The labels determine which user or roles can run which Playbooks.

For more information on assigning Playbooks to specific roles, see Assign Playbooks to Specific Roles.

Step 7

(Optional) After you enter the relevant details, click Dry Run to validate the parameters. A dialog box opens displaying the Playbook Details.

Note

 
Dry run does not commit the changes but just provides a platform to validate whether or not the Playbook would work with the parameters you entered.

Step 8

Click Create to create the Playbook.

The Playbook List window opens displaying your new custom Playbook in the list.

Export Playbooks

You can export any Playbook as a gzipped tar archive. This includes any Cisco-supplied Playbook, as well as custom Playbooks that you or another party have authored and have imported into Cisco Crosswork Change Automation.

The exported archive will contain only the user-customizable files listed in Playbook Components and Files. The archive will contain one or more .pb files (for example: router_config_bgp_rd.pb for the Playbook code), which are parsed and processed at the backend.

You can edit the exported files as needed, following the guidelines in the "Custom Playbooks" tutorial in the Change Automation Developer Guide on Cisco DevNet. You can then import them as explained in Import Playbooks.

Your user ID must have Change Automation read permission to export Playbooks.

Procedure

Step 1

From the main menu, choose Network Automation > Playbook List.

Step 2

(Optional) In the Playbook List window, filter the table as needed.

Step 3

Check the check boxes for the Playbooks you want to export. Check the check box at the top of the column to select all of the Playbooks for export.

Step 4

Click Export icon. Your browser will prompt you to select a path and the file name to use when saving the gzipped tar archive. Follow the prompts to save the file.

Import Playbooks

You can import any custom Playbook, provided it meets the following requirements:

  • The Playbook files must be packaged as a gzipped tar archive.

  • The archive must contain a .pb file, at minimum.

  • The archive file must have a unique name.

The individual files included in the archive must meet the additional validation requirements described in the "Custom Playbooks" tutorial in the Change Automation Developer Guide on Cisco DevNet.


Note

While you cannot overwrite a Cisco-supplied Playbook, you can overwrite a custom Playbook. The system will warn you when you are about to overwrite a custom Playbook, but will not prevent you from doing so. Take precautions to ensure that you do not overwrite your custom Playbooks accidentally.

You cannot re-import an exported Cisco-supplied Playbook with the same name as the original.

Before you begin

A user must have the write access to import Playbooks. For more information about granting read-writer role access to a user, see Assign proper level of access to other users in Configure Change Automation Settings.

Procedure

Step 1

From the main menu, choose Network Automation > Playbook List.

Step 2

Click Import icon. Your browser will prompt you to browse to and select the gzipped archive file containing the Playbooks you want to import.

Make sure there is no existing Playbook with the same name as the Playbook you intend to import, unless it is your intent to overwrite the existing Playbook.

If you are creating an improved version of a Playbook, it is recommend to use a version number or other indicator to make sure that the name is unique and does not overwrite the original Playbook until the replacement is completely tested.

Step 3

Follow the prompts to import the archive file.

Delete Custom Playbooks

You can delete user-defined Playbooks only. You cannot delete a Cisco-supplied Playbook.

Your user ID must have Change Automation delete permission to delete Playbooks.

Procedure

Step 1

From the main menu, choose Network Automation > Playbooks List.

Step 2

In the Playbooks List window, select the custom Playbook that you want to delete.

Step 3

Click Delete icon icon.

Step 4

When prompted, click Delete again to confirm.

Assign Playbooks to Specific Roles

This section gives the procedure to assign Playbook labels to specific roles for them to run and import the Playbooks with that particular label. Admin users can enable other users to run Playbooks with a specific label.

Before you begin

If required, create a new user to whom you would want to assign the Playbook. For more information, see Section: Create User Roles in the Cisco Crosswork Infrastructure 4.1 and Applications Administration Guide.

Procedure

Step 1

Go to Administration > Users and Roles > Roles > Playbook.

Step 2

Under Roles pane, select the role to whom you would want to assign the Playbook labels.

Step 3

Enable Permissions checkboxes for the Playbook Label(s) you would want to assign.

About Running Playbooks

A user must have the permission to run the Playbooks with particular Playbook label. For more information on assigning Playbooks to specific roles, see Assign Playbooks to Specific Roles.

Running any Playbook consists of five steps:

  1. Select the Playbook you want to run (see View the Playbook List).

  2. Select the device or devices you want to run it on.

  3. Enter the appropriate runtime parameters you want the Playbook to apply.

  4. Select the execution mode you want to use:

    1. Perform a Dry Run of a Playbook, where you can see what the Playbook will do before you commit to making changes to the network.

    2. Run Playbooks In Single Stepping Mode, so you have a chance to pause after each Playbook check or action, and roll back changes you did not intend.

    3. Run Playbooks In Continuous Mode and apply the changes immediately.

    While selecting the execution mode, you can also choose to:

    • Schedule Playbook Runs for another calendar date or time.

    • Collect syslogs during and after the run. Syslog collection is available only when running the Playbook in single-stepping or continuous execution mode, and only if you have already configured a syslog storage provider.

    • Specify a Failure Policy, where you decide what the system should do in case a failure occurs at any time during the Playbook run.

  5. Confirm your settings and run the Playbook in the execution mode you selected.

Depending on their complexity and on network factors, some Playbooks may take a lot of time to run. At any time during and after completion of a run, you can view the run details and status. If the Playbook is still running, you can also choose to abort it. For details, see View or Abort Playbook Jobs.

Playbook Execution Order

When it is running, every Playbook conducts checks and configuration changes in four phases, which correspond to sections of the Playbook code (identified using the tags discussed in Playbook Components and Files):

  1. Pre-Maintenance—This phase of the Playbook includes non-disruptive checks and any other operations on the device that prepare it for potentially traffic-impacting changes. For example:

    • Take snapshots of various routing protocol states.

    • Take snapshots of memory, CPU, and system health parameters.

    • Validate the capacity (storage, memory) on active and standby routers for the new software patch upgrade.

  2. Maintenance—This phase of the Playbook includes any task that may disrupt traffic flowing through the router or impact neighboring routers. For example:

    • Cost out the router and wait until traffic drains out completely.

    • Verify that the redundant router is healthy and carrying traffic.

    • Perform the upgrade procedure on the device.

    • Reconfigure the device(s) to support a new configuration or feature.

  3. Post-Maintenance—This phase of the Playbook includes verification tasks to perform on the router after any disruptive operation. For example:

    • Verify that the current state matches the desired state.

    • Cost in the router and wait for traffic to return to normal levels.

  4. Continuous—In addition to the three serial phases already described, Change Automation can also run check tasks that span the entire duration of Playbook execution. These tasks check the state of the router while the Playbook is being deployed, and cancel the Playbook execution if any catastrophic or undesirable state change occurs. The checks in the Playbook may also monitor a neighboring router to guarantee that there are no second-order failures in the network while the changes are being deployed.

Perform a Dry Run of a Playbook

A dry run lets you view configuration changes that the Playbook will send to the device, without performing the actual commit of the changes, as you would with a run in the single-stepping or continuous execution modes.

It is a best practice to perform a dry run and verify the configuration changes before you deploy those changes to the router. If the dry run fails, you may want to debug its parameter values using another dry run. You can also debug by performing a single-stepping run, which will allow you to abort and rollback changes after one or more of the plays, instead of only at the end, as part of a continuous run's Failure Policy.

Note that dry run mode is intended for use only with Playbooks that perform actual device configuration changes via Cisco NSO. See the "Playbooks" and "Plays" references in the Change Automation Developer Guide on Cisco Devnet for details on Playbooks that do not support dry run mode. These will include, for example, Node state snapshot, Install optional package or SMU, and Uninstall optional package or SMU.

Procedure

Step 1

From the main menu, choose Network Automation > Run Playbook.

Step 2

In the Select Playbook list on the left, click on the Playbook you want to dry run. On the right, the window displays the Playbook name, hardware and software compatibility information, and descriptions for all the plays in the selected Playbook.

Step 3

Click Next. The Select Devices window appears. Using this window:

  • You can toggle between the table view and topology map view by clicking and selecting the relevant option on the drop-down button on the upper left corner of the window. Choose Select Devices From List or Select Devices From Map to select the table view or topology map view respectively. By default, the table view is displayed.

  • With the topology map view displayed, you can toggle between the map's geographical and logical views by clicking on the Geographical View icon or the Logical View icon. You can also zoom, display bandwidth utilization, and change logical view layouts as you do with the topology map you see when you select Topology from the main menu.

  • You can select the devices using Static or Dynamic using Tags device selection options. Static selection allows you to select devices from the list using quick and advanced filters and filter by tags on the left. Dynamic using Tags selection targets you to select the relevant tag intead of devices from the table on the left side, and all devices associated with the relevant tag are selected. Hover the mouse pointer over the Info icon icon next to the options for more information. You can also view the selection criteria such as number of devices required for the selected playbook.

  • In Static selection mode, you can check the Allow Bulk Jobs check box to select multple devices and run the selected playbook on them at the simultaneously. Based on your selection, the system creates a static group of multiple jobs. Hover the mouse pointer over the Info icon icon next to the check box for more information. There is no limitation on the number of devices you can select for a bulk job.

    Note

     

    Allow Bulk Jobs option is enabled for playbooks that can be executed on a single device.

Step 4

The Select Devices window will prompt you to select one or more of the devices shown (depending on the Playbook). Click on the devices you want to select, then click Next. The Parameters window appears.

Step 5

In the fields provided in the Parameters window, enter the Playbook parameter values to use for this dry run.

With the Parameters window displayed, you can also:

  • Click JSON to enter the parameter values in JSON format. A popup text window displays the full list of JSON parameters, with empty values in quotes. Edit the values and, when you are finished, click Save.

  • Click Playbook Import JSON icon to upload a JSON file with the parameter values you want. You will be prompted to navigate to the JSON parameters file you have previously prepared (or downloaded from a previous Playbook run) and then upload it as appropriate for your browser and operating system.

  • Click + Add to add additional instances of a particular parameter, if required for the Playbook you are running. Click X Remove to delete instances added in this way.

  • Click Playbook Clear icon to clear all the parameter values entered so far.

Step 6

With the parameter values set, click Next. The Execution Policy window appears.

Step 7

Choose Dry Run and click Next. The Review your Job window appears, displaying a summary of all of your choices: playbook, devices, parameters, and execution policy. In this window:

  • You must provide a relevant Name for the job.

  • You can assign tags to your job. Click New Job Tag, provide a name and color and save your settings to create your own tag. You can also select from the list of existing job tags by clicking the corresponding checkboxes. Click Manage Job Tags to create, edit or delete job tags.

  • You can click on any of the Change links in the Review your Job window summary to modify your choices.

Step 8

(Optional) Enter the device credentials (name and password).

Note

 

This step is applicable only if Credential Prompt is enabled in the Change Automation settings. For more information, see Configure Change Automation Settings.

Step 9

When you are ready to continue, click Run Playbook.

Step 10

At the confirmation prompt, click Confirm. The Execution Mode window is displayed.

Step 11

After the dry run is complete:

  • Click the Dry Run tab and verify the configuration changes that would be pushed to the device had this not been a dry run. This tab will display a no config change message if no changes would have been made. Please note that this tab shows only cumulative configuration changes, not each individual change made. For example, if a Playbook configures set-overload-bit in one step and then unconfigures it using no set-overload-bit in a later step, the tab will show no config change.

  • Click the Events tab to see success and failure messages for each step of the Playbook. This can help you diagnose and correct problems with individual plays and the run as a whole.

  • Click the Console tab to see messages that are generated during the run.

As syslog collection is disabled for drying runs, the Syslog tab will contain only a message stating that.

Step 12

(Optional) If you want to perform a single-step debugging run, or are ready to commit the changes to the device, click Execute Now. The Execution Policy window will display, with all of your parameter values from the dry run pre-filled.

Run Playbooks In Single Stepping Mode

Single-stepping execution mode is a handy way to test a custom or modified Playbook, or diagnose problems with a pre-packaged Playbook that is not giving you the results you want. Unlike a dry run, a single-stepping execution commits configuration changes to the device as the Playbook runs. However, you can set breakpoints on or pauses after any Maintenance or Post-Maintenance action in the Playbook. Please note that, while you can set breakpoints on Pre-Maintenance actions, doing so will have no effect, and these actions will not pause.

Whenever the Playbook hits a breakpoint, it will stop, and will not continue until you issue the command to proceed. At each pause, you can also choose to abort the entire run and roll back all changes made, or rollback to any previous play.

Procedure

Step 1

From the main menu, choose Network Automation > Run Playbook.

Step 2

In the Select Playbook list on the left, click on the Playbook you want to run. On the right, the window displays the Playbook name, hardware and software compatibility information, and descriptions for all the plays in the selected Playbook.

Step 3

Click Next. The Select Devices window appears. Using this window:

  • You can toggle between the table view and topology map view by clicking and selecting the relevant option on the drop-down button on the upper left corner of the window. Choose Select Devices From List or Select Devices From Map to select the table view or topology map view respectively. By default, the table view is displayed.

  • With the topology map view displayed, you can toggle between the map's geographical and logical views by clicking on the Geographical View icon or the Logical View icon. You can also zoom, display bandwidth utilization, and change logical view layouts as you do with the topology map you see when you select Topology from the main menu.

  • You can select the devices using Static or Dynamic using Tags device selection options. Static selection allows you to select devices from the list using quick and advanced filters and filter by tags on the left. Dynamic using Tags selection allows you to select the relevant tag from the table on the left side, and all devices associated with the relevant tag are selected. Hover the mouse pointer over the Info icon icon next to the options for more information. You can also view the selection criteria such as number of devices required for the selected playbook.

  • In Static selection mode, you can check the Allow Bulk Jobs check box to select multple devices and run the selected playbook on them at the simultaneously. Based on your selection, the system creates a static group of multiple jobs. Hover the mouse pointer over the Info icon icon next to the check box for more information. There is no limitation on the number of devices you can select for a bulk job.

    Note

     

    Allow Bulk Jobs option is enabled for playbooks that can be executed on a single device.

Step 4

The Select Devices window will prompt you to select one or more of the devices shown (depending on the Playbook). Click on the devices you want. Click Next.

Step 5

Click Next. The Parameters window appears.

Step 6

In the fields provided in the Parameters window, enter the Playbook parameter values to use for this run.

With the Parameters window displayed, you can also:

  • Click JSON to enter the parameter values in JSON format. A popup text window displays the full list of JSON parameters, with empty values in quotes. Edit the values and, when you are finished, click Save.

  • Click Playbook Import JSON icon to upload a JSON file with the parameter values you want. You will be prompted to navigate to the JSON parameters file you have previously prepared (or downloaded from a previous Playbook run) and then upload it as appropriate for your browser and operating system.

  • Click + Add to add additional instances of a particular parameter, if required for the Playbook you are running. Click X Remove to delete instances added in this way.

  • Click Playbook Clear icon to clear all the parameter values entered so far.

Step 7

With the parameter values set, click Next. The Execution Policy window appears.

Step 8

Choose Single Stepping. The Execution Policy window displays additional features to customize the job:

  • Under Collect Syslogs, click Yes if you want syslogs to be collected during and immediately after the run, No if you do not. Yes is the default selection only if you have a syslog provider configured.
  • From the Failure Policy dropdown, select:

    • Abort to abort the entire run, without rolling back any changes, if a failure occurs at any point. This is the default. Any configuration changes made up to the point of failure will not be rolled back.

    • Pause to pause the run and allow you to decide how to handle the failure. This pause will be in addition to any breakpoints you set using the Single stepping breakpoints dropdown.

    • Complete Roll Back to abort the entire run and roll back all configuration changes made.

  • In the Schedule area, uncheck the default Run Now selection to schedule the job for a later time. See Schedule Playbook Runs for help on using the Schedule area features

Step 9

From the Single stepping breakpoints dropdown, select either

  • Every step to pause automatically after every step in the Playbook.
  • Customize to select the steps where you want the Playbook to pause.

If you select Customize, the Customize Breakpoints popup displays a list of all the plays in the Playbook, with a Playbook Pause iconat the step between each play. Click the Playbook Pause icon at each step where you want to set a breakpoint. When you are finished, click Done.

Step 10

Click Next. The Review your Job window appears, displaying a summary of all of your choices: playbook, devices, parameters, and execution policy. In this window:

  • You must provide a relevant Name for the job.

  • You can assign tags to your job. Click New Job Tag, provide a name and color and save your settings to create your own tag. You can also select from the list of existing job tags by clicking the corresponding checkboxes. Click Manage Job Tags to create, edit or delete job tags.

  • You can click on any of the Change links in the Review your Job window summary to modify your choices.

Step 11

(Optional) Enter the device credentials (name and password).

Note

 

This step is applicable only if Credential Prompt is enabled in the Change Automation settings. For more information, see Configure Change Automation Settings.

Step 12

When you are ready to continue, click Run Playbook.

Step 13

At the confirmation prompt, click Confirm. The Automation Job History window is displayed, with the details of the current job displayed on the right side. The job details include information such as job status, job set tags, title of selected playbook, execution parameters and policy, last updated date and update comments (if any). Click the Details icon icon next to the detail to view more information.

Step 14

While the run is executing, the blue Running tile at the top of the window will change to Paused for each step at which you have set a breakpoint. Your choices at each pause will be displayed as buttons below the blue tiles:

  • Click Resume to resume running from this point, with no changes. The Resume request includes the runtime parameters from the previous step; you can edit these, as needed, later.
  • Click Roll Back to roll back any changes made so far. You can choose how far to rollback:

    • Click Complete Roll Back to roll back all changes to the start of the Playbook run. Once you have rolled back to the start, you can choose to Resume from that point, Abort the run entirely, or Edit runtime parameters of the run.

    • Click Select Roll Back Point to roll back changes to the step you select. All the previous steps will then have a roll back point icon displayed next to them: Playbook Roll Back Point icon. Click the Playbook Roll Back Point icon for the step to which you want to roll back. Once you have selected the step, you can choose to Resume from that step, Roll Back further, Abort the run entirely, or Edit runtime parameters.

  • Click Abort to abort the run entirely. No changes made will be rolled back.
  • Click Edit runtime parameters to edit the parameters the run is using. You edit using a popup version of the Parameters window, just as you did in step 6. The parameters exposed for editing when resuming are specific to the task being resumed, which means that they are not the same global parameters you defined in step 6. Most of the time, they are a subset of the global parameters. When you are finished, click Apply. You can then choose to Resume execution with the changed parameters.

Step 15

While the run is executing, you can also use the following features of the progress window:

  • View the execution status of each play in the Playbook in the Maintenance play list at the left side of the window. Plays that fail are indicated with a red icon; plays that succeed are indicated with a green icon.

  • See reminders of your choices in the blue Playbook and Devices tiles at the top of the window.

  • See the current status of the run in the blue Running tile at the top of the window.

  • Click View in the Parameters tile to view the run's parameters. While viewing the parameters, you can click Download Parameters to save them in a JSON file. You will be prompted to name and save the file as appropriate for your browser and operating system.

  • Use the network topology in the map at the right side of the window to view the device and its connections to the rest of your network.

Step 16

After the run is complete:

  • Click the Events tab to see success and failure messages for each step of the Playbook. This can help you diagnose and correct problems with individual plays and the run as a whole.

  • Click the Syslogs tab to access syslog messages collected during and immediately after the run. If syslog collection is enabled, the tab will provide a pointer to the path on the syslog storage provider where collected syslogs are stored. If you chose not to collect syslogs, or no syslog storage provider has been configured, this tab will display a message indicating that syslog collection is disabled.

  • Click the Console tab to see relevant commands and responses from the device consoles that took place during the run. These messages can also help with diagnostics.

Run Playbooks In Continuous Mode

Continuous execution mode is the standard way to run Playbooks. Configuration changes are committed to the device during the run, with no checks or delays except those programmed into it for system resets or other purposes. The run continues until it succeeds or fails. If it fails, you can use the run's Failure Policy to abort, rollback all changes made to the device, or pause execution at the failure point.

It is always good practice to perform a dry run and verify the configuration changes before committing to a continuous run (see Perform a Dry Run of a Playbook). You can also run the Playbook in single-stepping mode, which will allow you to pause execution after any play you select, abort and rollback changes as needed, and even change runtime parameters in the middle of the run (see Run Playbooks In Single Stepping Mode).

Procedure

Step 1

From the main menu, choose Network Automation > Run Playbook.

Step 2

In the Select Playbook list on the left, click on the Playbook you want to run. On the right, the window displays the Playbook name, hardware and software compatibility information, and descriptions for all the plays in the selected Playbook.

Step 3

Click Next. The Select Devices window appears. Using this window:

  • You can toggle between the table view and topology map view by clicking and selecting the relevant option on the drop-down button on the upper left corner of the window. Choose Select Devices From List or Select Devices From Map to select the table view or topology map view respectively. By default, the table view is displayed.

  • With the topology map view displayed, you can toggle between the map's geographical and logical views by clicking on the Geographical View icon or the Logical View icon. You can also zoom, display bandwidth utilization, and change logical view layouts as you do with the topology map you see when you select Topology from the main menu.

  • You can select the devices using Static or Dynamic using Tags device selection options. Static selection allows you to select devices from the list using quick and advanced filters and filter by tags on the left. Dynamic using Tags selection allows you to select the relevant tag from the table on the left side, and all devices associated with the relevant tag are selected. Hover the mouse pointer over the Info icon icon next to the options for more information. You can also view the selection criteria such as number of devices required for the selected playbook.

  • In Static selection mode, you can check the Allow Bulk Jobs check box to select multple devices and run the selected playbook on them at the simultaneously. Based on your selection, the system creates a static group of multiple jobs. Hover the mouse pointer over the Info icon icon next to the check box for more information. There is no limitation on the number of devices you can select for a bulk job.

    Note

     

    Allow Bulk Jobs option is enabled for playbooks that can be executed on a single device.

Step 4

The Select Devices window will prompt you to select one or more of the devices shown (depending on the Playbook). Click on the devices you want to select them, then click Next. The Parameters window appears.

Step 5

In the fields provided in the Parameters window, enter the Playbook parameter values to use for this dry run.

With the Parameters window displayed, you can also:

  • Click JSON to enter the parameter values in JSON format. A popup text window displays the full list of JSON parameters, with empty values in quotes. Edit the values and, when you are finished, click Save.

  • Click Playbook Import JSON icon to upload a JSON file with the parameter values you want. You will be prompted to navigate to the JSON parameters file you have previously prepared (or downloaded from a previous Playbook run) and then upload it as appropriate for your browser and operating system.

  • Click + Add to add additional instances of a particular parameter, if required for the Playbook you are running. Click X Remove to delete instances added in this way.

  • Click Playbook Clear icon to clear all the parameter values entered so far.

Step 6

With the parameter values set, click Next. The Execution Policy window appears.

Step 7

Choose Continuous. The Execution Policy window displays additional features to customize the job:

  • Under Collect Syslogs, click Yes if you want syslogs to be collected during and immediately after the run, No if you do not. Yes is the default selection only if you have a syslog provider configured.
  • From the Failure Policy dropdown, select:

    • Abort to abort the entire run, without rolling back any changes, if a failure occurs at any point. This is the default. Any configuration changes made up to the point of failure will not be rolled back.

    • Pause to pause the run and allow you to decide how to handle the failure.

    • Complete Roll Back to abort the entire run and roll back all configuration changes made.

  • In the Schedule area, uncheck the default Run Now selection to schedule the job for a later time. See Schedule Playbook Runs for help on using the Schedule area features

Step 8

Click Next. The Review your Job window appears, displaying a summary of all of your choices: playbook, devices, parameters, and execution policy. In this window:

  • You must provide a relevant Name for the job.

  • You can assign tags to your job. Click New Job Tag, provide a name and color and save your settings to create your own tag. You can also select from the list of existing job tags by clicking the corresponding checkboxes. Click Manage Job Tags to create, edit or delete job tags.

  • You can click on any of the Change links in the Review your Job window summary to modify your choices.

Step 9

(Optional) Enter the device credentials (name and password).

Note

 

This step is applicable only if Credential Prompt is enabled in the Change Automation settings. For more information, see Configure Change Automation Settings.

Step 10

When you are ready to continue, click Run Playbook.

Step 11

At the confirmation prompt, click Confirm. The Automation Job History window is displayed, with the details of the current job displayed on the right side. The job details include information such as job status, job set tags, title of selected playbook, execution parameters and policy, last updated date and update comments (if any). Click the Details icon icon next to the detail to view more information.

Step 12

While the run is executing, the blue Running tile at the top of the window will change to Paused if you chose a Failure Policy of Pause. Your choices will be displayed as buttons below the blue tiles:

  • Click Resume to resume running from this point, with no changes.
  • Click Roll Back to roll back any changes made so far.
  • Click Abort to abort the run entirely. No changes made will be rolled back.

Step 13

While the run is executing, you can also use the following features of the progress window:

  • View the execution status of each play in the Playbook in the Maintenance play list at the left side of the window. Plays that fail are indicated with a red icon; plays that succeed are indicated with a green icon.

  • See reminders of your choices in the blue Playbook and Devices tiles at the top of the window.

  • See the current status of the run in the blue Running tile at the top of the window.

  • Click View in the Parameters tile to view the run's parameters. While viewing the parameters, you can click Download Parameters to save them in a JSON file. You will be prompted to name and save the file as appropriate for your browser and operating system.

  • Use the network topology in the map at the right side of the window to view the device and its connections to the rest of your network.

Step 14

After the run is complete:

  • Click the Events tab to see success and failure messages for each step of the Playbook. This can help you diagnose and correct problems with individual plays and the run as a whole.

  • Click the Syslogs tab to access syslog messages collected during and immediately after the run. If syslog collection is enabled, the tab will provide a pointer to the path on the syslog storage provider where collected syslogs are stored. If you chose not to collect syslogs, or no syslog storage provider has been configured, this tab will display a message indicating that syslog collection is disabled.

  • Click the Console tab to see relevant commands and responses from the device consoles that took place during the run. These messages can also help with diagnostics.

Schedule Playbook Runs

The Change Automation application's Execution Mode window allows you to schedule future Playbook runs as jobs, and view all the jobs that have been scheduled. Use the Schedule area on the left to schedule a job. Use the All Scheduled Jobs area on the right to view scheduled jobs on the calendar.


Note
Playbook Job Scheduling is only available if it was enabled when Change Automation was installed and initially configured. For more information, see Configure Change Automation Settings.

The Execution Mode window's scheduling features are only displayed when you have chosen to run a Playbook in continuous or single-stepping mode. You cannot schedule a dry run of a Playbook.

Figure 4. Execution Mode Scheduling Features
Item Description

1

Run Now: Running Playbooks immediately is the default for continuous and single-stepping execution modes. To schedule a run for a future time and date, you must uncheck this box.

2

Schedule Selectors: Use these fields to select the future time and date when the Playbook runs. Although it is the default for the Pre-Maintenance and Maintenance phases of a scheduled Playbook to start at the same time, you can use the upper Schedule Pre-check and lower Schedule Perform fields to schedule the start of Pre-Maintenance and the start of Maintenance independently. Note that the Schedule Perform time must always be greater than or equal to the Schedule Pre-check time.

3

Previous/Today/Next Selectors: Use these three selectors with the Month/Week/Day selectors to focus the calendar's display of scheduled jobs on the time range in which you are interested. For example: To show only those jobs scheduled for next week, click Next and Week.

4

Job Icons: Red, numbered icons in the squares representing each calendar date show how many jobs are scheduled for that date. Yellow circle icons represent each scheduled job.

5

Job Details Popup: Hover your mouse cursor over a yellow circle icon to see the details for the scheduled job represented by that icon. The popup shows the execution ID of the job and the name of the Playbook to be run.

6

Show jobs for selected devices only: Check this box to restrict the calendar display to only those jobs scheduled to run on the devices you have already selected. This is a handy way to see if the schedule you plan for your Playbook run will conflict with other scheduled jobs on the same devices.

7

Month/Week/Day Selectors: Use these three selectors with the Previous/Today/Nextselectors to focus the calendar's display of scheduled jobs on the time range in which you are interested. For example: To show only those jobs scheduled for last month, click Last and Month.


Note

Crosswork Change Automation Playbooks have a mop_timeout parameter, which is a user specified input needed to schedule any Playbook.

If you are scheduling a Playbook with Failure Policy set to Complete Roll Back, first dry run the Play and note the time taken. Then add a buffer time (for example, say 10 minutes) to the time taken during the dry run. After that, double the time value and enter it to the mop_timeout parameter as it can possibly take as much time to roll back the Playbook as it takes to run it until the last step. Without sufficient mop_timeout, the Playbook can end up in an incomplete state (in between transitions) if the timeout gets triggered while roll back is in progress. If this happens, you will have to manually revert the changes done or you will have to create a Playbook with the changes that you want to revert.

View or Abort Playbook Jobs

The Automation Job History window lets you click on any individual job in the list to see that job's detailed execution progress panel, which displays the name of the Playbook, its plays, the devices it ran on, the parameters used, and all event, syslog, console and other messages. These details are useful when diagnosing failures.

The Automation Job History window also allows you to abort running jobs.


Note

You can also navigate to Automation Job History window from theJobs panel in the Change Automation Dashboard.

Before you begin

A user must have the permission for specific Playbook label to run or abort a Playbook. For more information on assigning Playbooks to specific roles, see Assign Playbooks to Specific Roles.

Procedure

Step 1

From the main menu, select Network Automation > Automation Job History. The Automation Job History window is displayed with a list of Job Sets.

Job History window

The list in Automation Job History window is sorted by the last update time, with running or most recently executed jobs at the top. You can apply quick or advanced filters to the table as you would with columns in other table windows.

Step 2

To view information about a specific Playbook job, click the relevant job ID checkbox on the left. The job's status and execution details are displayed on the right side. Click on the Details icon icon next to each detail to get more information about the selected job set.

Step 3

You can abort a job set in running, paused or scheduled status, as follows:

  • To abort a specific job, click the check box next to it and then click Abort Selected.
  • To abort all jobs immediately, click Abort All.
When prompted, click Confirm. Jobs that are currently running, paused, or scheduled will abort once the current task has run to completion.

Troubleshoot Change Automation

The following table describes issues you may encounter when using the Change Automation application, and their solutions or workarounds.

Table 1. Change Automation Troubleshooting

Issue

Solution

Playbook run fails with messages indicating that Cisco Network Services Orchestrator (Cisco NSO) and the target device are out of sync or otherwise out of communication. Message text will vary, but may include "device out of sync", "NC client timeout", and other text indicating that there are connectivity or sync issues between Cisco NSO and the device.

Ensure that the Playbook does not include a sync operation. Get the device and Cisco NSO back in sync and then re-run the Playbook.

Alternatively, you can create a new Playbook that includes a sync operation to avoid future problems.

"Failed to end NSO transaction, 500:fatal:YClientError: Failed to send RPC:" error is displayed while running the playbook.

Include the below settings in the Cisco NSO configuration file (ncs.conf): 

<ssh>
<client-alive-interval>infinity</client-alive-interval> 
<client-alive-count-max>5</client-alive-count-max>
</ssh>

Note

 

This configuration could increase the load on Cisco NSO. So, it is better to do this only when necessary.

Playbook aborted due to failure in locking the device nodes.

In the Devices window, select the relevant devices and clear the lock by moving the device to DOWN and then UP. Go to Administration > Crosswork Manager, click the Change Automation tile and restart the robot-nca process. Once the protocols are reachable, you can schedule to run a new playbook.