Step 1

Download and verify the configuration validation script.

1. Ensure that you have Python installed on your local machine.

   The script requires Python 3 to run. You can check if Python is installed on your machine using the following command:

   $ python3 --version
   Python 3.9.6

2. Download and extract the validation script tarball.

   Navigate to https://software.cisco.com/download/home/285968390/type/286317465, select the target NDO version to which you want to upgrade, download the upgrade validation script (Final_ndo<version>-UpgradeValidationScript.tgz), then extract it, for example:

   $ tar -xzf Final_ndo<version>-UpgradeValidationScript.tgz
   $ ls
   ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM
   Final_ndo4.1.2h-UpgradeValidationScript.tgz
   UpgradeValidationScript.tgz
   UpgradeValidationScript.tgz.signature
   cisco_x509_verify_release.py3

3. Verify the validation script tarball signature.

   You can use the following command to verify the Cisco signature on the configuration validation script.

   $ ./cisco_x509_verify_release.py3 -e ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM \
     -i UpgradeValidationScript.tgz -s UpgradeValidationScript.tgz.signature -v dgst -sha512
   Retrieving CA certificate from https://www.cisco.com/security/pki/certs/crcam2.cer ...
   Successfully retrieved and verified crcam2.cer.
   Retrieving SubCA certificate from https://www.cisco.com/security/pki/certs/innerspace.cer ...
   Successfully retrieved and verified innerspace.cer.
   Successfully verified root, subca and end-entity certificate chain.
   Successfully fetched a public key from ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM.
   Successfully verified the signature of UpgradeValidationScript.tgz using ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM

   Note

   If signature verification fails, you will receive the following error: $ ./cisco_x509_verify_release.py3 -e ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM\ -i UpgradeValidationScript.tgz -s UpgradeValidationScript.tgz.signature.fail -v dgst -sha512 Retrieving CA certificate from https://www.cisco.com/security/pki/certs/crcam2.cer ... Successfully retrieved and verified crcam2.cer. Retrieving SubCA certificate from https://www.cisco.com/security/pki/certs/innerspace.cer ... Successfully retrieved and verified innerspace.cer. Successfully verified root, subca and end-entity certificate chain. Successfully fetched a public key from ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM. Error log: Failed to verify dgst signature of UpgradeValidationScript.tgz. Error log: Verification Failure In this case, we recommend you re-download the <ndo-version>-UpgradeValidationScript.tgz tarball from the Cisco Software Download portal.

4. Once the validation script signature is verified, extract the script itself.

   % tar -xzf UpgradeValidationScript.tgz
   $ ls
   ACI_4070389ff0d61fc7fbb8cdfdec0f38f30482c22e.PEM
   Final_ndo4.1.2h-UpgradeValidationScript.tgz
   README.md
   UpgradeValidationScript.tgz
   UpgradeValidationScript.tgz.signature
   cisco_x509_verify_release.py3
   ndo
   ndoCmd.py
   ndoCopy.py
   requirements.txt

Step 2

Validate your existing configuration before you create the backup.

1. Log in to your Nexus Dashboard and open the Nexus Dashboard Orchestrator service.

2. Download the tech support logs from your existing Orchestrator.

   While for the migration you will create and download the configuration backup using the standard procedure, the validation is done on the tech support information. Note that it is normal for the tech support archive to be significantly larger than your typical configuration backup.

   You can generate the tech support logs by navigating to Admin > Tech Support page in the Orchestrator UI. Then click the Download icon in the System Logs tile. This downloads the msc_report_<date>.zip archive to your machine.

3. Extract the tech support archive you downloaded.

   The tech support archive comes in a standard .zip format, so you can use any tool of your choice to extract the contents, for example:

   $ unzip msc_report_<date>.zip

   After you extract the archive, copy the msc-db-json-<date>_temp.tar.gz file inside into the directory where you extracted the validation script.

4. Run the validation script.

   The script requires a number of dependencies, which are all defined in the requirements.txt file that comes with the script, so we recommend creating a Python virtual environment before installing the dependencies and running the script:

   $ python -m venv ndo-upgrade
   $ source ndo-upgrade/bin/activate
   $ pip install -r requirements.txt

   After the virtual environment is set up and the required modules are installed, run the script using the tech support file you downloaded and extracted in a previous step, for example:

   • -f allows you to provide the file on which to run the validation.

   • -N specifies that no configuration will be deployed to any live system.

   • -C generates the JSON-formatted output at the end of the script.

   (ndo-upgrade)ndoCmd $ ./ndoCopy.py -f msc_report_20220617_181529/msc-db-json-20220617181553_temp.tar.gz -N -C
   11:49:56 Loading collection site2...4
   11:49:56 Loading collection tenant...12
   [...]
   11:49:56 Checking template versions
   11:49:56 Checking policy deployment dependencies
   11:49:56 Fixing template policy flow loops
   11:49:56 Fixing template dependency loops
   11:49:56 Fixing policies for upgrade
   11:49:56 Determine template ordering
   11:49:56 Analysis completed
   {
     "summaryStats": {
       "appTemplatePoliciesConverted": 139,
       "appTemplateSiteAssocMods": 7,
       "appTemplatePolicyEvictions": 2,
       "appTemplateSchemasConverted": 11,
       "appTemplatesConverted": 38,
       "appTemplatesCreated": 1,
       "tenantMods": 1
     },
   [...]
   }

   After the output is generated:

   • If there are no errors or warnings blocks at the end of the generated JSON, then your configuration is compliant with the migration requirements and you can proceed to the "Back up existing deployment configuration" step.

   • If there is only a number of warnings but no errors, it means the migration will complete successfully, but there's a number of things that you may want to resolve before or after the upgrade. We recommend reviewing any warnings before continuing with the next step.

       "warnings": [
         "dropped DHCP Relay policy dhcp-tn-epgOnRL-policy: invalid provider ip address: 141.1.141.2/24",
         "dropped Route Map policy sameContract: fromPrefixLen and toPrefixLen must be larger than prefix",
         "dropped Mulicast Route Map policy mCastRt.map: invalid RP ip: 12.13.14.15/23",
         "dropped DHCP Option policy dhcpBdMso-option: duplicate option id: 1; duplicate option id: 1",
         "removed dhcpLabels.0 from bd[tn-epgOnRL::Template 1::bdDhcpClient] for 
     unresolved policy ref key[dhcpRelayPolicies::tn-epgOnRL::dhcp-tn-epgOnRL-policy]",
         "removed dhcpLabels.0 from bd[dhcp-msite-mso::bd::bd-client-l3out] for 
     unresolved policy ref key[dhcpRelayPolicies::dhcp-msite-mso::dhcp-msite-mso-relay-policy-epg-cleint-l3out]"
       ], 

   • If there is 1 or more errors listed in the JSON, the migration would fail if you continue with the current configuration.

     Note	You must resolve any existing errors before creating the backup and proceeding with the upgrade. We recommend re-running the validation script after you resolve any existing errors to ensure that the backup will be ready for the migration.

     For example, the following sample shows 2 possible errors that can come up during validation:

     "errors": [
       "template appTemplate[<template>] version 6 is in state EDIT_CONFIG",
       "deployed policy bd[<bd>] requires vrf[<vrf>] which is not deployed",
     ] 

     • As mentioned in the Prerequisites and Guidelines section, any deployed templates must not have undeployed changes. You must either deploy the latest version of that template or revert to the deployed version (so it is the latest version) and re-deploy the template.

     • Objects must be deployed in order of their dependencies. In other words, you must not have a deployed bridge domain if the required VRF is not deployed.

5. Resolve any shown errors and repeat this step to re-validate the configuration.

Step 3

Back up existing deployment configuration.

1. From the left navigation pane, select Operations > Backups & Restore.

2. In the main window, click New Backup.

   A New Backup window opens.

3. In the Name field, provide the name for the backup file.

   The name can contain up to 10 alphanumeric characters, but no spaces or underscores (_).

4. From the Remote Location dropdown, choose the remote location you have configured previously.

5. In the Remote Path field, provide the path on the remote server where to store the backup.

6. Click Save to create the backup.

Step 4

Download the backup file.

Step 1

Deploy a new Nexus Dashboard cluster.

Step 2

After Nexus Dashboard and Nexus Dashboard Orchestrator have been successfully deployed, navigate to your Nexus Dashboard’s Admin Console.

Step 3

Onboard the fabrics managed by the Orchestrator service to Nexus Dashboard.

1. From the main navigation menu, choose Sites.

2. In the main pane, click Add Site.

3. Choose the type of site you want to on-board and provide the site's information, such as controller's IP address, username, and password.

   • For on-premises sites managed by Cisco APIC, choose ACI.

   • For cloud sites managed by Cloud Network Controller (previously Cloud APIC), choose Cloud Network Controller.

     Use this option for all Cloud APIC sites that were managed by your Orchestrator.

   • For on-premises sites managed by NDFC (previously DCNM), choose NDFC.

     Use this option for all DCNM sites that were managed by your Orchestrator.

   Note	You must use the same site Name as you did when on-boarding the site to your Orchestrator in the past. Adding a site with a different name will cause configuration restore to fail.

4. Click Save to add the site.

5. Wait for site to come up and show as Up in the Nexus Dashboard UI.

6. Repeat this step for all sites that were previously managed by your Orchestrator.

Step 4

Install Nexus Dashboard Orchestrator service.

1. In the Services screen, select the App Store tab.

2. In the Nexus Dashboard Orchestrator tile, click Upgrade.

3. In the License Agreement window that opens, click Agree and Download.

1. Browse to the Nexus Dashboard Orchestrator page on DC App Center:

   https://dcappcenter.cisco.com/nexus-dashboard-orchestrator.html

2. From the Version dropdown, choose the version you want to install and click Download.

3. Click Agree and download to accept the license agreement and download the image.

4. From the left navigation menu in Nexus Dashboard, select Services.

5. In the Nexus Dashboard's Services screen, select the Installed Services tab.

6. From the Actions menu in the top right of the main pane, select Upload Service.

7. In the Upload Service window, choose the location of the image

   If you downloaded the application image to your system, choose Local.

   If you are hosting the image on a server, choose Remote.

8. Choose the file.

   If you chose Local in the previous substep, click Select File and select the app image you downloaded.

   If you chose Remote, provide the full URL to the image file, for example http://<ip-address>:<port>/<full-path>/cisco-mso-<version>.nap.

9. Click Upload to add the app to the cluster.

   It may take a few minutes for the image to be uploaded to the cluster and initialized.

Step 5

Wait for the new image to initialize.

Step 6

In the Nexus Dashboard Orchestrator tile, click Enable.

Step 7

Launch the Orchestrator service.

Step 1

Ensure that the new Nexus dashboard cluster is up and running and the NDO service is installed.

Step 2

Open your new Nexus Dashboard Orchestrator service.

Step 3

Add remote location for configuration backups.

1. From the left navigation pane, select Admin > Backup & Restore.

2. Choose the Remote Locations tab.

3. Choose Create Remote Location.

   The Create Remote Location screen appears.

4. Provide the name for the remote location and an optional description.

   Two protocols are currently supported for remote export of configuration backups:

   • SCP

   • SFTP

   Note	SCP is supported for non-Windows servers only. If your remote location is a Windows server, you must use the SFTP protocol

5. Specify the host name or IP address of the remote server.

   Based on your Protocol selection, the server you specify must allow SCP or SFTP connections.

6. Provide the full path to a directory on the remote server where you will save the backups.

   The path must start with a slash (/) characters and must not contain periods (.) or backslashes (\). For example, /backups/ndo.

   Note	The directory must already exist on the remote server.

7. Specify the port used to connect to the remote server.

   By default, port is set to 22.

8. Specify the authentication type used when connecting to the remote server.

   You can configure one of the following two authentication methods:

   • Password—provide the username and password used to log in to the remote server.

   • SSH Private Files—provide the username and the SSH Key/Passphrase pair used to log in to the remote server.

9. Click Save to add the remote server.

Step 4

Import the backup file to your new Nexus Dashboard Orchestrator cluster.

1. From the left navigation pane, select Operations > Backups & Restore.

2. In the main pane, click Upload.

3. In the Upload to Remote window that opens, click Select File and choose the configuration backup file you created before the upgrade.

4. From the Remote Location dropdown menu, select the remote location.

5. (Optional) Update the remote location path.

   The target directory on the remote server, which you configured when creating the remote backup location, will be displayed in the Remote Path field.

   You can choose to append additional subdirectories to the path. However, the directories must be under the default configured path and must have been already created on the remote server.

6. Click Import to upload the file.

   Importing a backup will add it to the list of the backups displayed the Backups page. Note that even though the backups are shown on the NDO UI, the files are stored only on the remote server and not directly on the cluster nodes.

Step 5

Restore the configuration.

1. In the main window, click the actions (…) icon next to the backup you want to restore and select Rollback to this backup.

   

2. In the Restore from this backup dialog, read the warning and click Restore to confirm that you want to restore the backup you selected.

   The restore process imports the backup and checks for any issues, which may take several minutes to complete. After the initial backup import, you will be prompted for additional validation in the next step, which is required for database upgrades from releases prior to release 4.0(1).

3. After the backup import is complete, ensure there are no failures listed in the report, then click Restore Validation Required to proceed.

   Before the configuration database is updated for this release, the upgrade process performs a number of validations. The validation provides a summary of template and policy changes that will be performed during this final upgrade stage in the next step and includes the following:

   • Implicit template stretching – if one or more objects are implicitly stretched, the upgrade process will create new explicitly-stretched templates and move the objects into those templates.

     For example, if you have a template (t1) that contains vrf1 and is associated to site1 and another template (t2) that contains a BD that references vrf1 but is associated to two sites (site1 and site2), then vrf1 will be implicitly stretched between the two sites.

     This is no longer allowed starting with release 4.0(1) and the VRF must be explicitly stretched to both sites. In such cases during the upgrade, the VRF will be either moved to a different template which will be explicitly stretched between both sites or the original template will be associated with both sites, depending on whether the other policies in that template require stretching as well.

     Any templates that are created in this case will be named UpgradeTemplate%d, where %d is an incrementing number starting with 1 to ensure that all newly added templates are unique.

   • Global policy migration – all global tenant policies (such as DHCP relay or route maps) and fabric policies (such as QoS) will be moved into the new tenant and fabric policy templates that have been added in release 4.0(1).

   Note	At this stage, all the tenants have been imported from the backup and created in NDO, but the schemas and templates will be created in the next step.

4. In the Restore Validation Report window, click Restore and Continue to proceed.

   

   This is the stage of the upgrade where the schemas and the templates present in the backup are imported and recreated in your NDO configuration database according to the current best practices. These schemas and templates are then posted to the local NDO database as if it were a greenfield schema/template creation. Then the newly saved templates are deployed in the correct order that conforms to the current deployment requirements and best practices. The template deployment in this step uses a “local deploy” option to calculate the deployment plan and update the database, but does not send any configuration payload to the sites' controllers.

   The upgrade process also checks for any configuration drifts between the local NDO database (configuration that is correct from NDO's point of view) and what is actually deployed in the fabrics. If this release of NDO supports additional objects or properties compared to the release from which you are upgrading, the upgrade will automatically reconcile those drifts by importing the existing configuration from the site's controller.

   Note that if a template is automatically reconciled, two template versions are created – one before the automatic reconciliation and one after:

   

5. Review the report from the previous substep and click Ok to finish.

   The final stage of the database upgrade presents a full report of the performed actions for you to review. If you close the report but want to review it again, simply click the View Restore Report in the Backups page.

Step 6

Verify that backup was restored successfully and all objects and configurations are present.

1. In the Sites page, verify that all sites are listed as Managed.

2. In the Tenants and Schemas pages, confirm that all tenants and schemas from your previous Nexus Dashboard Orchestrator cluster are present.

3. Navigate to Infrastructure > Site Connectivity and confirm that intersite connectivity is intact.

   In the main pane, click Show Connectivity Status next to each site and verify that the existing tunnels are up and connectivity was not interrupted.

4. In the main pane, click Configure to open Fabric Connectivity Infra screen and verify External Subnet Pool addresses.

   You can view the external subnet pools by selecting General Settings > IPSec Tunnel Subnet Pools tab of the Fabric Connectivity Infra screen and verify that the External Subnet Pools previously configured in Cloud Network Controller have been imported from the cloud sites.

   These subnets are used to address the IPsec tunnel interfaces and loopbacks of the Cloud Routers used for on-premises connectivity and had to be configured directly in the Cloud Network Controller in earlier Nexus Dashboard Orchestrator releases.

Step 1

In your Nexus Dashboard Orchestrator, navigate to Operate > Tenant Templates.

Step 2

Choose the Applications tab.

Step 3

Select the first schema and check its templates for configuration drifts.

Step 4

If the template contains a configuration drift, resolve the conflicts.

1. Close the template deployment dialog to return to the Schema view.

   Deploying any templates at this point would push the values in the Orchestrator database and overwrite any existing settings in the fabrics.

2. From the template's Actions menu, select Drift Reconciliation.

   The Drift Reconciliation wizard opens.

3. In the Drift Reconciliation screen, compare the template-level configurations for each site and choose the one you want.

   Template-level properties are common across all sites associated to the template. You can compare the template level properties defined on Nexus Dashboard Orchestrator with the configuration rendered in each site and decide what should become the new configuration in the Nexus Dashboard Orchestrator template. Selecting the site configuration will modify those properties in the existing Nexus Dashboard Orchestrator template, whereas selecting the Nexus Dashboard Orchestrator configuration will keep the existing Nexus Dashboard Orchestrator template settings as is

4. Click Go to Site Specific Properties to switch to site-level configuration.

   You can choose a site to compare that specific site's configuration. Unlike template-level configurations, you can choose either the Nexus Dashboard Orchestrator-defined or actual existing configurations for each site individually to be retained as the template's site-local properties for that site.

   Even though in most scenarios you will make the same choice for both template-level and site-level configuration, the drift reconciliation wizard allows you to choose the configuration defined in the site's controller at the "Template Properties" level and the configuration defined in Nexus Dashboard Orchestrator at the "Site Local Properties" level or vice versa.

5. Click Preview Changes to verify your choices.

   The preview will display full template configuration adjusted based on the choices picked in the Drift Reconciliation wizard

6. Save the schema.

7. Click Deploy to sites to deploy the configuration and finish reconciling the drift for that template

Step 5

Repeat the above steps for every schema and template in your Nexus Dashboard Orchestrator.

Step 6

Check audit logs to verify that all templates have been re-deployed.