Troubleshooting and Reference

Troubleshooting Upgrade Packages

Table 1. Troubleshooting Upgrade Packages

Issue

Solution

No available upgrades even after I refresh.

You are already running the latest version available for your deployment, and you have no upgrade packages loaded/configured.

Suggested release is not marked.

The suggested release is listed only if you are eligible for it. It is not listed if you are already running the suggested release or higher, or if you cannot upgrade that far. Note that patches to suggested releases are not marked as suggested, although we do recommend you apply them.

I don't see the packages I want.

Only major, maintenance, and patch upgrades that apply to your deployment right now are listed and available for direct download. Unless you manually upload, the following are not listed:

  • Device upgrades (major and maintenance) to a particular version, unless you have a device that supports that version.

  • Device patches, unless you have at least one device at the appropriate maintenance release.

  • Hotfixes. You must manually upload these.

Troubleshooting Threat Defense Upgrade

Table 2. Troubleshooting Threat Defense Upgrade

Issue

Solution

Upgrade button missing for my target version.

Either of:

  • You still need the upgrade package.

  • You do not have anything that can be upgraded to that version right now.

Devices not listed in the upgrade wizard.

If you accessed the wizard directly from Devices > Threat Defense Upgrade, the workflow may be blank.

To begin, choose a target version from the Upgrade to menu. The system determines which devices can be upgraded to that version and displays them in the Device Details pane. Note that the choices in the Upgrade to menu correspond to the device upgrade packages on the management center. If your target version is not listed, click Manage Upgrade Packages to upload it; see Managing Upgrade Packages with the Management Center.

If you have a target version but the wizard still does not list any devices, you have no devices that can be upgraded to that version. If you still think you should see devices here, your user role could be prohibiting you from managing (and therefore upgrading) devices.

Copying upgrade packages from the management center to managed devices times out.

This often happens when there is limited bandwidth between the management center and its devices.

You can configure devices to get upgrade packages directly from an internal web server. Delete the upgrade package from the management center (optional but saves disk space), then re-add the upgrade package except this time specify a pointer (URL) to its location instead. See Copying Upgrade Packages to Devices from an Internal Server.

Unresponsive and Failed Threat Defense Upgrades


Caution

Do not reboot or shut down at any point during upgrade, even if the system appears inactive. You could place the system in an unusable state and require a reimage.

Table 3. Unresponsive and Failed Threat Defense Upgrades

Issue

Solution

Cannot reach the device.

Devices can stop passing traffic during the upgrade or if the upgrade fails. Before you upgrade, make sure traffic from your location does not have to traverse the device itself to access the device's management interface.

You should also able to access the management center's management interface without traversing the device.

Upgrade or patch appears hung/device appears inactive.

If device upgrade status has stopped updating on the management center but there is no report of upgrade failure, you can try canceling the upgrade; see below. If you cannot cancel or canceling does not work, contact Cisco TAC.

Tip: You can monitor upgrade logs on the device itself using expert mode and tail or tailf: tail /ngfw/var/log/sf/update.status.

Upgrade failed.

If an upgrade fails and:

  • The device reverted to its pre-upgrade state (auto-cancel is enabled), correct any issues and try again from the beginning.

  • The device is still in maintenance mode, correct any issues and resume the upgrade. Or, cancel and try again later.

If you cannot retry or cancel, or if you continue to have issues, contact Cisco TAC.

Patch failed.

You cannot cancel in-progress or failed patches. However, if a patch fails early, for example, during validation stages, the device may remain up and running normally. Simply correct any issues and try again.

If a patch fails after the device has entered maintenance mode, check for an uninstaller. If one exists, you can try running it to remove the failed patch; see Uninstall a Threat Defense Patch. After the uninstall finishes, you can correct any issues and try again.

If there is no uninstaller, if the uninstall fails, or if you continue to have issues, contact Cisco TAC.

Upgrade or patch on a clustered device failed, and I want to reimage instead of retrying the upgrade.

If a cluster node upgrade fails and you choose to reimage the node, reimage it to the current version of the control node before you add it back to the cluster. Depending on when and how the upgrade failed, the current version of the control node can be the old version or the target version.

We do not support mixed-version clusters except temporarily during upgrade. Deliberately creating a mixed-version cluster can cause outages.

Tip

 

Remove the failed node from the cluster and reimage it to the target version. Upgrade the rest of the cluster to the target version, then add your reimaged node.

I want to cancel an upgrade.

Canceling reverts the device to its pre-upgrade state. You can cancel failed and in-progress upgrades on the upgrade status pop-up, accessible from the Upgrade tab on the Device Management page. You cannot cancel patches.

If you cannot cancel or canceling does not work, contact Cisco TAC.

I want to retry (resume) a failed upgrade.

You can resume an upgrade on the upgrade status pop-up, accessible from the Upgrade tab on the Device Management page.

If you continue to have issues, contact Cisco TAC.

I want to change what happens when upgrade fails.

Part of the upgrade process is choosing what happens if it fails. This is done with the Automatically cancel on upgrade failure... (auto-cancel) option:

  • Auto-cancel enabled (default): If upgrade fails, the upgrade cancels and the device automatically reverts to its pre-upgrade state. This returns you to normal operations as quickly as possible while you regroup and try again.

  • Auto-cancel disabled: If upgrade fails, the device remains as it is. This allows you to correct any issues and resume the upgrade.

For high availability and clustered devices, auto-cancel applies to each device individually. That is, if the upgrade fails on one device, only that device is reverted.

Traffic Flow and Inspection

Schedule maintenance windows when upgrade will have the least impact, considering any effect on traffic flow and inspection.

Traffic Flow and Inspection for Threat Defense Upgrades

Software Upgrades for Standalone Devices

Devices operate in maintenance mode while they upgrade. Entering maintenance mode at the beginning of the upgrade causes a 2-3 second interruption in traffic inspection. Interface configurations determine how a standalone device handles traffic both then and during the upgrade.

Table 4. Traffic Flow and Inspection: Software Upgrades for Standalone Devices

Interface Configuration

Traffic Behavior

Firewall interfaces

Routed or switched including EtherChannel, redundant, subinterfaces.

Switched interfaces are also known as bridge group or transparent interfaces.

Dropped.

For bridge group interfaces on the ISA 3000 only, you can use a FlexConfig policy to configure hardware bypass for power failure. This causes traffic to drop during software upgrades but pass without inspection while the device completes its post-upgrade reboot.

IPS-only interfaces

Inline set, hardware bypass force-enabled: Bypass: Force

Passed without inspection until you either disable hardware bypass, or set it back to standby mode.

Inline set, hardware bypass standby mode: Bypass: Standby

Dropped during the upgrade, while the device is in maintenance mode. Then, passed without inspection while the device completes its post-upgrade reboot.

Inline set, hardware bypass disabled: Bypass: Disabled

Dropped.

Inline set, no hardware bypass module.

Dropped.

Inline set, tap mode.

Egress packet immediately, copy not inspected.

Passive, ERSPAN passive.

Uninterrupted, not inspected.

Software Upgrades for High Availability and Clustered Devices

You should not experience interruptions in traffic flow or inspection while upgrading high availability or clustered devices. For high availability pairs, the standby device upgrades first. The devices switch roles, then the new standby upgrades.

For clusters, the data security module or modules upgrade first, then the control module. During the control security module upgrade, although traffic inspection and handling continues normally, the system stops logging events. Events for traffic processed during the logging downtime appear with out-of-sync timestamps after the upgrade is completed. However, if the logging downtime is significant, the system may prune the oldest events before they can be logged.

Note that hitless upgrades are not supported for single-unit clusters. Interruptions to traffic flow and inspection depend on interface configurations of the active unit, just as with standalone devices.

Software Revert (Major/Maintenance Releases)

You should expect interruptions to traffic flow and inspection during revert, even in a high availability/scalability deployment. This is because revert is more successful when all units are reverted simultaneously. Simultaneous revert means that interruptions to traffic flow and inspection depend on interface configurations only, as if every device were standalone.

Software Uninstall (Patches)

For standalone devices, interruptions to traffic flow and inspection during patch uninstall are the same as for upgrade. In high availability/scalability deployments, you must explicitly plan an uninstall order that minimizes disruption. This is because you uninstall patches from devices individually, even those that you upgraded as a unit.

Traffic Flow and Inspection for Chassis Upgrades

Upgrading FXOS reboots the chassis. For FXOS upgrades to Version 2.14.1+ that include firmware upgrades, the device reboots twice—once for FXOS and once for the firmware. This includes Version 7.4.1+ chassis upgrades for the Secure Firewall 3100/4200 in multi-instance mode.

Even in high availability or clustered deployments, you upgrade FXOS on each chassis independently. To minimize disruption, upgrade one chassis at a time; see Upgrade Order for Threat Defense with Chassis Upgrade and High Availability/Clusters.

Table 5. Traffic Flow and Inspection: FXOS Upgrades

Threat Defense Deployment

Traffic Behavior

Method

Standalone

Dropped.

—

High availability

Unaffected.

Best Practice: Update FXOS on the standby, switch active peers, upgrade the new standby.

Dropped until one peer is online.

Upgrade FXOS on the active peer before the standby is finished upgrading.

Inter-chassis cluster

Unaffected.

Best Practice: Upgrade one chassis at a time so at least one module is always online.

Dropped until at least one module is online.

Upgrade chassis at the same time, so all modules are down at some point.

Intra-chassis cluster (Firepower 9300 only)

Passed without inspection.

Hardware bypass enabled: Bypass: Standby or Bypass‑Force.

Dropped until at least one module is online.

Hardware bypass disabled: Bypass: Disabled.

Dropped until at least one module is online.

No hardware bypass module.

Traffic Flow and Inspection when Deploying Configurations

Snort typically restarts during the first deployment immediately after upgrade. This means that for management center upgrades, Snort could restart on all managed devices. Snort does not restart after subsequent deployments unless, before deploying, you modify specific policy or device configurations.

Restarting the Snort process briefly interrupts traffic flow and inspection on all devices, including those configured for high availability/scalability. Interface configurations determine whether traffic drops or passes without inspection during the interruption. When you deploy without restarting Snort, resource demands may result in a small number of packets dropping without inspection.

Table 6. Traffic Flow and Inspection: Deploying Configuration Changes

Interface Configuration

Traffic Behavior

Firewall interfaces

Routed or switched including EtherChannel, redundant, subinterfaces.

Switched interfaces are also known as bridge group or transparent interfaces.

Dropped.

IPS-only interfaces

Inline set, Failsafe enabled or disabled.

Passed without inspection.

A few packets might drop if Failsafe is disabled and Snort is busy but not down.

Inline set, Snort Fail Open: Down: disabled.

Dropped.

Inline set, Snort Fail Open: Down: enabled.

Passed without inspection.

Inline set, tap mode.

Egress packet immediately, copy not inspected.

Passive, ERSPAN passive.

Uninterrupted, not inspected.

Time and Disk Space

Time to Upgrade

We recommend you track and record your own upgrade times so you can use them as future benchmarks. The following table lists some things that can affect upgrade time.


Caution

Do not make or deploy configuration changes during upgrade. Even if the system appears inactive, do not manually reboot or shut down. In most cases, do not restart an upgrade in progress. You could place the system in an unusable state and require a reimage. If you encounter issues with the upgrade, including a failed upgrade or unresponsive appliance, see Unresponsive and Failed Threat Defense Upgrades.

Table 7. Upgrade Time Considerations

Consideration

Details

Versions

Upgrade time usually increases if your upgrade skips versions.

Models

Upgrade time usually increases with lower-end models.

Virtual appliances

Upgrade time in virtual deployments is highly hardware dependent.

High availability and clustering

In a high availability or clustered configuration, devices upgrade one at a time to preserve continuity of operations, with each device operating in maintenance mode while it upgrades. Upgrading a device pair or entire cluster, therefore, takes longer than upgrading a standalone device.

Configurations

Upgrade time can increase with the complexity of your configurations.

Components

You may need additional time to perform operating system or virtual hosting upgrades, upgrade package transfers, readiness checks, VDB and intrusion rule (SRU/LSP) updates, configuration deployment, and other related tasks.

Disk Space to Upgrade

You must have enough space on the management center (in either /Volume or /var) for device upgrade packages. Or, you can use an internal server to store them. After you copy upgrade packages to the devices, readiness checks should indicate whether you have enough disk space to perform the upgrade. Without enough free disk space, the upgrade fails.

Table 8. Checking Disk Space

Platform

Command

Management center

Choose System (system gear icon) > Monitoring > Statistics and select the management center.

Under Disk Usage, expand the By Partition details.

Threat defense

Choose System (system gear icon) > Monitoring > Statistics and select the device you want to check.

Under Disk Usage, expand the By Partition details.

Upgrade Feature History

Table 9. 20240808

Feature

Minimum Threat Defense

Details

Threat Defense Upgrade

Chassis upgrade for the Secure Firewall 3100 in multi-instance mode.

7.4.1

For the Secure Firewall 3100 in multi-instance mode, you upgrade the operating system and the firmware (chassis upgrade) separately from the container instances (threat defense upgrade).

New/modified screens:

  • Upgrade the chassis: Devices > Chassis Upgrade

  • Upgrade threat defense: Devices > Threat Defense Upgrade

Generate and download post-upgrade configuration change reports from the threat defense and chassis upgrade wizards.

Any

You can now generate and download post-upgrade configuration change reports from the threat defense and chassis upgrade wizards, as long as you have not cleared your upgrade workflow.

Previously, you used the Advanced Deploy screens to generate the reports and the Message Center to download them. Note that you can still use this method, which is useful if you want to quickly generate change reports for multiple devices, or if you cleared your workflow.

New/modified screens:

  • Devices > Threat Defense Upgrade > Configuration Changes

  • Devices > Chassis Upgrade > Configuration Changes

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Cloud-Delivered Firewall Management Center

Deprecated: Copy upgrade packages ("peer-to-peer sync") from device to device.

7.6.0

You can no longer use the threat defense CLI to copy upgrade packages between devices over the management network. If you have limited bandwidth between the management center and its devices, configure devices to get upgrade packages directly from an internal web server.

Deprecated CLI commands: configure p2psync enable , configure p2psync disable , show peers , show peer details , sync-from-peer , show p2p-sync-status

Table 10. 20240203

Feature

Minimum Threat Defense

Details

Threat Defense Upgrade

Improved upgrade starting page and package management.

Any

A new upgrade page makes it easier to choose, download, manage, and apply upgrades to your entire deployment. The page lists all upgrade packages that apply to your current deployment, with suggested releases specially marked. You can easily choose and direct-download packages from Cisco, as well as manually upload and delete packages.

Patches are not listed unless you have at least one appliance at the appropriate maintenance release (or you manually uploaded the patch). You must manually upload hotfixes.

New/modified screens:

  • System (system gear icon) > Product Upgrades is now where you upgrade devices, as well as manage upgrade packages.

  • System (system gear icon) > Content Updates is now where you update intrusion rules, the VDB, and the GeoDB.

  • Devices > Threat Defense Upgrade takes you directly to the threat defense upgrade wizard.

Deprecated screens/options:

  • System (system gear icon) > Updates is deprecated. All threat defense upgrades now use the wizard.

  • The Add Upgrade Package button on the threat defense upgrade wizard has been replaced by a Manage Upgrade Packages link to the new upgrade page.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Cloud-Delivered Firewall Management Center

Enable revert from the threat defense upgrade wizard.

Any, if upgrading to 7.1+

You can now enable revert from the threat defense upgrade wizard.

Other version restrictions: You must be upgrading threat defense to Version 7.2+.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Cloud-Delivered Firewall Management Center

View detailed upgrade status from the threat defense upgrade wizard.

Any

The final page of the threat defense upgrade wizard now allows you to monitor upgrade progress. This is in addition to the existing monitoring capability on the Upgrade tab on the Device Management page, and on the Message Center. Note that as long as you have not started a new upgrade flow, Devices > Threat Defense Upgrade brings you back to this final wizard page, where you can view the detailed status for the current (or most recently complete) device upgrade.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Cloud-Delivered Firewall Management Center

Suggested release notifications.

Any

The management center now notifies you when a new suggested release is available. If you don't want to upgrade right now, you can have the system remind you later, or defer reminders until the next suggested release. The new upgrade page also indicates suggested releases.

See: Cisco Secure Firewall Management Center New Features by Release

Firmware upgrades included in FXOS upgrades.

Any

Chassis/FXOS upgrade impact. Firmware upgrades cause an extra reboot.

For the Firepower 4100/9300, FXOS upgrades to Version 2.14.1 now include firmware upgrades. Secure Firewall 3100 in multi-instance mode (new in Version 7.4.1) also bundles FXOS and firmware upgrades. If any firmware component on the device is older than the one included in the FXOS bundle, the FXOS upgrade also updates the firmware. If the firmware is upgraded, the device reboots twice—once for FXOS and once for the firmware.

Just as with software and operating system upgrades, do not make or deploy configuration changes during firmware upgrade. Even if the system appears inactive, do not manually reboot or shut down during firmware upgrade.

See: Cisco Firepower 4100/9300 FXOS Firmware Upgrade Guide

Updated internet access requirements for direct-downloading software upgrades.

Any

The management center has changed its direct-download location for software upgrade packages from sourcefire.com to amazonaws.com.

See: Internet Access Requirements

Scheduled tasks download patches and VDB updates only.

Any

The Download Latest Update scheduled task no longer downloads maintenance releases; now it only downloads the latest applicable patches and VDB updates. To direct-download maintenance (and major) releases to the management center, use System (system gear icon) > Product Upgrades.

See: Software Update Automation
Table 11. December 13, 2022

Feature

Minimum Threat Defense

Details

Select devices to upgrade from the threat defense upgrade wizard.

Any

Use the wizard to select devices to upgrade.

You can now use the threat defense upgrade wizard to select or refine the devices to upgrade. On the wizard, you can toggle the view between selected devices, remaining upgrade candidates, ineligible devices (with reasons why), devices that need the upgrade package, and so on. Previously, you could only use the Device Management page and the process was much less flexible.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Management Center

Unattended threat defense upgrades.

Any

The threat defense upgrade wizard now supports unattended upgrades, using a new Unattended Mode menu. You just need to select the target version and the devices you want to upgrade, specify a few upgrade options, and step away. You can even log out or close the browser.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Cloud-Delivered Firewall Management Center

Simultaneous threat defense upgrade workflows by different users.

Any

We now allow simultaneous upgrade workflows by different users, as long as you are upgrading different devices. The system prevents you from upgrading devices already in someone else's workflow. Previously, only one upgrade workflow was allowed at a time across all users.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Management Center

Skip pre-upgrade troubleshoot generation for threat defense devices.

Any

You can now skip the automatic generating of troubleshooting files before major and maintenance upgrades by disabling the new Generate troubleshooting files before upgrade begins option. This saves time and disk space.

To manually generate troubleshooting files for a threat defense device, choose System (system gear icon) > Health > Monitor, click the device in the left panel, then View System & Troubleshoot Details, then Generate Troubleshooting Files.

See: Cisco Secure Firewall Threat Defense Upgrade Guide for Management Center

Auto-upgrade to Snort 3 after successful threat defense upgrade is no longer optional.

Any

Upgrade impact. All eligible devices upgrade to Snort 3 when you deploy.

When you upgrade threat defense to Version 7.3+, you can no longer disable the Upgrade Snort 2 to Snort 3 option.

After the software upgrade, all eligible devices will upgrade from Snort 2 to Snort 3 when you deploy configurations. Although you can switch individual devices back, Snort 2 will be deprecated in a future release and we strongly recommend you stop using it now.

For devices that are ineligible for auto-upgrade because they use custom intrusion or network analysis policies, we strongly recommend you manually upgrade to Snort 3 for improved detection and performance. For migration assistance, see the Cisco Secure Firewall Management Center Snort 3 Configuration Guide for your version.

Combined upgrade and install package for Secure Firewall 3100.

7.3.0

Reimage Impact.

In Version 7.3, we combined the threat defense install and upgrade package for the Secure Firewall 3100, as follows:

  • Version 7.1–7.2 install package: cisco-ftd-fp3k.version.SPA

  • Version 7.1–7.2 upgrade package: Cisco_FTD_SSP_FP3K_Upgrade-version-build.sh.REL.tar

  • Version 7.3+ combined package: Cisco_FTD_SSP_FP3K_Upgrade-version-build.sh.REL.tar

Although you can upgrade threat defense without issue, you cannot reimage from older threat defense and ASA versions directly to threat defense Version 7.3+. This is due to a ROMMON update required by the new image type. To reimage from those older versions, you must "go through" ASA 9.19+, which is supported with the old ROMMON but also updates to the new ROMMON. There is no separate ROMMON updater.

To get to threat defense Version 7.3+, your options are:

Content Updates

Automatic VDB downloads.

Any

The initial setup on the management center schedules a weekly task to download the latest available software updates, which now includes the latest vulnerability database (VDB). We recommend you review this weekly task and adjust if necessary. Optionally, schedule a new weekly task to actually update the VDB and deploy configurations.

New/modified screens: The Vulnerability Database check box is now enabled by default in the system-created Weekly Software Download scheduled task.

Install any VDB.

Any

Starting with VDB 357, you can now install any VDB as far back as the baseline VDB for that management center.

After you update the VDB, deploy configuration changes. If you based configurations on vulnerabilities, application detectors, or fingerprints that are no longer available, examine those configurations to make sure you are handling traffic as expected. Also, keep in mind a scheduled task to update the VDB can undo a rollback. To avoid this, change the scheduled task or delete any newer VDB packages.

New/modified screens: On System (system gear icon) > Updates > Product Updates > Available Updates, if you upload an older VDB, a new Rollback icon appears instead of the Install icon.