The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the Solid State Drive (SSD) lifespan exhaustion behavior on Application Centric Infrastructure (ACI) switch platforms.
If you have landed on this page as a result of an SSD fault raised against Application Policy Infrastructure Controller (APIC) (not a switch node), refer to FN64329.
APIC SSD Replacement documents the APIC SSD replacement procedure.
ACI Leaf and Spine nodes leverage SSDs for storage and event logging. All SSDs, irrespective of the vendor type, have a fixed lifespan based on specific attributes defined by the manufacturer. Some examples include the number of writes and erases as well as the volume of data transferred to the drives over a period of time. The SSD lifespan on Leaf and Spine nodes can become exhausted due to regular wear and tear. If the user does not take proper action against a switch that has an SSD approaching exhaustion, the switch could crash due to the SSD going into read-only mode. For this reason, it is imperative to prioritize SSD replacement in the event the F3073 fault is raised against a given node. SSD are field replaceable by Cisco certified Field Engineer.
F3073 was implemented in releases 2.1(4), 2.2(4), 2.3(1o), and 3.1(2m). Any subsequent software versions have the functionality to raise F3073.
Note: If you run an ACI software version that does not have SSD monitoring functionality, it is strongly advised to upgrade to a version where such monitoring is available.
1. When the switch fails to boot, on the console you can see these errors:
/dev/hd-cfg0: ********** WARNING: Filesystem still has errors **********
e2fsck 1.42.1 (17-Feb-2012)
/dev/hd-cfg1: recovering journal
/sbin/e2fsck: unable to set superblock flags on /dev/hd-cfg1
2. The system is able to proactively raise faults related to the SSD (F3073 or F3074 based on the SSD lifespan usage) on later code releases (covered in a later section).
F3074: fltEqptFlashFlash-minor-alarm (80% lifetime)
F3073: fltEqptFlashFlash-worn-out (90% lifetime)
3. You might also see fault F3525 raised for high SSD usage. This fault is often confused with F3073/F3074.
# fault.Inst
code : F3525
ack : no
annotation :
cause : equipment-flash-warning
changeSet : deltape (New: 21), peCycles (New: 1678), tbw (New: 32.465179), warning (New: yes)
childAction :
created : 2019-08-05T18:22:01.455-07:00
delegated : no
descr : High SSD usage observed. Please check switch activity and contact Cisco Technical Support about high SSD usage.
dn : topology/pod-1/node-206/sys/ch/supslot-1/sup/flash/fault-F3525
domain : infra
extMngdBy : undefined
highestSeverity : warning
lastTransition : 2019-08-05T18:24:02.029-07:00
lc : raised
modTs : never
occur : 1
origSeverity : warning
prevSeverity : warning
rn : fault-F3525
rule : eqpt-flash-flash-warning-alarm
severity : warning
status :
subject : flash-warning-alarm
type : operational
Fault F3525 is raised if P/E (program-erase) cycles increment by more than 21 in seven days. It does not mean that the SSD is worn out, simply that there is a lot of churn which might eventually cause the SSDs to be worn out.
You need to work with the Technical Assistance Center (TAC) to understand what is causing this churn and address it. There is also a policy to change the thresholds that cause F3525 to be raised for a specific type of SSD (See Access Policies > Policies > Switch > Equipment Flash Config Policies).
There are two ways to check the SSD lifespan status:
SSD lifetime monitoring was added as part of Cisco bug ID CSCve88634 in these versions of code:
A minor fault is raised when the drive approaches 80% of its lifetime and a major fault is raised when it exceeds 90% of its lifetime.
This allows network operators the capability to monitor and proactively replace any switch before it fails due to SSD lifetime exceeded.
Earlier versions of code did not have an SSD lifetime check for switch SSDs. Therefore, even if the switch was approaching the SSD expiration, there would not be a fault raised.
SSD usage logging was introduced starting in these releases: 3.2(5d), 3.2(6i), 3.2(7f), and 4.1(1i).
a-leaf101# cat /mnt/pss/ssd_log_amp.log (Tested on 4.2.3l)
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 16 19:00:01 EST 2019 16531370 0 29398 12350 100 0 122396965772 3825894828 60841125365 16 0.00 % 0 0
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 17 19:00:01 EST 2019 16531725 0 29422 12355 100 0 122447755358 3827482016 60858831240 16 0.00 % 12355 61606629993
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Dec 18 19:00:01 EST 2019 16532222 0 29446 12359 100 0 122495197374 3828964594 60875667807 16 0.00 % 12359 61636366134
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Dec 19 19:00:01 EST 2019 16533056 0 29470 12364 100 0 122542269672 3830435617 60892533342 16 0.00 % 12364 61666601865
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Fri Dec 20 19:00:01 EST 2019 16534010 0 29494 12367 100 0 122587126224 3831837398 60908759296 16 0.00 % 12367 61694592882
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sat Dec 21 19:00:01 EST 2019 16535311 0 29518 12372 100 0 122631804318 3833233605 60925205662 16 0.00 % 12372 61723045022
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sun Dec 22 19:00:01 EST 2019 16536727 0 29542 12376 100 0 122676829388 3834640654 60941741722 16 0.00 % 12376 61751623726
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 23 19:00:01 EST 2019 16538321 0 29566 12380 100 0 122722137414 3836056544 60958504969 16 0.00 % 12380 61780395692
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 24 19:00:01 EST 2019 16539858 0 29591 12384 100 0 122767087802 3837461258 60975050343 16 0.00 % 12384 61808582833
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Dec 25 19:00:01 EST 2019 16541121 0 29615 12388 100 0 122812238580 3838872234 60991578411 16 0.00 % 12388 61837188237
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Dec 26 19:00:01 EST 2019 16542653 0 29639 12393 100 0 122857007868 3840271286 61008101155 16 0.00 % 12393 61865429457
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Fri Dec 27 19:00:01 EST 2019 16544352 0 29663 12397 100 0 122901807026 3841671272 61024757511 16 0.00 % 12397 61893705871
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sat Dec 28 19:00:01 EST 2019 16546213 0 29687 12401 100 0 122946927200 3843081289 61041459287 16 0.00 % 12401 61922169689
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Sun Dec 29 19:00:01 EST 2019 16549522 0 29711 12405 100 0 122991584018 3844476827 61058889999 16 0.00 % 12405 61950124731
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Mon Dec 30 19:00:01 EST 2019 16551885 0 29735 12410 100 0 123036252528 3845872732 61075846564 16 0.00 % 12410 61977362529
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Tue Dec 31 19:00:01 EST 2019 16553858 0 29759 12414 100 0 123098133082 3847806568 61093085451 16 0.00 % 12414 62022286518
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Wed Jan 1 19:00:01 EST 2020 16555127 0 29783 12418 100 0 123142018610 3849178024 61109758713 16 0.00 % 12418 62048933159
Model SerialNo Software-Version Date RawReadError(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attribute(210) TBW(246) HostPageCount(247) BackgroundPageCount(248) WAF Delta_WAF Delta_PE Delta_TBW
Micron_M600_MTFDDAT064MBF MSA195207V1 Thu Jan 2 19:00:01 EST 2020 16556532 0 29808 12422 100 0 123186637556 3850572390 61126687626 16 0.00 % 12422 62076878843
Model SerialNo SW-Version Date R.Error(1) GBB(5) UpTime(9) PE(173) LifeTime(202) Attr(210) TBW(246) WAF DWAF DPE DTBW
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-03 16557977 0 29832 12428 100 0 123278298184 16 0.00 % 0 0
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-04 16558440 0 29856 12433 100 0 123323818606 16 0.00 % 5 45520422
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-05 16559031 0 29880 12438 100 0 123369348610 16 0.00 % 5 45530004
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-06 16561567 0 29904 12443 100 0 123415316270 16 0.00 % 5 45967660
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-07 16564701 0 29928 12448 100 0 123460483898 16 0.00 % 5 45167628
Micron_M600_MTFDDAT064MBF MSA195207V1 14.2(3l) 2020-01-08 16565955 0 29952 12453 100 0 123505550532 16 0.00 % 5 45066634
ssd_log_amp.log file is in the 1of3 switch tech supports.
For any additional manufacturer-related commands outputs would need privileges of a root user which would require a TAC service request.
There have been many enhancements to optimize the writes to maintain serviceability, while decreasing the amount of data written to disk. These changes were introduced across different versions.
Releases 3.2.4d and later and 4.x and later have all the fault monitoring and optimization enhancements required to prolong the SSD lifespan. This of course does not mean that upgrading to these releases absolutely prevents the SSD lifespan issue.
Cisco bug ID CSCvt36458 changed the expected behavior of an ACI Switch node ungraceful reload:
Reload Type | Before Cisco bug ID CSCvt36458 | After Cisco bug ID CSCvt36458 (4.2(6d) and later and 5.1(1h) and later) |
Upgrades | Stateless | Stateless |
Graceful Reloads (Manual reloads, hap-resets) | Stateful | Stateful |
Ungraceful Reloads (Kernel panic, power cycle) | Stateful | Stateless |
In summary, graceful switch reloads continue to use their DB on bringup. Ungraceful switch reloads now require that all policies are pulled from the APIC, similar to an upgrade.
Can the SSD be replaced in the field?
Yes. Cisco is now replacing SSDs in the field with the help of a certified Field Engineer. On modular Spines, the entire supervisor is replaced.
Should you proactively replace switch hardware?
TAC engineers can check smartctl values and evaluate which switches have more lifetime, so that replacements can be made for the most urgent switches immediately, and then gradually move to replace those switches that are not as urgent.
Are there specific hardware PIDs that are more susceptible to SSD failures?
No. The SSD lifespan usage can vary across different devices in the same environment. Usage of the SSD disks depends on how often the device needs to write the logs for event keeping. A device that registers a large number of events contributed by network instability has the potential of reaching its SSD lifespan sooner than others. Hence Cisco advocates the need to use later releases to ensure that code optimization helps with controlling the SSD writes and prolongs the SSD lifespan.
Revision | Publish Date | Comments |
---|---|---|
3.0 |
18-Jan-2024 |
Updated the Miscellaneous FAQ section due to recent changes in the SSD replacement process. |
1.0 |
09-Jan-2020 |
Initial Release |