IoT-FND: Unable to check database connection. Exception: java.sql.SQLException: ORA-28001: the password has expired

Introduction

This document describes how to unlock Oracle user account of Cisco IoT Field Network Director (IoT-FND) and change password expiration setting to unlimited. 

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

• Basic Linux shell operations (view logs, start a process etc.)
• Basic understanding of SQL queries 

Components Used

This document is not restricted to specific software and hardware versions.

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Background Information

This document assumes that you have two separate Linux servers:

• IoT-FND-SRV - Server that hosts IoT FND application.

• IoT-FND-Oracle - Server that hosts the Oracle database for IoT-FND application.

Problem 

SYMPTOM 1: IoT-FND is not starting, database connection verification failed. 

[root@IoT-FND-SRV ~]# service cgms status
IoT-FND Version X.X.XX
08-01-2018 15:14:58 CEST: INFO: IoT-FND database server: XX.XX.XX.XX
08-01-2018 15:15:01 CEST: ERROR: IoT-FND database connection verification failed.
08-01-2018 15:15:02 CEST: ERROR: IoT-FND application server is not running.

SYMPTOM 2: cgms_db_connection_test.log  contains entries similar to:

[root@IoT-FND-SRV ~]# tail -n 50 /opt/cgms/server/cgms/log/cgms_db_connection_test.log
2018-08-01 12:27:22,767:INFO:main:TestDBConnection: Checking database connection. Please wait ...
2018-08-01 12:27:22,856:INFO:main:TestDBConnection: Database URL: jdbc:oracle:thin:@XX.XX.XX.XX:1522:cgms
2018-08-01 12:27:22,856:INFO:main:TestDBConnection: Database user: cgms_dev
2018-08-01 12:27:23,156:ERROR:main:TestDBConnection: Unable to check database connection. Exception: 
java.sql.SQLException: ORA-28001: the password has expired

Note: IoT-FND application uses CGMS_DEV database user to access the Oracle database.  

Note:  By default, Oracle database has Password Expiration enabled so after some time, account passwords get expired and it is not possible to use them without changing the password.

SYMPTOM 3: Oracle CGMS_DEV user account status is set to EXPIRED.  

[oracle@IoT-FND-Oracle]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Aug 1 16:02:25 2018

Copyright (c) 1982, 2014, Oracle. All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> SELECT USERNAME,ACCOUNT_STATUS FROM DBA_USERS WHERE USERNAME LIKE 'cg%';

USERNAME
--------------------------------------------------------------------------------
ACCOUNT_STATUS
--------------------------------
CGMSDBA
OPEN

CGMS_DEV
EXPIRED

Solution

Step 1. Login to the IoT-FND server with SSH.

Step 2. Stop FND services.

[root@IoT-FND-SRV]$ service cgms stop

Note: If you do not stop IoT-FND services, FND will try to connect to the database using an old password and the account will get blocked until you are done with this procedure. 

Step 3. Log into the Oracle server with SSH.

Step 4. Switch to oracle system user:

[root@IoT-FND-Oracle]$ su - oracle
[oracle@IoT-FND-Oracle]$

Step 5. Connect to the root instance of the Oracle database as sysdba user:

[oracle@IoT-FND-Oracle]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Wed Aug 1 16:19:23 2018

Copyright (c) 1982, 2014, Oracle. All rights reserved.


Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL>

Step 6. Modify password_life_limit setting to unlimited:

sql> ALTER PROFILE DEFAULT LIMIT PASSWORD_LIFE_TIME UNLIMITED;

Profile altered.

Step 7. Check which accounts have expired passwords. 

SQL> SELECT USERNAME,ACCOUNT_STATUS FROM DBA_USERS WHERE ACCOUNT_STATUS='EXPIRED';

USERNAME
--------------------------------------------------------------------------------
ACCOUNT_STATUS
--------------------------------
CGMSDBA
OPEN

CGMS_DEV
EXPIRED

Step 8. Set password for CGMS_DEV user:

sql> ALTER USER CGMS_DEV IDENTIFIED BY <new-password>;  

User altered.

Note: If you know the old password you can reuse it. If no, you will need to run cgmsSetup.sh script on IoT-FND server to modify database passwords set in FND. 

Step 9. Unlock the CGMS_DEV user account:

SQL> ALTER USER CGMS_DEV ACCOUNT UNLOCK;

User altered.

Step 10. Verify the account status of CGMS_ users:

 SQL> SELECT USERNAME,ACCOUNT_STATUS FROM DBA_USERS WHERE USERNAME LIKE 'cg%';

USERNAME
--------------------------------------------------------------------------------
ACCOUNT_STATUS
--------------------------------
CGMSDBA
OPEN

CGMS_DEV
OPEN

Step 11. Login to the IoT-FND server with SSH.

Step 12. Start /opt/cgms/bin/setupCgms.sh script to change the database password setting. Leave all other settings with their default values:

[root@IoT-FND-SRV]# /opt/cgms/bin/setupCgms.sh
08-01-2018 16:40:43 CEST: INFO: ========== IoT-FND Setup Started - 2018-08-01-16-40-43 ==========
08-01-2018 16:40:43 CEST: INFO: Log file: /opt/cgms/bin/../server/cgms/log/cgms_setup.log

Are you sure you want to setup IoT-FND (y/n)? y

08-01-2018 16:40:46 CEST: INFO: User response: y

Do you want to change the database settings (y/n)? y

08-01-2018 16:40:54 CEST: INFO: User response: y

Enter database server hostname or IP [XX.XX.XX.XX]:
08-01-2018 16:40:55 CEST: INFO: Database server: XX.XX.XX.XX

Enter database server port [1522]:
08-01-2018 16:40:57 CEST: INFO: Database server port: 1522

Enter database SID [cgms]:
08-01-2018 16:40:59 CEST: INFO: Database SID: cgms

Do you wish to configure another database server for this IoT-FND ? (y/n)? n

08-01-2018 16:41:02 CEST: INFO: User response: n
08-01-2018 16:41:02 CEST: INFO: Configuring database settings. This may take a while. Please wait ...
08-01-2018 16:41:03 CEST: INFO: Database settings configured.

Do you want to change the database password (y/n)? y

08-01-2018 16:41:06 CEST: INFO: User response: y

Enter database password: <enter cgms_dev database user password>
Re-enter database password: <repeat cgms_dev database user password>

08-01-2018 16:41:10 CEST: INFO: Configuring database password. This may take a while. Please wait ...
08-01-2018 16:41:13 CEST: INFO: Database password configured.

Do you want to change the keystore password (y/n)? n

08-01-2018 16:41:16 CEST: INFO: User response: n

Do you want to change the web application 'root' user password (y/n)? n

08-01-2018 16:41:19 CEST: INFO: User response: n

Do you want to change the FTP settings (y/n)? n

08-01-2018 16:41:22 CEST: INFO: User response: n

Do you want to change router CGDM protocol settings (y/n)? n

08-01-2018 16:41:24 CEST: INFO: User response: n

Do you want to change log file settings)? (y/n)? n

08-01-2018 16:41:25 CEST: INFO: User response: n
08-01-2018 16:41:25 CEST: INFO: ========== IoT-FND Setup Completed Successfully ==========

Step 13. Verify database connectivity:

[root@IoT-FND-SRV ~]# service cgms status
IoT-FND Version X.X.X-XXX
08-01-2018 16:46:14 CEST: INFO: IoT-FND database server: XX.XX.XX.XX
08-01-2018 16:46:15 CEST: INFO: IoT-FND database connection verified.
08-01-2018 16:46:15 CEST: ERROR: IoT-FND application server is not running.

Step 14. Start IoT-FND application:

[root@IoT-FND-SRV ~]# service cgms start

Step 15. Verify IoT-FND operations:

[root@IoT-FND-SRV ~]# service cgms status
IoT-FND Version X.X.X-XXX
08-01-2018 16:49:04 CEST: INFO: IoT-FND database server: XX.XX.XX.XX
08-01-2018 16:49:04 CEST: INFO: IoT-FND database connection verified.
08-01-2018 16:49:05 CEST: INFO: IoT-FND application server is up and running.
08-01-2018 16:49:06 CEST: INFO: IoT-FND is up and running.