Troubleshoot Jabber Log in Problems

Introduction

This document describes how to troubleshoot the Jabber Log in when it fails on an internal or corporate network.

Background Information

Jabber Log in is comprised of two stages; Cisco Unified Communications Manager server (CUCM) Log in and IM and Presence server (IM&P) Log in.

• CUCM Log in involves Service discovery to identify the CUCM server to which Jabber must log in.
  
  
• Jabber uses Authentication with CUCM to retrieve the Service profile details which contains IMP server, Voicemail, Conferencing, CTI server details and also Device Configuration file for Phone services.
  
  
• Once the CUCM Log in is successful, Jabber logs in to the IMP server to authenticate and retrieve the contact list and other IM services.
  
  
• The IMP Log in has two stages, namely: SOAP Log in which deals with user authentication, and then XMPP Log in which deals with XMPP session creation and Stream Management.

How to Collect Logs

Clear the cache on the PC and collect the clean Jabber Problem Report (PRT). 

Step 1. Sign out and exit the Jabber application.
Step 2. Delete all the logs located at

%AppData%\Local\Cisco\Unified Communications\Jabber\
%AppData%\Roaming\Cisco\Unified Communications\Jabber\

Step 3. Restart Jabber and recreate the problem.

Step 4. Collect the Problem report. (From the Jabber Help menu, select Report a problem option to launch the problem report tool. The instructions are found there.)

Link to these resources:

• How to Collect a Jabber Problem
• How to Collect Logs from Expressway (When Jabber is over MRA)

Keywords to Search in Logs

IMPStackCap::Log in::OnLog inError 
ServiceDiscoveryHandlerResult
@CupSoapCli: log in cup succeeds - shows when the SOAP log in was successful.
[CTriTPConnectionClient::OnConnectSuccess] - @XmppSDK: - shows when the XMPP log in was successful.
LERR - shows the Log in Errors when the Jabber fails to log in to the IM&P Server.

Stages to Troubleshoot

Stage 1. CUCM Service Discovery

Sample Log Snippet

017-03-19 17:55:00,422 WARN  [0x000050ac] [src\dnsutils\win32\win32DnsUtils.cpp(52)] [csf.dns]   [csf::dns::mapFromWindowsDNSResult] - *-----* DNS query _cisco-uds._tcp.applab has failed: DNS name does not exist.  (9003).

2017-03-19 17:55:00,438 WARN  [0x000050ac] [src\dnsutils\win32\win32DnsUtils.cpp(52)] [csf.dns]   [csf::dns::mapFromWindowsDNSResult] - *-----* DNS query _cuplogin._tcp.applab has failed: DNS name does not exist. (9003).

Steps to Resolve

Step 1. Start the command prompt (on a Windows Client) and then enter nslookup.

Step 2. Set the query type to SRV
set type = SRV

Step 3. Insert the SRV record you need to check
_cisco-uds._tcp.example.com

Step 4. This returns the DNS A records which point to the CUCM servers.

This is an example of the Successful _cisco-uds SRV record.

If no records are returned, contact your DNS administrator to configure the SRV records

Steps to Resolve

Step 1. Validate that the CUCM nodes configured as TFTP Servers are up and running.

Step 2. Restart these services on all the CUCM nodes.

• Cisco TFTP
• Cisco UDS Service

Stage 2. CUCM User Authentication

Sample Log Snippet

2017-01-09 08:59:10,652 INFO  [0x00001740] [vices\impl\DiscoveryHandlerImpl.cpp(460)] [service-discovery] [CSFUnified::DiscoveryHandlerImpl::evaluateServiceDiscoveryResult] - ServiceDiscoveryHandlerResult return code FAILED_UCM90_AUTHENTICATION

Steps to Resolve

Step 1. Ensure that the User is configured as an enduser in CUCM. Navigate to CUCM Administration > Enduser page.

Step 2. Verify that credentials are correct and the user is active. Log in to the CUCM Self-care Portal.

This image refers to the scenario where the LDAP is unable to authenticate the user either because the user is not a valid user or the password supplied is incorrect.

Step 3. If this issue is seen for all users, verify if the LDAP synchronization and LDAP Authentication settings on CUCM Administration > System > LDAP are correct.

Tip: From the LDAP Server perspective, ensure that the account is not locked, that the passwords are not expired, and that all the users are synchronized with the CUCM server.

Sample Log Snippet

2017-08-28 12:04:00,282 INFO  [0x00004290] [vices\impl\DiscoveryHandlerImpl.cpp(452)] [service-discovery] [CSFUnified::DiscoveryHandlerImpl::evaluateServiceDiscoveryResult] - ServiceDiscoveryHandlerResult return code FAILED_UCM90_CONNECTION

Steps to Resolve

Step 1. Test if you are able to open this URL in the browser on the PC https://<CUCM IP/FQDN> :8443/cucm-uds/version

Unsuccessful                                                                                                                                       

Successful

Step 2. If the response is unsuccessful, verify that the DNS is configured correctly to resolve them, and also if no Network Elements like Firewall/ASA block the port 8443.

Step 3. This URL must be tested for all CUCM Servers in the cluster. For a list of Servers, navigate to CUCM Administration > System > Server.

Sample Log Snippet

2016-08-18 13:14:49,943 INFO  [0x000036e4] [vices\impl\DiscoveryHandlerImpl.cpp(367)] [service-discovery] [DiscoveryHandlerImpl::evaluateServiceDiscoveryResult] - ServiceDiscoveryHandlerResult return code FAILED_USER_LOOKUP

Steps to Resolve

Step 1. Verify that you are able to open this URL in the browser on the PC https://CUCM:8443/cucm-uds/clusterUser?username=<userid>

Step 2. Verify that the userid that is entered in Jabber matches the userid on CUCM End-user page.

Tip: Jabber has UPN discovery enabled by default and hence gets the userid prepopulated from the LDAP UPN field. Check if UPN is the same as configured in CUCM. If you need to disable UPN discovery, set UPN_DISCOVERY_ENABLED=false during installation.

Stage 3. SOAP Log in (IM and Presence Log in)

Sample Log Snippet

2017-01-14 15:55:09,615 INFO  [0x00000dc0] [ts\adapters\imp\components\Login.cpp(99)] [imp.service] [IMPStackCap::Login::OnLoginError] - ****************************************************************
2017-01-14 15:55:09,615 INFO  [0x00000dc0] [s\adapters\imp\components\Login.cpp(100)] [imp.service] [IMPStackCap::Login::OnLoginError] - OnLoginError: (data=0) LERR_CUP_AUTH <12>:
201-01-14 15:55:09,615 INFO  [0x00000dc0] [s\adapters\imp\components\Login.cpp(101)] [imp.service] [IMPStackCap::Login::OnLoginError] - ****************************************************************

Steps to Resolve

Step 1. Verify that the user is assigned to a Presence node and there are no duplicates for the user (IM and presence Administration > Diagnostics > System troubleshooter).

Step 2. Verify that the High Availability (HA) state is Normal and no Failover has occurred.

If you have tried to assign the user during an abnormal HA state, Users are not assigned to any IMP node and log in fails.

Recover the HA state first and re-assign the user.

Step 3. Verify that the credentials are valid.

1. In the case of an LDAP user, verify if the user is able to log in to the CUCM Selfcare portal.
   
   
2. If ccmenduser page log in fails, check the LDAP Authentication settings in CUCM and also verify the same settings are replicated to IMP.
   
     run sql select * from ldapauthentication
      run sql select * from ldapauthenticationhost
   
   
3. Verify that the account is not locked in LDAP.
   
   
4. If the user was recently enabled for Presence, restart the Cisco Sync Agent service on IMP Publisher.

Step 4. Check the server has high TOMCAT CPU consumption.

• show process load
• utils diagnose test

Step 5. Set these services logs to DEBUG and then recreate the Log in issue and collect the logs.

• Client Profile Agent
• Cisco Tomcat
• Event Viewer-Application Log
• Event Viewer-System Log

Sample Log Snippet

[IMPServices] [CSFUnified::IMPStackCap::Log in::OnLog inError] - ****************************************************************
[IMPServices] [CSFUnified::IMPStackCap::Log in::OnLog inError] - OnLog inError: LERR_CUP_AUTH <10>: 
[IMPServices] [CSFUnified::IMPStackCap::Log in::OnLog inError] - ****************************************************************
[http-bio-443-exec-15] handlers.Log inHandlerAbstract - preLog in:PRELOGIN reasoncode=FAILURE. User either not CUP licensed or not found in database.

Tip: For this error, it is also recommended to retrieve the Cisco Tomcat logs from the CUCM and IM&P servers.

From the Cisco Tomcat logs:

2019-10-27 18:33:40,373 DEBUG [http-bio-443-exec-5] impl.LDAPHostnameVerifier - check : inside check with X509 cert
2019-10-27 18:33:40,373 DEBUG [http-bio-443-exec-5] impl.Certificates - getCNs :
2019-10-27 18:33:40,373 DEBUG [http-bio-443-exec-5] impl.LDAPHostnameVerifier - check : cns = [ldap.ciscolab.com]
2019-10-27 18:33:40,373 DEBUG [http-bio-443-exec-5] impl.Certificates - getDNSSubjectAlts :
2019-10-27 18:33:40,374 DEBUG [http-bio-443-exec-5] impl.LDAPHostnameVerifier - check : subjectAlts = [ldap.ciscolab.com, ldap2.ciscolab.com]
2019-10-27 18:33:40,374 ERROR [http-bio-443-exec-5] impl.AuthenticationLDAP - verifyHostName:Exception.javax.net.ssl.SSLPeerUnverifiedException: hostname of the server 'ldapdc.ciscolab.com' does not match the hostname in the server's certificate.
2019-10-27 18:33:40,374 DEBUG [http-bio-443-exec-5] impl.AuthenticationLDAP - value of hostnameverifiedfalse
2019-10-27 18:33:40,374 INFO [http-bio-443-exec-5] impl.AuthenticationLDAP - verifyHostName: Closing LDAP socket

Steps to Resolve

Two situations are encountered here if the Cisco Tomcat logs do not show any certificate error, this must be validated.

Step 1. Validate that the user is associated with an IM&P server.

• Navigate to the CUCM Administration webpage > User Management > User Management > Assign Presence Users. Look for the userid and Click Find.

Step 2. If the user is associated with an IM&P server, bounce the user from the Home Node cluster

• Navigate to CUCM Administration webpage > User Management > End User > Look for the end-user and click Find under Service Settings. Uncheck the Home Cluster checkbox,click Save, check the Home Cluster checkbox, and click Save.

In case the Cisco Tomcat logs show the error from the Snippet previously showed, perform these steps:

Step 1. Confirm if the Cisco Jabber is configured to use Secure LDAP.

Step 2. If Secure LDAP is in use, confirm the certificates' information associated with them like the Fully Qualified Domain Name (FQDN), Hostname and Common Name (CN).

Step 3. Validate how the CUCM and IM&P are configured. If with IP address or FQDN, then compare that the information is contained within the certificate.

• Navigate to the  CUCM Administration webpage > System > Server.

Step 4. If the servers are configured with IP address, and the LDAP certificates are configured with FQDN, the next command to be executed on all the CUCM and IM&P nodes is

• utils ldap config ipaddr
• The definition of the servers is required to be changed as FQDN. Refer to. Change CUCM Server Definition from IP Address or Hostname to FQDN guide.

Sample Log Snippet

2017-11-08 16:03:20,051 DEBUG [0x00003a0c] [s\adapters\imp\components\Login.cpp(127)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - ************************************************
2017-11-08 16:03:20,051 INFO  [0x00003a0c] [s\adapters\imp\components\Login.cpp(128)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - OnLoginError: LERR_CUP_UNREACHABLE 
2017-11-08 16:03:20,051 DEBUG [0x00003a0c] [s\adapters\imp\components\Login.cpp(129)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - *************************************************

Steps to Resolve

Step 1. Verify that IMP FQDN/Hostnames are resolvable from the client PC.

Step 2. Verify that you can open this URL in browser https://<IMP SERVER FQDN/IP>:8443/EPASSoap/service/v105.

Successful       

Unsuccessful

Step 3. Verify that firewall/VPN does not block the connectivity to IMP server ( Port 8443,5222).

Step 4. Verify if this service runs in IMP server: Cisco Client profile Agent.

Step 5. Set these services logs to DEBUG, recreate the Log in issue, and then collect the logs if the previous steps do not resolve the problem.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Client Profile Agent

Tip: If the problem persists for only one user, un-assign and re-assign the user for presence in CUCM. If it is a system-wide problem, collect the logs and check the status of the services 

Sample Log Snippet

2017-11-08 16:03:20,051 DEBUG [0x00003a0c] [s\adapters\imp\components\Login.cpp(127)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - ************************************************
2017-11-08 16:03:20,051 INFO  [0x00003a0c] [s\adapters\imp\components\Login.cpp(128)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - OnLoginError: LERR_CUP_INTERNAL_ERROR 
2017-11-08 16:03:20,051 DEBUG [0x00003a0c] [s\adapters\imp\components\Login.cpp(129)] [IMPServices] [CSFUnified::IMPStackCap::Login::OnLoginError] - *************************************************

Steps to Resolve

Step 1. Perform Mandatory checks.

Step 2. Verify that these services are running in the IM&P server.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Cisco Log in datastore

Step 3. Check if this Field notice is applicable.

Field Notice: FN - 64267 - Cisco Unified Communications Manager IM & Presence causes Cisco Jabber log in failures - Software Upgrade Recommended.

Step 4. Set these services logs to DEBUG, recreate the Log in issue, and then collect the logs if the previous steps do not resolve the problem.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Client Profile Agent
• Cisco Log in Datastore
• Event Viewer-Application Log
• Event Viewer-System Log

Step 5. Reboot the cluster to recover the situation.

Stage 4. XMPP Log in (IM and Presence Log in)

Sample Log Snippet

2019-05-03 15:19:32,225 DEBUG [0x0000000109732f80] [s/adapters/imp/components/Log in.cpp(128)] [IMPServices] [OnLog inError] - ****************************************************************
2019-05-03 15:19:32,225 INFO  [0x0000000109732f80] [s/adapters/imp/components/Log in.cpp(129)] [IMPServices] [OnLog inError] - OnLog inError: LERR_JABBER_AUTH <14>: Authentication error with server, resource bind, TLS, create session or SASL error
2019-05-03 15:19:32,225 DEBUG [0x0000000109732f80] [s/adapters/imp/components/Log in.cpp(130)] [IMPServices] [OnLog inError] - ****************************************************************

Steps to Resolve

Step 1. Verify that port 5222 is opened between the IM&P servers and the Expressways.

Step 2. Verify that these services are running in the IM&P server and restart them once.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service

Step 3. Disable the High Availability from the CUCM Presence Redundancy Groups.

Step 4. Restart the Cisco XCP Router service on all the IM&P nodes, first with the IM&P Publisher and then in the Subscribers.

• utils service restart Cisco XCP Router Service

Step 5. Reenable the High Availability from the CUCM Presence Redundancy Groups.

Sample Log Snippet

2017-10-27 10:56:47,396 DEBUG [0x00007fff8b3d7340] [s/adapters/imp/components/Login.cpp(127)] [IMPServices] [OnLoginError] - ****************************************************************
2017-10-27 10:56:47,396 INFO  [0x00007fff8b3d7340] [s/adapters/imp/components/Login.cpp(128)] [IMPServices] [OnLoginError] - OnLoginError: LERR_JABBER_AUTH <17>: Authentication error with server, for example, resource bind, TLS, create session or SASL error
2017-10-27 10:56:47,396 DEBUG [0x00007fff8b3d7340] [s/adapters/imp/components/Login.cpp(129)] [IMPServices] [OnLoginError] - ****************************************************************

Steps to Resolve

Step 1. Verify that the cup-xmpp Certificates are valid.

Step 2. Verify that Port 5222 is open.

Step 3. Set these services logs to DEBUG and then recreate the Log in issue and collect the logs before step 4. 

If Root cause to be identified as Reboot of the server is the only fix known.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Client Profile Agent
• Event Viewer-Application Log
• Event Viewer-System Log

Step 4. Reboot the server to resolve the issue.

Sample Log Snippet

2014-12-15 12:07:31,600 INFO  [0x00001670] [ts\adapters\imp\components\Login.cpp(96)] [imp.service] [IMPStackCap::Login::OnLoginError] - ****************************************************************
2014-12-15 12:07:31,600 INFO  [0x00001670] [ts\adapters\imp\components\Login.cpp(97)] [imp.service] [IMPStackCap::Login::OnLoginError] - OnLoginError: (data=0) LERR_JABBER_UNREACHABLE <16>:
2014-12-15 12:07:31,600 INFO  [0x00001670] [ts\adapters\imp\components\Login.cpp(98)] [imp.service] [IMPStackCap::Login::OnLoginError] - ****************************************************************

Steps to Resolve

Step 1. Check if IMP FQDN/Hostnames are resolvable.

Step 2. Verify that the firewall/VPN does not block the connectivity to the IM&P server ( Port 8443,5222).

Step 3. Verify if these services are running in the IM&P server and restart them once.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service

Step 4. Perform Mandatory checks.

Step 5. Set these services logs to DEBUG, recreate the Log in issue, and then collect the logs if the previous steps do not resolve the problem.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Client Profile Agent
• Event Viewer-Application Log
• Event Viewer-System Log

Step 6. In case all users experience the same error, a server Reboot can be done for quick recovery.

Sample Log Snippet

2020-03-12 19:55:01,283 DEBUG [0x000000010b71d800][apters/imp/components/Log inUtils.cpp(96)][IMPServices][Log inErrortoErrorCode] - Log inErrortoErrorCode: 27 mapped to: UnknownLog inError
2020-03-12 19:55:01,283 DEBUG [0x000000010b71d800][isteners/Log inEventListenerImpl.cpp(148)][IMPServices][OnLog inError] - errCode: UnknownLog inError
2020-03-12 19:55:01,283 INFO [0x000000016b61f000][ers/imp/lifecycle/Log inExecutor.cpp(314)][IMPServices][signOn] - logged in using User ID: 35309769, failed
2020-03-12 19:55:01,478 INFO [0x000000010b71d800][pp/tahiti/ui/log in/YLCLog inBaseVC.m(500)][UI.Action.System] [-[YLCLog inBaseVC getPresenceErrorMessgaWithCode:]] - Jabber log in failed and show errorcode:200 string: Cannot Sign in your account. Contact your administrator.

Steps to Resolve

Step 1. Validate that the user is assigned to the IM&P.

Step 2. Validate that the certificates are correctly exchanged between the nodes and the Jabber.

Step 3. Validate the OAuth Signing and Encryption keys are correctly configured on all the nodes. See this document under Verify section.

Step 4. Perform Mandatory checks.

Step 5. Set these services logs to DEBUG, recreate the Log in issue, and then collect the logs if the previous steps do not resolve the problem.

• Cisco XCP Router
• Cisco XCP Connection Manager
• Cisco XCP Authentication Service
• Client Profile Agent
• Event Viewer-Application Log
• Event Viewer-System Log
• Cisco SSO
• Cisco Tomcat
• Cisco Tomcat Security

Mandatory Checks

Step 1. Verify that the user is assigned to a Presence node (navigate to IM and Presence Administration > System > Topology) and there are no duplicates for the user (navigate to IM and Presence Administration > Diagnostics > System troubleshooter).

Step 2. If High Availability is enabled, navigate to CUCM Administration > Server > Presence Redundancy Group and check if they are in Normal state. This is the image of how Normal state looks. For more information about High Availability,refer to here.

Abnormal State

Note: These services are used by the Jabber to log in: Cisco Tomcat, Cisco Tomcat Security, Cisco Client Profile Agent, Cisco XCP Connection Manager, Cisco XCP Router and Cisco XCP Authentication.

Normal State

Step 3. Check High Availability Replication status.

   a.utils dbreplication runtimestate

If you encounter issues on the database replication, navigate to this link.

b.run pe sql ttlog in select count(*) from typesysreplication

or utils imdb_replication status ( 10.5.2 SU2a and later)

The three datastores need to show PASSED, and the command needs to be run on all the IM&P nodes, as sometimes on one node all the datastores' replication can show Passed, but on another node, it can show Failed. 

The implications if the IMDB (In-memory Database) replication is not correct can imply that some or all the users are unable to log in or that their presence status cannot be shown correctly.

The steps to resolve the IMDB replication issues are:

Step 1. Disable the High Availability (HA) for the IM&P Subcluster that is affected.

Step 2. Stop the Cisco Presence Engine on all nodes

utils service stop Cisco Presence Engine 

Step 3. Verify all the Datastore Services are running: Cisco Log in Datastore, Cisco Route Datastore, Cisco Presence Datastore, Cisco SIP Registration Datastore.

utils service list

Step 4. Restart Cisco Config Agent on each node one at a time.

utils service restart Cisco Config Agent

Step 5. Start Cisco Presence Engine.

utils service start Cisco Presence Engine

Step 6. Enable HA for the Sub-cluster.

How to Set Logs to DEBUG

Step 1.       Navigate to Navigation > Unified serviceability > Trace > Configuration.
Step 2.       From the Server drop-down list, choose the server (for example, IMP node) that runs the service to configure trace and click Go.
Step 3.       From the Service Group drop-down list box, choose the service group for the service to configure trace; then click Go.
Step 4.       From the Service drop-down list box, choose the service for which you want to configure trace; then click Go.
Step 5.       Check box Apply to All Nodes and select the trace level to DEBUG.
Step 6.       To save your trace parameters configuration, click Save.

For more information on how to set trace levels, refer to the Cisco Unified Serviceability Administration Guide.

Helpful videos:

• Collect Logs from the RTMT

Logs to Collect

Collect Logs from RTMT