Troubleshoot Finesse Agent State Change Delayed

Available Languages

Updated:September 14, 2017

Document ID:212077

Bias-Free Language

The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.

Introduction

Prerequisites

Requirements

Components Used

Problem

Troubleshoot

Conclusion

Resolution

Introduction

This document provides information on how to troubelshoot the agent status delays seen on the Finesse Agent Desktop.

Contributed by Rohit Harsh, Cisco TAC Engineer.

Prerequisites

Requirements

Cisco recommends that you have knowledge of these topics:

Cisco Unified Contact Center Enterprise (UCCE)
Cisco Finesse

Components Used

The information in this document is based on these software versions:

Finesse 11.5
UCCE 11.5

The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.

Problem

When the Agent receive a call (Inbound / Outbound), the change of state from Ready --> Reserved --> Talking is delayed.

Troubleshoot

To troubleshoot this type of issue, you can pull these logs from the Finesse server and client:

Finesse Desktop WebServices Logs
Finesse Client Side Logs

Finesse Client logs

CTI Message received for the agent State Change:

 0051203603: 10.106.81.82: Apr 14 2017 14:01:17.196 -0700: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIAgentStateEvent [skillGroupState=7 (BUSY_OTHER), stateDuration=0, skillGroupNumber=29808, skillGroupPriority=0, agentState=8 (RESERVED), eventReasonCode=0, numFltSkillGroups=0, CTIClientSignature=Finesse, agentID=1234, agentExtension=1234, agentInstrument=1234, agentID_Long=null, duration=null, nextAgentState=null, fltSkillGroupNumberList=[], fltSkillGroupIDList=[], fltSkillGroupPriorityList=[], fltSkillGroupStateList=[], MRDId=1, agentMode=1]CTIMessageBean [invokeID=null, cti_sequence_id=865941, msgID=30, timeTracker={"id":"AgentStateEvent","CTI_MSG_RECEIVED":1492203677196,"CTI_MSG_DISPATCH":1492203677196}, msgName=AgentStateEvent, deploymentType=CCE]][cti_response_time=0][dispatch_phase=DnD-CHECKPOINT-3B]: Decoded Message to Finesse from backend cti server

However only after 14 seconds, Finesse puts the agent in Reserved State:

001345281: 10.106.81.82: Apr 14 2017 14:01:31.936 -0700: %CCBU_CoreImpl-worker187-6-XMPP_PUBLISH_ASYNCHRONOUS: %[NodeId=/finesse/api/User/1234][Payload=&lt;Update&gt;

  &lt;data&gt;

    &lt;user&gt;

      &lt;dialogs&gt;/finesse/api/User/1234/Dialogs&lt;/dialogs&gt;

      &lt;extension&gt;1234&lt;/extension&gt;

      &lt;firstName&gt;Test&lt;/firstName&gt;

      &lt;lastName&gt;Test&lt;/lastName&gt;

      &lt;loginId&gt;1234&lt;/loginId&gt;

      &lt;loginName&gt;Test&lt;/loginName&gt;

      &lt;mediaType&gt;1&lt;/mediaType&gt;

      &lt;pendingState&gt;&lt;/pendingState&gt;

      &lt;roles&gt;

        &lt;role&gt;Agent&lt;/role&gt;

      &lt;/roles&gt;

      &lt;settings&gt;

        &lt;wrapUpOnIncoming&gt;OPTIONAL&lt;/wrapUpOnIncoming&gt;

      &lt;/settings&gt;

      &lt;state&gt;RESERVED&lt;/state&gt;

      &lt;stateChangeTime&gt;2017-04-14T21:01:31.935Z&lt;/stateChangeTime&gt;

      &lt;teamId&gt;5056&lt;/teamId&gt;

      &lt;teamName&gt;Test&lt;/teamName&gt;

      &lt;uri&gt;/finesse/api/User/1234&lt;/uri&gt;

    &lt;/user&gt;

  &lt;/data&gt;

  &lt;event&gt;PUT&lt;/event&gt;

  &lt;requestId&gt;ad6996dc-db0b-4f45-8e3a-0e975519f4db&lt;/requestId&gt;

  &lt;source&gt;/finesse/api/User/1234&lt;/source&gt;

&lt;/Update&gt;]: Publishing XMPP Message Asynchronously

Now if you see what that the agent was doing at that time, you can see that the agent tried to change the state to Not Ready State just 1 minutes 14 seconds before the call arrival on the Desktop and it was delayed by Finesse.

2017-04-14T14:00:22.128 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:22.137 -0700: Header : _changeStateCallback(): User changed state to NOT_READY with a reason code of: Break

2017-04-14T14:00:22.129 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:22.138 -0700: Header : Agent selected or went NOT_READY -- Linda flag cleared.

2017-04-14T14:00:22.130 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:22.139 -0700: Header : [ClientServices] User: requestId='ad6996dc-db0b-4f45-8e3a-0e975519f4db', Making REST request: method=PUT, url='https://finesse.cisco.com:/finesse/api/User/1234'

2017-04-14T14:00:22.159 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:22.168 -0700: Header : [ClientServices] User: requestId='ad6996dc-db0b-4f45-8e3a-0e975519f4db', Returned with status=202, content='null'

2017-04-14T14:00:27.129 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:27.138 -0700: Header : _showOperationError(): Failed to lookup error message in resource bundle(errorBundle=null, error=TypeError: response is undefined).

2017-04-14T14:00:27.130 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:27.139 -0700: Header : _showOperationError(): Failed to get error message from response(error=TypeError: response is undefined).

2017-04-14T14:00:27.130 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:27.139 -0700: Header : _showOperationError(): Displayed generic error message.

Here agent tried again.

2017-04-14T14:00:39.093 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:39.102 -0700: Header : _changeStateCallback(): User changed state to NOT_READY with a reason code of: Training

2017-04-14T14:00:39.093 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:39.102 -0700: Header : Agent selected or went NOT_READY -- Linda flag cleared.

2017-04-14T14:00:39.095 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:39.104 -0700: Header : [ClientServices] User: requestId='eb6659c6-06f2-4f02-a244-95839c450b48', Making REST request: method=PUT, url='https://finesse.cisco.com:/finesse/api/User/1234'

2017-04-14T14:00:39.109 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:39.118 -0700: Header : [ClientServices] User: requestId='eb6659c6-06f2-4f02-a244-95839c450b48', Returned with status=202, content='null'

2017-04-14T14:00:39.120 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:39.128 -0700: Header : Client: 2017-04-14T21:00:39.115Z, Server: 2017-04-14T21:00:39.123Z, Drift: 8ms, Network Latency (round trip): 8ms

2017-04-14T14:00:44.093 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:44.101 -0700: Header : _showOperationError(): Failed to lookup error message in resource bundle(errorBundle=null, error=TypeError: response is undefined).

2017-04-14T14:00:44.094 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:44.102 -0700: Header : _showOperationError(): Failed to get error message from response(error=TypeError: response is undefined).

2017-04-14T14:00:44.094 -07:00: 3ADF8F: finesse.cisco.com: Apr 14 2017 14:00:44.102 -0700: Header : _showOperationError(): Displayed generic error message.

Now in the Desktop WebServices Logs:

Finesse received the Not Ready request from the agent.

0051191445: 10.106.81.82: Apr 14 2017 14:00:22.148 -0700: %CCBU_http-apr-8445-exec-13-6-REQUEST_START: %[method_name=PUT][parameter_name={  }][resource_name=/User/1234][usr=1234]: Request start 


0051191446: 10.106.81.82: Apr 14 2017 14:00:22.149 -0700: %CCBU_http-apr-8445-exec-13-6-API_REQUEST: %[REQUEST_URL=User/1234][agent_id=1234][requestId=ad6996dc-db0b-4f45-8e3a-0e975519f4db][request_method=user.PUT][request_parameters= state:NOT_READY]: Request from client to webservice api 


0051191447: 10.106.81.82: Apr 14 2017 14:00:22.149 -0700: %CCBU_http-apr-8445-exec-13-7-API.REQUEST:  {Thrd=http-apr-8445-exec-13} Received handleRequest for [requestedObjectId=1234], [requestId=ad6996dc-db0b-4f45-8e3a-0e975519f4db] 



0051191449: 10.106.81.82: Apr 14 2017 14:00:22.150 -0700: %CCBU_http-apr-8445-exec-13-7-DISPATCHER:  {Thrd=http-apr-8445-exec-13} queueing command SetAgentState for sequencer: [1234] 


0051191450: 10.106.81.82: Apr 14 2017 14:00:22.150 -0700: %CCBU_http-apr-8445-exec-13-7-FinesseApiResponseFilter:  {Thrd=http-apr-8445-exec-13} FinesseApiResponseFilter : Added header Access-Control-Allow-Origin: null to the response of the REST API request=https://finesse.cisco.com/finesse/api/User/1234

Again it is the same set of events for the second Not Ready request:

0051195056: 10.106.81.82: Apr 14 2017 14:00:39.110 -0700: %CCBU_http-apr-8445-exec-19-6-REQUEST_START: %[method_name=PUT][parameter_name={  }][resource_name=/User/1234][usr=1234]: Request start 


0051195060: 10.106.81.82: Apr 14 2017 14:00:39.113 -0700: %CCBU_http-apr-8445-exec-19-7-DISPATCHER:  {Thrd=http-apr-8445-exec-19} queueing command SetAgentState for sequencer: [1234] 


0051195061: 10.106.81.82: Apr 14 2017 14:00:39.113 -0700: %CCBU_http-apr-8445-exec-19-7-FinesseApiResponseFilter:  {Thrd=http-apr-8445-exec-19} FinesseApiResponseFilter : Added header Access-Control-Allow-Origin: null to the response of the REST API request=https://finesse.cisco.com/finesse/api/User/1234 


0051195060: 10.106.81.82: Apr 14 2017 14:00:39.113 -0700: %CCBU_http-apr-8445-exec-19-7-DISPATCHER:  {Thrd=http-apr-8445-exec-19} queueing command SetAgentState for sequencer: [1234] 


0051195061: 10.106.81.82: Apr 14 2017 14:00:39.113 -0700: %CCBU_http-apr-8445-exec-19-7-FinesseApiResponseFilter:  {Thrd=http-apr-8445-exec-19} FinesseApiResponseFilter : Added header Access-Control-Allow-Origin: null to the response of the REST API request=https://finesse.cisco.com/finesse/api/User/1234

Only after 1 min 2 secs, Finesse started processing the request:

0051220500: 10.106.81.82: Apr 14 2017 14:01:24.585 -0700: %CCBU_CoreImpl-worker118-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :26553 , agentstate : 2, workmode : 0, reason code: 4, forceflag :1, agentcapacity: 0, agentext: 1234, agentid: 1234, supervisorid: null, ssoFlag=false][cti_message_name=SetAgentStateReq]: Message going to the backend cti server 


0051223431: 10.128.224.141: Apr 14 2017 14:01:24.766 -0700: %CCBU_CoreImpl-worker118-7-DISPATCHER:  {Thrd=CoreImpl-worker118} completed command SetAgentState for sequencer: [1234] 

 

0051227244: 10.106.81.82: Apr 14 2017 14:01:27.078 -0700: %CCBU_CoreImpl-worker130-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :26563 , agentstate : 2, workmode : 0, reason code: 4, forceflag :1, agentcapacity: 0, agentext: 1234, agentid: 1234, supervisorid: null, ssoFlag=false][cti_message_name=SetAgentStateReq]: Message going to the backend cti server 


0051229958: 10.128.224.141: Apr 14 2017 14:01:27.274 -0700: %CCBU_CoreImpl-worker130-7-DISPATCHER:  {Thrd=CoreImpl-worker130} completed command SetAgentState for sequencer: [1234]

Now if dig further into the Desktop WebServices logs, yoiu can see that system events are frequently received with systemEventId=10 which indicates (SYS_INSTRUMENT_ OUT_OF_SERVICE - An Enterprise Agent device target has been removed from service. SystemEventArg1 contains the PeripheralID of the peripheral, and SystemEventText contains the AgentInstrument that was removed from service. )

0051220500: 10.106.81.82: Apr 14 2017 14:01:24.585 -0700: %CCBU_CoreImpl-worker118-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :26553 , agentstate : 2, workmode : 0, reason code: 4, forceflag :1, agentcapacity: 0, agentext: 1234, agentid: 1234, supervisorid: null, ssoFlag=false][cti_message_name=SetAgentStateReq]: Message going to the backend cti server 


0051223431: 10.128.224.141: Apr 14 2017 14:01:24.766 -0700: %CCBU_CoreImpl-worker118-7-DISPATCHER:  {Thrd=CoreImpl-worker118} completed command SetAgentState for sequencer: [1234] 

 

0051227244: 10.106.81.82: Apr 14 2017 14:01:27.078 -0700: %CCBU_CoreImpl-worker130-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :26563 , agentstate : 2, workmode : 0, reason code: 4, forceflag :1, agentcapacity: 0, agentext: 1234, agentid: 1234, supervisorid: null, ssoFlag=false][cti_message_name=SetAgentStateReq]: Message going to the backend cti server 


0051229958: 10.128.224.141: Apr 14 2017 14:01:27.274 -0700: %CCBU_CoreImpl-worker130-7-DISPATCHER:  {Thrd=CoreImpl-worker130} completed command SetAgentState for sequencer: [1234]

In Finesse System Events are critical Events and which are processed if no other events are in execution queue. All other pending events will be backed up due to this condition and it will result in general slowness of event processing.

Also Finesse is randomly seeing a delay in CLEAR_CONNECTION command of about one minute, which are timing out (CF_REQUEST_TIMEOUT_REJECTION). This is for the Same extension for which Finesse is getting a SystemEvent.

Here is the specific example what happened in Finesse for one of the instance:

Finesse Web service received the request and sent it to the ICM CTI server

0051049979: 10.106.81.82: Apr 14 2017 13:56:20.806 -0700: %CCBU_CoreImpl-worker146-6-MESSAGE_TO_CTI_SERVER: %[cti_message=Invoke id :26488 , callId: 54473438, extension: 8888, peripheralId: 5000, connectionDeviceIdType : 0, connectionDeviceId: 8888][cti_message_name=ClearConnectionReq]: Message going to the backend cti server

0051061608: 10.106.81.82: Apr 14 2017 13:57:21.758 -0700: %CCBU_CTIMessageEventExecutor-0-6-DECODED_MESSAGE_FROM_CTI_SERVER: %[cti_message=CTIControlFailureConf [failureCode=78, peripheralErrorCode=0, text=null]CTIMessageBean [invokeID=26488, cti_sequence_id=863946, msgID=35, timeTracker={"id":"ControlFailureConf","CTI_MSG_NOTIFIED":1492203441758,"CTI_MSG_RECEIVED":1492203441758}, msgName=ControlFailureConf, deploymentType=CCE]][cti_response_time=0][dispatch_phase=DnD-CHECKPOINT-3A]: Decoded Message to Finesse from backend cti server

PG CTI server received the request

13:56:20:784 cg1A-ctisvr Trace: *** ClearConnectionRequest DefaultCallInstrument - callID:54473438.+9999(d) ... inst:"8888" 

13:56:20:784 cg1A-ctisvr Trace: CSTAClearConnection:  InvokeID=0x2114a2fb Call(CID=54473438,Dev=8888,Type=Static) 

13:56:20:784 cg1A-ctisvr Trace: PRIVATE_DATA: CompoundRequestType=Unknown CSTA Event Type SubjectDevice=8888

However due to an JTAPI exception, the call didn't get disconnected normally.

13:56:20:846 PG1A-jgw1 Trace: MsgClearConnection: CID: 54473438 ConnDevID: 8888/0 CompReqType: -1 InvID: 8285054. 

13:56:35:862 PG1A-jgw1 Trace: Call to connection[54473438,8888].disconnect() encountered exception.  com.cisco.jtapi.PlatformExceptionImpl: Could not meet post conditions of disconnect().

Eventually the call gets cleared on ICM side.

14:08:05:647 PG1A-opc Trace: CSTA_CONNECTION_CLEARED (PID=5000):  DroppedConn=(CID=54473438,Dev=8888,Type=Static) ReleasingDev=(DEV)8888 localConnInfo(-1)=CS_NONE cause=-1 

14:08:05:647 PG1A-opc Trace: CSTAConnectionClearedEvent_PD:: CorrData=102785 UniqueID=00000000003F32DE0309F16000000000 

14:08:05:647 PG1A-opc Trace: Call:SetCallState (PID=5000) - CallID=54473438 CallObj=(0x6c33110) NewState=NONE OldState=CONNECTED. 

14:08:05:647 PG1A-opc Trace: Cleared(PID=5000): CallObj=(0x6c33110)  SrcConn(CID=54473438,Dev=+13003001858,DevType=Dynamic) DestConn(CID=54473438,Dev=8888,DevType=Static) 

14:08:05:647 PG1A-opc Trace: AgentStateUpdate (OPC Time=04/14 21:08:05) - Updating 2 Agent Records on Peripheral 5000 

14:08:05:647 PG1A-opc Trace:  - PID=5000 AGSTID=7295 Periph#=8888 Ext=8888 SG#=29808 Dur=0 State=AS_NOT_READY CurrentLine=0 Reason=8 NetworkTID=0 NumLines=0 AgentObj=0x52f50f8 

14:08:05:647 PG1A-opc Trace: SetAgentState: ASTID=7295 Periph#=8888 MRDomainID=1 SGSTID=5000 SG#=29808(0x7470) OldState=AS_BUSY_OTHER NewState=AS_NOT_READY Duration=0 CurLine=0 ReasonCode=8 AgentObj=0x52f50f8 

14:08:05:647 PG1A-opc Trace:  - PID=5000 AGSTID=7295 Periph#=8888 Ext=8888 SG#=3957053 Dur=0 State=AS_NOT_READY CurrentLine=0 Reason=8 NetworkTID=0 NumLines=0 AgentObj=0x52f50f8

Conclusion

The reason why all the agent State Event is delayed is because when Finesse sends the 'ClearConnectionReq' to CTI Server, CTI Server sends the request to Periheral Gateway (PG) Open Peripheral Controller (OPC) service, which sends the request to CUCM via the PG Peripheral Interface Manager (PIM) / PG JTAPI Gateway (JGW). CTI Server waits for the response from CUCM (which eventually failed), Finesse starts getting the SystemEvents which takes priority and hence the Delay to respond to the agent state request.

Resolution

The resolution of this problem is tracked by these defects which are fixed in Finesse 11.5(1) ES4

CSCve12017 For ICM (pg.cucm)

CSCve11977 For Finesse

Revision History

Revision	Publish Date	Comments
1.0	14-Sep-2017	Initial Release

Contributed by Cisco Engineers

Rohit Harsh
Cisco TAC Engineer

Was this Document Helpful?

Feedback

Contact Cisco

Open a Support Case
(Requires a Cisco Service Contract)

This Document Applies to These Products

Finesse

Troubleshoot Finesse Agent State Change Delayed

Available Languages

Bias-Free Language

Contents

Introduction

Prerequisites

Requirements

Components Used

Problem

Troubleshoot

Conclusion

Resolution

Revision History

Contributed by Cisco Engineers

Was this Document Helpful?

Contact Cisco

This Document Applies to These Products