The documentation set for this product strives to use bias-free language. For the purposes of this documentation set, bias-free is defined as language that does not imply discrimination based on age, disability, gender, racial identity, ethnic identity, sexual orientation, socioeconomic status, and intersectionality. Exceptions may be present in the documentation due to language that is hardcoded in the user interfaces of the product software, language used based on RFP documentation, or language that is used by a referenced third-party product. Learn more about how Cisco is using Inclusive Language.
This document describes the steps to troubleshoot the Security Services Exchange (SSE) Connector process when it becomes disabled on the Firepower Management Center (FMC) or Firepower Threat Defense (FTD) devices for the integration with Cisco Threat Response (CTR).
Cisco recommends that you have knowledge of these topics:
The information in this document is based on these software and hardware versions:
The information in this document was created from the devices in a specific lab environment. All of the devices used in this document started with a cleared (default) configuration. If your network is live, ensure that you understand the potential impact of any command.
SSEConnector is a process on the Firepower devices after 6.4.0 which enrolls the devices into the SSE portal. The FMC broadcasts to all the managed FTDs when the Cisco Cloud configuration is set to On or Off. Once the Cisco Cloud is enabled, the SSEConnector service starts the communication between the SSE portal and the Firepower devices. Each FTD requests the FMC for a registration token which allows the devices to be integrated into the SSE portal. After this integration, the SSE context is activated on the devices and the EventHandler is reconfigured to send Intrusion Events to the Cisco Cloud.
Threat Response is a threat incident response orchestration hub, which supports and automates integrations across multiple Cisco Security products. Threat Response accelerates key security tasks: detection, investigation, and remediation, and is a keystone in our integrated security architecture.
The goal of Threat Response is to help network operations teams and incident responders understand threats on their network by all of the threat intelligence gathered and combined available from Cisco and third parties.
But more than anything else, Threat Response is designed to reduce the complexity of security tools, help identify threats, and speed up incident response.
Threat Response is an integration platform (https://visibility.amp.cisco.com/). The system works via “modules”, which are independent pieces of code that handle communications with different integrated systems (eg Threat Grid, or AMP). These modules handle all 3 of the functions that an integrated system can provide (enrichment, local context, and response).
What can CTR be used for?
When you search for an observable, all of your configured modules ask the systems for which they are responsible to search for any record of those observables. They then take the provided responses and pass them back to Threat Response, then it takes the collected results from all modules (in this case the Stealthwatch module), and sorts and organizes the data and display it in a graph.
To integrate CTR with different products are involved two more portals “https://castle.amp.cisco.com/” (Castle) and “https://admin.sse.itd.cisco.com/app/devices” (Security Services Exchange)
Here you can manage the Cisco Security Accounts:
A Cisco Security account allows you to manage multiple applications within the Cisco Security portfolio. In accordance on your licensing entitlements, this can include:
This portal is an extension of the CTR portal, where you can manage the devices that have been registered in the CTR portal, so here you can create the tokens needed to integrate the products.
Security Services Exchange provides device, service, and event management when you integrate certain Cisco security products with Cisco Threat Response, including these products and features:
On the FMC, first, verify on System > Licenses > Smart Licenses you are not on evaluation mode.
Verify now under System > Integration on the Smart Software Satellite tab that the selected option is Connect directly to Cisco Smart Software Manager as this feature is not supported on an air-gapped environment.
Navigate to System > Integration on the Cloud Services tab and check that Cisco Cloud Event Configuration option is turned on.
These next URLs needs to be allowed as IPs can change:
US Region
EU Region
APJ Region
Both FMC and FTD need a connection to the SSE URLs on their management interface, to test the connection, enter these commands on the Firepower CLI with root access:
curl -v https://api-sse.cisco.com/providers/sse/services/registration/api/v2/clients --cacert /ngfw/etc/ssl/connectorCA.pem
curl -v https://est.sco.cisco.com --cacert /ngfw/etc/ssl/connectorCA.pem
curl -v https://eventing-ingest.sse.itd.cisco.com --cacert /ngfw/etc/ssl/connectorCA.pem
curl -v https://mx01.sse.itd.cisco.com --cacert /ngfw/etc/ssl/connectorCA.pem
After each command is run, you must see this line around the end of the connection: Connection #0 to host "URL" left intact.
If the connection times out or you don't receive this line on the output, please verify that the management interfaces are allowed access to these URLs and that there are no upstream devices that block or modify the connection between the devices and these URLs.
The certificate check can be bypassed with this command:
root@ftd01:~# curl -v -k https://api-sse.cisco.com
* Rebuilt URL to: https://api-sse.cisco.com/
* Trying 52.4.85.66...
* Connected to api-sse.cisco.com (52.4.85.66) port 443 (#0)
* ALPN, offering http/1.1
* Cipher selection: ALL:!EXPORT:!EXPORT40:!EXPORT56:!aNULL:!LOW:!RC4:@STRENGTH
* successfully set certificate verify locations:
* CAfile: none
CApath: /etc/ssl/certs
* TLSv1.2 (OUT), TLS header, Certificate Status (22):
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Request CERT (13):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Certificate (11):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Client hello (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES128-GCM-SHA256
* ALPN, server accepted to use http/1.1
* Server certificate:
* subject: C=US; ST=California; L=San Jose; O=Cisco Systems, Inc.; CN=api -sse.cisco.com
* start date: 2019-12-03 20:57:56 GMT
* expire date: 2021-12-03 21:07:00 GMT
* issuer: C=US; O=HydrantID (Avalanche Cloud Corporation); CN=HydrantID S SL ICA G2
* SSL certificate verify result: self signed certificate in certificate c hain (19), continuing anyway.
> GET / HTTP/1.1
> Host: api-sse.cisco.com
> User-Agent: curl/7.44.0
> Accept: */*
>
< HTTP/1.1 403 Forbidden
< Date: Wed, 08 Apr 2020 01:27:55 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 9
< Connection: keep-alive
< Keep-Alive: timeout=5
< ETag: "5e17b3f8-9"
< Cache-Control: no-store
< Pragma: no-cache
< Content-Security-Policy: default-src 'self'
< X-Content-Type-Options: nosniff
< X-XSS-Protection: 1; mode=block
< Strict-Transport-Security: max-age=31536000; includeSubdomains;
Note: You get the 403 Forbidden message as the parameters sent form the test is not what SSE expects but this proves enough to validate connectivity.
You can verify the connector properties as below.
# more /ngfw/etc/sf/connector.properties
registration_interval=180
connector_port=8989
connector_fqdn=api-sse.cisco.com
In order to check the connectivity between the SSConnector and the EventHandler you can use this command, this is an example of a bad connection:
root@firepower:/etc/sf# netstat -anlp | grep EventHandler_SSEConnector.sock
unix 2 [ ACC ] STREAM LISTENING 3022791165 11204/EventHandler /ngfw/var/sf/run/EventHandler_SSEConnector.sock
In the example of an established connection you can see that the stream status is connected:
root@firepower:/etc/sf# netstat -anlp | grep EventHandler_SSEConnector.sock
unix 2 [ ACC ] STREAM LISTENING 382276 7741/EventHandler /ngfw/var/sf/run/EventHandler_SSEConnector.sock
unix 3 [ ] STREAM CONNECTED 378537 7741/EventHandler /ngfw/var/sf/run/EventHandler_SSEConnector.soc
In order to send events from the FTD device to SEE a TCP connection needs to be established with https://eventing-ingest.sse.itd.cisco.com This is an example of a connection not established between the SSE portal and the FTD:
root@firepower:/ngfw/var/log/connector# lsof -i | grep conn
connector 60815 www 10u IPv4 3022789647 0t0 TCP localhost:8989 (LISTEN)
connector 60815 www 12u IPv4 110237499 0t0 TCP firepower.cisco.com:53426->ec2-100-25-93-234.compute-1.amazonaws.com:https (SYN_SENT)
In the connector.log logs:
time="2020-04-13T14:34:02.88472046-05:00" level=error msg="[firepower.cisco.com][events.go:90 events:connectWebSocket] dial tcp 18.205.49.246:443: getsockopt: connection timed out"
time="2020-04-13T14:38:18.244707779-05:00" level=error msg="[firepower.cisco.com][events.go:90 events:connectWebSocket] dial tcp 100.25.93.234:443: getsockopt: connection timed out"
time="2020-04-13T14:42:42.564695622-05:00" level=error msg="[firepower.cisco.com][events.go:90 events:connectWebSocket] dial tcp 18.205.49.246:443: getsockopt: connection timed out"
time="2020-04-13T14:47:48.484762429-05:00" level=error msg="[firepower.cisco.com][events.go:90 events:connectWebSocket] dial tcp 100.25.93.234:443: getsockopt: connection timed out"
time="2020-04-13T14:52:38.404700083-05:00" level=error msg="[firepower.cisco.com][events.go:90 events:connectWebSocket] dial tcp 100.25.93.234:443: getsockopt: connection timed out"
Note: Noticed that the IP addresses displayed 18.205.49.246 and 18.205.49.246 belong to https://eventing-ingest.sse.itd.cisco.com might change, this is why the recommendation is to allow the traffic to SSE Portal based on URL instead of IP addresses.
If this connection is not established, the events are not sent to the SSE portal, this is an example of an established connection between the FTD and the SSE portal:
root@firepower:# lsof -i | grep conn
connector 13277 www 10u IPv4 26077573 0t0 TCP localhost:8989 (LISTEN)
connector 13277 www 19u IPv4 26077679 0t0 TCP 192.168.1.200:56495->ec2-35-172-147-246.compute-1.amazonaws.com:https (ESTABLISHED)
After the upgrade to 6.4 the SSE connector does not communicate with the SSE portal. Connector.log provides errors similar to events:(*Service).Start] Could not connect to ZeroMQ PUSH endpoint: could not dial to \"ipc:///ngfw/var/sf/run/EventHandler_SSEConnector.sock\": dial unix /ngfw/var/sf/run/EventHandler_SSEConnector.sock: connect: no such file or directory\n"
Restart the SSEConnector Service:
1) sudo pmtool disablebyid SSEConnector
2) sudo pmtool enablebyid SSEConnector
3) Restart the device. Upon restart, the device communicates to the cloud.
Debug logs - Shows successful connection or failure messages
/ngfw/var/log/connector/connector.log
Configuration Settings
/ngfw/etc/sf/connector.properties
Configuration Settings
curl localhost:8989/v1/contexts/default