• X86 Module Development

X86 Module Development

Overview

An Intel Skylake x86 Processor Module is available for installation into one of the internal module bays within the Cisco Nexus 3550-F Fusion (formerly ExaLINK Fusion). This processor is available for you to run your own applications on, allowing construction and deployment of any number of custom appliances. The CPU is not required by Cisco for running or management of the rest of the Nexus 3550-F, so you are not virtualized or sandboxed in any way - you have complete control over the processor.

Specifications

The specifications of the processor module are as follows:

• Intel Core-i7 Skylake 6820EQ CPU, 4 cores @ 2.8GHz, 3.5GHz turbo, QM170 chipset
• 32GB DDR4-2133MT/s (PC4-17000)
• Dual mSATA SSDs, typically shipped with Samsung 850 EVO 250GB as RAID1 for operating system
• M.2 PCIe NVMe SSD, typically shipped with Samsung 950 PRO 512GB capable of sustained write at 1,500MByte/s
• Onboard low latency Cisco Nexus SmartNIC K35-Q (formerly ExaNIC X40), providing 8x10Gbe (2x40G via firmware update) interfaces which can be connected to other Nexus 3550-F objects (mux, switch, front panel ports etc)
• Onboard 1Gb LAN, typically patched through to front panel
• Trusted Platform Module (TPM)
• Intel AMT allows remote KVM access, including operating system install and BIOS config
• Multiple serial ports available
• Multiple FPGA development options available
• Access to PPS from front panel or Nexus 3550-F GPS receiver for highly accurate time synchronization

Top and bottom view of x86 Processor Module, showing drive bays

Architecture

This module consists of a carrier board designed by Cisco, and a 3rd party x86 processor board in a COM Express form factor. This design allows for potential upgrading of the x86 processor board to newer microarchitectures - for example Intel's Kaby Lake or Cannonlake, without changing the underlying carrier board.

The 3rd party processor board currently shipping is the SH960 from DFI.

As can be seen below the carrier board features the SmartNIC K35-Q (X40), connectors for mounting mSATA's and the M.2 drive, along with all the required power supply, support and management hardware.

Block diagram of x86 Processor Module

x86 Processor Module consists of a COM Express module installed on Cisco's carrier board

x86 Processor Module installed into Nexus 3550-F

System Power Management

Power State

The Nexus 3550-F CLI is used to enable/disable power to the processor module. Assuming the processor is installed into module Y, the module can be powered up/down using the following commands:

admin@N3550-F> configure module Y power on
Module Y powered on
admin@N3550-F> configure module Y power off
Module Y powered off

This is a hard power off, ie the equivalent of removing all power feeds to a server.

When operating, the power consumption of the processor module can be displayed using the show power command as follows:

admin@N3550-F> show power
             Voltage  Current
             -------  -------
Line card A  12.3 V   0.3 A
Line card B  12.3 V   0.3 A
Line card C  12.2 V   0.5 A
Module X     12.3 V   0.9 A
Module Y     12.2 V   2.7 A

The temperature of the processor module can be read using the show temperature command, as follows:

admin@N3550-F> sh temp
             Temperature
             ------------------------------
Crosspoint   36.9 C  36.9 C  34.3 C  34.3 C
Line card A  34.0 C
Line card B  35.4 C
Line card C  35.8 C
Mainboard    30.9 C  34.9 C
Module X     34.0 C  33.0 C
Module Y     52.0 C  51.0 C

The first temperature listed above for module Y is the carrier board temperature. The second is the temperature of the SmartNIC K35-Q (X40) FPGA. There is no way to directly read the CPU temperature from the Nexus 3550-F CLI, however the CPU and FPGA are thermally coupled by the heatsink. The CPU temperature can be read by logging into the processor module itself and running an application such as lm_senors, i7z etc.

Installed Operating System

The default configuration ships with Linux CentOS 7 installed onto a RAID1 partition created from the dual mSATA drives. The M.2 NVMe drive is mounted at /data. The login for this system as it ships is:

username: root
password: exablaze

Refer below for console access and Network Interfaces 459647 options for logging into the operating system.

Primary Serial Port

There are 2 serial ports provided on the x86 processor module, both of which are connected through to the Nexus 3550-F management processor. One of these can be used to gain console access to the x86 processor module. This is done by enabling a serial-server on the Nexus 3550-F and connecting to it remotely via the Nexus 3550-F management port. Using this feature allows you to access the x86 processor module for management purposes without sacrificing a high speed front panel port

As can be seen in the diagram above, the remote host connects to a serial server running on the Nexus 3550-F management processor. Traffic received/transmitted on this connection is routed to the x86 processor module's serial port, where a getty session would typically be running, enabling console access to the system. Commands to configure this would be as follows:

admin@N3550-F> configure module Y serial-server port 1444
Serial server enabled for module Y on TCP port 1444

---

WarningNo authentication or other security is provided on this connection (other than the Access Control List for the whole management interface)

---

Beginning with Nexus 3550-F Fusion Software v1.16.0, you can restrict access to the connection to only the localhost by using the localhost-only option.

To restrict connection access only to the localhost, use the following command:

admin@N3550-F> configure module x serial-server port 9999 localhost-only
Serial server enabled for module X on TCP port 9999 (localhost only)

Once the serial server is enabled on the Nexus 3550-F, various applications can be used to pass traffic to/from the connection. For example, telnet can be used on the remote host to login to the x86 processor module as follows:

$ telnet N3550-F 1444
Trying 192.168.220.11...
Connected to N3550-F.
Escape character is '^]'.

CentOS Linux 7 (Core)
Kernel 4.6.2-1.el7.elrepo.x86_64 on an x86_64

fusion_x86 login: root
Password:
Last login: Mon Aug  8 13:22:02 on tty1
[root@fusion_x86 ~]#

To disable the serial-server, the standard no form of the command can be issued:

admin@N3550-F> config module Y no serial-server
Serial server disabled for module Y

Secondary Serial Port

The second serial port on the x86 processor module can be used to gain access to the CLI of the Nexus 3550-F the processor module resides in.

To enable this, the Nexus 3550-F must first be instructed to enable CLI access via this serial port:

admin@N3550-F> configure module Y serial-console
Serial console access enabled for module Y

We can then use a standard terminal application on the x86 module to open the serial port and gain access to the CLI login on the Nexus 3550-F:

[root@fusion_x86 ~]# picocom /dev/ttyS0 --baud 9600
picocom v1.7

port is        : /dev/ttyS0
flowcontrol    : none
baudrate is    : 9600
parity is      : none
databits are   : 8
escape is      : C-a
local echo is  : no
noinit is      : no
noreset is     : no
nolock is      : no
send_cmd is    : sz -vv
receive_cmd is : rz -vv
imap is        :
omap is        :
emap is        : crcrlf,delbs,

Terminal ready

N3550-F login: admin
Password:
admin@N3550-F>
admin@N3550-F>

The no form of the command can be used to disable to console server:

admin@N3550-F> config module Y no serial-console
Serial console disabled for module Y

Network Interfaces

Cisco Nexus SmartNIC

There is an onboard SmartNIC K35-Q (X40) embedded into the processor module. Whilst this is not in the same physical form factor as the SmartNIC K35-Q (X40) (i.e. it's not a PCIe card), the functionality and performance is exactly the same. The SmartNIC K35-Q (X40) provides 8 ultra low latency network interfaces to the host processor module, each of which can be configured for operation at 10G, 1G or 100M (with 40G firmware in development). The standard features of all Cisco Nexus SmartNICs (formerly ExaNIC) apply, including hardware timestamping, flow steering and kernel bypass for low latency transparent acceleration of sockets applications.

When the Nexus 3550-F detects an installed processor module, nine new ports are added to the system, for example Y1-Y9 assuming the module was installed into bay Y.

These interfaces/ports can be patched directly through to the front panel, or used to monitor & capture data flowing elsewhere throughout the Nexus 3550-F. The ports can be displayed with the show port command, for example we can see the following on the Nexus 3550-F CLI:

admin@N3550-F> sh port Y*
Port Status  Description
---- ------- -----------
Y1   unused
Y2   unused
Y3   unused
Y4   unused
Y5   unused
Y6   unused
Y7   unused
Y8   unused
Y9   unused

Port Y9 is the onboard LAN (described below), and Y1-Y8 are the 8 SmartNIC interfaces, from exanic0:0 through to exanic0:7. As with any other port, they can have an alias and description set, get patched to another port, be added to a mux object etc.

Note: Setting the speed of a port in this case just configures the signal recovery circuitry on the Nexus 3550-F for optimal performance, it does not change the speed the SmartNIC is configured for on the Host - the user must ensure this is done independently.

In order to patch one of these SmartNIC ports to the front panel, the patch command should be used:

admin@N3550-F> conf patch Y1 B2
Patch created between port "Y1" and port "B2"

This would allow, for example, a remote host to connect to the processor module as follows:

Onboard LAN

The motherboard within the processor module has an Intel(R) i217 Ethernet controller capable of running at 1G and 100M. This is exposed as port Y9, and would typically be patched out to allow for remote connection and management:

The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. Refer to the section on AMT for further details.

Intel AMT

The onboard Ethernet controller supports Intel Active Management Technology (AMT), which allows for a number of handy IPMI/remote management capabilities. For example, you are able to establish a remote KVM over IP session with the processor, so you can interact with the system as a local user, for example reboot, enter the BIOS, install operating systems etc. Users might be familiar with similar technologies from Dell (iDRAC), HP (iLO) etc.

Cisco ships the processor modules with AMT enabled with a password set which is Exablaze1!. AMT is only available via the onboard LAN (ie port Y9), not any of the SmartNIC interfaces. It can be disabled via the KVM methods described below.

Intel AMT can be connected to via a number of different methods. It is configured to be assigned an IP address from a DHCP server on the network. Once the IP has been assigned, it can be discovered by searching for open ports on port 16992, for example:

$ sudo nmap --open -sT -p 16992 192.168.220.1/24
[sudo] password for donald:

Starting Nmap 6.40 ( http://nmap.org ) at 2016-08-09 18:17 AEST
Nmap scan report for 192.168.220.174
Host is up (-0.088s latency).
PORT      STATE SERVICE
16992/tcp open  amt-soap-http
MAC Address: 00:01:29:5D:EA:6B (DFI)

Nmap done: 256 IP addresses (122 hosts up) scanned in 8.14 seconds

As can be seen above, the local DHCP server has assigned the module an IP address of 192.168.220.174. AMT has a fairly simple web interface for interacting with the system, so browsing to http://192.168.0.174:16992 and logging in as user admin with password Exablaze1! yields the following:

The default admin password can be changed by selecting User Accounts, and clicking the Change Admin... button.

A number of software utilities are available the allow for greater control over the system, including event & access logs, full KVM access (including into the BIOS), storage redirection, serial-over-lan etc.

Mesh Commander (compatible with connecting from Windows, Linux and Mac hosts) is known to work well, and the somewhat older Intel Platform Solution Manager is another option.

Editing the x86 Module BIOS using Mesh Commander from a remote PC

Intel AMT can be completely disabled in the BIOS, however users should be aware that once this is done, the only way to re-enable it is to remove the lid of the Nexus 3550-F chassis and connect an HDMI monitor and USB keyboard to the connectors provided on the x86 module, and entering the MEBX extension within the BIOS menus.