Cisco UCS C890 M5 Rack Server for SAP HANA with SUSE Linux Enterprise Server for SAP

 

 

Executive Summary

Organizations in every industry are generating and using more data than ever before, from customer transactions and supplier delivery information to real-time user-consumption statistics. Without reliable infrastructure that can store, process, and analyze big data sets in real time, companies are unable to use this information to their advantage.

The Cisco® solution for SAP HANA® using the high-performance Cisco UCS C890 M5 server is well-suited for mission-critical business scenarios such as for medium- to large-size SAP HANA database management systems including high availability and secure multitenancy. The Cisco solution provides a robust platform for SAP HANA workloads as a single node (scale-up) SAP HANA instance.

The SAP HANA platform provides a scalable database with advanced analytical capabilities and intuitive application-development tools in an in-memory data platform. SAP HANA Tailored Data Center Integration (TDI) supports nonvolatile Intel® Optane™ Persistent Memory (PMem) that brings together the low latency of memory and the persistence of storage. PMem modules provide faster access to data and retain data across power cycles.

SUSE Linux Enterprise Server (SLES) for SAP Applications is the leading Linux platform for SAP HANA, SAP NetWeaver, and SAP S/4H solutions. It provides optimized performance, it can reduce downtimes, and it provides faster SAP landscape deployments. Features include secure SAP HANA systems that can ease the transition to SAP S/4HANA for systems administrators.

 

Solution Overview

This section introduces the SAP HANA on Cisco UCS C890 M5 Rack Server solution.

Introduction

The Cisco Unified Computing System™ (Cisco UCS®) C890 M5 Server for SAP HANA solution provides a prevalidated, ready-to-deploy infrastructure, reducing the time and complexity involved in configuring and validating a traditional data center deployment. The reference architecture detailed in this document highlights the resiliency and ease of deployment of an SAP HANA solution using the SUSE Linux Enterprise for SAP Applications operating system.

The SAP HANA in-memory database combines transactional and analytical SAP workloads and hereby takes advantage of the low-cost main memory, data-processing capabilities of multicore processors, and faster data access. The Cisco UCS C890 M5 rack server is equipped with the second-generation Intel® Xeon™ scalable processors, and it supports DDR4-only memory modules or mixture with Intel Optane Persistent Memory.

The solution combines Cisco hardware and software components optimized for SAP HANA workloads as preconfigured and validated architecture. The Cisco UCS C890 M5 provides the advantages of an integrated computing and storage stack while the standard configuration already enables the server for SAP HANA TDI deployments. SAP HANA TDI enables organizations to run multiple SAP HANA production systems on a shared infrastructure. It also enables customers to run SAP application servers and the SAP HANA database hosted on the same infrastructure.

Audience

The audience for this document includes, but is not limited to, field consultants, professional services staff, partner engineers, and customers who want to deploy SAP HANA on a Cisco UCS C890 M5 rack server. External references are provided wherever applicable, but readers are expected to be familiar with the technology, infrastructure, and database security policies of the customer installation.

Purpose of this document

This document provides high-level steps to deploy and configure SAP HANA on a Cisco C890 M5 rack server. It showcases one of the variants of this Cisco solution for SAP HANA. Although readers of this document are expected to have access to the knowledge bases of SUSE and SAP and sufficient knowledge to install and configure the products used, configuration details important to the solution deployment are provided.

Solution summary

The Cisco C890 M5 rack server for SAP HANA scale-up solution is prevalidated with local storage technology. Details about the system architecture and design considerations are available in the Cisco UCS C890 M5 rack server for SAP HANA white paper accompanying this document. It is a defined set of hardware and software that serves as an integrated foundation for nonvirtualized workloads.

All components are connected and configured according to best practices of Cisco, SUSE, and SAP SE. The Cisco UCS C890 M5 rack server data sheet summarizes the server specifications. Table 1 shows proposed configurations for the specific SAP HANA use cases.

 

 

 

Table 1.        Cisco UCS C890 M5 rack server configuration overview

Technology Overview

This section provides a technical overview of the computing and storage components and the solution requirements key for SAP HANA operation in this environment.

Solution architecture

The high-performance 8-socket Cisco C890 M5 rack server is based on the second-generation Intel Xeon scalable processors. It is well-suited for mission-critical business scenarios such as for medium- to large-size SAP HANA database management systems including high availability and secure multi-tenancy.

The baseline server configuration uses eight 25-GE uplink network ports and four 32-GB Fibre Channel (FC) Host Bus Adapters (HBAs). One 12-GBps MegaRAID SAS controller manages two internal, RAID-1-protected SAS drives to host the SUSE Linux Enterprise for SAP Applications operating system. A second 12-GBps MegaRAID SAS controller manages at a minimum 10 (up to 16) internal, RAID-5 protected and hot-swappable SAS disks to host the persistent data of the SAP HANA database.   

Solution requirements

Several hardware and software requirements must be fulfilled before you set up and install an SAP HANA server. SAP increases flexibility and allows alternative configurations to the SAP HANA appliances with the six phases of SAP HANA TDI, which are described in SAP HANA Tailored Data Center Integration – Overview.

CPU architecture

The 8-socket Cisco UCS C890 M5 rack server is available with the second-generation Intel Xeon Scalable Platinum processors (either the 8276 or the 8280L) with 28 cores per CPU socket each. Both CPU models are certified for SAP HANA but are specified for a different amount of addressable memory size.

The smaller platinum 8276 processor addresses a maximum memory size of 1 TB and is available for the 6-TB main memory configuration only. The recommended platinum 8280L processor addresses a maximum memory size of 4.5 TB and is available for all memory configurations listed in Table 1.

Memory platform support

SAP HANA supports the following DDR4 DRAM memory configurations:

●      SAP HANA 2.0 memory per-socket ratio is up to 768 GB for SAP NetWeaver Business Warehouse (BW) and DataMart.

●      SAP HANA 2.0 memory per-socket ratio is up to 1536 GB for SAP Business Suite on SAP HANA (SoH).

With SAP HANA 2.0 SPS 03 or later, Intel Optane Persistent Memory modules are supported. In App Direct mode, the PMem modules appear as byte-addressable memory resources that are controlled by SAP HANA.

The use of Intel Optane PMem requires a homogeneous, symmetrical assembly of DRAM and PMem modules with maximum use of all memory channels per processor. Presently the total usable memory capacity for SAP HANA is up to 24 TB. 

Various capacity ratios between PMem and DRAM memory modules are supported. The ratios are heavily dependent on the data model and data distribution. A correct SAP sizing exercise is strongly recommended before you consider Intel Optane PMem for SAP HANA.

Ratio examples include the following:

●      1:1 ratio: A single 128-GB Intel Optane PMem is matched with a single 128-GB DDR4 DIMM.

●      1:2 ratio: A single 256-GB Intel Optane PMem is matched with a single 128-GB DDR4 DIMM.

●      1:4 ratio: A singe 512-GB Intel Optane PMem is matched with a single 128-GB DDR4 DIMM.

You can use different-sized Intel Optane PMem and DDR4 DIMMs together as long as you maintain supported ratios. The FAQ: SAP HANA Persistent Memory provides more information if you are interested in using PMem in SAP HANA environments.

Operating system

The Cisco C890 M5 rack server is certified for SAP HANA starting with SUSE Linux Enterprise Server (SLES) for SAP Applications 15 SP2. To evaluate compatibility information between Linux operating system release and SAP HANA platform releases, refer to SAP Note 2235581 – SAP HANA: Supported Operating Systems.

Disk size and file-system requirements

Since SUSE Linux Enterprise 12, Btrfs is the default file system for the root directory “/”, and XFS is the default for all other use cases such as the SAP instance directory “/usr/sap”, which contains the SAP HANA executable programs, instance profiles, and SAP HANA configuration files. The minimum space requirement for the SAP instance directory is 52 GB.

Configure the SAP HANA data, log, and shared directories according to the recommendations detailed in the SAP HANA Storage Requirements. If no application-specific sizing program data is available, the recommended size for the SAP HANA volumes is directly related to the total memory required for the system.

For example, for an SAP HANA scale-up appliance with 12-TB main memory, the recommended SAP HANA file system sizes are as follows:

●      /hana/data = 3 x memory = 3 x 12 TB = 36 TB

●      /hana/log = 512 GB

●      /hana/shared = 1 x memory = 12 TB

Server hardware configuration

It is beyond the scope of this document to explain the Cisco UCS infrastructure installation and connectivity. All physical hardware needs to be racked according to the specific hardware installation guides. This deployment guide assumes the cabling is complete and the IPMI network setup is done.

The server contains a security feature for the BMC firmware stack and is shipped with an individual, unique, and preprogrammed default password for the ADMIN user. Cisco recommends that you change the password after the first logon to the IPMI Console (IPMI-Over-LAN) management web interface.

The server is powered off prior to starting the initial configuration.

Server deployment

This section provides directions for the deployment of the SAP HANA scale-up environment on a Cisco UCS C890 M5 rack server using SLES for SAP Applications 15 SP2.

Naming convention

This guide uses the following conventions for commands that you enter at the command-line interface (CLI). Commands to enter at a CLI prompt:

hostname# hostname

Angle brackets (<>) indicate a mandatory character string that you need to enter like you would enter a variable pertinent to the customer environment or a password.

hostname# ip add <ip_address/mask> dev <interface>

For example, to assign 192.168.1.200/255.255.255.0 to network interface eth0:

hostname# ip add 192.168.1.200/255.255.255.0 dev eth0

BIOS configuration

The following configuration procedure is recommended for the Cisco UCS C890 M5 server for SAP HANA appliance bare-metal installation.

Connect to the KVM console

1.     Connect the keyboard, video, mouse (KVM) cable available in the accessory box to a laptop and the KVM connector on the low right rear side of the server for the initial basic input/output (BIOS) and IPMI configuration.

2.     When physically connected to the server press the Power button located in the top right corner on the front side of the server.

3.     When the Cisco logo appears, press <DEL> to enter the AMI BIOS setup utility in the KVM console.

Main Menu

Set the system date and system time in the main screen of the BIOS.

IPMI menu (BMC network configuration)

4.     Change to the IPMI top menu and use the arrow keys to select the BMC Network Configuration sub-menu.

5.     Update the IPMI LAN Configuration and provide an IP address, subnet mask, and gateway IP address.

6.     Move to the Save & Exit top menu, Save the changes; then reset the server.

7.     When the Cisco logo appears, press <DEL> to enter the AMI BIOS setup utility.

IPMI Console management

Disconnect the KVM cable and connect to the IPMI LAN IP address with your preferred browser. Log in with the user ADMIN and the unique password that came with the server.

1.     To modify the initial password, select the menu Configuration > Users.

2.     Select the row with the ADMIN user and press the Modify User button to change the initial password.

3.     Select the menu Remote Control > Remote Console.

4.     Press the Launch Console button to access the remote console.

5.     The remote console opens in a new window displaying the AMI BIOS setup utility.

Advanced menu (CPU configuration)

1.     Use the arrow keys to select the CPU Configuration sub-menu.

2.     For the bare-metal installation, disable Intel Virtualization Technology.

3.     Select Advanced Power Management Configuration at the bottom of the processor configuration page.

4.     Change Power Technology to Custom.

5.     Select CPU C State Control and disable the Autonomous Core C-State and CPU C6 report.

6.     Select Package C State Control and change the Package C State value to C0/C1 state.

Advanced menu (memory RAS configuration)

Modern servers, including Cisco UCS M5 servers, provide increased memory capacities that run at higher bandwidths and lower voltages. These trends, along with higher application memory demands and advanced multicore processors, contribute to an increasing probability of experiencing memory errors.

Fundamental Error-Correcting Code (ECC) capabilities and scrub protocols were historically successful at handling and mitigating memory errors. As memory and processor technologies advance, reliability, availability, and serviceability (RAS) features must evolve to address new challenges. Improve the server resilience with the Adaptive Double Device Data Correction (ADDDC Sparing) feature, which mitigates correctable memory errors.

7.     Use the arrow keys to select the Chipset Configuration sub-menu.

8.     Continue with North Bridge and Memory Configuration.

9.     Keep all values and select Memory RAS Configuration at the bottom of the screen.

10.  Enable ADDDC Sparing.

Advanced menu (PCIe/PCI/PnP configuration)

In the default configuration, the Option Rom Settings are set to Legacy. You can access configuration utilities of the add-on cards by pressing a key combination such as <CTRL>-<R> for the MegaRAID cards during initialization of the card itself. Changing the BIOS settings to EFI will enable access to the configuration utilities directly from the BIOS after the next reboot.

11.  Use the arrow keys to select the PCIe/PCI/PnP Configuration sub-menu.

12.  Change the PCI Devices Option Rom from Legacy to EFI.

13.  Press <ESC> and select optionally the network stack configuration to disable the PXE and HTTP boot support if not required.

Boot menu

14.  Switch to the Boot screen in the top menu. Confirm that the entry for boot mode select is UEFI.

Save and exit menu

15.  Save all changes and reset the server to activate the changes.

After the system initialization, the Cisco logo displays. Press <DEL> to enter the AMI BIOS setup utility again.

MegaRAID 3108 configuration

1.     In the Advanced menu of the Setup Utility, select the AVAGO 3108 MegaRAID Configuration Utility.

2.     Select Configure from the Actions section and create a virtual drive.

3.     Select RAID level 1 and the two unconfigured SATA drives to be added.

4.     Apply the changes and provide a virtual drive name like, for example, system. Keep all other settings on default. Save the configuration.

MegaRAID 9361 configuration

1.     From the Advanced screen, select the AVAGO 9361-16i MegaRAID Configuration Utility.

2.     Select Configure from the Actions section and create a virtual drive.

3.     Select RAID level 5 and all unconfigured SAS drives to be added.

4.     Apply the changes and provide a virtual drive name like, for example, hana. Change the strip size to 128 KB and the Write Policy to Write Back. Keep all other settings on default and save the configuration.

Intel Optane DC PMem configuration

If the Cisco UCS C890 M5 server is not equipped with Intel PMem modules, the BIOS menu entry is hidden.

1.     From the Advanced screen select Intel Optane DC Persistent Memory Configuration.

2.     Confirm all 48 DIMMs have been detected and are healthy.

For SAP HANA bare-metal installations, no further BIOS configuration is required. The Intel Optane PMem modules are managed using the management utilities from the Linux command line. Return to the main screen and keep the server in the AMI BIOS setup utility.

 

SAP HANA with Intel Optane PMem on VMware vSphere

For SAP HANA VMware-based installations, the Intel Optane PMem modules require initial configuration in the BIOS to enable ESXi to detect the PMem namespaces properly. Further configuration in VMware vCenter (VMware KB article 78094) and configuration changes using the management utilities from the Linux command line within the virtual machine are required. Additional information is available from SAP note 2913410 – SAP HANA about VMware vSphere with PMem.

For the initial configuration of the PMem modules for VMware, follow these steps:

1.     In the Advanced screen, select Intel Optane DC Persistent Memory Configuration.

2.     Select Regions and scroll to the end of the screen. Select Create goal config and press <Enter>.

3.     Keep the defaults to create a goal for App Direct mode. Select Create goal config and press <Enter>.

4.     Save the changes and reset the server to initialize the Intel Optane PMem modules.

5.      When the Cisco logo appears, press <DEL> to enter the AMI BIOS setup utility.

6.     Move to the Advanced screen and select Intel Optane DC Persistent Memory Configuration.

7.     Select Namespaces and then Create namespace.

8.     Select Name and press <Enter> to provide a name for the new namespace and select the region ID accordingly; for example, namespace1.0 for Region ID 0x0001.

9.     Keep the mode on None, thereby creating a namespace in raw mode.

10.  Select Create namespace.

11.  Repeat steps 7 through 9 and create a namespace for each of the eight region IDs.

12.  Select Save from the Save & Exit top menu and press <Enter> to save the changes.

13.  Mount the ESXi ISO installation image to proceed with the VMware installation after the next reboot.

SUSE Linux Enterprise Server for SAP Applications installation

SUSE Linux Enterprise Server for SAP Applications is the reference platform for the software development of SAP. It is optimized for SAP applications like SAP HANA. The SUSE documentation provides an Installation Quick Start Guide that describes the installation workflow procedure. Consult the SLES for SAP Applications 15.x Configuration Guide attached to SAP Note 1944799 - SAP HANA Guidelines for SLES Operating System Installation and follow the installation workflow.

The following supplement SUSE information is available from the SAP notes system:

●      SAP Note 2369910 – SAP Software on Linux: General Information

●      SAP Note 2578899 - SUSE Linux Enterprise Server 15: Installation Note

●      SAP Note 2684254 - SAP HANA DB: Recommended OS settings for SLES 15 for SAP Applications 15

●      SAP Note 1275776 - Linux: Preparing SLES for SAP environments

●      SAP Note 2382421 – Optimizing the Network Configuration on HANA- and OS-Level

Download the appropriate SLES for SAP Applications 15 SP2 ISO installation image from the SUSE customer portal.

1.     In the remote console window, select Virtual Media in the top menu.

2.     Keep the device type ISO image and choose the SLES for SAP Applications 15 SP2 ISO installation image.

3.     Click the Mount button to mount the image and close the window.

4.     In the BIOS Setup Utility, Save & Exit top menu, select Save Changes, and Reset to restart the server.

The Cisco UCS C890 M5 server will automatically boot from the mapped ISO installation image, and the SUSE installation wizard will start. Follow the installation workflow of the SUSE Linux Enterprise Server 15.x for SAP Applications Configuration Guide for SAP HANA with the following changes:

●      Use English as the installation and system default language and a keyboard layout according to your local preferences.

●      In the network & hostname configuration screen, enter the short name of the server <hostname> and not the fully qualified hostname.

●      Register the server and enable update repositories during installation.

●      Add the Legacy Module 15 SP2 during the module selection.

●      Modify the proposed partitioning layout. Create a dedicated /usr/sap partition (or logical volume) with xfs as the file system.

●      Select the correct time zone and ensure that the date and time are set correctly. If applicable, configure a local Network Time Protocol (NTP) server.

Installation settings

The final screen of the SUSE installation workflow is Installation Settings, which offers the review and change of the installation settings. The following changes are recommended for an SAP HANA server installation (refer to Figure 1).

Adapt the software selection and remove the software packages:

●      Gnome

●      SAP Application Service Base

In the same selection screen, add the following software packages:

●      X-Windows

●      SAP HANA Server Base

●      High Availability

In the Security settings section, disable the firewall. Disable Kdump and change the default systemd target to Text Mode before you start the SLES installation.

SUSE installer: Installation Settings

Initial setup

After the first login, modify the system to fit into the existing infrastructure and to follow your company rules. Change the hostname and adapt the network configuration according to your requirements. Proceed with the recommended OS settings provided in SAP note 2684254 – SAP HANA DB: Recommended OS settings for SLES for SAP Applications 15.

SAP HANA partitions

The SAP HANA disk size requirements in the following configuration example can vary based on the available main memory or backup/snapshot configuration, for example.

From the command line, list the available block devices to check for empty devices:

# lsblk

NAME            MAJ:MIN RM   SIZE RO TYPE MOUNTPOINT

sda               8:0    0  26.2T  0 disk

sdb               8:16   0 223.1G  0 disk

├─sdb1            8:17   0   500M  0 part /boot/efi

└─sdb2            8:18   0 222.6G  0 part

  ├─system-root 254:0    0    60G  0 lvm  /

  ├─system-swap 254:1    0     2G  0 lvm  [SWAP]

  ├─system-home 254:2    0    50G  0 lvm  /home

  └─system-sap  254:3    0    50G  0 lvm  /usr/sap

In this example, the operating system uses /dev/sdb, and device /dev/sda is available. Create a Logical Volume Management (LVM) volume group and add three logical volumes from the available space in the group following the SAP HANA sizing requirements.

1.     Use the YaST2 Expert Partitioner to create and manage file systems and RAID devices.

# yast partitioner

2.     Select Volume Management and <ALT-A> to add a volume group.

3.     Provide a volume group name <hanavg> and add device /dev/sda to the volume group. Press <ALT-N> for Next.

4.     In the Volume Management screen, select the new volume group /dev/hanavg.

5.     Press <ALT-L> to open the Logical Volume sub-menu and <ALT-A> to add a logical volume.

6.     Enter the logical volume name <logvol> and press <ALT-N> for Next.

7.     Specify the custom size of 0.5 TiB.

8.     Change the strip size to 128 KiB. Press <ALT-N> for Next.

9.     Keep the logical volume unformatted. Press <ALT-N> for Next.

10.  Do not format and do not mount the device. Press <ALT-N> for Next.”

Repeat the same steps to create the <datavol> and the <sharedvol> with their custom sizes. Then press <ALT-N> for Next. Apply the changes with <ALT-N> and exit YAST.

Create a XFS file system for the three new logical volumes:

# mkfs.xfs -f -b size=4096 /dev/hanavg/logvol

# mkfs.xfs -f -b size=4096 /dev/hanavg/datavol

# mkfs.xfs -f -b size=4096 /dev/hanavg/sharedvol

Create the mount points:

# mkdir -p /hana/log

# mkdir -p /hana/data

# mkdir -p /hana/shared

 

From the command line, list the available block devices to confirm the configuration:

# lsblk -f

NAME               FSTYPE      LABEL UUID       FSAVAIL FSUSE% MOUNTPOINT

sda                LVM2_member       JIl5YZ…

├─hanavg-sharedvol xfs               39efac…        6T    

├─hanavg-datavol   xfs               676e1b…       18T    

└─hanavg-logvol    xfs               0531ce…    511.2G    

sdb

├─sdb1             vfat              4A24-A…    492.1M     2% /boot/efi

└─sdb2             LVM2_member       PED5J9…

  ├─system-root    btrfs             5191a0…     52.2G    10% /

  ├─system-swap    swap              1fa4e8…                  [SWAP]

  ├─system-home    xfs               895fc2…     50.0G     0% /home

  └─system-sap     xfs               3198e4…     49.6G     1% /usr/sap

By default, SLES uses the Universally Unique Identifiers (UUIDs) instead of device names. Create persistent mount points by adding the volumes to /etc/fstab (end the stream with <CTRL-D>).

# cat >> /etc/fstab

UUID=0531ce…                    /hana/log         xfs   defaults   0  0

UUID=676e1b…                     /hana/data        xfs   defaults    0  0

UUID=39efac…                    /hana/shared      xfs   defaults    0  0

Mount the logical volumes:

# mount -a

Disable OS-based memory error monitoring

Linux supports two features related to error monitoring and logging, Error Detection and Correction (EDAC) and mcelog. Both are common in most recent Linux distributions. Cisco recommends disabling EDAC-based error collection to allow all error reporting to be handled in firmware.

Disable EDAC by adding the kernel option edac_report=off. To add the kernel option, use:

# yast bootloader

Mcelog is enabled by default in most recent Linux distributions such as SLES for SAP Applications 15. For customers who prefer to collect all diagnostic and fault information from OS resident tools, mcelog is recommended. Firmware logs may be incomplete when OS logging is enabled.

SAP HANA performance settings

Apply SAP-recommended operating system settings and tuning options.

# saptune solution apply HANA

# saptune daemon start

Recommended software packages

Enable system usage monitoring:

# systemctl enable sysstat

# systemctl start sysstat

Install IPv4 and IPv6 networking utilities:

# zypper in iputils

Install the supportutils, including the latest supportconfig:

# zypper in supportutils

Storage Command Line tool installation

The solution has been tested and certified with the SUSE inbox driver. Use the Storage Command Line Tool (STorCLI) to monitor and manage the two virtual RAID groups. Select Management Software and Tools in the download area of https://www.broadcom.com/products/storage/raid-controllers/megaraid-sas-9361-16i and download the current StorCLI software package. Use the RPM package manager to install the MegaRAID StorCLI tool. Show a summary of controller and controller-associated information using the following command:

# /opt/MegaRAID/storcli/storcli64 show

Fix the BIOS boot order priority

Reboot the server from the CLI. Connect to the IPMI address of the Cisco UCS C890 M5 server with your preferred browser. Use the ADMIN user to log on to IPMI.

1.     From the top menu, select Remote Control and Remote Console from the drop-down menu. The Launch Console button opens a new window with the console redirection.

2.     After the system initialization, the Cisco logo displays. Press <DEL> to enter the AMI BIOS setup utility.

3.     In the BIOS setup utility, change to the boot top menu and select UEFI Hard Disk Drive BBS Priorities.

4.     Open Boot Option #1 and select SUSE Linux Enterprise from the drop-down menu.

5.     If the Boot Option UEFI Hard Disk: SUSE Linux Enterprise is not listed at the first position of the boot order, move it to the first position.

6.     Change to the Save & Exit top menu and quit the setup utility with Save Changes and Reset.

Persistent Memory configuration

Configure and manage Intel Optane PMem modules from the Linux command line. First install the management utilities ipmctl and ndctl required to manage the libnvdimm (non-volatile memory device) sub-system in the Linux kernel. For additional information, refer to the IPMCTL User Guide and the NDCTL User Guide.

Open a Secure Shell (SSH) Protocol prompt as root to install the host tools and follow these steps:

1.     Install the ipmctl host utility.

# zypper in ipmctl

2.     Install the ndctl utility:

# zypper in ndctl

3.     Verify that the PMem modules have been discovered and the software can communicate with them.

# ipmctl show -dimm

DimmID  | Capacity    | LockState | HealthState | FWVersion

===============================================================

 0x0001 | 502.599 GiB | Disabled  | Healthy     | 01.02.00.5435

 0x0011 | 502.599 GiB | Disabled  | Healthy     | 01.02.00.5435

 0x0021 | 502.599 GiB | Disabled  | Healthy     | 01.02.00.5435

 .

 .

 .

 0x7121 | 502.599 GiB | Disabled  | Healthy     | 01.02.00.5435

4.     Add a UDEV rule to ensure persistent device naming:

# vi /etc/udev/rules.d/60-persistent-storage.rules

# PMEM devices

KERNEL=="pmem*", ENV{DEVTYPE}=="disk", ATTRS{uuid}=="?*", SYMLINK+="disk/by-id/pmem-$attr{uuid}"

5.     Create the goal:

# ipmctl create -goal

The following configuration will be applied:

 SocketID | DimmID | MemorySize | AppDirect1Size | AppDirect2Size

==================================================================

 0x0000   | 0x0001 | 0.000 GiB  | 502.000GiB    | 0.000 GiB

 0x0000   | 0x0011 | 0.000 GiB  | 502.000GiB    | 0.000 GiB

 0x0000   | 0x0021 | 0.000 GiB  | 502.000GiB    | 0.000 GiB

 .

 .

 .

Do you want to continue? [y/n]

6.     Confirm with Y and reboot the server to apply the new memory allocation goals.

7.     Verify that regions have been created:

# ipmctl show -region

SocketID | ISetID    | Persistent | Capacity     | FreeCapacity | HealthState

         |           | MemoryType |              |              |

===============================================================================

 0x0000   | 0x03927… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0001   | 0x7e2a7… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0002   | 0xf78a7… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0003   | 0xe5607… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0004   | 0xf3967… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0005   | 0xc5f27… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0006   | 0x7e007… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

 0x0007   | 0x558c7… | AppDirect  | 3012.000 GiB | 0.000 GiB    | Healthy

8.     Create a namespace for each of the eight regions.

# ndctl create-namespace --continue

9.     The default syntax creates a namespace with direct access (DAX) support, mode fsdax, required by SAP HANA to use Intel Optane PMem properly. Verify that the namespace has been created successfully:

# ndctl list

[

  {

    "dev":"namespace1.0",

    "mode":"fsdax",

    "map":"dev",

    "size":3183575302144,

    "uuid":"de793a2f-5763-4d82-b2c7-59b73721a3e9",

    "sector_size":512,

    "align":2097152,

    "blockdev":"pmem1"

  },

  .

  .

  .

]

10.  The assigned device name for the namespaces is usually /dev/pmemX, where X denotes the NUMA node mapping of the underlying hardware. Construct an XFS file system on the block devices:

# for i in {0..7}; do mkfs.xfs -f -d su=2m,sw=1 -m reflink=0 /dev/pmem$i; done

11.  Create directories and mount the block devices using the DAX file system option:

# for i in {0..7}; do mkdir -p /hana/pmem/nvmem$i; done

# for i in {0..7}; do mount -t xfs -o dax,lazytime /dev/pmem$i \

  /hana/pmem/nvmem$i; done

12.  Change the permission of the mount points:

# chmod 755 /hana/pmem/nvmem*

13.  Add the mount points to /etc/fstab to persist them:

# vi /etc/fstab

/dev/pmem0 /hana/pmem/nvmem0 xfs dax,lazytime 1 2

/dev/pmem1 /hana/pmem/nvmem1 xfs dax,lazytime 1 2

/dev/pmem2 /hana/pmem/nvmem2 xfs dax,lazytime 1 2

/dev/pmem3 /hana/pmem/nvmem3 xfs dax,lazytime 1 2

/dev/pmem4 /hana/pmem/nvmem4 xfs dax,lazytime 1 2

/dev/pmem5 /hana/pmem/nvmem5 xfs dax,lazytime 1 2

/dev/pmem6 /hana/pmem/nvmem6 xfs dax,lazytime 1 2

/dev/pmem7 /hana/pmem/nvmem7 xfs dax,lazytime 1 2

 

The device names chosen by the kernel are subject to creation order and discovery. For static configurations they usually don’t change. To ensure persistent naming of the PMem namespaces, exchange the device name in /etc/fstab as follows:

# ll /dev/disk/by-id/pmem*

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-1466813e-b789-452d-bb27-464795f9a32f -> ../../pmem0

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-24057307-8d32-4aa5-b1dc-858006ed4c83 -> ../../pmem7

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-2a2b109c-9b3d-463a-8d12-6a51237b4251 -> ../../pmem3

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-2b0ade1b-8b64-4a55-90d6-1f69ee0695c0 -> ../../pmem6

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-98d295c1-8bf9-4b75-a62c-1e42f337f9f0 -> ../../pmem2

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-b0b8f716-e23a-4e02-8d11-b471b348d373 -> ../../pmem4

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-d7276772-af1d-496e-a5b2-3f9d2a19f6c4 -> ../../pmem5

lrwxrwxrwx 1 root root 11 Jul 15 13:59 /dev/disk/by-id/pmem-de793a2f-5763-4d82-b2c7-59b73721a3e9 -> ../../pmem1

 

The persistent name for PMem namespace 0 in /etc/fstab will then look like this:

# vi /etc/fstab

/dev/disk/by-id/pmem-1466813e-b789-452d-bb27-464795f9a32f /hana/pmem/nvmem0 xfs dax,lazytime 1 2

.

.

.

The Linux-specific setup of Intel Optane PMem is now completed. After you finish the SAP HANA installation, you must configure SAP HANA in order to use Intel Optane PMem.

 

SAP HANA installation

All version-specific SAP installation and administration documentation is available from the SAP HANA Help portal: https://help.sap.com/hana. Please refer to the official SAP documentation that describes the different SAP HANA installation options.

Note:      Review all relevant SAP notes related to the SAP HANA installation on recent changes.

SAP HANA 2.0 platform installation

The official SAP documentation describes in detail how to install the HANA software and its required components. All required file systems are mounted before the installation.

SAP HANA scale-up installation

Download and extract the SAP HANA Platform 2.0 software to an installation sub-folder of your choice.

SAP HANA installation

Follow the installation workflow of the SAP HANA Database Lifecycle Manager (hdblcm) and provide the user passwords when asked for:

1.     Change to the folder <path>/DATA_UNITS/HDB_LCM_Linux_X86_64.

2.     Adapt the following command for the SAP SID, SAP system ID number, hostname, and required components:

# ./hdblcm --action install --components=server,client --install_hostagent \

 --number <SAP System ID> --sapmnt=/hana/shared --sid=<SID> --hostname=<hostname>

 --certificates_hostmap=<hostname>=<map name>

3.     Switch user to <sid>adm.

4.     Confirm that SAP HANA is started:

# sapcontrol -nr <SAP System ID> -function GetProcessList

Configure SAP HANA to use Intel Optane PMem

In a multi-tenant database architecture, create sub-directories in the mounted /hana/pmem/nvmem* directories. For example:

# mkdir /hana/pmem/nvmem*/DB1

# mkdir /hana/pmem/nvmem*/DB2

 

Adapt the directory permissions and assign the database user and group. For example:

# chmod 755 /hana/pmem/nvmem*/DB*

# chown db1adm:sapsys /hana/pmem/nvmem*/DB1

# chown db2adm:sapsys /hana/pmem/nvmem’/DB2

For a single instance, assign the database user and group to the /hana/pmem/nvmem* mount points.

Specify the HANA PMem base path

Enable SAP HANA to use the nonvolatile memory and add the DAX volumes to the SAP HANA configuration file global.ini. Configure the required SAP HANA parameter basepath_persistent_memory_volumes in the persistence section. Separate multiple locations by a semicolon.

(<sid>adm) # cdglo

 

(<sid>adm) # vi hdb/custom/config/global.ini

[persistence]

basepath_datavolumes = /hana/data/<SID>

basepath_logvolumes = /hana/log/<SID>

basepath_persistent_memory_volumes = /hana/pmem/nvmem0;/hana/pmem/nvmem1;/hana/pmem/nvmem2;/hana/pmem/nvmem3; hana/pmem/nvmem4;/hana/pmem/nvmem5;/hana/pmem/nvmem6;/hana/pmem/nvmem7

 

 

For a multi-tenant database server, you must configure the parameter separately for both example instances DB1 and DB2:

(db1adm) # cdglo

 

(db1adm) # vi hdb/custom/config/global.ini

[persistence]

basepath_datavolumes = /hana/data/DB1

basepath_logvolumes = /hana/log/DB1

basepath_persistent_memory_volumes = /hana/pmem/nvmem0/DB1;/hana/pmem/nvmem1/DB1;/hana/pmem/nvmem2/DB1;/hana/pmem/nvmem3/DB1;/hana/pmem/nvmem4/DB1;/hana/pmem/nvmem5/DB1;/hana/pmem/nvmem6/DB1;/hana/pmem/nvmem7/DB1

 

(db2adm) # cdglo

 

(db2adm) # vi hdb/custom/config/global.ini

[persistence]

basepath_datavolumes = /hana/data/DB2

basepath_logvolumes = /hana/log/DB2

basepath_persistent_memory_volumes = /hana/pmem/nvmem0/DB2;/hana/pmem/nvmem1/DB2;/hana/pmem/nvmem2/DB2;/hana/pmem/nvmem3/DB2;/hana/pmem/nvmem4/DB2;/hana/pmem/nvmem5/DB2;/hana/pmem/nvmem6/DB2;/hana/pmem/nvmem7/DB2

 

Restart the SAP HANA database instance to activate the parameter change. SAP HANA automatically verifies the DAX-enabled file system during startup; all tables will use persistent memory by default afterward.

SAP HANA system replication

The SAP HANA system replication (HSR) mechanism helps ensure high availability of the SAP HANA database; you can combine it with a disaster recovery setup. In system replication mode, the content of the SAP HANA database is transferred synchronously to a secondary location in the same or a different data center.

The SAP HANA System Replication - SAP Help Portal provides detailed information about the functions and configuration of SAP HANA system replication. SUSE Linux Enterprise Server for SAP Applications is optimized in various ways for SAP applications. Based on the high-availability solution scenario “SAP HANA Scale Up System Replication Performance Optimized”, SUSE automates the installation procedure and enables the management of SAP HANA by SUSE cluster resources.

One critical element of the SUSE cluster implementation is the configuration of a node fencing mechanism that avoids a cluster split-brain situation when the two cluster nodes are not able to communicate with each other any longer. For two cluster nodes with dedicated storage like the SAP HANA appliance configuration, the SUSE default shared STONITH block device (SBD) is not available as a node fencing mechanism. Instead configure an IPMI interface as the STONITH resource.

IPMI STONITH resource

First, you need to create an IPMI user in the management web interface (IPMI Console). Connect to the IPMI address of the Cisco UCS C890 M5 server with your preferred browser. Use the ADMIN user to log on to IPMI.

1.     From the top menu, select Configuration and Users from the drop-down menu.

2.     Select an empty slot and press the Add User button. Add a new user and provide that user with the privilege level operator. This level allows all baseboard-management-control (BMC) commands except commands that can modify out-of-band interface settings.

Configure the IPMI interface as a STONITH resource and use the IPMI user credentials to authenticate.

For example:

# crm configure edit

...

primitive STONITH-<hostname> stonith:external/ipmi \

        params hostname=<hostname> ipaddr=<IPMI IP address> userid=cisco \

  passwd=<IPMI cisco password> interface=lanplus \

...

IPMI watchdog driver

SUSE Linux Enterprise High Availability Extension ships with several modules that provide hardware-specific watchdog drivers and uses the SBD daemon as the software component that “feeds” the watchdog. Configure the correct hardware watchdog if you plan to use SBD instead of the IPMI STONITH resource. The default Intel driver module iTCO_wdt doesn’t interact with the baseboard management controller (BMC) and needs to be replaced.

The BMC runs independent of the server processors, so if a processor, memory, or any other hardware failure occurs, BMC still provides its services unless there is a full power outage.

Loading the correct kernel module

List the watchdog modules that are currently loaded in the kernel:

# lsmod | egrep "(wd|dog)"

  iTCO_wdt               16384  0

  iTCO_vendor_support    16384  1 iTCO_wdt

Unload the kernel module from the system:

 # rmmod iTCO_wdt iTCO_vendor_support

Blacklist the iTCO watchdog driver to fully disable it (end the stream with <CTRL-D>).

# cat >> /etc/modprobe.d/50-blacklist.conf

 

# unload the iTCO watchdog kernel module

blacklist iTCO_wdt

blacklist iTCO_vendor_support

Next install the OpenIPMI software package and configure the IPMI watchdog timer.

# zypper install OpenIPMI ipmitool

 

# cp -p /etc/sysconfig/ipmi /etc/sysconfig/ipmi.orig

 

# vi /etc/sysconfig/ipmi

IPMI_SI=yes

DEV_IPMI=yes

IPMI_WATCHDOG=yes

IPMI_WATCHDOG_OPTIONS="timeout=20 action=reset nowayout=0 panic_wdt_timeout=15"

IPMI_POWEROFF=no

IPMI_POWERCYCLE=no

IPMI_IMB=no

Note:      If a kernel core dump capture is active, increase the panic_wdt_timeout value to allow the core dump to be written to disk.

Configure the systemd watchdog timer and uncomment the following lines in the file system.conf:

# vi /etc/systemd/system.conf

…

RuntimeWatchdogSec=10

ShutdownWatchdogSec=60

Enable the change and restart the systemd daemon:

# systemctl daemon-reexec

Start the IPMI driver and enable startup during boot time:

# systemctl start ipmi

# systemctl enable ipmi

 

Verify that the IPMI kernel modules are loaded:

# lsmod |grep ipmi

ipmi_watchdog          32768  1

ipmi_ssif              40960  0

ipmi_si                69632  1

ipmi_devintf           20480  0

ipmi_msghandler       114688  4 ipmi_devintf,ipmi_si,ipmi_watchdog,ipmi_ssif

Verify that the watchdog timer is running and active:

# ipmitool mc watchdog get

Watchdog Timer Use:     SMS/OS (0x44)

Watchdog Timer Is:      Started/Running

Watchdog Timer Logging: On

Watchdog Timer Action:  Hard Reset (0x01)

Pre-timeout interrupt:  None

Pre-timeout interval:   0 seconds

Timer Expiration Flags: (0x10)

                        * SMS/OS

Initial Countdown:      10.0 sec

Present Countdown:      8.4 sec

Now a watchdog device is present on the system as well:

# ls -l /dev/watchdog

crw------- 1 root root 10, 130 Apr 28 18:09 /dev/watchdog

Finally, test the watchdog configuration and trigger a kernel crash. The system should reboot either after the time set by the parameter panic_wdt_timeout runs out or after 5 minutes (BMC timeout).

# echo c > /proc/sysrq-trigger

Conclusion

The Cisco solution for SAP HANA using the high-performance Cisco UCS C890 M5 Server is well-suited for mission-critical business scenarios such as for medium- to large-size SAP HANA database management systems including high availability and secure multitenancy. The integrated infrastructure solution provides a robust platform for SAP HANA workloads as a single-node (scale-up) SAP HANA appliance.

The 8-socket server delivers exceptional performance and is sized, configured, and deployed to match the SAP HANA appliance key performance metrics that SAP SE demands. With standard configuration options including 16 hot-swappable 2.5-inch SAS-3 drive bays (with RAID cards), eight 25-Gbps SFP Ethernet ports, and four 32-Gbps Fibre Channel ports, this server is well-suited for mission-critical SAP HANA workloads where performance and memory size are key attributes.

SUSE Linux Enterprise Server (SLES) for SAP Applications is the leading Linux platform for SAP HANA, SAP NetWeaver, and SAP S/4H solutions. It provides optimized performance, can reduce downtimes, and provides faster SAP landscape deployments. Features include secure SAP HANA systems, and they ease the transition to SAP S/4HANA for systems administrators.

The SAP HANA platform provides a scalable database with advanced analytical capabilities and intuitive application-development tools in an in-memory data platform. SAP HANA Tailored Data Center Integration (TDI) supports nonvolatile Intel Optane Persistent Memory that brings together the low latency of memory and the persistence of storage. Persistent memory modules provide faster access to data and retain data across power cycles.

Software- and hardware-based design options to meet your high-availability and disaster recovery requirements are available, and you can combine them with SAP application monitoring using Cisco AppDynamics® technology to detect problems in real time before they affect customers.

For more information

For additional information, refer to the resources listed in this section.

Cisco Unified Computing System

●      Cisco UCS C890 M5 Rack Server for SAP HANA

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/ucs-c890m5-sap-hana-wp.html

●      Cisco UCS C890 M5 Rack Server data sheet

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/ucs-c-series-rack-servers/datasheet-c78-744647.html

●      Cisco UCS for SAP HANA with Intel Optane DC Persistent Memory Module

https://www.cisco.com/c/dam/en/us/products/servers-unified-computing/ucs-b-series-blade-servers/whitepaper-c11-742627.pdf

●      Cisco UCS M5 Memory Technical Overview – Memory RAS Features

https://www.cisco.com/c/en/us/products/collateral/servers-unified-computing/technical-overview-c17-743902.html

●      Configure Cisco AppDynamics Monitoring Solution for SAP Applications:

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/unified-computing/appd-sap-monitoring-wp.html

SAP HANA platform

●      SAP HANA platform on SAP Help portal:

https://help.sap.com/viewer/p/SAP_HANA_PLATFORM

●      SAP HANA TDI: Overview

https://www.sap.com/documents/2017/09/e6519450-d47c-0010-82c7-eda71af511fa.html

●      SAP HANA TDI storage requirements:

https://www.sap.com/documents/2015/03/74cdb554-5a7c-0010-82c7-eda71af511fa.html

●      SAP HANA network requirements:
https://www.sap.com/documents/2016/08/1cd2c2fb-807c-0010-82c7-eda71af511fa.html

●      SAP HANA System Replication – SAP Help Portal

https://help.sap.com/viewer/4e9b18c116aa42fc84c7dbfd02111aba/2.0.04/en-US/afac7100bc6d47729ae8eae32da5fdec.html

●      SAP HANA Business Continuity with SAP HANA System Replication and SUSE Cluster on Cisco Unified Computing System

https://www.cisco.com/c/en/us/solutions/collateral/data-center-virtualization/sap-applications-on-cisco-ucs/whitepaper-C11-734028.html#_Toc525044037

Interoperability matrixes

●      Cisco UCS hardware compatibility matrix:
https://ucshcltool.cloudapps.cisco.com/public/

●      SAP Note 2235581: SAP HANA: Supported Operating Systems:
https://launchpad.support.sap.com/#/notes/2235581

●      SAP certified and supported SAP HANA hardware:
https://www.sap.com/dmc/exp/2014-09-02-hana-hardware/enEN/index.html

Feedback

For comments and suggestions about this guide and related guides, join the discussion on Cisco Community at https://cs.co/en-cvds.