FlexPod Datacenter for SAP Solution with Cisco UCS 3rd Generation Fabric and NetApp AFF A–Series

FlexPod Datacenter for SAP Solution with Cisco UCS 3^rd Generation Fabric and NetApp AFF A–Series

Deployment Guide for FlexPod Datacenter for SAP Solution with NetApp ONTAP 9.6 on NetApp AFF A-Series and Cisco UCS M5 Servers with 2^nd Generation Intel Xeon Scalable Processors

NOTE: Works with document’s Advanced Properties “First Published” property. Click File | Properties | Advanced Properties | Custom.

Published: November 2020 NOTE: Works with document’s Advanced Properties “Last Updated” property. Click File | Properties | Advanced Properties | Custom.

In partnership with:

About the Cisco Validated Design Program

The Cisco Validated Design (CVD) program consists of systems and solutions designed, tested, and documented to facilitate faster, more reliable, and more predictable customer deployments. For more information, go to:

http://www.cisco.com/go/designzone.

ALL DESIGNS, SPECIFICATIONS, STATEMENTS, INFORMATION, AND RECOMMENDATIONS (COLLECTIVELY, "DESIGNS") IN THIS MANUAL ARE PRESENTED "AS IS," WITH ALL FAULTS.  CISCO AND ITS SUPPLIERS DISCLAIM ALL WARRANTIES, INCLUDING, WITHOUT LIMITATION, THE WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT OR ARISING FROM A COURSE OF DEALING, USAGE, OR TRADE PRACTICE.  IN NO EVENT SHALL CISCO OR ITS SUPPLIERS BE LIABLE FOR ANY INDIRECT, SPECIAL, CONSEQUENTIAL, OR INCIDENTAL DAMAGES, INCLUDING, WITHOUT LIMITATION, LOST PROFITS OR LOSS OR DAMAGE TO DATA ARISING OUT OF THE USE OR INABILITY TO USE THE DESIGNS, EVEN IF CISCO OR ITS SUPPLIERS HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES.

THE DESIGNS ARE SUBJECT TO CHANGE WITHOUT NOTICE.  USERS ARE SOLELY RESPONSIBLE FOR THEIR APPLICATION OF THE DESIGNS.  THE DESIGNS DO NOT CONSTITUTE THE TECHNICAL OR OTHER PROFESSIONAL ADVICE OF CISCO, ITS SUPPLIERS OR PARTNERS.  USERS SHOULD CONSULT THEIR OWN TECHNICAL ADVISORS BEFORE IMPLEMENTING THE DESIGNS.  RESULTS MAY VARY DEPENDING ON FACTORS NOT TESTED BY CISCO.

CCDE, CCENT, Cisco Eos, Cisco Lumin, Cisco Nexus, Cisco StadiumVision, Cisco TelePresence, Cisco WebEx, the Cisco logo, DCE, and Welcome to the Human Network are trademarks; Changing the Way We Work, Live, Play, and Learn and Cisco Store are service marks; and Access Registrar, Aironet, AsyncOS, Bringing the Meeting To You, Catalyst, CCDA, CCDP, CCIE, CCIP, CCNA, CCNP, CCSP, CCVP, Cisco, the Cisco Certified Internetwork Expert logo, Cisco IOS, Cisco Press, Cisco Systems, Cisco Systems Capital, the Cisco Systems logo, Cisco Unified Computing System (Cisco UCS), Cisco UCS B-Series Blade Servers, Cisco UCS C-Series Rack Servers, Cisco UCS S-Series Storage Servers, Cisco UCS Manager, Cisco UCS Management Software, Cisco Unified Fabric, Cisco Application Centric Infrastructure, Cisco Nexus 9000 Series, Cisco Nexus 7000 Series. Cisco Prime Data Center Network Manager, Cisco NX-OS Software, Cisco MDS Series, Cisco Unity, Collaboration Without Limitation, EtherFast, EtherSwitch, Event Center, Fast Step, Follow Me Browsing, FormShare, GigaDrive, HomeLink, Internet Quotient, IOS, iPhone, iQuick Study,  LightStream, Linksys, MediaTone, MeetingPlace, MeetingPlace Chime Sound, MGX, Networkers, Networking Academy, Network Registrar, PCNow, PIX, PowerPanels, ProConnect, ScriptShare, SenderBase, SMARTnet, Spectrum Expert, StackWise, The Fastest Way to Increase Your Internet Quotient, TransPath, WebEx, and the WebEx logo are registered trademarks of Cisco Systems, Inc. and/or its affiliates in the United States and certain other countries.

All other trademarks mentioned in this document or website are the property of their respective owners. The use of the word partner does not imply a partnership relationship between Cisco and any other company. (0809R)

© 2019 Cisco Systems, Inc. All rights reserved.

 

Table of Contents

Executive Summary. 7

Solution Overview.. 8

Introduction. 8

Audience. 8

Purpose of this Document 8

What’s New in this Release?. 8

Reference. 8

Deployment Hardware and Software. 9

Architecture. 9

Topology. 10

Software Revisions. 10

Configuration Guidelines. 11

Physical Infrastructure. 11

Network Switch Configuration. 13

Physical Connectivity. 13

FlexPod Cisco Nexus Base. 13

Set Up Initial Configuration. 14

FlexPod Cisco Nexus Switch Configuration. 15

Enable Licenses. 15

Create VLANs. 16

Create Port Channels and assign interfaces. 16

Configure Port Channel Parameters. 18

Configure Virtual Port Channels. 19

Uplink into Existing Network Infrastructure. 20

Cisco UCS Solution for SAP HANA TDI 21

Cisco UCS Server Configuration. 21

Initial Setup of Cisco UCS 6332 Fabric Interconnect 21

Cisco UCS 6332 Fabric Interconnect A. 21

Cisco UCS 6332 Fabric Interconnect B. 22

Log into Cisco UCS Manager 22

Upgrade Cisco UCS Manager Software to Version 4.0(4d) 22

Add Block of IP Addresses for KVM Access. 22

Synchronize Cisco UCS to NTP. 23

Cisco UCS Blade Chassis Connection Options. 23

Edit Chassis Discovery Policy. 23

Enable Server and Uplink Ports. 24

Acknowledge Cisco UCS Chassis and Rack-Mount Servers. 25

Create Uplink Port Channels to Cisco Nexus Access layer Switches. 25

Create New Organization. 29

Create MAC Address Pools. 29

Create UUID Suffix Pool 31

Create IQN Pools for iSCSI Boot 32

Create IP Pools for iSCSI Boot 33

Power Policy. 35

Power Control Policy. 35

Create Network Control Policy. 36

Create Host Firmware Package. 37

Create Local Disk Configuration Policy (Optional) 38

Create Server BIOS Policy. 39

Create Serial Over LAN Policy. 43

Update Default Maintenance Policy. 44

Adapter Policy Configuration – HANA. 44

Network Configuration. 46

LAN Tab Configurations. 46

Create vNIC Templates. 52

Create vNIC/vHBA Placement Policy. 62

Create LAN Connectivity Policy. 64

Create iSCSI Boot Policy. 70

Create Service Profile Template for SAP HANA Node. 72

Configure Storage Provisioning. 73

Configure Networking Options. 73

Configure Storage Options. 75

Configure Zoning Options. 75

Configure vNIC/HBA Placement 75

Configure vMedia Policy. 77

Configure Server Boot Order 78

Configure Maintenance Policy. 83

Configure Server Assignment 83

Configure Operational Policies. 84

Create Service Profiles for Single-Host System.. 84

Create Service Profiles for Multi-Host System.. 85

Storage Configuration. 87

Complete Configuration Worksheet 87

Configure ONTAP Nodes. 87

Configure Node 01. 87

Configure Node 02. 88

Set Up ONTAP Cluster 90

Set Auto-Revert on Cluster Management 95

Set Up Management Broadcast Domain. 95

Create Aggregates. 96

Verify Storage Failover 97

Disable Flow Control on 100GbE Ports. 97

Configure AutoSupport 98

Enable Cisco Discovery Protocol 98

Create Broadcast Domains. 98

Create Interface Groups. 99

Create VLANs. 100

Configure HTTPS Access. 100

Configure SVM for the Infrastructure. 102

Create SVM for the Infrastructure. 104

Create Load-Sharing Mirrors. 104

Create Export Policies for the Root Volumes. 104

Add Infrastructure SVM  Management LIF. 105

Create iSCSI LIFs. 105

Create Block Protocol (iSCSI) Service. 105

Create FlexVol Volumes. 105

Configure LUNs for iSCSI Boot 106

Configure SVM for HANA. 106

Create SVM for SAP HANA. 107

Create Load-Sharing Mirrors. 108

Create Export Policies for the Root Volumes. 108

Add HANA SVM Management Interface and Administrator 109

Create Export Policies for the HANA SVM.. 109

Create NFS LIF for SAP HANA Data. 109

Create NFS LIF for SAP HANA Log. 110

HANA Node Preparation. 111

SAP HANA Node OS Preparation – RHEL for SAP HANA 7.6. 111

OS Installation. 111

Post Installation Tasks. 119

SAP HANA Node OS Preparation – SLES for SAP 15. 127

OS Installation. 127

Post Installation Steps. 157

System Provisioning for SAP HANA. 174

Configuring SAP HANA Single-Host Systems. 175

Configuration Example for a SAP HANA Single-Host System.. 177

Persistent Memory Configuration. 180

Configure SAP HANA Multiple-Host Systems. 184

Configuration Example for a SAP HANA Multiple-Host Systems. 186

SAP HANA Installation. 194

Important SAP Notes. 194

SAP HANA IMDB Related Notes. 194

Linux Related Notes. 194

SAP Application Related Notes. 195

Third Party Software. 195

NetApp Technical reports. 195

High-Availability Configuration for Multiple-host 196

SAP HANA Installation Preparations for NFSv4. 196

SAP HANA Hosts. 196

Storage Controllers. 196

SAP HANA Data Volume Size. 196

Monitor SAP HANA with AppDynamics. 198

Introduction. 198

SAP Landscape Monitoring. 198

Trial Registration. 199

Agent Installation. 200

Prerequisites. 200

Java Agent Installation. 200

ABAP Agent Installation. 201

Server Visibility Agent Installation. 203

About the Authors. 204

Acknowledgements. 204

 

 

This document describes the deployment methodology of Cisco and NetApp® FlexPod Datacenter for SAP HANA based on 2nd Generation Intel Xeon Scalable Processors supported Cisco UCS Computing System (Cisco UCS).

Cisco UCS Manager (UCSM) 4.0(4) provides consolidated support of all current Cisco UCS Fabric Interconnect models (6200, 6300, 6324 and 6454), 2200/2300 series IOM, Cisco UCS B-Series Blade and Cisco UCS C-Series Rack Formfactor servers. FlexPod Datacenter with Cisco UCS unified software release 4.0(4d) and NetApp ONTAP 9.6, is a predesigned, best-practice data center architecture built on the Cisco UCS, the Cisco Nexus® 9000 family of switches, and NetApp AFF A-Series storage arrays.

Introduction

Industry trends indicate a vast data center transformation toward shared infrastructure and cloud computing. Business agility requires application agility, so IT teams must provision applications quickly and resources must scale up (and out) as needed.

FlexPod Datacenter is a best practice data center architecture that was designed and validated by Cisco and NetApp to meet the needs of enterprise customers and service providers. FlexPod Datacenter is built on NetApp AFF enterprise storage, the Cisco UCS, and the Cisco Nexus family of switches. These components combine to create management synergy across a business’s IT infrastructure. FlexPod Datacenter has been proven to be the optimal platform for a wide variety of workloads, including bare metal and virtualized systems, which enables enterprises to standardize their IT infrastructure.

Audience

The audience for this document includes, but is not limited to: field consultants, professional services, IT managers, partner engineers, and customers who want to take advantage of an infrastructure built to deliver IT efficiency and enable IT innovation.

Purpose of this Document

This document provides a step by step configuration and implementation guide for the FlexPod Datacenter with Cisco UCS Fabric Interconnects, NetApp AFF storage and Cisco Nexus 9000 solution.

What’s New in this Release?

The following design elements distinguish this version of FlexPod from previous FlexPod models:

·      Support for the Cisco UCS 4.0(4) unified software release, Cisco UCS B200-M5 servers, Cisco UCS B480-M5 servers with Cascade Lake CPUs, and Cisco 1400 Series Virtual Interface Cards (VICs)

·      Validation with cloud-scale FX Series Nexus switches

·      Support for the NetApp AFF A320 storage controller

·      Support for the latest release of NetApp ONTAP® 9.6 storage software

·      Support for NFS v4.1

·      Support for NetApp SnapCenter® 4.2

·      NFS and iSCSI storage design

Reference

The design guide counterpart for this deployment guide can be accessed here: https://www.cisco.com/c/en/us/td/docs/unified_computing/ucs/UCS_CVDs/flexpod_datacenter_sap_netappaffa_design.html

Architecture

FlexPod is a defined set of hardware and software that serves as an integrated foundation for both virtualized and non-virtualized workloads. The design is flexible enough that the networking, computing, and storage can fit in one data center rack or be deployed according to a customer's data center design. Port density enables the networking components to accommodate multiple configurations of this kind.

One benefit of the FlexPod architecture is the ability to customize or "flex" the environment to suit a customer's requirements. A FlexPod can easily be scaled as requirements and demand change. The unit can be scaled both up (adding resources to a FlexPod unit) and out (adding more FlexPod units). The reference architecture detailed in this document highlights the resiliency, cost benefit, and ease of deployment of an IP-based storage solution.

Figure 1 shows the solution built on FlexPod components and the network connections for a configuration with the Cisco UCS 6300 series Fabric Interconnects. This design has port-channeled 40 Gb Ethernet connections between the Cisco UCS 5108 Blade Chassis and the Cisco UCS Fabric Interconnects, port-channeled 40 Gb Ethernet connections between the C-Series rackmounts and the Cisco UCS Fabric Interconnects, and 40 Gb Ethernet connections between the Cisco UCS Fabric Interconnect and Cisco Nexus 9000, and 40/100 Gb Ethernet connection between Cisco Nexus 9000 and NetApp AFF A320 storage array. The reference architecture reinforces the "wire-once" strategy, because as additional storage is added to the architecture, no re-cabling is required from the hosts to the Cisco UCS fabric interconnect.

Topology

The reference end-to-end 40Gb based hardware configuration includes:

·       Two Cisco Nexus 9336C-FX2 switches

·       Two Cisco UCS 6300 series fabric interconnects managing Cisco UCS blade/rack formfactor servers.

·       One NetApp AFF A320 (HA pair) running ONTAP 9.6 with external disk shelf NS224 using NVMe SSD disks

Software Revisions

Table 1  lists the software revisions for this solution.

Table 1    Software Revisions

Configuration Guidelines

This document explains how to configure a fully redundant, highly available configuration for a FlexPod unit with ONTAP storage. Therefore, reference is made to which component is being configured with each step, either 01 or 02 or A and B. For example, node01 and node02 are used to identify the two NetApp storage controllers that are provisioned with this document, and Cisco Nexus A or Cisco Nexus B identifies the pair of Cisco Nexus switches that are configured. The Cisco UCS fabric interconnects are similarly configured. Additionally, this document details the steps for provisioning multiple Cisco UCS hosts, and these examples are identified as: VM-Host-Infra-01, VM-Host-Infra-02 to represent infrastructure hosts deployed to each of the fabric interconnects in this document. Finally, to indicate that you should include information pertinent to your environment in a given step, <text> appears as part of the command structure. See the following example for the network port vlan create command:

Usage:

network port vlan create ?

 [-node] <nodename>                  Node

 { [-vlan-name] {<netport>|<ifgrp>}  VLAN Name

 |  -port {<netport>|<ifgrp>}        Associated Network Port

 [-vlan-id] <integer> }              Network Switch VLAN Identifier

Example:

network port vlan create -node <node01> -vlan-name a0a-<vlan id>

This document is intended to enable you to fully configure the customer environment. In this process, various steps require you to insert customer-specific naming conventions, IP addresses, and VLAN schemes, as well as to record appropriate MAC addresses. Table 2  describes the VLANs necessary for deployment as outlined in this guide.

Table 2    Necessary VLANs

Physical Infrastructure

FlexPod Cabling

The information in this section is provided as a reference for cabling the physical equipment in a FlexPod environment. To simplify cabling requirements, a cabling diagram was used.

The cabling diagram in this section contains details for the prescribed and supported configuration of the NetApp AFF A320 running NetApp ONTAP® 9.6.

This document assumes that out-of-band management ports are plugged into an existing management infrastructure at the deployment site. These interfaces will be used in various configuration steps.

The NetApp storage controller and disk shelves should be connected according to best practices for the specific storage controller and disk shelves. For disk shelf cabling, refer to NetApp Support.

Figure 22 illustrates the cable connections used in the validation lab for the FlexPod topology based on the Cisco UCS 6300 Series Fabric Interconnects. Also, 40Gb links connect the Cisco UCS Fabric Interconnects to the Cisco Nexus Switches and 40/100Gb links connect the NetApp AFF controllers to the Cisco Nexus Switches. Additional 1Gb management connections will be needed for an out-of-band network switch that sits apart from the FlexPod infrastructure.  Each Cisco UCS fabric interconnect and Cisco Nexus switch is connected to the out-of-band network switch, and each AFF controller has two connections to the out-of-band network switch. Layer 3 network connectivity is required between the Out-of-Band (OOB) and In-Band (IB) Management Subnets.

This section provides a detailed procedure for configuring the Cisco Nexus 9000s for use in a FlexPod environment. Follow these steps precisely because failure to do so could result in an improper configuration.

Physical Connectivity

Follow the physical connectivity guidelines for FlexPod as explained in the section FlexPod Cabling.

FlexPod Cisco Nexus Base

The following procedures describe how to configure the Cisco Nexus switches for use in a base FlexPod environment.  This procedure assumes the use of Cisco Nexus 9000 7.0(3)I7(6), the Cisco suggested Nexus switch release at the time of this validation.

See Table 2  for the VLANs necessary for deployment as outlined in this guide.

Table 3    Configuration Variables

Set Up Initial Configuration

Cisco Nexus A

To set up the initial configuration for the Cisco Nexus A switch on <nexus-A-hostname>, follow these steps:

1.     Configure the switch.

2.     Review the configuration summary before enabling the configuration.

Use this configuration and save it? (yes/no) [y]: Enter

Cisco Nexus B

To set up the initial configuration for the Cisco Nexus B switch on <nexus-B-hostname>, follow these steps:

1.     Configure the switch.

2.     Review the configuration summary before enabling the configuration:

Use this configuration and save it? (yes/no) [y]: Enter

FlexPod Cisco Nexus Switch Configuration

Enable Licenses

Cisco Nexus A and Cisco Nexus B

To license the Cisco Nexus switches, follow these steps:

1.     Log in as admin.

2.     Run the following commands to enable the required features:

Create VLANs

Cisco Nexus A and Cisco Nexus B

To create the necessary virtual local area networks (VLANs), follow this step on both switches:

1.     From the global configuration mode, run the following commands:

Create Port Channels and assign interfaces

Cisco Nexus A

To create the necessary port channels between devices, follow these steps:

1.     From the global configuration mode, run the following commands:

Cisco Nexus B

To create the necessary port channels between devices, follow these steps:

1.     From the global configuration mode, run the following commands:

Configure Port Channel Parameters

Cisco Nexus A and Cisco Nexus B

To configure port channel parameters, follow this step on both switches:

1.     From the global configuration mode, run the following commands:

Configure Virtual Port Channels

Cisco Nexus A

To configure virtual port channels (vPCs) for switch A, follow this step:

1.     From the global configuration mode, run the following commands:

 

Cisco Nexus B

To configure vPCs for switch B, follow this step:

1.     From the global configuration mode, run the following commands:

 

Uplink into Existing Network Infrastructure

Depending on the available network infrastructure, several methods and features can be used to uplink the FlexPod environment. If an existing Cisco Nexus environment is present, we recommend using vPCs to uplink the Cisco Nexus switches included in the FlexPod environment into the infrastructure. The previously described procedures can be used to create an uplink vPC to the existing environment. Make sure to run copy run start to save the configuration on each switch after the configuration is completed.

The SAP HANA TDI option enables multiple SAP HANA production systems to run on the same infrastructure. In this configuration, the existing blade servers used by different SAP HANA systems share the same network infrastructure and storage systems. In addition, the SAP application server can share the same infrastructure as the SAP HANA database.

Cisco UCS Server Configuration

This section describes the specific configurations on Cisco UCS servers to address SAP HANA requirements.

See Table 2  for the VLANs necessary for deployment as outlined in this guide.

Table 4    Configuration Variables

Initial Setup of Cisco UCS 6332 Fabric Interconnect

This section provides the detailed procedures to configure the Cisco Unified Computing System (Cisco UCS) for use in FlexPod Datacenter Solution for SAP HANA environment. These steps are necessary to provision the Cisco UCS C-Series and B-Series servers to meet SAP HANA requirements.

Cisco UCS 6332 Fabric Interconnect A

To configure the Cisco UCS Fabric Interconnect A, follow these steps:

1.     Connect to the console port on the first Cisco UCS 6300 series Fabric Interconnect.

2.     Review the settings printed to the console. If they are correct, answer yes to apply and save the configuration.

3.     Wait for the login prompt to make sure that the configuration has been saved.

Cisco UCS 6332 Fabric Interconnect B

To configure the Cisco UCS Fabric Interconnect B, follow these steps:

1.     Connect to the console port on the second Cisco UCS 6248 fabric interconnect.

2.     Wait for the login prompt to make sure that the configuration has been saved.

Log into Cisco UCS Manager

To log into the Cisco Unified Computing System (UCS) environment, follow these steps:

1.     Open a web browser and navigate to the Cisco UCS 6332 Fabric Interconnect cluster address.

2.     Click the Launch UCS Manager link to download the Cisco UCS Manager software.

3.     If prompted to accept security certificates, accept as necessary.

4.     When prompted, enter admin as the user name and enter the administrative password.

5.     Click Login to log into Cisco UCS Manager.

Upgrade Cisco UCS Manager Software to Version 4.0(4d)

This document assumes the use of Cisco UCS Manager Software version 4.0(4d). To upgrade the Cisco UCS Manager software and the Cisco UCS 6332 Fabric Interconnect software to version 4.0(4d), refer to Cisco UCS Manager Install and Upgrade Guides.

Add Block of IP Addresses for KVM Access

To create a block of IP addresses for server Keyboard, Video, Mouse (KVM) access in the Cisco UCS environment, follow these steps:

1.     This block of IP addresses should be in the same subnet as the management IP addresses for the Cisco UCS Manager.

2.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

3.     Select Pools > root > IP Pools > IP Pool ext-mgmt.

4.     In the Actions pane, select Create Block of IP Addresses.

5.     Enter the starting IP address of the block and the number of IP addresses required, and the subnet and gateway information.

6.     Click OK to create the IP block.

7.     Click OK in the confirmation message.

Synchronize Cisco UCS to NTP

To synchronize the Cisco UCS environment to the NTP server, follow these steps:

1.     In Cisco UCS Manager, click the Admin tab in the navigation pane.

2.     Select All > Timezone Management.

3.     In the Properties pane, select the appropriate time zone in the Timezone menu.

4.     Click Save Changes and then click OK.

5.     Click Add NTP Server.

6.     Enter <global-ntp-server-ip> and click OK.

7.     Click OK.

Cisco UCS Blade Chassis Connection Options

For the Cisco UCS 2300 Series Fabric Extenders, two configuration options are available: pinning and port-channel.

SAP HANA node communicates with every other SAP HANA node using multiple I/O streams and this makes the port-channel option a highly suitable configuration. SAP has defined a single-stream network performance test as part of the hardware validation tool (TDINetServer/TDINetClient).

However, with the new 40Gb network speed it is also possible to stay with the default setting of Cisco UCS which is Port-Channel as connection policy.

Edit Chassis Discovery Policy

Setting the discovery policy simplifies the addition of Cisco UCS B-Series chassis and of additional fabric extenders for further Cisco UCS C-Series connectivity.

To modify the chassis discovery policy, follow these steps:

1.     In Cisco UCS Manager, click the Equipment tab in the navigation pane and select Equipment in the list on the left.

2.     In the right pane, click the Policies tab.

3.     Under Global Policies, set the Chassis/FEX Discovery Policy to match the number of uplink ports that are cabled between the chassis or fabric extenders (FEXes) and the fabric interconnects.

4.     Set the Link Grouping Preference to “Port Channel” for Port Channel.

5.     Click Save Changes.

6.     Click OK.

Enable Server and Uplink Ports

To enable server and uplink ports, follow these steps:

1.     In Cisco UCS Manager, click the Equipment tab in the navigation pane.

2.     Select Equipment > Fabric Interconnects > Fabric Interconnect A (primary) > Fixed Module.

3.     Expand Ethernet Ports.

4.     Select the ports that are connected to the chassis and / or to the Cisco C-Series Server (two per FI), right-click them, and select Configure as Server Port.

5.     Click Yes to confirm server ports and click OK.

6.     Verify that the ports connected to the chassis and / or to the Cisco C-Series Server are now configured as server ports.

7.     In the validation setup, Eth ports 1/19 and 1/20 on FI-A and FI-B are connected to the chassis. Right click these ports and “Configure as Server Port”.

8.     Select ports that are connected to the Cisco Nexus switches, right-click them, and select “Configure as Uplink Port”.

9.     In the validation setup, this is done for the eth ports 1/27-28 and 1/31-34.

10.  Click Yes to confirm uplink ports and click OK.

11.  Select Equipment > Fabric Interconnects > Fabric Interconnect B (subordinate) > Fixed Module.

12.  Expand Ethernet Ports.

13.  Select the ports that are connected to the chassis or to the Cisco C-Series Server (two per FI), right-click them, and select Configure as Server Port.

14.  Click Yes to confirm server ports and click OK.

15.  Select ports that are connected to the Cisco Nexus switches, right-click them and select Configure as Uplink Port.

16.  Click Yes to confirm the uplink ports and click OK.

Acknowledge Cisco UCS Chassis and Rack-Mount Servers

To acknowledge all Cisco UCS chassis and Rack Mount Servers, follow these steps:

1.     In Cisco UCS Manager, click the Equipment tab in the navigation pane.

2.     Expand Chassis and select each chassis that is listed.

3.     Right-click each chassis and select Acknowledge Chassis.

4.     Click Yes and then click OK to complete acknowledging the chassis.

5.     If C-Series servers are part of the configuration, expand Rack Mounts and FEX.

6.     Right-click each Server that is listed and select Acknowledge Server.

7.     Click Yes and then click OK to complete acknowledging the Rack Mount Servers

Create Uplink Port Channels to Cisco Nexus Access layer Switches

An uplink port channel 25 on Fi-A and port channel 26 on FI-B with 4 ports eth 1/31-1/34 is defined to carry all the SAP HANA networks traffic. We are using 4 x 40G ports for this port channel providing 160Gbps operational speed and is good for carrying all the HANA networks traffic.

Optionally another port channel with two ports each on FI- A and FI-B can be configured for backup traffic usage, if bandwidth isolation is warranted. This 2 x 40G port channel providing 80Gbps operational speed should suffice for the same.

To configure the necessary port channels as planned above for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     In this procedure, two port channels are created: one from fabric A to both Cisco Nexus switches and one from fabric B to both Cisco Nexus switches.

3.     Under LAN > LAN Cloud, expand the Fabric A tree.

4.     Right-click Port Channels.

5.     Select Create Port Channel.

6.     Enter 25 as the unique ID of the port channel.

7.     Enter Uplink-A as the name of the port channel.

8.     Click Next.

9.     Select the following ports to be added to the port channel:

·       Slot ID 1 and port 31

·       Slot ID 1 and port 32

·       Slot ID 1 and port 33

·       Slot ID 1 and port 34

10.  Click >> to add the ports to the port channel.

11.  Click Finish to create the port channel.

12.  Click OK.

13.  In the navigation pane, under LAN > LAN Cloud, expand the fabric B tree:

a.     Right-click Port Channels.

b.     Select Create Port Channel.

c.     Enter 26 as the unique ID of the port channel.

d.     Enter Uplink-B as the name of the port channel.

14.  Click Next.

15.  Select the following ports to be added to the port channel:

·       Slot ID 1 and port 31

·       Slot ID 1 and port 32

·       Slot ID 1 and port 33

·       Slot ID 1 and port 34

16.  Click >> to add the ports to the port channel.

17.  Click Finish to create the port channel.

18.  Click OK.

19.  To create optional additional port-channel for segregated network, follow these steps:

20.  In Cisco UCS Manager, click the LAN tab in the navigation pane.

21.  Under LAN > LAN Cloud, expand the Fabric A tree.

22.  Right-click Port Channels.

23.  Select Create Port Channel

24.  Enter 35 as the unique ID of the port channel.

25.  Enter Uplink2-A as the name of the port channel.

26.  Click Next.

27.  Select the following ports to be added to the port channel:

·       Slot ID 1 and port 27

·       Slot ID 1 and port 28

28.  Click >> to add the ports to the port channel.

29.  Click Finish to create the port channel.

30.  Click OK.

31.  In the navigation pane, under LAN > LAN Cloud, expand the fabric B tree.

32.  Right-click Port Channels.

33.  Select Create Port Channel.

34.  Enter 36 as the unique ID of the port channel.

35.  Enter Uplink2-B as the name of the port channel.

36.  Click Next.

37.  Select the following ports to be added to the port channel:

·       Slot ID 1 and port 27

·       Slot ID 1 and port 28

38.  Click >> to add the ports to the port channel.

39.  Click Finish to create the port channel.

40.  Click OK.

Create New Organization

For secure multi-tenancy within the Cisco UCS domain, a logical entity is created as Organizations.

To create organization unit, follow these steps:

1.     In Cisco UCS Manager, from the Servers bar, select Servers and right-click root and select Create Organization.

2.     Enter the Name as HANA.

3.     Optional Enter the Description as Org for HANA.

4.     Click OK to create the Organization.

Create MAC Address Pools

To configure the necessary MAC address pools for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Select Pools > root.

3.     In this procedure, two MAC address pools are created, one for each switching fabric.

4.     Right-click MAC Pools under the root organization.

5.     Select Create MAC Pool to create the MAC address pool.

6.     Enter HANA-FAB-A as the name of the MAC pool.

7.     Optional: Enter a description for the MAC pool.

8.     Choose Assignment Order Sequential.

9.     Click Next.

10.  Click Add.

11.  Specify a starting MAC address.

12.  The recommendation is to place 0A in the fourth octet of the starting MAC address to identify all the MAC addresses as Fabric Interconnect A addresses.

13.  Specify a size for the MAC address pool that is enough to support the available blade or server resources.

14.  Click OK.

15.  Click Finish.

16.  In the confirmation message, click OK.

17.  Right-click MAC Pools under the HANA organization.

18.  Select Create MAC Pool to create the MAC address pool.

19.  Enter HANA-FAB-B as the name of the MAC pool.

20.  Optional: Enter a description for the MAC pool.

21.  Click Next.

22.  Click Add.

23.  Specify a starting MAC address.

24.  Specify a size for the MAC address pool that is enough to support the available blade or server resources.

25.  Cisco UCS - Create MAC Pool for Fabric B.

26.  Click OK.

27.  Click Finish.

28.  In the confirmation message, click OK.

Create UUID Suffix Pool

To configure the necessary universally unique identifier (UUID) suffix pool for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Pools > root.

3.     Right-click UUID Suffix Pools.

4.     Select Create UUID Suffix Pool.

5.     Enter UUID_Pool as the name of the UUID suffix pool.

6.     Optional: Enter a description for the UUID suffix pool.

7.     Keep the Prefix as the Derived option.

8.     Select Sequential for Assignment Order.

9.     Click Next.

10.  Click Add to add a block of UUIDs.

11.  Keep the From field at the default setting.

12.  Specify a size for the UUID block that is enough to support the available blade or server resources.

13.  Click OK.

14.  Click Finish.

15.  Click OK.

Create IQN Pools for iSCSI Boot

To configure the necessary IQN pools for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click SAN.

2.     Expand Pools > root>Sub-Organizations>HANA.

3.     Right-click IQN Pools.

4.     Select Create IQN Suffix Pool to create the IQN pool.

5.     Enter IQN-HANA for the name of the IQN pool

6.     Optional: Enter a description for the IQN pool

7.     Enter iqn.2019-09.com.flexpod as the prefix.

8.     Select Sequential for Assignment Order

9.     Click Next.

10.  Click Add.

11.  Enter hana-node as the suffix.

12.  Enter 1 in the From field.

13.  Specify the size of the IQN block sufficient to support the available server resources.

14.  Click OK.

15.  Click Finish and OK to complete creating the IQN pool.

Create IP Pools for iSCSI Boot

To configure the necessary IP pools iSCSI boot for the Cisco UCS environment within the HANA Organization, follow these steps:

1.     In Cisco UCS Manager, click LAN.

2.     Expand Pools > root > Sub-Organizations > HANA Organization.

3.     Right-click IP Pools under the HANA Organization.

4.     Select Create IP Pool.

5.     Enter iSCSI-IP-Pool-A as the name of IP pool.

6.     Optional: Enter a description for the IP pool.

7.     Select Sequential for the assignment order.

8.     Click Next.

9.     Click Add to add a block of IP addresses.

10.  In the From field, enter the beginning of the range to assign as iSCSI boot IP addresses on Fabric A.

11.  Set the size to enough addresses to accommodate the servers.

12.  Enter the appropriate Subnet Mask.

13.  Click OK.

14.  Click Next.

15.  Click Finish and OK to complete creating the Fabric A iSCSI IP Pool.

16.  Right-click IP Pools under the FlexPod Organization.

17.  Select Create IP Pool.

18.  Enter iSCSI-IP-Pool-B as the name of IP pool.

19.  Optional: Enter a description for the IP pool.

20.  Select Sequential for the assignment order.

21.  Click Next.

22.  Click Add to add a block of IP addresses.

23.  In the From field, enter the beginning of the range to assign as iSCSI IP addresses on Fabric B.

24.  Set the size to enough addresses to accommodate the servers.

25.  Enter the appropriate Subnet Mask.

26.  Click OK.

27.  Click Next.

28.  Click Finish and OK to complete creating the Fabric B iSCSI IP Pool.

Power Policy

To run Cisco UCS with two independent power distribution units, the redundancy must be configured as Grid. Follow these steps:

1.     In Cisco UCS Manager, click the Equipment tab in the navigation pane and select Equipment in the list on the left.

2.     In the right pane, click the Policies tab.

3.     Under Global Policies, set the Power Policy to “Grid.”

4.     Click Save Changes.

5.     Click OK.

Power Control Policy

The Power Capping feature in Cisco UCS is designed to save power with a legacy data center use cases. This feature does not contribute much to the high-performance behavior of SAP HANA. By choosing the option “No Cap” for power control policy, the SAP HANA server nodes will not have a restricted power supply. It is recommended to have this power control policy to ensure sufficient power supply for high performance and critical applications like SAP HANA.

To create a power control policy for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root.

3.     Right-click Power Control Policies.

4.     Select Create Power Control Policy.

5.     Enter HANA as the power control policy name.

6.     Change the power capping setting to No Cap.

7.     Click OK to create the power control policy.

8.     Click OK.

Create Network Control Policy

To create a network control policy that enables CDP and LLDP on server virtual network controller (vNIC) ports, follow these steps:

1.     In Cisco UCS Manager, click LAN.

2.     Expand Policies > root>Sub-Organizations>HANA.

3.     Right-click Network Control Policies.

4.     Select Create Network Control Policy.

5.     Enter Enable-CDP-LLDP as the policy name.

6.     For CDP, select the Enabled option.

7.     For LLDP, scroll down and select Enabled for both Transmit and Receive.

8.     Click OK to create the network control policy.

9.     Click OK.

Create Host Firmware Package

Firmware management policies allow the administrator to select the corresponding packages for a given server configuration. These policies often include packages for adapter, BIOS, board controller, FC adapters, host bus adapter (HBA) option ROM, and storage controller properties.

To create a firmware management policy for a given server configuration in the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root.

3.     Right-click Host Firmware Packages.

4.     Select Create Host Firmware Package.

5.     Enter HANA-FW as the name of the host firmware package.

6.     Leave Simple selected.

7.     Select the version 4.0(4d) packages for both the Blade and Rack Packages.

8.     Click OK to create the host firmware package.

9.     Click OK.

Create Local Disk Configuration Policy (Optional)

A local disk configuration for the Cisco UCS environment is necessary if the servers in the environment do not have a local disk.

To create a local disk configuration policy, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root.

3.     Right-click Local Disk Config Policies.

4.     Select Create Local Disk Configuration Policy.

5.     Enter No-Local as the local disk configuration policy name.

6.     Change the mode to No Local Storage.

7.     Click OK to create the local disk configuration policy.

8.     Click OK.

Create Server BIOS Policy

To get the best performance for HANA it is required to configure the Server BIOS accurately. To create a server BIOS policy for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click BIOS Policies. Choose Create BIOS Policy.

4.     Enter HANA as the BIOS policy name.

5.     Click OK.

6.     Under BIOS Policies, click the newly created HANA Policy.

7.     In the Main pane, under BIOS Setting choose Disabled for Quiet Boot.

8.     Click the Advance tab.

9.     Under Processor choose Custom for Power Technology and enable “C0 C1 State” for Package C State Limit option.

10.  Set IO sensitive for Workload Configuration.

11.  Click RAS Memory.

12.  Choose Platform Default for Memory RAS Configuration and Enabled for NUMA optimized.

13.  Click Serial Port.

14.  Choose Enabled for Serial Port A enable.

15.  Click Server Management.

16.  Choose 115.2k for BAUD Rate, Enabled for Legacy OS redirection, VT100-PLUS for Terminal type. This is used for Serial Console Access over LAN to all SAP HANA servers.

17.  Click Save Changes.

Create Serial Over LAN Policy

The Serial over LAN policy is required to get console access to all the SAP HANA servers through SSH from the management network. This is used if the server hangs or there is a Linux kernel crash, where the dump is required. To configure the speed in the Server Management tab of the BIOS Policy, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root > Sub-Organization > HANA.

3.     Right-click Serial over LAN Policies.

4.     Select Create Serial over LAN Policy.

5.     Enter SoL-Console as the Policy name.

6.     Select Serial over LAN State to enable.

7.     Change the Speed to 115200.

8.     Click OK.

Update Default Maintenance Policy

It is recommended to update the default Maintenance Policy with the Reboot Policy “User Ack” for the SAP HANA server. This policy will wait for the administrator to acknowledge the server reboot for the configuration changes to take effect.

To update the default Maintenance Policy, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root.

3.     Select Maintenance Policies > default.

4.     Change the Reboot Policy to User Ack.

5.     Click Save Changes.

6.     Click OK to accept the change.

Adapter Policy Configuration – HANA

This section describes the Ethernet Adapter Policy with optimized Interrupts values. This policy must be used for the SAP HANA internal network to provide best network performance.

To create an Ethernet Adapter Policy, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root > Sub-Organization > HANA.

3.     Right-click Adapter Policies.

4.     Select Create Ethernet Adapter Policy.

5.     Enter HANA as the Ethernet Adapter policy name.

6.     Expand Resources:

a.     Change the Transmit Queues to 8

b.     Change Ring size to 4096

c.     Change the Receive Queues to 8

d.     Change Ring size to 4096

e.     Change the Completion Queue to 16

f.      Change the Interrupts to 32

7.     Expand Options > Change Receive Side Scaling (RSS) to Enabled

8.     Change Accelerated Receive Flow Steering to Disabled

9.     Click OK to create the Ethernet Adapter policy.

10.  Click OK.

Network Configuration

The core network requirements for SAP HANA are covered by Cisco UCS defaults. Cisco UCS is based on 40-GbE and provides redundancy via the Dual Fabric concept. The Service Profile is configured to distribute the traffic across Fabric Interconnect A and B. During normal operation, the traffic in the Internal Zone and the Data base NFS data traffic is on FI A and all the other traffic (Client Zone and Database NFS log) is on FI B. The inter-node traffic flows from a Blade Server to the Fabric Interconnect A and back to other Blade Server. All the other traffic must go over the Cisco Nexus switches to storage or to the data center network. With the integrated algorithms for bandwidth allocation and quality of service the Cisco UCS and Cisco Nexus distributes the traffic in an efficient way.

LAN Tab Configurations

Within Cisco UCS, all the network types for an SAP HANA system are reflected by defined VLANs. Network design from SAP has seven SAP HANA related networks and two infrastructure related networks. The VLAN IDs can be changed if required to match the VLAN IDs in the data center network – for example, ID 224 for backup should match the configured VLAN ID at the data center network switches. Even though nine VLANs are defined, VLANs for all the networks are not necessary if the solution will not use the network. For example, if the Replication Network is not used in the solution, then VLAN ID 225 does not have to be created.

Create VLANs

To configure the necessary VLANs for the Cisco UCS environment, follow these steps:

In Cisco UCS Manager, click the LAN tab in the navigation pane.

1.     Select LAN > LAN Cloud.

2.     Right-click VLANs.

3.     Select Create VLANs.

4.     Enter iSCSI_A as the name of the VLAN to be used for iSCSI -A network.

5.     Keep the Common/Global option selected for the scope of the VLAN.

6.     Enter <iscsi-a-vlan-id>> as the ID of the HANA node admin network.

7.     Keep the Sharing Type as None.

8.     Click OK and then click OK again.

9.     Repeat steps 1-8 to configure the rest of the VLANs as planned in the table below.

Table 5    VLANs used in this CVD

Create VLAN Groups

For easier management and bandwidth segregation on the Fabric Interconnect, VLAN Groups are created within the Cisco UCS. We could bundle client zone networks, inter-node and IP storage networks in a VLAN group and create a separate VLAN group consisting of Backup and System replication networks.

VLAN group with HANA networks is assigned to 160Gbps bandwidth port channels with 4 x 40G ports such as port channels 25 and 26 on FI-A and FI-B respectively. VLAN group of backup and replication networks is assigned to 80Gbps bandwidth capable port channels with 2 x 40G ports, in other words, port channels 35 and 36 on FI-A and FI-B respectively.

To configure the necessary VLAN Groups for the Cisco UCS environment, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.      Select LAN > LAN Cloud.

3.     Right-click VLAN Groups.

4.     Select Create VLAN Groups.

5.     Enter HANA-nws as the name of the VLAN Group used for this bundle of networks.

6.     Select all the created VLANs except for backup and system replication

7.     Click Next.

8.     Click Next on Add Uplink Ports, since you will use port-channel.

9.     Choose port-channels created for uplink network. Click >>.

10.  Click Finish.

11.  Similarly, create bkp-sysrep vlan-group including the Backup and System Replication networks.

12.  Click Next.

13.  Click Next on Add Uplink Ports, since you will use port-channel.

14.  Choose port-channels created for uplink network. Click >>.

15.  Click Finish.

Create vNIC Templates

Each VLAN is mapped to a vNIC template to specify the characteristic of a specific network. The vNIC template configuration settings include MTU size, Failover capabilities and MAC-Address pools.

Create vNIC Template for iSCSI via Fabric A

To create iSCSI virtual network interface card (vNIC) templates for the Cisco UCS environment within the FlexPod Organization, follow these steps:

1.     Select LAN.

2.     Expand Policies > root > Sub-Organizations > FlexPod Organization.

3.     Right-click vNIC Templates under the HANA Organization.

4.     Select Create vNIC Template.

5.     Enter iSCSI-A as the vNIC template name.

6.     Select Fabric A. Do not select the Enable Failover checkbox.

7.     Leave Redundancy Type set at No Redundancy.

8.     Under Target, make sure that only the Adapter checkbox is selected.

9.     Select Updating Template for Template Type.

10.  Under VLANs, select only iSCSI_A.

11.  Select iSCSI_A as the native VLAN.

12.  Leave vNIC Name set for the CDN Source.

13.  Under MTU, enter 9000.

14.  From the MAC Pool list, select HANA-Fab-A.

15.  From the Network Control Policy list, select Enable-CDP-LLDP.

16.  Repeat steps1-15 to create vNIC template for each Network Interface.

Create vNIC Template for iSCSI with Fabric B

To create a vNIC template for iSCSI with fabric B, follow these steps:

1.     Right-click vNIC Templates.

2.     Select Create vNIC Template.

3.     Enter iSCSI-Template-B as the vNIC template name.

4.     Select Fabric B. Do not select the Enable Failover checkbox.

5.     Leave Redundancy Type set at No Redundancy.

6.     Under Target, make sure that only the Adapter checkbox is selected.

7.     Select Updating Template for Template Type.

8.     Under VLANs, select only iSCSI-B.

9.     Select iSCSI-B as the native VLAN.

10.  Leave vNIC Name set for the CDN Source.

11.  Under MTU, enter 9000.

12.  From the MAC Pool list, select HANA-Fab-B.

13.  From the Network Control Policy list, select Enable-CDP-LLDP.

14.  Click OK to complete creating the vNIC template.

15.  Click OK.

Create a vNIC Template for Inter-node Network

To create a vNIC template for Inter-node network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter Server as the vNIC template name.

6.     Keep Fabric B selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for Server.

11.  Set Server as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-B.

14.  For Network Control Policy, choose default from drop-down list

15.  Click OK to create the vNIC template.

Create a vNIC Template for NFS-Data Network

To create a vNIC template for NFS-Data network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter NFS-Data as the vNIC template name.

6.     Keep Fabric A selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for NFS_Data.

11.  Set NFS_Data as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-A.

14.  For Network Control Policy, choose Enable-CDP-LLDP from drop-down list

15.  Click OK to create the vNIC template.

Create a vNIC Template for NFS-Log Network

To create a vNIC template for an NFS-Log network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter NFS-Log as the vNIC template name.

6.     Keep Fabric A selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for NFS_Log.

11.  Set NFS_Log as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-B.

14.  For Network Control Policy, choose Enable-CDP-LLDP from drop-down list.

15.  Click OK to create the vNIC template.

Create a vNIC Template for Application Network

To create a vNIC template for an application network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter Application as the vNIC template name.

6.     Keep Fabric B selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for Application.

11.  Set Application as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-B.

14.  For Network Control Policy, choose default from drop-down list.

15.  Click OK to create the vNIC template.

Create a vNIC Template for DataSource Network

To create vNIC template for a DataSource network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter DataSource as the vNIC template name.

6.     Keep Fabric A selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for DataSource.

11.  Set DataSource as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-A.

14.  For Network Control Policy, choose default from drop-down list.

15.  Click OK to create the vNIC template.

Create a vNIC Template for Backup Network

To create vNIC template for a backup network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates

4.     Choose Create vNIC Template.

5.     Enter Backup as the vNIC template name.

6.     Keep Fabric B selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for Backup.

11.  Set Backup as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-B.

14.  For Network Control Policy, choose Enable-CDP-LLDP from drop-down list.

15.  Click OK to create the vNIC template.

Create a vNIC Template for System Replication Network

To create a vNIC template for a system replication network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates

4.     Choose Create vNIC Template.

5.     Enter SysRep as the vNIC template name.

6.     Keep Fabric A selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for SysRep.

11.  Set SysRep as the native VLAN.

12.  For MTU, enter 9000.

13.  In the MAC Pool list, choose HANA-Fab-A.

14.  For Network Control Policy, choose default from drop-down list.

15.  Click OK to create the vNIC template.

Create a vNIC Template for Management Network

To create a vNIC template for a management network, follow these steps:

1.     In Cisco UCS Manager, click the LAN tab in the navigation pane.

2.     Choose Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC Templates.

4.     Choose Create vNIC Template.

5.     Enter Mgmt as the vNIC template name.

6.     Keep Fabric A selected.

7.     Check the Enable Failover checkbox.

8.     Under Target, make sure that the VM checkbox is unchecked.

9.     Choose Updating Template as the Template Type.

10.  Under VLANs, check the checkboxes for Management.

11.  Set Management as the native VLAN.

12.  For MTU, enter 1500.

13.  In the MAC Pool list, choose HANA-Fab-A.

14.  For Network Control Policy, choose default from drop-down list.

15.  Click OK to create the vNIC template.

Create vNIC/vHBA Placement Policy

Cisco UCS assigns virtual network interface connections (vCons) to the PCIe adapter cards in the server. Each vCon is a virtual representation of a physical adapter that can be assigned vNICs and vHBAs.

For blade or rack servers that contain one adapter, Cisco UCS assigns all vCons to that adapter.

For blade or rack servers that contain two or three adapters, Cisco UCS assigns the vCons based on the selected virtual slot mapping scheme. This can be one of the following:

·       Round Robin— In a server with two adapter cards, Cisco UCS assigns vCon1 and vCon3 to Adapter1, then assigns vCon2 and vCon4 to Adapter2.

·       In a server with three adapter cards, Cisco UCS assigns vCon1 to Adapter1, vCon2 and vCon4 to Adapter2, and vCon3 to Adapter3.

·       This is the default scheme.

·       Linear Ordered— In a server with two adapter cards, Cisco UCS assigns vCon1 and vCon2 to Adapter1, then assigns vCon3 and vCon4 to Adapter2.

·       In a server with three adapter cards, Cisco UCS assigns vCon1 to Adapter1 and vCon2 to Adapter2, then assigns vCon3 and vCon4 to Adapter3.

To create a vNIC/vHBA placement policy for the SAP HANA hosts, follow these steps:

1.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

2.     Select Policies > root > Sub-Organization > HANA.

3.     Right-click vNIC/vHBA Placement Policies.

4.     Select Create Placement Policy.

5.     Enter HANA as the name of the placement policy and select Liner Ordered for Virtual Slot Mapping Scheme.

 

Create LAN Connectivity Policy

With LAN connectivity policy we are pre-defining the vNICs that a system needs to have in order to cater to the specific networks the use-case demands.

For both the single host system and multiple-host use-cases, apart from the admin/management the node may need backup and application server network connections, at a minimum. It could have other networks depending on the usage.

However, for the multiple-host scenario, inter-node communication is mandatory and that is what distinguishes it from a single host system’s policy.

In the following steps we create separate LAN connectivity policies for the two standard use-cases. This further simplifies the service profile template creation by having the pre-determined vNICs already part of the network configuration.

Single-Host Systems Use-Case

1.     To configure the necessary LAN Connectivity Policy within the HANA Organization, follow these steps:

2.     In Cisco UCS Manager, click LAN.

3.     Expand LAN > Policies > root > Sub-Organizations > HANA Organization.

4.     Right-click LAN Connectivity Policies under the HANA Organization.

5.     Select Create LAN Connectivity Policy.

6.     Enter iSCSI-Boot-SclUp as the name of the policy.

7.     Click Add to add a vNIC.

8.     In the Create vNIC dialog box, enter HANA-node-admin as the name of the vNIC.

9.     Select the Use vNIC Template checkbox.

10.  In the vNIC Template list, select Mgmt.

11.  In the Adapter Policy list, select Linux.

12.  Click OK to add this vNIC to the policy.

13.  Click Add to add another vNIC to the policy.

14.  In the Create vNIC box, enter HANA-data as the name of the vNIC.

15.  Select the Use vNIC Template checkbox.

16.  In the vNIC Template list, select NFS-Data.

17.  In the Adapter Policy list, select HANA.

18.  Click OK to add the vNIC to the policy.

19.  Click Add to add another vNIC to the policy.

20.  In the Create vNIC dialog box, enter HANA-log as the name of the vNIC.

21.  Select the Use vNIC Template checkbox.

22.  In the vNIC Template list, select NFS-Log.

23.  In the Adapter Policy list, select HANA.

24.  Click OK to add this vNIC to the policy.

25.  Click Add to add a vNIC.

26.  In the Create vNIC dialog box, enter HANA-AppServer as the name of the vNIC.

27.  Select the Use vNIC Template checkbox.

28.  In the vNIC Template list, select Application.

29.  In the Adapter Policy list, select Linux.

30.  Click OK to add this vNIC to the policy.

31.  Click Add to add a vNIC.

32.  In the Create vNIC dialog box, enter HANA-node-backup as the name of the vNIC.

33.  Select the Use vNIC Template checkbox.

34.  In the vNIC Template list, select Backup.

35.  In the Adapter Policy list, select Linux.

36.  Click OK to add this vNIC to the policy.

37.  Click Add to add a vNIC.

38.  In the Create vNIC dialog box, enter iSCSI-A as the name of the vNIC.

39.  Select the Use vNIC Template checkbox.

40.  In the vNIC Template list, select iSCSI-A.

41.  In the Adapter Policy list, select Linux.

42.  Click OK to add this vNIC to the policy.

43.  Click Add to add a vNIC to the policy.

44.  In the Create vNIC dialog box, enter iSCSI-B as the name of the vNIC.

45.  Select the Use vNIC Template checkbox.

46.  In the vNIC Template list, select iSCSI-B.

47.  In the Adapter Policy list, select Linux.

48.  Click OK to add this vNIC to the policy.

49.  Expand Add iSCSI vNICs.

50.  Select Add in the Add iSCSI vNICs section.

51.  Set the name to iSCSI-Boot-A.

52.  Select iSCSI-A as the Overlay vNIC.

53.  Set the iSCSI Adapter Policy to default.

54.  Leave the VLAN set to iSCSI_A (native).

55.  Leave the MAC Address set to None.

56.  Click OK.

57.  Select Add in the Add iSCSI vNICs section.

58.  Set the name to iSCSI-Boot-B.

59.  Select iSCSI-B as the Overlay vNIC.

60.  Set the iSCSI Adapter Policy to default.

61.  Leave the VLAN set to iSCSI_B (native).

62.  Leave the MAC Address set to None

63.  Click OK.

Multiple-Host Systems Use-Case

To configure the necessary LAN Connectivity Policy within the HANA Organization, follow these steps:

1.     In Cisco UCS Manager, click LAN.

2.     Expand LAN > Policies > root > Sub-Organizations > HANA Organization.

3.     Right-click LAN Connectivity Policies under the HANA Organization.

4.     Select Create LAN Connectivity Policy.

5.     Enter iSCSI-Boot-ScOut as the name of the policy.

6.     Click Add to add a vNIC.

7.     In the Create vNIC dialog box, enter HANA-node-admin as the name of the vNIC.

8.     Select the Use vNIC Template checkbox.

9.     In the vNIC Template list, select Mgmt.

10.  In the Adapter Policy list, select Linux.

11.  Click OK to add this vNIC to the policy.

12.  Click Add to add another vNIC to the policy.

13.  In the Create vNIC box, enter HANA-data as the name of the vNIC.

14.  Select the Use vNIC Template checkbox.

15.  In the vNIC Template list, select NFS-Data.

16.  In the Adapter Policy list, select HANA.

17.  Click OK to add the vNIC to the policy.

18.  Click Add to add a vNIC.

19.  In the Create vNIC dialog box, enter HANA-log as the name of the vNIC.

20.  Select the Use vNIC Template checkbox.

21.  In the vNIC Template list, select NFS-Log.

22.  In the Adapter Policy list, select HANA.

23.  Click OK to add this vNIC to the policy.

24.  Click Add to add a vNIC.

25.  In the Create vNIC dialog box, enter HANA-AppServer as the name of the vNIC.

26.  Select the Use vNIC Template checkbox.

27.  In the vNIC Template list, select Application.

28.  In the Adapter Policy list, select Linux.

29.  Click OK to add this vNIC to the policy.

30.  Click Add to add a vNIC.

31.  In the Create vNIC dialog box, enter HANA-node-backup as the name of the vNIC.

32.  Select the Use vNIC Template checkbox.

33.  In the vNIC Template list, select Backup.

34.  In the Adapter Policy list, select Linux.

35.  Click OK to add this vNIC to the policy.

36.  Click Add to add a vNIC.

37.  In the Create vNIC dialog box, enter HANA-internode as the name of the vNIC.

38.  Select the Use vNIC Template checkbox.

39.  In the vNIC Template list, select Server.

40.  In the Adapter Policy list, select HANA.

41.  Click OK to add this vNIC to the policy

42.  Click Add to add a vNIC.

43.  In the Create vNIC dialog box, enter iSCSI-A as the name of the vNIC.

44.  Select the Use vNIC Template checkbox.

45.  In the vNIC Template list, select iSCSI-A.

46.  In the Adapter Policy list, select Linux.

47.  Click OK to add this vNIC to the policy.

48.  Click Add to add a vNIC to the policy.

49.  In the Create vNIC dialog box, enter iSCSI-B as the name of the vNIC.

50.  Select the Use vNIC Template checkbox.

51.  In the vNIC Template list, select iSCSI-B.

52.  In the Adapter Policy list, select Linux.

53.  Click OK to add this vNIC to the policy.

54.  Expand Add iSCSI vNICs.

55.  Select Add in the Add iSCSI vNICs section.

56.  Set the name to iSCSI-Boot-A.

57.  Select iSCSI-A as the Overlay vNIC.

58.  Set the iSCSI Adapter Policy to default.

59.  Leave the VLAN set to iSCSI_A (native).

60.  Leave the MAC Address set to None.

61.  Click OK.

62.  Select Add in the Add iSCSI vNICs section.

63.  Set the name to iSCSI-Boot-B.

64.  Select iSCSI-B as the Overlay vNIC.

65.  Set the iSCSI Adapter Policy to default.

66.  Leave the VLAN set to iSCSI_B (native).

67.  Leave the MAC Address set to None

68.  Click OK

Create iSCSI Boot Policy

This procedure applies to a Cisco UCS environment in which two iSCSI logical interfaces (LIFs) are on cluster node 1 (iscsi_lif_1a and iscsi_lif_1b) and two iSCSI LIFs are on cluster node 2 (iscsi_lif_2a and iscsi_lif_2b). Also, it is assumed that the A LIFs are connected to Fabric A (Cisco UCS Fabric Interconnect A) and the B LIFs are connected to Fabric B (Cisco UCS Fabric Interconnect B).

To create a boot policy for the Cisco UCS environment within the HANA Organization, follow these steps:

1.     In Cisco UCS Manager, click Servers.

2.     Expand Policies > root > Sub-Organizations > HANA Organization.

3.     Right-click Boot Policies under the HANA Organization.

4.     Select Create Boot Policy.

5.     Enter iSCSI-HANA as the name of the boot policy.

6.     Optional: Enter a description for the boot policy.

7.     Do not select the Reboot on Boot Order Change checkbox.

8.     Leave Enforce on vNIC/vHBA/iSCSI Name checkbox selected.

9.     Select the Legacy Mode.

10.  Expand the Local Devices drop-down menu and select Add CD/DVD.

11.  Expand the iSCSI vNICs drop-down menu and select Add iSCSI Boot.

12.  In the Add iSCSI Boot dialog box, enter iSCSI-Boot-A.

13.  Click OK.

14.  Select Add iSCSI Boot.

15.  In the Add iSCSI Boot dialog box, enter iSCSI-Boot-B.

16.  Click OK.

  

17.  Click OK then click OK again to create the policy.

Create Service Profile Template for SAP HANA Node

This section details the service profile template creation procedure. The steps to create service profile template to instantiate HANA nodes for single-host or multiple-host use-case depends on the LAN connectivity policy you select and corresponding placement of vNICs per vCONs.

To create the service profile template, follow these steps:

1.     In Cisco UCS Manager, click Servers.

2.     Expand Service Profile Templates > root > Sub-Organizations > HANA Organization.

3.     Right-click the HANA Organization.

4.     Select Create Service Profile Template to open the Create Service Profile Template wizard.

5.     Enter iSCSI-HANA-node as the name of the service profile template. This service profile template is configured to boot from storage node 1 on fabric A.

6.     Select the “Updating Template” option.

7.     Under UUID Assignment, select UUID_Pool.

8.     Click Next.

Configure Storage Provisioning

To configure the storage provisioning, follow these steps:

1.     If you have servers with no physical disks, click on the Local Disk Configuration Policy tab select the default Local Storage Policy.

2.     Click Next.

Configure Networking Options

To configure the network options for HANA node intended to be ScaleUp system, follow these steps:

1.     Keep the default setting for Dynamic vNIC Connection Policy.

2.     Select the “Use Connectivity Policy” option to configure the LAN connectivity.

3.     Select iSCSI-Boot-SclUp from the LAN Connectivity Policy drop-down list.

4.     Select IQN-HANA in Initiator Name Assignment.

 

5.     Click Next.

OR

To configure the network options for HANA node intended to be part of multiple-host cluster, follow these steps:

1.     Keep the default setting for Dynamic vNIC Connection Policy.

2.     Select the “Use Connectivity Policy” option to configure the LAN connectivity.

3.     Select iSCSI-Boot-SclOut from the LAN Connectivity Policy drop-down list.

4.     Select IQN-HANA in Initiator Name Assignment.

 

5.     Click Next.

Configure Storage Options

To configure the storage options, follow these steps:

1.     Select No vHBAs for the “How would you like to configure SAN connectivity?” field.

2.     Click Next.

Configure Zoning Options

To configure the zoning options, follow this step:

1.     Make no changes and click Next.

Configure vNIC/HBA Placement

To configure the vNIC/HBA placement in case of Single system, follow these steps:

1.     In the “Select Placement” list, Select HANA.

2.     Assign the vNICs to vCon1 and vCon3 as below:

3.     Click Next.

To configure the vNIC/HBA placement in case of Multiple-host system, follow these steps:

1.     In the “Select Placement” list, Select HANA.

2.     Assign the vNICs to vCon1 and vCon3 as below:

3.     Click Next.

Configure vMedia Policy

To configure the vMedia policy, follow these steps:

1.     Do not select a vMedia Policy.

2.     Click Next.

Configure Server Boot Order

To configure the server boot orders, follow these steps:

1.     Select Boot-iSCSI-A for Boot Policy.

2.     In the Boor order, select iSCSI-Boot-A.

3.     Click Set iSCSI Boot Parameters.

4.     In the Set iSCSI Boot Parameters pop-up, leave Authentication Profile to <not set> unless you have in-dependently created one appropriate to your environment.

5.     Leave the “Initiator Name Assignment” dialog box <not set> to use the single Service Profile Initiator Name defined in the previous steps.

6.     Set iSCSI-IP-Pool-A as the “Initiator IP address Policy.”

7.     Select iSCSI Static Target Interface option.

8.     Click Add.

9.     Enter the iSCSI Target Name. To get the iSCSI target name of Infra-SVM, log into the storage cluster management interface and run the “iscsi show” command”.

10.  Enter the IP address of iscsi_lif_1a for the IPv4 Address field.

11.  Click OK to add the iSCSI static target.

12.  Click Add.

13.  Enter the iSCSI Target Name.

14.  Enter the IP address of iscsi_lif_2a for the IPv4 Address field.

15.  Click OK to add the iSCSI static target.

 

16.  Click OK to complete setting the iSCSI Boot Parameters.

17.  In the Boot order, select iSCSI-Boot-B.

18.  Click Set iSCSI Boot Parameters.

19.  In the Set iSCSI Boot Parameters pop-up, leave Authentication Profile to <not set> unless you have in-dependently created one appropriate to your environment.

20.  Leave the “Initiator Name Assignment” dialog box <not set> to use the single Service Profile Initiator Name defined in the previous steps.

21.  Set iSCSI-IP-Pool-B as the “Initiator IP address Policy”.

22.  Select the iSCSI Static Target Interface option.

23.  Click Add.

24.  Enter the iSCSI Target Name. To get the iSCSI target name of Infra-SVM, login into storage cluster management interface and run “iscsi show” command”.

25.  Enter the IP address of iscsi_lif_1b for the IPv4 Address field.

26.  Click OK to add the iSCSI static target.

27.  Click Add.

28.  Enter the iSCSI Target Name.

29.  Enter the IP address of iscsi_lif_2b for the IPv4 Address field.

30.  Click OK to add the iSCSI static target.

31.  Click OK to complete setting the iSCSI Boot Parameters. Click Next.

Configure Maintenance Policy

To configure the maintenance policy, follow these steps:

1.     Change the Maintenance Policy to default.

2.     Click Next.

Configure Server Assignment

To configure the server assignment, follow these steps:

1.     Select Assign Later for the Pool Assignment

2.     Under Firmware Management select HANA-FW for Host Firmware Package from the dropdown list. Click Next.

Configure Operational Policies

To configure the operational policies, follow these steps:

1.     In the BIOS Policy list, select HANA.

2.     Expand Management IP Address: On Outband IPv4 tab, select Outband-Mgmt policy from the drop-down list for Management IP Address Policy.

3.     Expand Power Control Policy Configuration and select HANA in the Power Control Policy list.

4.     Choose default from drop-down list for the rest.   

5.     Click Finish to create the service profile template.

6.     Click OK in the confirmation message.

Create Service Profiles for Single-Host System

To create service profiles from the service profile template created using the LAN connectivity policy iSCSI-Boot-SclUp, intending to create a single host system based on RHEL7.6, follow these steps:

1.     Connect to Cisco UCS Manager and click Servers.

2.     Select Service Profile Templates > root > Sub-Organizations > HANA Organization > Service Template iSCSI-HANA-node

3.     Right-click iSCSI-HANA-node and select Create Service Profiles from Template.

4.     Enter HANA-ScaleUp-RHEL as the service profile prefix.

5.     Enter 76 as “Name Suffix Starting Number.”

6.     Enter 1 as the “Number of Instances.”

7.     Click OK to create the service profile.

8.     Click OK in the confirmation message.

Create Service Profiles for Multi-Host System

To create service profiles from the service profile template created using the LAN connectivity policy iSCSI-Boot-ScOut, intending to create a 4-node scale-out cluster based on SLES 15, follow these steps:

1.     Connect to Cisco UCS Manager and click Servers.

2.     Select Service Profile Templates > root > Sub-Organizations > HANA Organization > Service Template iSCSI-HANA-node

3.     Right-click iSCSI-HANA-node and select Create Service Profiles from Template.

4.     Enter HANA-SO-SLES as the service profile prefix.

5.     Enter 1 as “Name Suffix Starting Number.”

6.     Enter 4 as the “Number of Instances.”

7.     Click OK to create the service profiles.

8.     Click OK in the confirmation message.

Complete Configuration Worksheet

Before running the setup script, complete the cluster setup worksheet in the ONTAP 9.6 Software Setup Guide located in the NetApp® ONTAP® 9 Documentation Center.

Configure ONTAP Nodes

Before running the setup script, review the configuration worksheets in the ONTAP 9.6 Software Setup Guide to learn about configuring ONTAP software. Table 6  lists the information needed to configure two ONTAP nodes. Customize the cluster detail values with the information applicable to your deployment.

Table 6    ONTAP Software Installation Prerequisites

Configure Node 01

To configure node 01, follow these steps:

1.     Connect to the storage system console port. You should see a Loader-A prompt. However, if the storage system is in a reboot loop, press Ctrl-C to exit the autoboot loop when the following message displays:

2.     Allow the system to boot up.

3.     Press Ctrl-C when prompted.

4.     To install new software, select option 7.

5.     Enter y to perform an upgrade.

6.     Select e0M for the network port you want to use for the download.

7.     Enter y to reboot now.

8.     Enter the IP address, netmask, and default gateway for e0M.

9.     Enter the URL where the software can be found.

10.  Press Enter for the user name, indicating no user name.

11.  Enter y to set the newly installed software as the default to be used for subsequent reboots.

12.  Enter y to reboot the node.

13.  Press Ctrl-C when the following message displays:

14.  Select option 4 for Clean Configuration and Initialize All Disks.

15.  Enter y to zero disks, reset config, and install a new file system.

16.  Enter y to erase all the data on the disks.

Configure Node 02

To configure node 02, follow these steps:

1.     Connect to the storage system console port. You should see a Loader-A prompt. However, if the storage system is in a reboot loop, press Ctrl-C to exit the autoboot loop when the following message displays:

2.     Allow the system to boot up.

3.     Press Ctrl-C when prompted.

4.     To install new software, select option 7.

5.     Enter y to perform an upgrade.

6.     Select e0M for the network port you want to use for the download.

7.     Enter y to reboot now.

8.     Enter the IP address, netmask, and default gateway for e0M.

9.     Enter the URL where the software can be found.

10.  Press Enter for the username, indicating no username.

11.  Enter y to set the newly installed software as the default to be used for subsequent reboots.

12.  Enter y to reboot the node.

13.  Press Ctrl-C when you see this message:

14.  Select option 4 for Clean Configuration and Initialize All Disks.

15.  Enter y to zero disks, reset config, and install a new file system.

16.  Enter y to erase all the data on the disks.

Set Up ONTAP Cluster

Table 7  lists all the parameters required to set up the ONTAP cluster.

Table 7    ONTAP Cluster Prerequisites

To set up an ONTAP cluster, follow these steps:

1.     From a console port program attached to the storage controller A (node 01) console port, run the node setup script. This script appears when ONTAP 9.6 software boots on the node for the first time.

2.     Follow the prompts to set up node 01:

3.     To complete the cluster setup, access https://<node01-mgmt-ip> with your web browser
Start the “Guided Cluster Setup.”

4.     Provide the cluster name, the password of the user admin, and add all relevant licenses. Click “Submit and Continue”

5.     Provide the IP address, netmask, and gateway for the Cluster management and choose the desired port, for example select e0M for the first controller. Retain the settings for the Node management by checking the box.

6.     Provide the IP addresses for the Service Processor Management. Check the boxes “Override the Default Values” and “Retain Networks and Gateway Configuration of the Cluster Management”.
Finally enter the DNS and NTP details.

7.     Enter the information for any needed event notifications.

8.     Skip the fourth step (Storage) without creating any aggregate. You will perform this step manually later.

9.     The Cluster has been successfully configured.

To log into the cluster, follow these steps:

1.     Open an SSH connection to either the cluster IP or host name.

2.     Log in to the admin user with the password you provided earlier.

Set Auto-Revert on Cluster Management

To set the auto-revert parameter on the cluster management interface, run the following command:

Set Up Management Broadcast Domain

By default, all network ports are included in the default broadcast domain. Network ports used for data services (for example, e0d, e1a, and e1e) should be removed from the default broadcast domain, leaving just the management network ports (e0c and e0M).

To make the changes, the following commands must be executed for each storage node. Storage nodes are named after the cluster name with an appended number.

Create Aggregates

An aggregate containing the root volume for each storage controller is created during the ONTAP software setup process. To create additional aggregates, determine the aggregate name, the node on which to create it, and the number of disks it should contain.

To create new aggregates, follow these steps:

1.     Run the following commands:

2.     (Optional) Rename the root aggregate on node 01 to match the naming convention for this aggregate on node 02. The aggregate is automatically renamed if system-guided setup is used.

Verify Storage Failover

To confirm that storage failover is enabled, run the following commands for a failover pair:

1.     Verify the status of storage failover.

2.     Enable failover on one of the two nodes.

3.     Verify the HA status for a two-node cluster.

4.     Continue with step 6 if high availability is configured.

5.     Only enable HA mode for two-node clusters. Do not run this command for clusters with more than two nodes because it causes problems with failover.

6.     Verify that hardware assist is correctly configured and, if needed, modify the partner IP address.

Disable Flow Control on 100GbE Ports

NetApp recommends disabling flow control on all the 100GbE ports that are connected to external devices. To disable flow control, follow these steps:

1.     Run the following commands to configure node 1:

2.     Run the following commands to configure node 2:

Configure AutoSupport

NetApp AutoSupport sends support summary information to NetApp through HTTPS.

To configure AutoSupport, run the following command:

Enable Cisco Discovery Protocol

To enable the Cisco Discovery Protocol (CDP) on the NetApp storage controllers, run the following command:

Create Broadcast Domains

Figure 4 shows the physical network connection and the virtual LANs (VLANs) used for this setup.

Table 8    Cluster Networking Requirements

All broadcast domains must be created with an MTU size of 9000 (jumbo frames):

Create Interface Groups

To create the Link Aggregation Control Protocol (LACP) interface groups for the 40GbE data interfaces, run the following commands:

Create VLANs

To create VLANs, follow these steps:

1.     Set the MTU size of the interface groups.

2.     Create HANA data VLAN ports and add them to the NFS-Data broadcast domain.

3.     Create HANA log VLAN ports and add them to the NFS-Log broadcast domain.

4.     Create the iSCSI-a VLAN ports and add them to the iSCSI-a broadcast domain.

5.     Create the iSCSI-b VLAN ports and add them to the iSCSI-b broadcast domain.

6.     Create backup VLAN ports and add them to the backup domain.

Configure HTTPS Access

For each of the SVMs and the cluster node, create a certificate to allow secure communication with HTTPS. For each of the certificates, specify the individual values listed in Table 9  .

Table 9     ONTAP Software Parameter to Enable HTTPS

To configure secure access to the storage controller, follow these steps:

1.     Increase the privilege level to access the certificate commands.

2.     Generally, a self-signed certificate is already in place. Verify the certificate and obtain parameters (for example the <serial-number>) by running the following command:

3.     For each SVM shown, the certificate common name should match the DNS FQDN of the SVM. Delete the two default certificates and replace them with either self-signed certificates or certificates from a certificate authority (CA). To delete the default certificates, run the following commands:

4.     To generate and install self-signed certificates, run the following commands as one-time commands. Generate a server certificate for the Infra-SVM, the HANA SVM, and the cluster SVM. Use tab completion to aid in the completion of these commands.

5.     To obtain the values for the parameters required in step 5 (<cert-ca> and <cert-serial>), run the security certificate show command.

6.     Enable each certificate that was just created by using the –server-enabled true and –client-enabled false parameters. Use TAB completion to aid in the completion of these commands.

7.     Disable HTTP cluster management access.

8.     Change back to the normal admin privilege level and set up the system to allow SVM logs to be available by web.

Configure SVM for the Infrastructure

Table 10  and Figure 5 describe the infrastructure SVM together with all required storage objects (volumes, export-policies, and LIFs).

Table 10      ONTAP Software Parameters for Infrastructure SVMs

Create SVM for the Infrastructure

To create an infrastructure SVM, follow these steps:

1.     Run the vserver create command.

2.     Select the SVM data protocols to configure, keeping iSCSI and NFS.

3.     Add the data aggregates to the Infra-svm aggregate list .

4.     Enable and run the NFS protocol in the Infra-svm.

Create Load-Sharing Mirrors

To create a load-sharing mirror of an SVM root volume, follow these steps:

1.     Create a volume to be the load-sharing mirror of the infrastructure SVM root volume on each node.

2.     Create the mirroring relationships.

3.     Initialize the mirroring relationship.

Create Export Policies for the Root Volumes

To configure to export policies on the SVM, follow these steps:

1.     Create a new rule for the infrastructure NFS subnet in the default export policy.

2.     Assign the export policy to the infrastructure SVM root volume.

Add Infrastructure SVM  Management LIF

To add the infrastructure SVM administration LIF in the out-of-band management network, follow these steps:

1.     Run the following commands:

2.     Create a default route to allow the SVM management interface to reach the outside world.

Create iSCSI LIFs

To create the four iSCSI LIFs (two on each node), run the following commands:

Create Block Protocol (iSCSI) Service

Run the following command to create the iSCSI service. This command also starts the iSCSI service and sets the iSCSI Qualified Name (IQN) for the SVM. Not required, if SVM creation has been done via ONTAP System Manager

Create FlexVol Volumes

To create the FlexVol volumes, run the following commands:

Configure LUNs for iSCSI Boot

Create Boot LUNs for Servers

To create boot LUNs, run the following commands:

The example above created boot LUNs for two servers. Repeat the command with a different LUN name to create additional boot LUNs for additional servers

Create Portset

To create a portset that includes all iSCSI LIFs, run the following commands:

Create igroups

To create igroups, run the following commands:

Repeat this command by using the iqn name of additional servers to create additional igroups for additional servers

Map Boot LUNs to igroups

To map server boot LUNs to igroups, run the following commands:

Repeat this command to map additional boot LUNs to additional servers

Configure SVM for HANA

Table 11  and Figure 6 describe the HANA SVM together with all the required storage objects (volumes, export-policies, and LIFs).

Table 11      ONTAP Software Parameter for HANA SVM

Create SVM for SAP HANA

To create an SVM for SAP HANA volumes, follow these steps:

1.     Run the vserver create command.

2.     Select the SVM data protocols to configure, keeping NFS.

3.     Add the two data aggregates to the hana-svm aggregate list.

4.     Enable and run the NFS protocol in the Infra-SVM.

5.     Enable a large NFS transfer size.

6.     Set the NFSv4 ID domain.

7.     Set the NFSv4 lease time.

8.     Set the group ID of the user root to 0.

Create Load-Sharing Mirrors

To create a load-sharing mirror of an SVM root volume, follow these steps:

1.     Create a volume to be the load-sharing mirror of the HANA SVM root volume on each node.

2.     Create the mirroring relationships.

3.     Initialize the mirroring relationship.

Create Export Policies for the Root Volumes

To configure the NFS export policies on the SVM, follow these steps:

1.     Create a new rule for the infrastructure NFS subnet in the default export policy.

2.     Assign the FlexPod export policy to the infrastructure SVM root volume.

Add HANA SVM Management Interface and Administrator

To add the HANA SVM administrator and SVM administration LIF in the out-of-band management network, follow these steps:

1.     Run the following commands:

2.     Create a default route to allow the SVM management interface to reach the outside world.

3.     Set a password for the SVM vsadmin user and unlock the user.

Create Export Policies for the HANA SVM

To create an export policy for the HANA SVM, follow these steps:

1.     Create a new export policy for the HANA data and log subnet.

2.     Create a rule for this policy.

Create NFS LIF for SAP HANA Data

To create the NFS LIFs for SAP HANA data, run the following commands:

Create NFS LIF for SAP HANA Log

To create an NFS LIF for SAP HANA log, run the following commands:

This section details the preparation of HANA nodes based on SLES 15 and RHEL 7.6.

SAP HANA Node OS Preparation – RHEL for SAP HANA 7.6

This section details the RHEL 7.6 installation and configuration.

OS Installation

To install the OS, follow these steps:

This section provides the procedure for Red Hat Enterprise Linux 7.6 Operating System and customizing for SAP HANA requirement.

To install the RHEL 7.6 system, follow these steps:

1.     Prepare the iSCSI LUN like described in the Storage part of this CVD for the OS.

2.     In Cisco UCS Manager, click the Servers tab in the navigation pane.

3.     Select Service Profiles > root > HANA-Server01.

4.     Click KVM Console.

5.     When the KVM Console is launched, click Boot Server.

6.     If you using CD click Virtual Media > Activate Virtual Devices.

7.     Select Accept this Session for Unencrypted Virtual Media Session then click Apply.

8.     Click Virtual Media and Choose Map CD/DVD.

9.     Click Browse to navigate ISO media location.

10.  Click Map Device.

11.  Normally a reboot is necessary to activate this virtual drive.

12.  During the reboot the iSCSI targets must be shown. If not check the iSCSI configuration.

13.  After the POST the system will boot from the RHEL ISO.

14.  At the prompt of the Installation options

15.  Press the Tab key to alter the command line options. Append parameter rd.iscsi.ibft=1 to the kernel command line as shown below:

This is added to make sure the iSCSI targets are discovered appropriately.

16.  Choose Keyboard and configure your layout.

17.  Configure the appropriate Timezone and Time.

18.  Click the ‘Security Policy’ to set the security policy to OFF.

 

19.  Leave the Software section selections as default (Minimal Installation).

20.  Disable KDUMP.

 

21.  Click Installation destination.

The next screen lists all the local standard disks, if any.

22.  Click Specialized and Network Disks section – “Add a Disk”

23.  Select the discovered iSCSI boot LUN disk and click “Done.”

24.  Click Done. Select “I will configure partitioning’. Click Done.

25.  Select Standard Partition and then select “Click here to create them automatically.”

 

26.  Confirm the default Partition table. Click Done. Accept Changes.

 

27.  Click Begin Installation and then setup the root password.

28.  Installation completes. Click Reboot.

Post Installation Tasks

Configure the Network

In RHEL 7, system and udev support a number of different naming schemes. By default, fixed names are assigned based on firmware, topology, and location information: for example, enp72s0.

With this naming convention, although names remain fixed even if hardware is added or removed, names often are more difficult to read than with traditional kernel-native ethX naming: that is, eth0, etc. 

You could change to traditional device names by setting these Kernel command line parameters net.ifnames=0 biosdevname=0. At this time, you can disable IPv6 support with ipv6.disable=1.

1.     Log into the newly installed system as root.

2.     Add the following to the line starting with GRUB_CMDLINE_LINUX in /etc/default/grub:

3.     Rebuild the /boot/grub2/grub.cfg file.

 

4.     Reboot for effect the disabling of consistent network device naming. Log back in and check the interfaces.

5.     Configure the admin/management network, hostname and default GW. Identify the Admin/Mgmt vNIC using the MAC address to eth interface mapping at the OS level with “ip address” command [as above] and on the Network tab of service profile under the vNICs.

6.     Create the configuration file for the interface as below. Default Route and Gateway setting are done at the interface level.

7.     Update the hostname with hostnamectl command

8.     Restart the Network.

Update the Red Hat System

Refer to the Red Hat KB article https://access.redhat.com/solutions/65300 to prepare the system to access the Red Hat Subscription Manager through proxy.

To register the system to Red Hat portal and attach SAP HANA subscription, follow the instructions in the Red Hat KB articles:

https://access.redhat.com/solutions/2318061 and https://access.redhat.com/solutions/2334521

To update the Red Hat System, follow these steps:

1.     Set the release to the right minor release and clear the yum cache and subscribe to the right channels.

2.     Check for the available repositories.

3.     Install the base package group:

4.     Install other additional required packages required for running SAP HANA on RHEL 7:

5.     Disable SELinux:

6.     Disabling the firewall:

7.     Edit the file /etc/ntp.conf to reflect the appropriate ntp servers for the region and start the ntp service:

8.     Starting with Red Hat Enterprise Linux 7.2 the kernel parameters must be set in a configuration file in the /etc/sysctl.d directory. For example, a configuration file with the name 91-NetApp-HANA.conf must be created.

9.     Disable ABRT and Core dumps:

10.  Disable core file creation. To disable core dumps for all users, open /etc/security/limits.conf, and add the lines:

11.  Increase the maximum number of processes a user is allowed to run. For the same, create the file /etc/security/limits.d/99-sapsys.conf with the following content

12.  Add symbolic links - Since SAP HANA is built on a different Linux Distribution, some of the library names used during the build process doesn't match with the library names used on RHEL7.

13.  Reboot the system to effect kernel switch and SELinux settings.

Implement SAP Notes Recommendations

To configure optimal settings for running HANA or HANA2 on RHEL for SAP HANA 7.6, follow the instructions in the SAP Note 2292690.

To implement the SAP notes recommendations, follow these steps:

1.     The tuned profile "sap-hana", which is provided by Red Hat as part of RHEL 7 for SAP HANA, contains many of the settings mentioned in the SAP note and also configures some additional settings. Therefore the "sap-hana" tuned profile must be activated on all systems running SAP HANA.

a.     Install tuned-profiles-sap-hana

b.     Start and enable the tuned:

c.     Apply the profile for sap-hana

d.     Verify the solution applied

Install Cisco VIC Drivers

To download the Cisco UCS Drivers ISO bundle, which contains most Cisco UCS Virtual Interface Card drivers, follow these steps:

1.     In a web browser, navigate to http:/​/​www.cisco.com.

2.     Under Support, click All Downloads.

3.     In the product selector, click Products, then click Server - Unified Computing.

4.     If prompted, enter your Cisco.com username and password to log in.

5.     Cisco UCS drivers are available for both Cisco UCS B-Series Blade Server Software and Cisco UCS C-Series Rack-Mount UCS-Managed Server Software.

6.     Click UCS B-Series Blade Server Software.

7.     Click Cisco Unified Computing System (UCS) Drivers. 

8.     Click 4.0(4d) Version.

9.     Download ISO image of Cisco UCS-related drivers ucs-bxxx-drivers-linux.4.0.4. 

10.  Choose your download method and follow the prompts to complete your driver download.

11.  Browse the downloaded drivers iso -> ucs-bxxx-drivers-linux.4.0.4\Network\Cisco\VIC\RHEL\RHEL7.6 and copy  kmod-enic-3.2.210.18-738.12.rhel7u6.x86_64.rpm to  /opt of the HANA node.

12.  Update the enic driver:

13.  Browse the downloaded drivers iso -> ucs-bxxx-drivers-linux.4.0.4\ Storage\Cisco\VIC\RHEL\RHEL7.6 and copy  kmod-fnic-2.0.0.42-77.0.rhel7u6.x86_64.rpm to  also /opt of the HANA node.

14.  Update the enic driver:

SAP HANA Node OS Preparation – SLES for SAP 15

In this section, SLES for SAP 15 installation and configuration is detailed.

OS Installation

To install the OS, follow these steps:

1.     Refer the 2578899 - SUSE Linux Enterprise Server 15: Installation Note for installation instructions.

2.     On the UCSM page, Servers -> Service Profiles -> root -> Sub-Organizations -> HANA – Right-click HANA-node01 and select KVM console.

3.     After the KVM console is launched, click Boot Server.

4.     Choose Virtual Media > Activate Virtual Devices.

a.     For Unencrypted Virtual Media Session, select Accept this Session and then click Apply.

b.     Click Virtual Media and choose Map CD/DVD.

c.     Click Browse to navigate to the ISO media location. Select SLE-15-Installer-DVD-x86_64-GM-DVD1.iso. Click Open.

d.     Click Map Device.

5.     At server boot time, during the check the of VIC FC boot driver version, it recognizes the NetApp iSCSI target via four paths; two each per VIC. This verifies the server to storage connectivity.

6.     The System will automatically boot from the ISO image. Scroll down on the selection menu to Installation option and key in the boot options- rd.iscsi.ibft=1 (Turn on iBFT autoconfiguration for the interfaces), rd.iscsi.firmware=1 (to read the iscsi parameter from the BIOS firmware) and rd.neednet=1 (bring up network even without netroot set). Click Return.

7.     On the first “Language, Keyboard and License Agreement” page, select the Language of choice and Keyboard Layout. Select “SUSE Linux Enterprise Server for SAP Applications 15” for Product to Install and click Next.

8.     SUSE Linux Enterprise Server for SAP Applications 15 License Agreement – Select “I Agree to the License Terms” . Click Next.

9.     Disk Activation – Click Configure iSCSI Disks.

10.  On the iSCSI Initiator tab, ensure the automatically picked up Initiator Name is correct. [If not update the Initiator Name defined in the service profile.]

11.  Click the Connected Targets tab, verify the interface can login to the target via all 4 ports.

 

12.  If not by default, click Add perform iSCSI Initiator Discovery. Key in the IP address of the remaining iSCSI_LIF Ip Address. Click Next.

13.  The remaining paths are discovered and added. Click Connect selecting newly added paths one at a time [with startup value False]

14.  Select Continue for the Warning. Set Startup to “automatic.”

15.  Do the same for remaining path.

16.  Have all paths discovered and startup set to value True.

17.  Click Next

18.  Click Next on Disk Activation page.

19.  System Probing – Select No for the prompt to activate multipath. We will activate the multipath post installation.

20.  Registration – Select Skip Registration. We will register the system post installation. Click OK for the Warming.

21.  Add On Product: Select “I would like to install an additional Add On Product. And select DVD as the source. lick Next.

 

22.  At this stage, unmap the currently mapped Installer DVD1 drive. Map the SLE-15-Packages-x86_64-GM-DVD1.iso  Click Next.

23.  Extension and Module Selection -  Select as appropriate. Click Next.

24.  Click Next on the Add-On Product Installation page.

25.  System Role – Keep the default selection of SLES for SAP Applications.

26.   Choose Operating System Edition – Unselect both options. Click Next.

27.  Suggested Partitioning – Click Expert Partitioner and Select Start with Existing Partitions.

28.  Expert Partitioner – Select one out of the four iSCSi NetApp devices listed. Click Add Partition.

29.  Create a 100 MiB size BIOS_Boot volume.

30.  Select Raw Volume (unformatted) for the Role.

31.  Select BIOS Boot Partition under Partition ID and leave the rest with default selections.

32.  Add New Partition with Maximum Size, using up the rest of the space available on drive. Click Next.

33.  Select Operating System Role and click Next.

34.  Select Ext3 for Filesystem. Click Next.

35.  Click Accept.  Click Yes for the warning, to continue. We will create the swap file later.

36.  Suggested partitioning – Click Next.

37.  Clock and Time Zone – choose the appropriate time zone and select Hardware clock set to UTC.

38.  Password for the System Administrator “root” – Key in appropriate password <<var_sys_root-pw>>. Click Next.

39.  On the Installation Settings screen, review the default information.

40.  Click Install and select Install again for the subsequent ‘Confirm Installation’ prompt. The installation is started and you can monitor the status.

41.  When prompted during the installation process, remap installation DVD1.

42.  After the installation is complete, a reboot alert appears. The system will reboot and boot from disk on startup. Login using the root.

Post Installation Steps

Configure the Network

As part of the post-install configuration, you will define the hostname, network configuration, kernel/packages update, and tuning as per the SAP Notes recommendations.

To configure the network, follow these steps:

1.     Configure the hostname and disable IPV6.

 

2.     System > Network Settings and select Run > Alt+s to select Hostname/DNS tab.

3.     Input the <<var_hostname.domain>>. Also, key in DNS server address of your network for resolution, if any and select Alt+o.

4.     On the Global Options tab with Alt+g, you can choose to disable IPV6, by unselecting the Enable IPV6 option as shown in the figure below. Changing the IPV6 setting requires a reboot to effect the changes.

5.     Select Alt+o to save the Network Configuration. Select Alt+q to quit the YaST Control center.

6.     Perform a reboot to effect the IPV6 selection and also the hostname settings.

7.     Host networking configuration:

a.     The vNIC to MAC address mapping information for a host can be obtained from the network tab of that host’s Service Profile.

b.     At the host OS level, the Ethernet interface to MAC address mapping can be ascertained with the ‘ip address’ command.

c.     Co-relating the outputs in step a and b above, we are able to determine the right IP address/network that need to should be assigned to the Ethernet interface. For the same an IP addressing scheme cheat sheet, as below, can be quite handy.

d.     Assign the IP address and subnet mask for the ethernet interfaces based on the al the information we have so far.

e.     Add the default gateway.

f.      Update the /etc/hosts with IP address of all networks and their alias hostnames:

g.     Create SWAP partition.

h.     Update the /etc/fstab with swap filesystem information by appending this line.

8.     Set up a proxy service, so that the appliance can reach the Internet.

a.     YaST2 – Key in the proxy server and port details. Select OK and then quit YaST to save the configuration.

b.     Select “Enable Proxy” > key in the <<proxy server IP address:port >> information and select “use same proxy for all Protocols” option.

c.     Test the Proxy Settings to make they are working.

Update the SLES System

To update the SLES system, follow these steps:

1.     Register the system with SUSE to get the latest patches. The system must have access to the Internet to proceed with this step.

2.     Update the system with the following command. Again, the system must have access to the Internet to proceed with this step.

3.     Follow the on-screen instructions to complete the update process. Reboot the server and log in to the system again.

Set Kernel Parameters

Starting with SUSE Linux Enterprise Server12 SP1 the kernel parameter must be set in a configuration file in the /etc/sysctl.d directory. For example, a configuration file with the name 91-NetApp-HANA.conf must be created.

Install Cisco VIC Drivers

To install the Cisco VIC drivers, follow these steps:

1.     Update fnic and enic drivers:

a.     Based on the serer type/model, processor version, OS release and version information download the Firmware bundle corresponding to the UCS Server firmware installed from the Cisco UCS Hardware and Software Compatibility site

b.     Extract the rpm files of the fnic and enic drivers from the bundle over to the node.

2.     Multipath configuration:

We disabled multipath during install time. It is recommended to install OS using a single path referencing the iSCSi device and enable the multipath configuration post OS installation and zypper update.

a.     At first check with multipath – ll may not return any result.

b.     Enable and start the multipath daemon and verify it active status.

c.     Check with iscsiadm commands to check for iscsi logins. If some paths are missing, discover the target and make sure the initiators are logged in to all ports.

d.     Restart multipathd and Enable and start the multipath daemon and verify it active status.

e.     Create a Dracut configuration file exclusively for multipath.

f.      Rebuild the initrd ad reboot the system.

Implement SAP Notes recommendations

To optimize the HANA DB with SLES for SAP 15, follow the instructions in the SAP Note 2684254:

1.     SAP Note 1275776 describes how to apply recommended operating system settings for running SAP applications on SLES. There are three ways to implement the same – sapconf, sapture or manually. I tis important to note When using sapconf or saptune, verify that parameters handled by these tools  are not configured elsewhere (e.g. boot parameter, sysctl.conf, and so on). This can cause inconsistent system behavior and makes debugging very hard.

2.     This CVD uses the saptune (version 2) method which can can prepare the operating system for SAP applications based on implementing specific SAP Notes

3.     Install saptune:

 

4.     Configuration – activate saptune

 

5.     All available solutions and notes can be listed with:

 

6.     Apply SAP HANA solution/notes:

7.     Verify the solution applied:

 

This chapter describes the sequence of steps required to provision nodes for SAP HANA installation starting with the storage volume configuration, OS configuration needed to mount the storage volumes and subsequent use-case specific preparation tasks. The undelaying infrastructure configuration has already been defined in the earlier sections of this document.

The configuration steps are identical for SAP HANA running on bare metal servers and on VMware virtual machines.

Table 12  shows the required variables used in this section.

Table 12      Required Variables

 

Each SAP HANA host, either bare metal or VMware virtual machine, has two network interfaces connected to the storage network. One network interface is used to mount the log volumes, and the second interface is used to mount the data volumes for SAP HANA. The data and log volumes of the SAP HANA systems must be distributed to the storage nodes, as shown in Figure 7, so that a maximum of six data and six log volumes are stored on a single storage node.

The limitation of having six SAP HANA hosts per storage node is only valid for production SAP HANA systems for which the storage-performance key performance indicators defined by SAP must be fulfilled. For nonproduction SAP HANA systems, the maximum number is higher and must be determined during the sizing process.

Configuring SAP HANA Single-Host Systems

Figure 8 illustrates the volume configuration of four single-host SAP HANA systems. The data and log volumes of each SAP HANA system are distributed to different storage controllers. For example, volume SID1_data_mnt00001 is configured on controller A, and volume SID1_log_mnt00001 is configured on controller B.

 

Configure a data volume, a log volume, and a volume for /hana/shared for each SAP HANA host. Table 13   lists an example configuration for single-host SAP HANA systems.

Table 13      Volume Configuration for SAP HANA Single-Host Systems

Table 14  lists an example of the mount point configuration for a single-host system. To place the home directory of the sidadm user on the central storage, you should mount the /usr/sap/SID file system from the SID_shared volume.

Table 14      Mount Points for Single-Host Systems

Configuration Example for a SAP HANA Single-Host System

The following examples show a SAP HANA database with SID=NF2 and a server RAM size of 1TB. For different server RAM sizes, the required volume sizes are different.

For a detailed description of the capacity requirements for SAP HANA, see the SAP HANA Storage Requirements white paper.

Figure 9 shows the volumes that must be created on the storage nodes and the network paths used.

Create Data Volume and Adjust Volume Options

To create data volume and adjust the volume options, run the following commands:

Create a Log Volume and Adjust the Volume Options

To create a log volume and adjust the volume options, run the following commands:

Create a HANA Shared Volume and adjust the Volume Options

To create a HANA shared volume and qtrees, and adjust the volume options, run the following commands:

Create Directories for HANA Shared Volume

To create the required directories for the hana shared volume mount the shared volume temporally and create the required directories

Update the Load-Sharing Mirror Relation

To update the load-sharing mirror relation, run the following command:

Create Mount Points on the node

To create the required mount-point directories, take one of the following actions:

Synchronize Domain information

To be able to mount the volumes inside the HANA nodes, the v4-id-domain information of NFS enabled hana-svm providing the HANA persistence access should tally with domain information in /etc/idmapd.conf of the HANA nodes.

On the NetApp command line fetch the information about the configured v4 doamin as shown below:

Make sure the same is updated in the ‘Domain’ filed on /etc/idmapd.conf file of HANA node.

Mount File Systems

The mount options are identical for all file systems that are mounted to the host:

·       /hana/data/NF2/mnt00001

·       /hana/log/NF2/mnt00001

·       /hana/shared

·       /usr/sap/NF2

Table 15  lists the required mount options.

This example uses NFSv4.1 to connect the storage. However, NFSv3 is supported for SAP HANA single host systems.

For NFSv3, NFS locking must be switched off to avoid NFS lock cleanup operations in case of a software or server failure.

With NetApp^® ONTAP^® 9, the NFS transfer size can be configured up to 1MB. Specifically, with connections to the storage system over 10GbE, you must set the transfer size to 1MB to achieve the expected throughput values.

Table 15      Mount Options

To mount the file systems during system boot using the /etc/fstab configuration file, follow these steps:

1.     Add the file systems to the /etc/fstab configuration file.

2.     Run mount –a to mount the file systems on the host.

Persistent Memory Configuration

For detailed information, see: Cisco UCS for SAP HANA with Intel Optane DC Persistent Memory Module white paper

This section addresses the use-case of Single-host system implementation on a node configured with Intel Persistent Memory and extends configuration scope to include the same.

The utility ipmctl is used for configuring and managing Intel Optane DC persistent memory modules (DCPMM) and the ndctl utility library is required for managing the libnvdimm (non-volatile memory device) sub-system in the Linux kernel.

To configure the persistent memory, follow these steps:

1.     ssh to the Server as root.

2.     Install the ipmctl host utility

3.     Install the ndctl utility library

4.     Confirm the persistent memory modules are discovered in the system and verify the software can communicate with them.

5.     Create the goal

6.     Reboot the server for the new memory allocations.

7.     Verify the regions created

8.     Create a name space for each region; on a server with a total of 4 CPU invoke the command four times.

ndctl create-namespace

9.     List the active name spaces created before

10.  Create a file system and mount the persistent memory modules

 

11.  Add the mount points to the /etc/fstab file to make them permanent

Configure the Base Path to Use Persistent Memory

Post the SAP HANA Single-host system install the directory that SAP HANA uses as its base path must point to the XFS file system. Define the base path location with the configuration parameter basepath_persistent_memory_volumes in the [persistence] section of the SAP HANA global.ini file. This section can contain multiple locations separated by semicolons.

Changes to this parameter require a restart of SAP HANA services.

Configure SAP HANA Multiple-Host Systems

The figure below shows the volume configuration of a 4+1 SAP HANA system. The data and log volumes of each SAP HANA host are distributed to different storage controllers. For example, volume SID1_data1_mnt00001 is configured on controller A, and volume SID1_log1_mnt00001 is configured on controller B.

 

For each SAP HANA host, a data volume and a log volume are created. /hana/shared is used by all hosts of the SAP HANA system. Table 16  shows an example configuration for a multiple-host SAP HANA system with four active hosts.

Table 16      Volume Configuration for SAP HANA Multiple-Host Systems

 

Table 17  lists the configuration and mount points of a multiple-host system with four active SAP HANA hosts. To place the home directories of the sidadm user of each host on the central storage, the /usr/sap/SID file systems are mounted from the SID_shared volume.

Table 17      Mount Points for Multiple-Host Systems

Configuration Example for a SAP HANA Multiple-Host Systems

The following examples show a 4+1 SAP HANA multiple-host database with SID=NF3 and a server with a RAM size of 2TB. For different server RAM sizes, the required volume sizes are different.

For a detailed description of the capacity requirements for SAP HANA, see the SAP HANA Storage Requirements white paper.

The figure below shows the volumes that must be created on the storage nodes and the network paths used.

Create Data Volumes and Adjust Volume Options

To create data volumes and adjust the volume options, run the following commands:

Create Log Volume and Adjust Volume Options

To create a log volume and adjust the volume options, run the following commands:

Create HANA Shared Volume and Adjust Volume Options

To create a HANA shared volume and adjust the volume options, run the following commands:

Create Directories of HANA Shared Volume

To create the required directories for the HANA shared volume, mount the shared volume temporally and create the required directories

Update Load-Sharing Mirror Relation

To update the load-sharing mirror relation, run the following command:

Create Mount Points

For a multiple-host system, create mount points and set the permissions on all worker and standby hosts as follows.

1.     Create mount points for the first worker host.

2.     Create mount points for the second worker host.

3.     Create mount points for the third  worker host.

4.     Create mount points for the fourth worker host.

5.     Create mount points for the standby host.

Synchronize Domain information

To be able to mount the volumes inside the HANA nodes, the v4-id-domain information of NFS enabled hana-svm providing the HANA persistence access should tally with domain information in /etc/idmapd.conf of the HANA nodes.

On the NetApp command line fetch the information about the configured v4 doamin as below:

Make sure the same is updated in the ‘Domain’ filed on /etc/idmapd.conf file of HANA node.

Mount File Systems

The mount options are identical for all file systems that are mounted to the hosts:

·       /hana/data/NF3/mnt00001

·       /hana/data/NF3/mnt00002

·       /hana/data/NF3/mnt00003

·       /hana/data/NF3/mnt00004

·       /hana/log/NF3/mnt00001

·       /hana/log/NF3/mnt00002

·       /hana/log/NF3/mnt00003

·       /hana/log/NF3/mnt00004

·       /hana/shared

·       /usr/sap/NF3

Table 18  lists the required mount options.

For NFSv3, you must switch off NFS locking to enable failover capabilities in multiple-host installations. Also, NFS locking must be switched off in single-host setups to avoid NFS lock cleanup operations in case of a software or server failure.

With the ONTAP 9, the NFS transfer size can be configured up to 1MB. Specifically, with 40GbE connections to the storage system, you must set the transfer size to 1MB to achieve the expected throughput values.

Table 18      Mount Options

The following examples show a SAP HANA database with SID=NF3 using NFSv4.1 and an NFS transfer size of 1MB. To mount the file systems during system boot using the /etc/fstab configuration file, follow these steps:

1.     For a multiple-host system, add the required file systems to the /etc/fstab configuration file on all hosts.

2.     Run mount –a on each host to mount the file systems.

3.     For scale-out system, all nodes should be able to resolve Internal network IP address. Below is an example of 4 node scale-out system host file with all the network defined in the /etc/hosts file:

Passwordless Authentication

The SSH Keys must be exchanged between all nodes in a SAP HANA Scale-Out system for user ‘root’ and user <SID>adm.

1.     Generate the rsa public key by executing the command ssh-keygen -b 2048

 

2.     Exchange the rsa public key by executing the below command from First server to rest of the servers in the scale-out system.

“ssh-copy-id -i /root/.ssh/id_rsa.pub fphana02”

3.     Repeat steps 1-2 for all the servers in the scale-out system so that each node has the keys copied over from all other nodes.

A sample global.ini file from the installed multi-host SAP HANA system is as below:

For information about the SAP HANA installation, please use the official SAP documentation, which describes the installation process with and without the SAP unified installer.

SAP HANA Server Installation Guide

All other SAP installation and administration documentation is available here: http://service.sap.com/instguides

Important SAP Notes

Read the following SAP Notes before you start the installation. These SAP Notes contain the latest information about the installation, as well as corrections to the installation documentation.

The latest SAP Notes can be found here: https://service.sap.com/notes.

SAP HANA IMDB Related Notes

SAP Note 1514967   - SAP HANA: Central Note

SAP Note 1523337   - SAP HANA Database: Central Note

SAP Note 2000003   - FAQ: SAP HANA

SAP Note 1730999 - Configuration changes in SAP HANA appliance

SAP Note 1514966 - SAP HANA 1.0: Sizing SAP In-Memory Database

SAP Note 1780950 - Connection problems due to host name resolution

SAP Note 1743225 - SAP HANA: Potential failure of connections with scale out nodes

SAP Note 1755396 - Released DT solutions for SAP HANA with disk replication

SAP Note 1890444 - HANA system slow due to CPU power save mode

SAP Note 1681092   - Support for multiple SAP HANA databases on a single SAP HANA appliance

SAP Note 1514966   - SAP HANA: Sizing SAP HANA Database

SAP Note 1637145   - SAP BW on HANA: Sizing SAP HANA Database

SAP Note 1793345   - Sizing for Suite on HANA

Linux Related Notes

SAP Note 2235581   - SAP HANA: Supported Operating Systems

2578899 - SUSE Linux Enterprise Server 15: Installation Note

SAP Note 2009879   - SAP HANA Guidelines for Red Hat Enterprise Linux (RHEL)

SAP Note 2292690 - SAP HANA DB: Recommended OS settings for RHEL 7

SAP Note 1731000 - Non-recommended configuration changes

SAP Note 2382421 - Optimizing the Network Configuration on HANA- and OS-Level

SAP Note 1557506 - Linux paging improvements

SAP Note 1740136 - SAP HANA: wrong mount option may lead to corrupt persistency

SAP Note 1829651 - Time zone settings in SAP HANA scale out landscapes

SAP Application Related Notes

SAP Note 1658845 - SAP HANA DB hardware check

SAP Note 1637145 - SAP BW on SAP HANA: Sizing SAP In-Memory Database

SAP Note 1661202 - Support for multiple applications on SAP HANA

SAP Note 1681092 - Support for multiple SAP HANA databases one HANA aka Multi SID

SAP Note 1577128 - Supported clients for SAP HANA 1.0

SAP Note 1808450 - Homogenous system landscape for on BW-HANA

SAP Note 1976729   - Application Component Hierarchy for SAP HANA

SAP Note 1927949   - Standard Behavior for SAP Logon Tickets

SAP Note 1577128   - Supported clients for SAP HANA

SAP Note 2186744   - FAQ: SAP HANA Parameters

SAP Note 2267798   - Configuration of the SAP HANA Database during Installation Using hdbparam

SAP Note 2156526   - Parameter constraint validation on section indices does not work correctly with hdbparam

SAP Note 2399079   - Elimination of hdbparam in HANA 2

Third Party Software

SAP Note 1730928 - Using external software in a SAP HANA appliance

SAP Note 1730929 - Using external tools in an SAP HANA appliance

SAP Note 1730930 - Using antivirus software in an SAP HANA appliance

SAP Note 1730932 - Using backup tools with Backint for SAP HANA

NetApp Technical reports

TR-4435-SAP HANA on NetApp AFF Systems with NFS

TR-3580-NFSv4 Enhancements and Best Practices

TR-4614-SAP HANA Backup and Recovery with SnapCenter

TR-4646-SAP HANA Disaster Recovery with Asynchronous Storage Replication

High-Availability Configuration for Multiple-host

Since you are using NFSv4.1, a specific HA configuration for multiple hosts SAP HANA databases is not necessary.

NFSv4 locking ensures that only one hosts can access the SAP HANA data and log files at a time.

SAP HANA Installation Preparations for NFSv4

NFS version 4 and higher requires user authentication. This authentication can be accomplished by using a central user management tool such as an LDAP server or local user accounts. The following sections describe how to configure local user accounts.

The administration user <sidadm> and the sapsys group must be created manually on the SAP HANA hosts and the storage controllers before the installation of SAP HANA software begins.

SAP HANA Hosts

If the sapsyy group doesn’t exist, you must create it on the SAP HANA host. You must choose a unique group ID that does not conflict with the existing group IDs on the storage controllers.

Create the user <sidadm> on the SAP HANA host. You must choose a unique ID that does not conflict with existing user IDs on the storage controllers.

For a multiple-host SAP HANA system, the user and group ID must be the same on all SAP HANA hosts. The group and user are created on the other SAP HANA hosts by copying the affected lines in /etc/group and /etc/passwd from the source system to all other SAP HANA hosts.

Storage Controllers

The user ID and group ID must be the same on the SAP HANA hosts and the storage controllers. To create the group and user, run the following commands on the storage cluster:

SAP HANA Data Volume Size

A default SAP HANA instance uses only one data volume per SAP HANA service. Due to the max file size limitation of the file system, NetApp recommends limiting max data volume size.

To do so automatically set the following parameter in the global.ini within section [persistence]:

This created a new data volume when the limit of 8000GB is reached.

SAP Note 240005 question 15 provides more information: https://launchpad.support.sap.com/#/notes/2400005

Introduction

AppDynamics is an Application Performance Monitoring (APM) Platform that helps you to understand and optimize the performance of your business, from its software to infrastructure to business journeys.

The AppDynamics APM Platform enables you to monitor and manage your entire application-delivery ecosystem, from the mobile app or browser client request through your network, backend databases and application servers and more. AppDynamics APM gives you a single view across your application landscape, letting you quickly navigate from the global perspective of your distributed application right down to the call graphs or exception reports generated on individual hosts.

AppDynamics has an agent-based architecture.  Once our agents are installed it gives you a dynamic flow map or topography of your application. It uses the concept of traffic lights to indicate the health of your application (green is good, yellow is slow and red indicates potential issues) with dynamic baselining.  AppDynamics measures application performance based on business transactions which essentially are the key functionality of the application. When the application deviates from the baseline AppDynamics captures and provides deeper diagnostic information to help be more proactive in troubleshooting and reduce the MTTR (mean time to resolution).

SAP Landscape Monitoring

AppDynamics has a one of its kind ABAP agent for monitoring SAP ABAP systems. We have comprehensive coverage of the SAP landscape with our ABAP, Java, .net and Server visibility agents. In addition, Datavard Insights extends the AppDynamics for SAP solution with system-level monitoring for the overall SAP systems and SAP HANA databases. While AppDynamics agents provides transaction-level visibility, Datavard Insights collects performance metrics, logs and events, including processes outside of the user business transactions, such as background jobs or IDocs processing.

The complex and proprietary nature of SAP applications makes it difficult to diagnose issues. AppDynamics allows enterprises to instrument SAP applications, monitor performance, and understand the root cause of performance bottlenecks.

 

Trial Registration

To register, follow these steps:

1.     Connect to https://www.appdynamics.com/free-trial/

2.     Provide the details to sign up for a free trial utilizing an AppDynamics SaaS controller.

3.     Once the AppDynamics SaaS Controller has been provisioned, you will receive an email with the information you need for you to login to the Controller.

4.     You can download the Java Agent and the Server Visibility Agent directly from the Controller

 

5.     You can use the email and password you provided to sign up for the trial to login to the agent download site at the URL listed below and download the ABAP Agent:

https://download.appdynamics.com

Agent Installation

AppDynamics has several types of agents to monitor different language applications to user Experience to Infrastructure monitoring. Based on the SAP landscape and the underlying technology of the SAP systems the agents are installed.

The most frequently installed agents are:

1.     Java Agent - For Java based SAP Systems

2.     ABAP Agent - For ABAP based SAP systems

3.     Server Visibility Agent - Provides extended hardware metrics and Service Availability Monitoring

Prerequisites

Please see the link below to verify the supported SAP environments:

https://docs.appdynamics.com/display/SAP/SAP+Supported+Environments

Java Agent Installation

The Java Agent must be installed on SAP Java application servers (e.g. Enterprise Portal and PO application servers).

The high-level steps for installing the Java Agent are listed below:

1.     Ensure you are logged into the host server as the appropriate <SID>adm OS user

2.     Create the permanent installation directory for the agent

3.     Unzip the file in the permanent directory (i.e. /usr/sap/appdyn/app)

4.     Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels

5.     Edit the configuration file in the agent installation directory to configure the agent to connect to the controller, provide the identity of the JVM, and so on.

6.     You will need to add parameters to the SAP JVM to start the Java agent when the SAP system is started up by logging into the SAP app server as the “sidadm” user

7.     Use the SAP NetWeaver Administrator or the AS Java Config Tool (depending on your SAP system) to edit the JVM startup parameters.  For more detailed information, see the link below:

Configuring AppDynamics Java Agent in SAP

8.     Restart the SAP JVM for the settings to take effect

9.     Validate the Java Agent is reporting to the controller by logging into the controller UI

For detailed information, see the link below:

Install the AppDynamics Java Agent

ABAP Agent Installation

The ABAP Agent needs to be installed on the SAP servers utilizing the ABAP stack.

There are four primary steps to perform, each with secondary steps involved.  The four primary steps are:

1.     Copy and unzip the ABAP Agent

2.     Import the ABAP Agent Transports

3.     Configure ABAP Agent and Install HTTP SDK

4.     Activate Datavard Insight Collectors

Copy and Unzip ABAP Agent

The high-level steps to copy and unzip the ABAP Agent are listed below:

1.     Ensure you are logged into the host server as the appropriate <SID>adm OS user

2.     Copy the agent binary to a temporary directory on the server

3.     Unzip the file into a temporary directory

4.     Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels

Import the ABAP Agent Transports

Install on SAP NetWeaver Systems

The high-level steps to import the ABAP agent transports are listed below:

1.     The ABAP Agents data files and cofiles should be copied from the temporary directories where they were unzipped over to the appropriate transport directories for the SAP module in question.

-      For example, for ECC we would copy the transports to "/usr/sap/trans/ECC2/cofiles" and "/usr/sap/trans/ECC2/data", respectively

2.     Set the permissions on the cofiles and data files to allow read/write access at the owner and group levels.

3.     Log into the SAP system, execute transaction STMS:

-      Go to the import queue of the system where you want to install the ABAP agent

-      Select "Extras > Other Requests > Add" from the menu bar and add the vendor transports to the import queue one at a time

4.     Import the transports in the appropriate order

-      The import order of the transports is specified in the “readme.txt” file of the ABAP Agent subdirectory that is specific to the version of NetWeaver in use

For more information please see the link to the related documentation below:

Install on SAP NetWeaver Systems

-      Make sure that when selecting the "Execution" tab in the "Import Transport Request" pop-up dialog box to select the option "Synchronous". When selecting the "Options" tab, put a checkmark next to "Ignore Invalid Component Version".

Configure ABAP Agent / Install HTTP SDK

Supported SAP Operating Systems

Installing HTTP SDK Automatically

The high-level steps to configure the ABAP agent and install the HTTP SDK are listed below:

1.     Log into the SAP system and execute transaction "/DVD/APPD_CUST".

2.     Switch to edit mode.

3.     Fill in the fields on the screen to configure the agent to connect to the controller, SDK settings, and Analytics API settings..

4.     Click the "Activate integration" button on the toolbar.

5.     Click the "SDK Installation" button on the toolbar. This will take you to the "AppDynamics HTTP SDK Installation Manager" screen.

6.     Select "Edit > Change Directory" from the menu bar.

a.     Enter the path that was used for the agents' permanent base install directory (i.e. /usr/sap/appdyn) in the field displayed in the pop-up dialog box shown below, and then click OK.

b.     Click the "Install SDK" button on the toolbar.

c.     Click the green checkmark to exit the screen and return to the "AppDynamics settings" screen.

d.     Click the "Status" button on the toolbar. This will take you to the "AppDynamics status check" screen.

e.     Click each "Start" button to start the HTTP SDK proxy on each SAP app server.

Activate Datavard Insight Collectors

Datavard Insights collect detailed performance data for an SAP system. It uses collector jobs that run as periodic background processes in the SAP system. These jobs must be scheduled to run.

Please refer to the links to the related documentation below:

Datavard Insights Integration

Performance Collector Settings

SAP Dashboards

Mapping Between AppDynamics Metrics and Datavard Insights KPIs

Server Visibility Agent Installation

The Server Visibility Agent must be installed on every application server and central services server that will be monitored.

The high-level steps for installing the Server Visibility agent are listed below:

1.     Ensure you are logged into the host server as the appropriate <SID>adm OS user.

2.     Create the permanent installation directory for the agent.

3.     Unzip the file in the permanent directory (i.e. /usr/sap/appdyn/machine).

4.     Change the permissions on the contents of the agents' installation directory to give full read/write privileges at the owner and group levels.

5.     Edit the configuration files in the agent installation directory to configure the agent to connect to the controller, provide the identity of the host, and so on.

6.     Start the server visibility agent with the script provided in the agents’ bin directory.

7.     Validate the server visibility is reporting to the controller by logging into the controller UI.

Pramod Ramamurthy, Technical Marketing Engineer, Cisco Systems, Inc.

Pramod is a Technical Marketing Engineer with Cisco UCS Solutions and Performance Group. Pramod has more than 15 years of experience in the IT industry focusing on SAP technologies. Pramod is currently focusing on the Converged Infrastructure Solutions design, validation and associated collaterals build for SAP HANA.

Marco Schoen, Technical Marketing Engineer, NetApp, Inc.

Marco is a Technical Marketing Engineer with NetApp and has over 20 years of experience in the IT industry focusing on SAP technologies. His specialization areas include SAP NetWeaver Basis technology and SAP HANA. He is currently focusing on the SAP HANA infrastructure design, validation and certification on NetApp Storage solutions and products including various server technologies.

Acknowledgements

For their support and contribution to the design, validation, and creation of this Cisco Validated Design, we would like to thank:

·       Shailendra Mruthunjaya, Cisco Systems Inc.

·       Sarath Gonuganta, Cisco Systems Inc.

NOTE: Available paragraph styles are listed in the Quick Styles Gallery in the Styles group on the Home tab. Alternatively, they can be accessed via the Styles window (press Alt + Ctrl + Shift + S).