SAP HANA Central

First things first.

Due to the fact that starting with the wave 2019.01 SAP Analytics Cloud (SAC) has stopped accepting the self-signed SSL certificates for HTTPS INA live connections I have ended up by replacing the self-signed HANA Express SSL certificate(s) with the equivalent CA-signed SSL certificate(s).

Synopsis:

In a nutshell, the SAP Analytics tools (the likes of SAC, Analysis for Office, BW4H, BOE with Lumira 2.x, Analysis for OLAP or WebIntelligence) rely heavily on the secured HTTPS INA protocol to consume the SAP HANA calculation views on-the-fly via the SAP HANA EPM-MDS engine.

This is being referred to as live/on-line/remote connectivity, where the data is always current, as opposed to acquired/off-line/import option where the data needs to be acquired first and then refreshed in a scheduled manner.

I have invested quite some time and effort in order to build a viable testbed with the focus on the front-end SAP analytical applications that rely on the secured HTTP INA protocol based connectivity [either with or without SSO to the backend HANA database] and that can consume the HANA HDI views (e.i. the views which are no longer tied into the default _SYS_BIC schema).

As you may know the SAP HANA Express 2.x ready-to-deploy images implement several self-signed SSL certificates for the secured domains of HTTPS INA (XS classic engine running on the tenant database), XSA (XS Advanced) and LCM access over the web.

As aforementioned, having recently come across the necessity to implement the trusted SSL certificates signed with a trusted CA (Certificate Authority) rather than relying on self-signed ones, I have decided to share my experience in “Taming your SAP HANA Express” series of blog posts.

Let me be clear, I have been very happy with the HXE SAPCAL image deployed on AWS; It takes 45 minutes to have this image deployed and running.

It is a great HANA appliance! Regards to the entire SAPCAL team!

For the sake of convenience and transparency most of the URLs in this article are not obfuscated. They refer to a stock HANA Express instance deployed on AWS and do not reveal any trade secrets.

Let’s begin:

First I got the HTTPS INA protocol up and running fine on the tenant database where both the index server and webdispatcher are running; (This is very important as with the multi-tenanted databases there is neither index server nor webdispatcher available to run on the system database)

This setup used to work against SAP Analytics Cloud waves 18 through 22 out of the box (aka with the self-signed SSL certificates in Chrome browser);

So what…?

1. The initial problem with the HANA Express self-signed certificates.

Initially one particular problem I faced was that a number (but not all) of front-end applications require the CN certificate name to match the hostname in the HTTPS service URL;

When a mismatch between the two names is detected these applications deny any further access.

In our case the CN name is sid-hxe and in the webdispatcher HTTPS URL service name is vhcalhxedb; Obviously sid-hxe != vhcalhxedb !

Eventually, after a thought, I narrowed down the problem to the following piece of advise:

Can someone advise the steps required to replace the system certificate (sid-hxe) with the default.root.crt.pem certicificate (vhcalhxedb) in the webdispatcher’s SAPSSLS.pse store ?

Let me explain it in more details.

1a.The below links to the HANA XS classic engine work perfectly fine on my testbed. 4390 is the SSL port number of the webdispatcher running on the HXE tenant database;

https://vhcalhxedb:4390/

https://vhcalhxedb:4390/sap/bc/ina/service/v2/GetServerInfo

The OS host name is sid-hxe;

The HANA host name is vhcalhxedb;

Both names resolve to the appliance IP address;

I have noticed the HXE webdispacther URL is using the OS system host sid-hxe SSL certificate as depicted below.

You may notice that the URL uses vhcalhxedb HXE appliance name [which is resolved into the elastic IP address in my laptop’s /etc/hosts file].

Still the webdispatcher/XS classic engine are configured to use the system certificate which shows the OS host name, namely sid-hxe:

SAP HANA Tutorial and Material, SAP HANA Certification, SAP HANA Guides, SAP HANA Study Materials

1b.Moreover, if I try to replace the appliance’s name vhcalhxedb with the hostname sid-hxe I get the 503 error as depicted below (of course it goes without saying sid-hxe has been added to the /etc/hosts file on my Mac so it resolves to the same IP address as vhcalhxedb):

in other words the name of the HANA host (vhcalhxedb) must not be replaced in the URL;

2. Further on all the XSA applications pre-installed during the appliance deployment or that I have installed myself on either SYSTEMDB or HXE tenant database use a different self-signed domain SSL certificate, namely vhcalhxedb

This vhcalhxedb certificate is in the default.root.crt.pem certicificate file found here: <installation_path>/HXE/xs/controller_data/controller/ssl-pub/router

3. I can access the webdispatcher admin console (on the HXE tenant database)

https://vhcalhxedb:4390/sap/hana/xs/wdisp/admin/public/default.html

(You may need to grant specific HANA user privileges in order to get access to the webdispatcher admin console and the user needs to be defined on the tenant database)

The PSE of interest is the SAPSSLS.pse which is configured with the system certificate (CN=sid-hxe);

What I did is that I have imported the aforementioned default.root.crt.pem certificate file into the SAPSSLS.pse store (as shown below).

What I did not know is what to do to replace the Own Certificate (CN=sid-hxe) with the default.root.crt.pem certicificate (CN=vhcalhxedb). Or with any other viable certificate ?

At the same time I do not want to damage my testbed. So again I narrowed down the problem to the following piece of advise:

Can someone advise the steps to replace the system certificate with the default.root.crt.pem certificate in the webdispatcher’s SAPSSLS.pse store ? Does it make any sense at all ?

But let me walk you through all the steps one by one.

2. SSL system certificate with HXE tenant database web dispatcher and XS classic

Long story short: Despite the above certificate/URL names mismatch I was able to use HXE SAPCAL image with SAP Analytics Cloud for some quite time…

Until recently the discrepancy of the HXE Webdispatcher CN name in the SLL certificate and the hostname in the Webispatcher URL was still manageable ; For whatever reason SAC including the the SAC wave 2018.22 disregarded the CN name mismatch and that the certificate was self-signed (not trusted).

However with the latest release of SAP Analytics Cloud (wave 2019.01+) it is no longer case as SAC requires the SSL certificate be both valid and trusted….

In order to find a manageable solution and a way out of this “cul-de-sac” I broke it down into three questions:

1. What needs to be done to get the Webdispatcher SSL certificate right?

2. Could I replace the self-signed sid-hxe certificate with the vcalhxedb certificate used elsewhere in the system ? I understand that might solve the name mismatch but would not necessarily fix the certificate trust.

3. Last but not least which CA authority could I be using with the SAPCAL images to generate trusted SSL certificates in order to replace the self-signed ones ?

The answer to question 3 would deliver both valid (name match) and trusted (CA-signed) certificates.

Let me walk you through the required steps as follows:

ad 1.What needs to be done to get the Webdispatcher SSL certificate right?

To fix the non secure, self-signed HTTPA INA URL ?

https://vhcalhxedb:4390/sap/bc/ina/service/v2/GetServerInfo

ad 2. Could I replace the self-signed sid-hxe system certificate with the XSA domain vcalhxedb certificate used elsewhere in the system ?

Here goes the OS host (sid-hxe) self-signed SSL certificate which is used by the webdispatcher admin webapp and more generally to secure access to the XS classic domain resources:

and here goes the self-signed vhcalhxedb certificate used to secure the access to the XSA domain:

ad 3. Last but not least which CA authority could I be using with the SAPCAL images

to generate trusted SSL certificates in order to replace the self-signed ones ?

If you are interested in more details about the HANA PSEs the following SAP note 2009483 – PSE Management in Web Administration Interface of SAP Web Dispatcher describes the PSE management techniques

Alternatively the below SQL statement allows to retrieve the HANA webdispatcher profile(s)

SELECT KEY, VALUE, LAYER_NAME

FROM SYS.M_INIFILE_CONTENTS

WHERE FILE_NAME = ‘webdispatcher.ini’

AND SECTION = ‘profile’ AND KEY LIKE ‘wdisp/system%’

which yields the following result:

GENERATED, SID=HXE, NAME=HXE, EXTSRV=localhost:39008, SRCVHOST=vhcalhxedb

The FQDN of the webispatcher FQDN is set to be vhcalhxedb.sap.corp; but this could be any name like ateam.sap.corp etc…

Let’s make sure the FQDN can be resolved into the IP address. The below excerpt shows the edited OS hosts system file:

vi /etc/hosts

# IP-Address Full-Qualified-Hostname Short-Hostname

#127.0.0.1 localhost sid-hxe.dummy.nodomain sid-hxe

127.0.0.1 localhost

10.0.0.11 sid-hxe.dummy.nodomain sid-hxe

10.0.0.11 vhcalhxedb.sap.corp vhcalhxedb

Additionally, in order to prevent the SAPCAL image to change the hostname

sid-hxe:~ # cat /etc/hostname

sid-hxe

Prevent the appliance change the hostname at boot time

sid-hxe:~ # vi /etc/init.d/updatehosts

sid-hxe:~ #

sid-hxe:~ # cat /etc/init.d/updatehosts

if [ “$1” == “start” ]; then

# Commented out the below line to prevent the hostname changes

# /sbin/updatehosts.sh

else

echo “$0 $1 does not do anything”

exit $?

sid-hxe:~ #

3. The hardening of the XS classic domain:

Following the piece of advice from the wiki the CA-signed trusted certificate has been implemented into the PSE SAPSSLS.pse as follows. (You will notice we still use the vhcalhxedb name in the service URL):

I recreated the PSE, created the CSR

and then imported the CA response (full chain of certificates)

The end result is as follows:

Still the dispatcher parameters reveal the appliance host name as vhcalhxedb and not the FQDN vhvalhxedb.sap.corp ? What has gone wrong ?

Initially I thought that I would have to rename the HANA appliance host, namely rename vhcalhxedb into the FQDNvhvalhxedb.sap.corp .

I do reckon, eventually, this might have been the best way of getting the FQDN work in the URL once for all. And, BTW, there is an excellent post that explains how to rename the HANA host. However, I did not resort to it;

Instead I used the HANA cockpit and searched for all the occurrences of vhcalhxedb in the HANA configuration files through the XSA HANA cockpit at https://vhcalhxedb.sap.corp:51045/sap/hana/cockpit/landscape/index.html

[I have to confess I have already renamed the XSA engine’s domain from vhcalhxedb into vhcalhxedb.sap.corp initially to prove that it can be done, but more importantly that XSA is world apart from classic XS and that the HANA appliance hostname (=vhcalhxedb) is also separate from either XSA and XS engines or the OS host name]

I have found that the public_urls of the index engine (xsengine.ini) were of the form http://vhcalhxedb:8090 and https://vhcalhxedb:4390 so I edited them as follows

Then I restarted the HANA DB appliance.

4. Hooray!!!

In the aftermath the FQDN URL to HTTPS INA GetServerInfo can reveal both trusted (CA Signed) and valid (CN==URL hostname) certificate for the XS classic domain:

and that the short-name URL does not work any more:

Equally the FQDN URL to the webdispatcher admin console reveals the valid and trusted certificate

and that the short-name URL does not work any more:

From this moment on I was able to start using the FQDN, namely vhcalhxedb.sap.corp in the SAC connection definition, in the BOE HANA trust definition etc….

I would like to show how to read data from Microsoft Office 365 using Microsoft Graph API and NodeJS and then write this data into SAP HANA XSA tables. Our scenario requirement was to pull data from Excel files on Sharepoint on Microsoft Office 365 and write this data to SAP HANA XSA container tables. We wanted to use the SAP HANA File Adapter which has a section for Sharepoint Configuration. However, the Excel adapter as of HANA 2.0 SPS 3 can only connect to Sharepoint 2013 on premise systems and not Sharepoint on the Microsoft Office 365 Cloud. So we had to come up with an approach to address this requirement and develop it. This Blog describes the approach which will hopefully help other folks needing to accomplish this type of scenarios for integrating Microsoft Office 365 data with SAP HANA XSA . There will be another Blog that describes how to trigger our custom developed NodeJS application from a virtual procedure through the file adapter through SAP HANA Smart Data Integration (SDI) layer.

Here is a screenshot of the ExcelAdapter with the Sharepoint URL we are trying to connect with and the error that is returned stating that the connection to the Sharepoint URL does not work:

SAP HANA XSA, SAP HANA Study Materials, SAP HANA Guides, SAP HANA Tutorial and Material

First thing first – we need to connect to Sharepoint on Microsoft Office 365. Microsoft provides the Graph API which allows users to invoke APIs across most of Office 365 functionality which is very impressive including the Sharepoint integration capabilities. Check out the Microsoft Graph API with the Microsoft Graph Explorer: Microsoft Graph Explorer

Once you get familiar with the API and see how simple, powerful and flexible it is, you will see that you can call the Graph API and pass the Sharepoint site, workbook, worksheet and cell ranges and get the data back in JSON format very easily. Very cool! For the Graph API, we need our Sharepoint site, GUID for the drive, workbook, worksheet and the range for the worksheet:

In my example, I have:

https://graph.microsoft.com/v1.0/sites//sites/mycompany.sharepoint.com,5304406e-30bd-4b4e-8bd0-704c8a2e6eaa,ba9d641d-14c9-41f8-a9d0-e8c6a2cda00e/drives/b%21bkAEU70wTkuL0HBMii5uqh1knbrJFPhBqdDoxqLNoA4HcPFP9eqPTIUQCSbiDtgZ/items/01HX4WN4RSH2GZKQXRWNEKR2S5YZLIUHOS/workbook/worksheets(%27{00000000-0001-0000-0000-000000000000}%27)/Range(address=%27Sheet1%21A1:C50%27)

Now in order to call this API, we need to register this application on the Azure portal.

Click on Add an app:

Give the application name

Get the application id and secret – we will need these for authentication to invoke the interface:

In our case, we have named this sharepoint_excel_outlook_graph_api_hana_integration – the name could have been shorter but we wanted to have it be self descriptive enough since there were many other applications for the company.

We need to set the permissions for the app to allow reading the sharepoint data:

In order for the service to be called with an OAuth token with just the application id and secret, we need to apply for admin consent by the Azure admin which will allow permissions for the application to be triggered in background mode.

So once we have the necessary setup done to read the data from Sharepoint, we will write the application using NodeJS.

Here is the main app.js module that allows the application to be invoked from the command line that uses our custom modules hdbutility.js and sharepointutility.js:

///////////////////////////////////////////////////////////////////////////////////

// app.js - main module to read the file from Sharepoint Office 365 and then save into HANA

// Author - Jay Malla @Licensed To Code

///////////////////////////////////////////////////////////////////////////////////

var hdbutility = require('./hdbutility');

var sharepointutility = require('./sharepointutility');

// Read all of the main configurable parameters

var config = require('config');

///////////////////////////////////////////////////////////////////////////////////

// Here is our main....

///////////////////////////////////////////////////////////////////////////////////

var sharepointExcelURL;

var schema;

var table;

var sqlArrayResults;

console.log("Let's start this journey");

//Command line usage (Note that the exact sequence is important)

//node.exe app.js -f sharepointurl -s schema -t table

// Let's extract the variables sharepointurl, schema, table from the command line

process.argv.forEach(function (value, index, array) {

console.log(index + ': ' + value);

if (array.length < 8) {

console.error("Not enough parameters supplied");

throw (new Error("Not enough parameters supplied"));

}

switch (index) {

case 3:

console.log('FileName' + ': ' + value);

sharepointExcelURL = value;

sharepointExcelURL.replace('\\', '');

break;

case 5:

console.log('Schema' + ': ' + value);

schema = value;

break;

case 7:

console.log('Schema' + ': ' + value);

table = value;

break;

}

});

//If not supplied through command line, then read from the config file

//if (!schema) {config.get('schema')};

//if (!table) {config.get('table')};

//if (!sharepointExcelURL) {config.get('sharepointExcelURL')};

var hdbConnectionDetails = config.get('hdbConnectionDetails');

var oauth_info = config.get('oauth_info');

mainlogic();

///////////////////////////////////////////////////////////////////////////////////

// mainlogic is the main function that runs the main logic

///////////////////////////////////////////////////////////////////////////////////

async function mainlogic() {

try {

// Set the credentials from the configuration module which has read the default.json

const credentials = {

client: {

id: oauth_info.client_id,

secret: oauth_info.client_secret

auth: {

tokenHost: oauth_info.tokenHost,

authorizePath: oauth_info.authorizePath,

tokenPath: oauth_info.tokenPath

options: {

bodyFormat: 'form',

authorizationMethod: 'body'

}

};

////////////////////////////////////////////////////////////////

// Use Sharepoint Utility to get Excel

var sharepointclient = sharepointutility.createClient(credentials);

sharepointclient.getExcelFileFromSharepoint(sharepointExcelURL, schema, table, oauth_info.scope)

// If Excel file is retrieved

.then(result => {

console.log(result);

console.log("Excel File retrieved as array of SQL statements");

sqlArrayResults = result;

////////////////////////////////////////////////////////////////

// Save to HANA Database

var hdbclient = hdbutility.createClient(hdbConnectionDetails);

hdbclient.setSchema(schema);

hdbclient.setTable(table);

hdbclient.insertIntoHANA_ReturningPromise(sqlArrayResults)

.then(result => {

console.log(result);

console.log("Data uploaded to SAP HANA Table");

})

.catch(error => {

console.error(error);

console.log("Could not upload the data to SAP HANA table. Please fix issues and try again. Check config file and input parameters.");

});

})

.catch(error => {

console.error(error);

console.log("Could not read the Excel file from Sharepoint");

});

} catch (err) {

console.log(err);

}

Here is the NodeJS code that uses the Microsoft Graph Client on NodeJS to connect to Sharepoint and then call the Graph API to read in the contents. From the contents, the code then creates an array of SQL statements that is later used to insert the data into the SAP HANA tables:

///////////////////////////////////////////////////////////////////////////////////

// sharepointutility.js - sharepoint module to integrate with Sharepoint

// Author - Jay Malla @Licensed To Code

////////////////////////////////////////////////////////////////////////////////////

var graph = require('@microsoft/microsoft-graph-client');

// Class sharepointutility - object constructor function

function sharepointutility(credentials) {

// Set the credentials Info

this.credentials = credentials;

// We need to store a reference to this - since we will need this later on

self = this;

///////////////////////////////////////////////////////////////////////////////////

// This method async function connects to Sharepoint

this.getExcelFileFromSharepoint = async function getExcelFileFromSharepoint(sharepointExcelURL, schema, table, inputscope) {

return new Promise(async function (resolve, reject) {

self.sharepointExcelURL = sharepointExcelURL;

self.schema = schema;

self.table = table;

self.inputscope = inputscope;

const oauth2 = require('simple-oauth2').create(credentials);

var accessToken;

const tokenConfig = {

scope: inputscope // also can be an array of multiple scopes, ex. ['<scope1>, '<scope2>', '...']

};

// Get the access token object for the client

try {

const result = await oauth2.clientCredentials.getToken(tokenConfig);

accessToken = result.access_token;

} catch (error) {

console.log('Access Token error', error.message);

reject(error);

return;

}

// Initialize Graph client

const client = graph.Client.init({

authProvider: (done) => {

done(null, accessToken);

}

});

////////////////////////////////////////////////////////////////

// Get the Sharepoint Excel file

var sharepointurl = sharepointExcelURL;

const result = await client

.api(sharepointurl)

.get();

////////////////////////////////////////////////////////////////

// Let's do an HTTP post for the same:

var request = require('request');

var bearer = "Bearer " + accessToken;

var httpurl = "https://graph.microsoft.com/v1.0";

// Set the headers

var headers = {

'Content-Type': 'application/json',

'Authorization': bearer

}

// Configure the request

var options = {

url: httpurl + sharepointurl,

method: 'GET',

headers: headers

}

// Start the request

request(options, function (error, response, body) {

if (!error && response.statusCode == 200) {

// Print out the response body

console.log(body);

var sqlArrayResults;

sqlArrayResults = self.generateSQLarrayFromResults(result.formulas);

resolve(sqlArrayResults);

return;

}

})

////////////////////////////////////////////////////////////////

// Convert the Excel file results to an array of SQL

var sqlArrayResults;

sqlArrayResults = self.generateSQLarrayFromResults(result.formulas);

resolve(sqlArrayResults);

return;

////////////////////////////////////////////////////////////////

});

};

this.generateSQLarrayFromResults = function generateSQLarrayFromResults(sharepointTable) {

var columnsString;

var sqlArray = [];

sharepointTable.forEach((element, index) => {

//Assumption - the first row has the table headings

if (index == 0) {

var processArray = (array) => {

var sqlString = "(";

array.forEach((element, index) => {

console.log(element);

if (index < (array.length - 1)) {

sqlString = sqlString + element + ",";

} else {

sqlString = sqlString + element + ")";

}

});

return sqlString;

}

columnsString = processArray(element);

} else {

if (element[0] != '') { //As long as there are other entries

var valuesArrayString;

var tempString = "insert into \"" + this.schema + "\".\"" + this.table + "\"" + columnsString + " values "; // + element[0] + "," + "'" + element[1] + "'" + "," + "'" + element[2] + "'" + ")";

var processValuesArray = (array) => {

var sqlString = "(";

array.forEach((element, index) => {

console.log(element);

if (index < (array.length - 1)) {

if (typeof (element) == "number") {

sqlString = sqlString + element + ",";

} else {

sqlString = sqlString + "'" + element + "'" + ",";

}

} else {

if (typeof (element) == "number") {

sqlString = sqlString + element + ")";

} else {

sqlString = sqlString + "'" + element + "'" + ")";

}

});

return sqlString;

}

var valuesArrayString;

valuesArrayString = processValuesArray(element);

tempString = tempString + valuesArrayString;

console.log(tempString);

sqlArray.push(tempString);

}

});

return sqlArray;

}

///////////////////////////////////////////////////////////////////////////////////

exports.createClient = function (credentials) {

return new sharepointutility(credentials);

}

///////////////////////////////////////////////////////////////////////////////////

Here is the code that connects to the HANA database and inserts the data into the tables – note that the schema and table names are parameterized:

///////////////////////////////////////////////////////////////////////////////////

// hdbutility.js - database module save into HANA

// Author - Jay Malla @Licensed To Code

////////////////////////////////////////////////////////////////////////////////////

// Class hdbutility - object constructor function

function hdbutility(hdbConnectionInfo) {

// Set the hdbConnection Info

this.hdbConnectionInfo = hdbConnectionInfo;

//property method to set schema name

this.setSchema = function (schema) {

this.schema = schema;

};

//property method to set the table name

this.setTable = function (table) {

this.table = table;

};

// We need to store a reference to this - since we will need this later on

self = this;

//////////////////////////////////////////////////////////////////////////////////

This method async function runs the SQL Array of statements in the HANA database - but order is not gauranteed

this.insertIntoHANA_ReturningPromise = async function insertIntoHANA_ReturningPromise(sqlArray) {

return new Promise(function (resolve, reject) {

var inputSQLArray = sqlArray;

var results = [];

var hdb = require('hdb');

var hdbclient = hdb.createClient(self.hdbConnectionInfo);

hdbclient.on('error', function (err) {

reject(err);

return;

});

hdbclient.connect(function (err) {

if (err) {

reject(err);

return;

}

// First delete the entries from the table

var strQuery = 'delete from \"' + self.schema + '\".\"' + self.table + '\"';

hdbclient.exec(strQuery, function (err, rows) {

//hdbclient.end();

if (err) {

reject(err);

return;

}

console.log('Table Contents before SQL Inserts:', JSON.stringify(rows));

/////////////////////////////////////////////////////////////////////////

// Recursive approach to go through Array and execute SQL statements

var iterateOverArray = (index) => {

// if the end of Array reached..

if (index == inputSQLArray.length) {

// Read all of the the entries in that table and log this to see if all records inserted...

strQuery = 'select * from \"' + self.schema + '\".\"' + self.table + '\"';

hdbclient.exec(strQuery, function (err, rows) {

hdbclient.end();

if (err) {

reject(err);

return;

}

console.log('Table Contents After SQL Inserts:', JSON.stringify(rows));

resolve(JSON.stringify(rows));

return;

});

} else {

// If the end of the Array has not been reached....

// Execute the insert into the table

hdbclient.exec(inputSQLArray[index], (err, rows) => {

//hdbclient.end();

if (err) {

console.error('Execute error:', err);

//return callback(err);

reject(err);

return;

}

//otherwise capture the results and move to the next array member for the iteration

console.log('Results executing SQL ' + inputSQLArray[index] + ' = ' + JSON.stringify(rows));

results.push('Results executing SQL ' + inputSQLArray[index] + ' = ' + JSON.stringify(rows));

iterateOverArray(index + 1);

});

}

/////////////////////////////////////////////////////////////////////////

//Calling the recursive function...

iterateOverArray(0); // Initiate the recursive function that iterates through the array and executes the SQL

/////////////////////////////////////////////////////////////////////////

});

}

///////////////////////////////////////////////////////////////////////////////////

exports.createClient = function (hdbConnectionInfo) {

return new hdbutility(hdbConnectionInfo);

}

///////////////////////////////////////////////////////////////////////////////////

So once we have the NodeJS program that we can invoke from the command line to read the data from Sharepoint and write the data into HANA, we need HANA to trigger this on demand dynamically. So here is where HANA has this nice feature of a Virtual procedure using the file adapter that allows us to call our NodeJS program from the command line with dynamic parameters. These details will be in another Blog to follow which will be available very soon.

This article uses as To perform SLT based replication in hana with ecc system

Login to D1H and execute LTRC T-Code

SAP HANA Tutorial and Materials, SAP HANA Guides, SAP HANA Certification, SAP HANA Study Materials

2) Choose the respective System , in which replication needs to be done. Here we are doing in first (SR1)

3) Find the table lists whose count is greater than million records. Search that list of tables first to find out which is already present.

4) Verify tables are available for replication in HANA

Only 2 tables are in HANA

Out of 6 tables-2 tables are already replicated in HANA SIDE CAR, So we need to replicate 4 tables

We need to replicate only the below 4 tables

NAST

COBRB

COBRA

S115

Come out of LTRC T-Code and then again execute LTRC T-code

Click on Data Provisioning

Click on Start Replication

Click on execute button

Click on No.Of Tables and click on Descending

We will get the list of tables in scheduled state

We could see Replication is in stopped status. We need to start the replication in order the tables to replicate.

Once the replication is started, you can see the status Replication (Initial Load)

Once the Replication is completed, you will get the Current Action as “Replication”

Now, after replication. We have to create the scenario. Here the scneraio name is Z-FUSIONOPS2.

Before , Go to SE16 and give table name as RDA_CONFIG and give here scenario name. You will get that there is no entries in the table list.

Now, you have to create the scenario. Go to SE38 and give Program Name as RDA_MAINTAIN

This program is used to upload scnerio

You have to create the XML file for those listed tables. XML files creation reference take it from local path where you have saved

Give the path where xml file is located

Click on execute button, a message will come that Scenario Z_FUSIONOPS02 was added.

Click on Maintain Database Connection

Choose the scenario name. You will see the new scenario name added in the list.

Now Database Connection –SLT_SR1

Here now the database connection for Z_FUSIONOPS02 is assigned to SR1.

Now we have to activate the scenario

Click on execute and activate the scenario

Now to RDA_CONFIG table and give the scenario name and click on number of entries

Here you will get the number of entries

Suppose 6 tables then double of that i.e 12 because we have 2 programs using the same set of tables

Click on execute and you will see the list of tables entries

SAP’s column-oriented, in-memory database, that combines OLAP and OLTP operations into a single system, or in other words, SAP HANA originated from a research started in 2006 by SAP’s co-founder Hasso Plattner while he was a computer science professor at the Hasso Platner Institute in Postdam, Germany.

After several years of development at SAP, a prerelease version of SAP HANA was given to selected customers in October 2010. SAP released the first official version, SAP HANA 1.0, on June 18, 2011. After a few years of development and major architectural changes, SAP HANA 2.0 was released in 2017, with a variety of updated and new features. Below you can check a timeline with the major SAP HANA events:

SAP HANA Certification, SAP HANA Guides, SAP HANA Tutorial and Materials, SAP HANA Study Materials

SAP HANA is an enterprise platform designed to handle a huge amount of data and a large number of applications in support of mission-critical business. By platform, we mean that it is not only a database anymore. Application developers can build applications and data scientists can run advanced analytics and machine learning directly on SAP HANA, and database administrators can integrate their landscape within the SAP HANA database management tools. SAP HANA provides features for multi tenancy, high availability, disaster recovery, and dynamic tiering to help ensure 24×7 access and smooth horizontal and vertical scaling. SAP HANA also lets you access a wide variety of data sources and support Big Data and the Internet of Things.

Since its release, SAP HANA has had multiple versions and editions. The objective of this blog post is to clarify what is the most suited licensing option for you to leverage the full potential of SAP’s highest profile product – SAP HANA. The following options are going to be covered:

◈ SAP HANA Runtime Edition

◈ SAP HANA Full Use Edition

◈ SAP HANA Active/Active (Read Enabled)

◈ SAP HANA Express Edition

◈ SAP Cloud Platform, SAP HANA Service

SAP HANA Runtime Edition

SAP HANA Runtime Edition is limited to a runtime environment for SAP Applications. An SAP Application is any application that includes a NetWeaver Application Server, e.g. SAP Business Warehouse (BW), SAP BW/4HANA, SAP Business Suite, SAP S/4HANA and other related products.

SAP offers two options with SAP HANA Runtime Edition:

◈ SAP HANA Runtime Edition for SAP BW (HANA REB)

◈ SAP HANA Runtime Edition for Apps and SAP BW (HANA REAB), which includes all capabilities from REB, plus other SAP Apps.

The picture below shows the available features of both for these options.

The options marked with * have some restrictions regarding data processing.

SAP HANA Runtime Edition is considered an integral part of the application itself, so that any kind of data processing (modeling, administration, creation of data structures and use of advanced analytics) must be done by this application, via the application layer (where the interface can be either from SAP or a 3rd party provider). With SAP HANA Runtime Edition, the data from the application can only be displayed or graphically processed at the front-end application and any direct connection to the database bypassing the application is a license violation.

The picture below shows not only the available SAP HANA capabilities, but also the restricted (Δ) and not included (X) ones when SAP HANA Runtime Edition is licensed.

SAP HANA Full Use Edition

SAP HANA Full Use Edition offers an unrestricted platform for any combination of SAP, non-SAP, custom, third-party, and hybrid applications.

With SAP HANA Full Use Edition there are no limitations on data modeling, administration, creation of custom structures, and use of advanced engines via HANA Web IDE, HANA Studio (IDEs used to develop artifacts on a HANA server) or other applications. There are no limitations on loading and exporting of SAP & non-SAP data directly into and out of SAP HANA.

With the Full Use Edition it is possible to read data both from the application and database layers. You can consume the data of the SAP HANA system whichever way you need it.

SAP offers two versions of the SAP HANA Full Use Edition:

◈ SAP HANA Standard Edition

◈ SAP HANA Enterprise Edition (includes all capabilities from the Standard Edition). The picture below shows the available features for both editions.

It is also possible to acquire the Standard Edition and just add the services you need for your business. Here are the available options:

◈ Data Privacy Option, for enhanced protection of sensitive and confidential data (e.g. Data Masking);

◈ Information Management Option, for data integration, data quality management, and information stewardship;

◈ Predictive Option, for Predictive Analytics Library access, R engine, and TensorFlow integration for advanced analytics;

◈ Replication Option, for replicating data from any supported source system to the SAP HANA database;

◈ Search/Text Option, for search, text analysis, and text-mining functionality for you to gain real insights from unstructured textual data;

◈ Spatial/Graph Option, for advanced spatial and graph analytics capabilities;

◈ Streaming Analytics Option, for processing streams of events and messages in real time, allowing some or all of the data to be captured in the SAP HANA database and/or Hadoop.

Basically, if you license all available options on top of your SAP HANA Standard Edition, your SAP HANA system will have the same capabilities as the SAP HANA Enterprise Edition.

SAP HANA Enterprise Edition includes the full range of functions and features from SAP HANA, being the foundation of SAP’s data management solutions and the Intelligent Enterprise.

SAP HANA Full Use allows you to use widespread skills (such as JavaScript, Python, SQL, ABAP). ABAP developers have unrestricted code push-down possibilities.

The HANA platform allows for a development with no data transfers, no data duplication and no data latency, because the processing is done within the database itself; and the powerful technical capabilities of SAP HANA can be utilized. With this, we can have faster time to value and we can have lower development and maintenance costs.

SAP HANA Runtime Edition vs Full Use Edition

Conceptually speaking SAP HANA Runtime Edition offers a limited platform, centralized only for SAP applications and SAP HANA Full Use Edition offers an unrestricted platform for all systems and distributed data in modern, hybrid environments.

The picture below provides an overview of the biggest technical differences between these two editions. For SAP HANA Runtime Edition we’ve included restricted (Δ) and not available (X) capabilities.

In terms of functions and features, the picture below compares the difference between the two available editions.

To sum up, the decision of choosing which edition is best for you depends on whether you want to leverage SAP Applications, or you want to use the full power of the SAP HANA platform.

If you are just looking for a faster database that supports your SAP applications SAP HANA Runtime Edition is the right choice, but if you are looking for a long-term and strategic solution, you need SAP HANA Full Use Edition. Making the right choice here will depend on your company’s needs. Even if you license SAP HANA Runtime Edition based on your current requirements, but your long-term roadmap calls for SAP HANA Full Use Edition, SAP supports this transition both commercially and technically.

SAP HANA Active/Active (Read Enabled)

SAP HANA Active/Active (Read Enabled) is a solution that targets resiliency and better performance of your SAP HANA productive environment. With this option you can offload intensive read operations to the secondary system, freeing up more computational power to be used on the primary system for read and write operations.

As data is continuously being replicated from the primary to the secondary system using either synchronous or asynchronous communication protocols, you can leverage the secondary system to offset read-intensive workloads from the primary system. With SAP HANA Active/Active (Read Enabled) you can gain usage rights to access secondary system for productive read-enabled operation, the utilization of the underlying hardware can be increased because the workloads can be much more balanced, which also improves the performance of the operations on the primary system. Latency can also be reduced as read intensive users can now be located next to the secondary data center and there is no impact on your Recovery Time Objective (RTO) and Recovery Point Objective (RPO).

Without SAP HANA Active/Active (Read Enabled), the secondary system can’t be used for any operation, which means it’s only idle capacity for High Availability and Disaster Recovery purposes.

This option can either be used by SAP HANA Runtime and Full Use Editions as you can check below. Only SAP HANA 2.0 works with this option.

SAP HANA Express Edition

All the different versions of SAP HANA that we have introduced are enterprise-ready, meant to be run in productive environments. What if you are a programmer who wants to run SAP HANA on a local PC, or on a cloud service offered by a Hyperscaler (AWS, GCP, Azure, …) ?

SAP HANA, express edition is the best way to test out and get hands on with SAP HANA, being available to install on your local laptop or desktop computer, a local server, or in the cloud. This means you can now use the express edition of SAP HANA on devices other than existing, SAP-certified, hardware appliances, or access it from the SAP Cloud Appliance Library tool on either Amazon Web Services, Microsoft Azure or Google Cloud Platform.

This edition allows you to jump-start application development on top of SAP HANA today and move to other editions of SAP HANA (i.e. SAP HANA Runtime or Full Use Edition) as your needs to grow tomorrow. Developers and independent software vendors (ISVs) can start using the express edition of SAP HANA at no cost to build and deploy applications that use up to 32 GB of memory. You can expand the use of your existing SAP HANA Express Edition for a fee, paying incrementally for up to 128 GB of memory use.

To support application development on your personal computer, where data size and scalability are limited, we have streamlined the SAP HANA platform to fit within the constraints of PCs. Because of this, the following features are not available with the express edition of SAP HANA:

◈ Data warehousing foundation

◈ Disaster recovery

◈ Dynamic tiering

◈ High availability

◈ Multihosting

◈ Outward scaling for multiple hosts

◈ Remote data synchronization

◈ SAP Solution Manager

◈ Smart data integration

◈ Smart data quality

◈ Smart data streaming

◈ System replication

SAP Cloud Platform, SAP HANA Service

The SAP HANA service (officially called SAP Cloud Platform, SAP HANA service) allows you to leverage the in-memory data processing capabilities of SAP HANA in the cloud. As a managed database service, backups are fully automated and service availability guaranteed. Using the SAP HANA service, you can set up and manage SAP HANA databases and bind them to applications running on SAP Cloud Platform. You can access SAP HANA databases using a variety of languages and interfaces, as well as build applications and models using tools provided with SAP HANA.

There are multiple cloud deployment options available for SAP HANA service: it can be deployed in the Cloud Foundry environment or in the Neo Environment from SAP Cloud Platform. The Cloud Foundry environment is based on Amazon Web Services, Google Cloud Platform or Microsoft Azure while the Neo environment is hosted by SAP.

SAP HANA service is available in two editions: Standard and Enterprise (similar to the editions available for SAP HANA Full Use). Standard Edition only includes core database services with in-memory capabilities as well as smart data access. Enterprise Edition includes all SAP HANA Service Standard Edition capabilities including advanced analytics options and smart data integration.

This blog is for enthusiastic people who want to learn to create HDI container type Database using SAP WEB IDE and deploy the same in Cloud Foundry and can use SAP WEB IDE to perform DML/DDL operations on DB.

We will go through step by step processes to achieve the same.

Pre-requisites required:-

Accounts and infrastructure:-

1. SAP Cloud platform Cloud Foundry account.
2. SAP Web Ide Full stack service enabled.

Skills:-

1. Basic knowledge of Cloud Foundry and SAP Cloud Platform.
2. Basic knowledge of SQL.

Let’s start:

Login into Cockpit and navigate to your trial account and then to services and then open SAP WEB IDE full stack.

SAP WEB IDE, SAP HANA Study Materials, SAP HANA Guides, SAP HANA Certifications

Click on SAP Web IDE Full -Stack grid and then “Go to Service” link.

It will navigate to SAP web ide in that select my workbench which will navigate you to editor page where you start creating a project as below.

Create an MTAR project from the project template

Enter Basic details

It will create a project folder with the mta.yaml file.

Create a DB module inside the project. as shown below.

We are trying to create a DB for storing employee and their role details.

Enter basic details.

This will create a folder named Employee, Now create a new file named cdsAtrifcat, under src as mentioned below.

namespace FirstFullStackApp.Employee;

context cdsArtifact {

/*@@layout{"layoutInfo":{"x":32,"y":121.5}}*/

entity employeeDetails {

key EMPLOYEE_ID : String(10) not null;

DESCRIPTION : String(50) not null;

DEPARTMENT : String(20);

EMPLOYEE_NAME : String(50) not null;

association : association[1, 1..*] to cdsArtifact.roleEnrollments { EMPLOYEE_ID };

};

/*@@layout{"layoutInfo":{"x":-444,"y":105.5}}*/

entity roleEnrollments {

key EMPLOYEE_ID : String(20) not null;

key ROLE_ID : String(20) not null;

ROLE_NAME : String(20) not null;

EMPLOYEE_NAME : String(50) not null;

EMAIL : String(40) not null;

LOCATION : String(20);

};

Create a new file named employeeDetails.hdbtabledata

{

"format_version": 1,

"imports":

[ {

"target_table" : "FirstFullStackApp.Employee::cdsArtifact.employeeDetails",

"source_data" : { "data_type" : "CSV", "file_name" : "FirstFullStackApp.Employee::employeeDetails.csv", "has_header" : false },

"import_settings" : { "import_columns" : ["EMPLOYEE_ID","DESCRIPTION","EMPLOYEE_NAME" ] },

"column_mappings" : {"EMPLOYEE_ID" : 1,"DESCRIPTION" : 2,"EMPLOYEE_NAME" : 3}

}

]

}

Create a file named roleEnrollments.hdbtabledata

{

"format_version": 1,

"imports":

[ {

"target_table" : "FirstFullStackApp.Employee::cdsArtifact.roleEnrollments",

"source_data" : { "data_type" : "CSV", "file_name" : "FirstFullStackApp.Employee::roleEnrollments.csv", "has_header" : false },

"import_settings" : { "import_columns" : ["EMPLOYEE_ID", "ROLE_ID", "ROLE_NAME","EMPLOYEE_NAME","EMAIL"] },

"column_mappings" : {"EMPLOYEE_ID": 1, "ROLE_ID" : 2, "ROLE_NAME" : 3,"EMPLOYEE_NAME" :4,"EMAIL" :5}

}

]

}

Now for above-created tables, we will create data by using CSV file as shown below under the src folder.

Create employeeDetails.csv

"E100","desc1","Arun"

"E101","desc2","Anand"

"E102","desc3","Ram"

"E103","desc4","Ananya"

Create roleEnrollments.csv

"E100","J5136","JavaCodes","Arun","arun.rage@sap.com"

"E101","J5137","BusinessAnaylist","Anand","anand.kista@sap.com"

"E102","J5138","Designer","Ram","ram.raja@sap.com"

"E103","J5139","Fresher","Ananya",”ananya.sudheer@sap.com”

Folder structure will look like this

Right click on Project (FirstFullStackApp) and navigate to project setting to install cloud Foundry builder as shown below. This will help us to build and deploy our DB module to Cloud Foundry further.

Click on “install builder” button

Note: If you find default settings are not present or not as per your cloud Foundry trial account in the cockpit, then from your trail account space you can get the required API endpoint details and paste the same here and click on build installer button.

Once the builder is installed successfully click on the save button.

Now right click on Employee (DB module folder) and build it as shown below.

Once the build is successful, In cockpit->cloud foundry-> space check for the deployed DB module

Check for instance created.

Now your HDI container type DB is ready in Cloud Foundry. We can access it in another module if required by using data service.

Below are the steps to create OData service.

Create a Nodejs module.

Create service.xsodata under lib/xsodata/service.xsodata

service

{

"FirstFullStackApp.Employee::cdsArtifact.employeeDetails" as "employeeDetails" navigates ("role_Enrollments" as "roleEnrollments");

"FirstFullStackApp.Employee::cdsArtifact.roleEnrollments" as "roleEnrollments" ;

association "role_Enrollments" principal "employeeDetails"("EMPLOYEE_ID")

multiplicity "1" dependent "roleEnrollments"("EMPLOYEE_ID") multiplicity "*";

}

In MTA.yaml file add HDI as required for Odatajs

Build Odatajs module with cloud Foundry setting to the project.

Once build is successful then Run the application as Nodejs Application.

Once the application started successfully, we can check the metadata.

replace index.js in the URL from the browser it opened after running successfully, with /xsodata/service.xsodata/$metadata.

By using this URL we can create “Destination” in cloud Foundry which can be used further in other modules.

if you are not willing to use OData service then you can skip the step of the creation of Odata module itself and directly navigate to ” Accessing DB explorer ” and check for tables as shown below.

Accessing DB from SAP WEB IDE.

This will add DB explorer to you SAP Web IDE. Now you can see DB icon on the left side of the screen.

Click on the icon–> then click on “+” symbol where can add your DB.

Select the container and click on OK.

Onclick of tables in this list will list down all the tables available. Right click on the respective table and open data will display the data present inside tables.

Top right corner “SQL” image “on click” will open an editor where you can execute SQL queries.

I learned about SAP SQL Data Warehousing. This topic was even more exciting to me, because I’m an SAP BW Consultant, and I don’t knew before about possibility to build your own SAP Warehouse directly on HANA database. So why I can’t just install this add-on on my private Hana DB, to check this out by myself? This is so simple, and this article shows you how to do this, starting from fresh SAP HANA XSA installation.

Requirements:

◈ S-USER for DWF download,
◈ 4 cores of CPU (in my case), and 12 GB RAM for VM,
◈ SAP Hana Server + Applications (XSA),
◈ two hours of free time.

SAP HANA XSA INSTALLATION

First of all, you have to download SAP HANA Express, there are many possible variants to achieve this. My recommended way is just to download and run the full official VM image, which you can run on most popular operating systems

The most important thing is that you have to download option with applications, because of DWF is a plugin.

SAP HANA Express SLES registration

The second thing is registering your SLES machine to get the registration code. This code is necessary to register our copy of SUSE Linux Enterprise server. I propose to perform this step to avoid issues with the installation of your virtual machine tools responsible for access to host machine disc. You can skip this step if you don’t need to install any packages in SLES, and if you are able to move downloaded DWF plugin in another way ( i.e. you can download it using CURL, from local machine HTTP server ).

1. Check your system version by run:

cat /etc/os-release

SAP HANA Study Materials, SAP HANA Guides, SAP HANA Learning, SAP HANA Certifications

2. Go to SLES download page, and select your version of OS

3. Register yourself

4. When your registration code is ready, you will be able to log in at https://scc.suse.com/subscriptions?family=sles. There you should see something similar to:

5. Now you need to provide this code into Yast tool. To do this by copy/paste you can simply login by ssh into the machine, and run:

sudo yast

then go to Software -> Product registration

6. Here provide your e-mail and registration code

7. Repositories should be refreshed. After this, you should be able to install your VM tools or any additional software.

Data warehouse foundation download

If you have any troubles, please refer to this.

1. Please check your Hana version. Execute

HDB version

2. Go to SAP Launchpad software download,and next please folow:

3. Add to basket and download you version of DWF using SAP Download manager from official SAP site

4. In your Hana VM Machine, please start jobscheduler-broker

xs start jobscheduler-broker

5. Mount your external catalog, with downloaded plugin and run command dependend of your version:

xs install SAPDW<version>.ZIP

6. After succesfull instalation you should see similar confirmation:

7. For now you only need to enable plugin in SAP WEB IDE, go to

https://hxehost:53075/

8. Click on rack icon on the left panel (3 icon from the top), and go to Features. Find SAP Hana Data Warehousing foundation, click “ON” and then “Save”.

Voilla ! Eveything is ready for now. You can test DWF by yourself, I recomend offical tutorials like:

In section 34 Changed Data Capture of the SAP Data Services Designer Guide you can find a very good description of the various delta load capabilities in SAP Data Services. Isn’t that worth an attentive read?

Delta loads are commonly used to reduce overall ETL processing time. When the number of new and modified source records is relatively low compared to the full data size, overall performance will increase significantly. Not only the time spent in extraction will be much lower, also the transformation and load steps will run much faster because those operations will apply to a (minor) subset of data only.

I would say that, in general, you may consider following options to implement an incremental extraction:

◈ Use last_modification timestamps from the source tables. Create a control table for keeping track of the timestamp(s) used in the previous incremental extraction. Have your jobs call an initialisation script to assign the timestamps to global variables. Use the variables in the where-clause of the Query transform in your extraction dataflows.
◈ If there’s no timestamp available, but the records are numbered sequentially, you can apply a similar approach using the sequence numbers.
◈ Use log tables. Many systems do not include timestamps in the actual tables modified. They use so-called log tables to keep track of changes. Join source and log tables and proceed as above.
◈ Use the built-in CDC-mechanisms of the source database.
◈ Use specific CDC-enabling software, like SLT.
◈ With SAP as a source, you can leverage delta-enabled Business Content Extractors.

Unfortunately, when none of those options are available, you probably will have to do a full extraction of your source data, not such a great idea when dealing with big data. Note that, in such a scenario, you still have the possibility to reduce transformation and load times by calculating the delta yourself. You achieve this by keeping a copy of the data extracted in the previous job run and running it thru a Table_Comparison transform to calculate the delta. Then you can apply the transformations to the delta only. And write the new and modified records only to the target.

This blog describes a solution around the two potential bottlenecks in that approach:

◈ Full extraction times are related to the size of the source data, the number of records time the record size. The larger the latter (multiple and wide columns), the more time it will take to extract the data.
◈ The time spent in the Table_comparison transform (make sure to always sort the input data and set the comparison method in the transform to Sorted input!) may negate the profit obtained from transforming and loading a reduced data set. The more columns, the longer the comparison process will run.

Both problems can be overcome by using hash functionality. The example below is for SAP HANA but can be applied to any database that has a built-in hash function or is able to call an external one.

Create two tables in HANA, first:

◈ CHECKSUM_MINUS1, to contain checksums from the previous job run. The table has 2 columns, one for the primary key and the other for the hash value of the record.
◈ CHECKSUM, to contain the same data from the current job run.

Create a 1st dataflow to copy the checksums from the previous run to CHECKSUM_MINUS1:

SAP HANA Study Materials, SAP HANA Guides, SAP HANA Tutorial and Materials

As both tables are in the same database, full logic is pushed to HANA and execution terminated almost instantly.

Create a 2nd dataflow to calculate the checksums. Use a SQL transform as source:

Open the SQL – Transform Editor. Enter the SQL text as:

SELECT PKCOL, hash_sha256(

to_binary(PKCOL)

, to_binary(ifnull(INT1,0))

, to_binary(ifnull(INT2,0))

, to_binary(ifnull(DATE1,''))

, to_binary(ifnull(INT3,0))

, to_binary(ifnull(INT4,0))

, to_binary(ifnull(INT5,''))

, to_binary(ifnull(TXT01,''))

, to_binary(ifnull(TXT02,''))

, to_binary(ifnull(INT6,0))

, to_binary(ifnull(TXT03,''))

, to_binary(ifnull(TXT04,''))

, to_binary(ifnull(INT7,0))

, to_binary(ifnull(DEC1,0))

, to_binary(ifnull(TXT05,''))

, to_binary(ifnull(DEC2,0))

, to_binary(ifnull(TXT06,''))

, to_binary(ifnull(DEC3,0))

, to_binary(ifnull(TXT07,''))

, to_binary(ifnull(INT8,0))

, to_binary(ifnull(TXT08,''))

, to_binary(ifnull(TXT09,''))

, to_binary(ifnull(INT9,0.0))

, to_binary(ifnull(DEC4,0.0))

, to_binary(ifnull(DEC5,0))

, to_binary(ifnull(TXT10,''))

, to_binary(ifnull(TXT11,''))

, to_binary(ifnull(DEC6,0))

, to_binary(ifnull(TXT12,''))

, to_binary(ifnull(TXT13,''))

, to_binary(ifnull(DATE2,''))

) as hash_value

FROM MY_SOURCE_TABLE

Column PKCOL is the primary key column of MY_SOURCE_TABLE that has several additional numeric, date and varchar columns.

Then select the Update schema button. And close the Editor window.

The SQL transform prevents full SQL-pushdown to HANA, so all data is passing thru SAP Data Services memory. Because the data size is relatively small (2 columns only) compared to the full source table size, performance gains may be high.

Create a 3rd dataflow to calculate the delta:

The Table_Comparison compares the input data to CHECKSUM_MINUS1 (current to previous data set).

The Map_Operation maps all Input row types to Normal. And registers the type of operation in the additional column.

Because there’s one comparison column only, the operation will run much quicker, too.

When you run the 3 dataflows in sequence, the DELTA table will contain the primary key column of all new, updated and deleted records, with the type of operation.

You can now join DELTA and MY_SOURCE_TABLE to limit the size of the input data set. Apply the necessary transformations to that subset. And use a Map_CDC_Operation to eventually apply the results to the target.

AI and machine learning are the hottest trends in the current IT market. Everyone is talking about it and customers are adopting these technologies in day to day processes. Because of this, there is a need to have systems that will enable the processes to be scaled, governed and compliant to current business needs.

As part of digital transformation efforts, customers currently running SAP ERP applications are implementing innovative solutions to enhance operations. These innovative solutions range from RPA-robotic process automation, machine learning and enhanced analytics leading to an intelligent ERP aka iERP.

This blog assumes that the audience is familiar with SAP HANA technology both as a database and application platform, along with the engines that are available to perform various tasks. The blog will cover use of SAP HANA as a scalable machine learning platform for enterprises.

We will cover the business applications and technical aspects of the following HANA components:
1) PAL – HANA Predictive Analytics Library

2) HANA- R – Integrated platform between HANA – R

3) HANA EML – Extended Machine Library

4) AFM – Application function modeler

Along with the components above, HANA also enables the use of external API’s via the XS engine.

The diagram below covers the components available within the HANA platform which can be utilized for machine learning.

SAP HANA Study Material, SAP HANA Tutorial and Material, SAP HANA Certification

SAP HANA Predictive Analytics Library (PAL)

Let’s understand some technical aspects of PAL before we see its applications. PAL is one of the components of the Application Function Library (AFL) in HANA. It can define functions that can be called from within SQL Script procedures in HANA to perform advanced analytic algorithms. SAP has provided host of classic and universal algorithms with PAL. When PAL is paired with HANA’s ability to host execution engines and perform local calculations in-memory and in parallel, it provides a unique capability to accelerate machine learning models. PAL is available on every HANA license (from HANA 1.0 SPS06 onward) and cloud platform in the AFL. SAP PAL algorithms are divided into 10 categories, some of them include clustering, classification, association and time series functions. All of the PAL procedures can be seen below.

PAL also includes several algorithms that learn and continuously update to enable dynamic predictions, allowing companies the ability to use current data and instantly adapt to the changing conditions and behaviors of their clients.

The goal of HANA PAL is to enable a majority of the most common predictive use cases . Paired with the in-memory and fast performance of HANA, many choose this as their predictive tool of choice. However, even with all of the algorithms offered, you may need an external R server for even more advanced algorithms.(The HANA-R integration is covered in the points below).

Use Case

For clients, it will be ideal to start with algorithms provided out of box with PAL and then explore external algorithms if there is an absolute need.

PAL alone is best for those who need to perform the above algorithms, already use SAP HANA and have SAP HANA Studio installed. For those who want flexibility and customization such as data scientists and mathematicians, deploy both PAL and R together. However, PAL requires experience in SQL Script and/or the predictive methods used.

HANA-R Integration

R is an open source programming language for statistical computing that is widely used for advanced data analysis. Providing R integration opens the door wide by enabling HANA to consume and execute all the open source algorithms available in R. HANA database interprets the R language and accordingly submits the script to R Server.

The goal of integrating SAP HANA with R is to ultimately customize algorithms even more than what is offered in the standard libraries via PAL. SAP HANA uses an external R environment to execute the R code. The application developer can then embed R function definition and calls within HANA SQLScript and submit the entire code as a database query.

This opens up all new possibilities because the extent of R’s capabilities can be utilized on your data in HANA.

Use Case

Integration of R code is suitable when an SAP HANA-based modeling and consumption application or developer wants to use the R environment for specific statistical functions. It allows a developer to use their creativity, choose from thousands of R packages and script some very agile data analysis and predictions.

SAP does not include the R environment with a SAP HANA license since R is open source and available under the General Public License. Similarly, SAP does not provide support for R. In order to use the SAP HANA integration with R, you need to download R from the open-source community and configure it. You also need Rserve, a TCP/IP server that allows other programs to use facilities of R without the need to initialize R or link with the R library. Note that this integration requires prior knowledge and expertise with R code.

HANA Extended Machine Learning Library (EML)

SAP has introduced HANA EML from HANA 2.0 SP02 onward. It gives ability to use HANA data for scoring (the process of applying a predictive model to a set of data is referred to as scoring) on pre-built machine learning models.

What does this mean? HANA will not be used in this case for training and building machine learning models. Models will be built using a python environment – Tensorflow, which is a python package offered by Google to build complex, deep learning models. Tensorflow is not only limited to deep learning but also can be used for preforming common machine learning models.

How does this work? Build models (train, test & validate) in Tensorflow and then enable the models to be consumed by Tensorflow serving (a separate server needed to enable the HANA EML and Tensoflow integration). Then build the HANA SQL scripts to call the models served in Tensorflow serving and enable data for the scoring (predicting) .

Use Case

EML is already included as part of SAP HANA in the AFL and users can easily connect TensorFlow serving to HANA. With HANA EML, TensorFlow models can now be easily integrated into enterprise applications and business processes for clients.

HANA Application Function Modeler (AFM)

HANA AFM is an extension of SAP HANA Studio used to create flowgraph models (tables, views, procedures, R scripts, PAL functions) without writing SQL script code. Think of it as a drag and drop interface that allows less experienced developers to build complex procedures quickly.

A major advantage of AFM is that it eliminates the need to code PAL and BFL (Business Function Library) algorithms. Another benefit for businesses is that model objects can be checked in and out of development so that multiple users can access and develop simultaneously.

Use Case

AFM is for users who are less experienced in SQL code but still want to perform predictive analytics using HANA database. The advantages are that the algorithms are available out of box, there’s no need to send data out of the system and it provides easier integration with in-process applications and transactions in SAP .

Key Takeaways

The table below compares the four products’ use cases, skillsets and prerequisites.

PAL and AFM can be used out of the box with HANA enterprise without needing any additional licenses, servers or components. This can jump start clients to start building machine learning use cases in SAP. For very complex use cases you can then enable R integration and EML .

In conclusion, both data scientists and business analysts should start their analysis by using SAP HANA automated predictive capabilities whenever possible. Automated machine learning can apply to a growing number of scenarios while producing valid results in seconds or minutes. Therefore, those who are not data scientists now have the ability to answer their own questions and quickly depict on the results while providing data scientists and mathematicians an automated way of quickly analyzing problems.

Introduction

I’m going to show you a chat-bot which can help HR search employee’s salary by name and employee number on specific date.

On the high-level, I’ll need to do:

1. Create SAP UI 5 on SAP Cloud Platform
2. Create SAP Conversational AI chat-bot
3. Create server REST APIs to help chat-bot query database.
4. Create Two tables(people and salary) in SAP HANA

For this post, I will focus on the chat-bot side.

SAP HANA Tutorial and Material, SAP Study Materials, SAP HANA Certifications

It’s very easy to build your bot by leveraging SAP Conversational AI. You just need four steps!

The first step is training your bot, helping it to know what do you say. Here, “Intents” means the concept of which is designed to trigger “Skills”.

For example, “I wanna know the salary” and “show me the salary” means the same intent — “salary ”. On the other hand, “Entities” means the words like “person”, “number”, “date-time”.

Second, “skills” — after your chat-bot understand what do you say, you need to create your conversational flow with builder tool. You can let chat-bot take actions here.

For here, we need to set up web-hook which is meaning your chat-bot can request to your APIs and get the responses!

If you choose using SAP HANA and developing with Python. You also need to open-db-tunnel at the same time when you query data from HANA.

Also, you need to make sure your REST APIs are good to respond. You can use software like Postman to check.

We can use ngrok to help us connect our localhost APIs via the Internet. (Don’t forget to set it in setting page)

Third, just choose the platform which you want to deploy!

Finally, the last step is helping you to trace what did your chat-bot receive and what kind of skills did it trigger.

Results:

Let me show the tables I created on SAP HANA here. We can see there’s an employee named tim cook and his employee_id is i000000 with several salary data.

Let’s focus on salary on January 2019, it’s 9999.

You can see payment date 2019–01–01 is 9999 in each query.

Let’s change the salary to 13579!

Woohoo, It shows 13579!

Due to the fact that starting with the wave 2019.01 SAP Analytics Cloud (SAC) has stopped accepting the self-signed SSL certificates for HTTPS INA live connections I have ended up by replacing the self-signed HANA Express SSL certificate(s) with the equivalent CA-signed SSL certificate(s).

Synopsis:

In a nutshell, the SAP Analytics tools (the likes of SAC, Analysis for Office, BW4H, BOE with Lumira 2.x, Analysis for OLAP or WebIntelligence) rely heavily on the secured HTTPS INA protocol to consume the SAP HANA calculation views on-the-fly via the SAP HANA EPM-MDS engine.

This is being referred to as live/on-line/remote connectivity, where the data is always current, as opposed to acquired/off-line/import option where the data needs to be acquired first and then refreshed in a scheduled manner.

I have invested quite some time and effort in order to build a viable testbed with the focus on the front-end SAP analytical applications that rely on the secured HTTP INA protocol based connectivity [either with or without SSO to the backend HANA database] and that can consume the HANA HDI views (e.i. the views which are no longer tied into the default _SYS_BIC schema).

As you may know the SAP HANA Express 2.x ready-to-deploy images implement several self-signed SSL certificates for the secured domains of HTTPS INA (XS classic engine running on the tenant database), XSA (XS Advanced) and LCM access over the web.

As aforementioned, having recently come across the necessity to implement the trusted SSL certificates signed with a trusted CA (Certificate Authority) rather than relying on self-signed ones, I have decided to share my experience in “Taming your SAP HANA Express” series of blog posts.

It is a great HANA appliance! Regards to the entire SAPCAL team!

For the sake of convenience and transparency most of the URLs in this article are not obfuscated. They refer to a stock HANA Express instance deployed on AWS and do not reveal any trade secrets.

Let’s begin:

First I got the HTTPS INA protocol up and running fine on the tenant database where both the index server and webdispatcher are running; (This is very important as with the multi-tenanted databases there is neither index server nor webdispatcher available to run on the system database)

This setup used to work against SAP Analytics Cloud waves 18 through 22 out of the box (aka with the self-signed SSL certificates in Chrome browser);

So what…?

1. The initial problem with the HANA Express self-signed certificates.

Initially one particular problem I faced was that a number (but not all) of front-end applications require the CN certificate name to match the hostname in the HTTPS service URL;

When a mismatch between the two names is detected these applications deny any further access.

In our case the CN name is sid-hxe and in the webdispatcher HTTPS URL service name is vhcalhxedb; Obviously sid-hxe != vhcalhxedb !

Eventually, after a thought, I narrowed down the problem to the following piece of advise:

Can someone advise the steps required to replace the system certificate (sid-hxe) with the default.root.crt.pem certicificate (vhcalhxedb) in the webdispatcher’s SAPSSLS.pse store ?

Let me explain it in more details.

1a.The below links to the HANA XS classic engine work perfectly fine on my testbed. 4390 is the SSL port number of the webdispatcher running on the HXE tenant database;

The OS host name is sid-hxe;

The HANA host name is vhcalhxedb;

Both names resolve to the appliance IP address;

I have noticed the HXE webdispacther URL is using the OS system host sid-hxe SSL certificate as depicted below.

You may notice that the URL uses vhcalhxedb HXE appliance name [which is resolved into the elastic IP address in my laptop’s /etc/hosts file].

Still the webdispatcher/XS classic engine are configured to use the system certificate which shows the OS host name, namely sid-hxe:

SAP HANA Express, SAP HANA Tutorial and Materials, SAP HANA Study Materials

in other words the name of the HANA host (vhcalhxedb) must not be replaced in the URL;

This vhcalhxedb certificate is in the default.root.crt.pem certicificate file found here: <installation_path>/HXE/xs/controller_data/controller/ssl-pub/router