Quantcast
Channel: SAP HANA Central
Viewing all 711 articles
Browse latest View live

HANA MDC cross-database access

$
0
0
In a multi tenant database setup for HANA, the tenants are isolated from each other and can not see the content in other tenants by default. But there can be need to access some data from the other tenant. This can be accomplished using a cross-database access functionality that is provided by HANA.

The first thing to know about cross-database access is that it only allows SELECT access to data from the source database.


Pre-Requisites:
  1. You are logged in as SYSTEMDB
  2. You have the system privilege INIFILE ADMIN
Configuration (from the administration editor in HANA Studio):
  1. Enable cross-database access by changing the parameter enable = true in [cross_database_access] under global.ini in SYSTEMDB.
  2. This will enable the feature but does not configure it
  3. For configuration add the following parameters: targets_for_<source_db_name> under global.ini-> [cross_database_access] and adding the <target_db_name> as value.
  4. Example: You have two databases DB1 and DB2 and you want to be able to access DB1 from DB2. So you add the parameter targets_for_DB2 with the value DB1.
  5. This will configure one way access to DB1 from DB2. But similar to previous configuration, access to DB2 from DB1 can also be configured at the same time by adding the parameter.
  6. This will complete the configuration
Concept of remote identities:

As we know, in single database, every user will need proper authorization/privileges on an object to access that object. This is the same in MDC cross-database access. But since the users are isolated, we need to create something like a “proxy” called remote identities.

Terminology I will be using from here on in my example(s):

Source Database: Where the object/data exists and hence the database you need access to.

Target database: Where you are running the query and want the results. Hence the database you need access from.

DB1: Target tenant database

DB2: Source tenant database

USER1: User in source database that needs access to object(s) in target database

USER2: Remote Identity of USER1 in source database

Example:

USER1 needs to run a Select query on table T1 that exists in DB2. You need to add a remote identity in source database for USER1

1. Create USER2 (remote identity for USER1) in DB2:
CREATE USER2 WITH REMOTE IDENTITY USER1 AT DATABASE DB1.
or
Add remote identity to an already existing user in DB2
ALTER USER2 ADD REMOTE IDENTITY USER1 AT DATABASE DB1
Here USER1 will ask USER2 to run the query on database DB2 and provide the results. But for USER2 to itself get result from the table T1, it needs access (SELECT) to table T1.

Therefore we provide USER2 the necessary acess:

GRANT SELECT ON “<SCHEMA>”.”T1″ TO USER2;

But what if you want to create a calculation view which needs access to table T1 from DB2?

We still need the USER2 to get data from table T1. But since the “activation guy” for the calculation view is the _SYS_REPO of DB1 (every tenant has its own _SYS_REPO user and are isolated from each other).

The database does not allow the administrator to add a remote identity to SYS_REPO in any tenant. Instead, the administrator should create a dedicated user (for example, REPO_DB1_DB2), which you can use exclusively for privileges in the remote database. This user only needs the privileges for tables that are used as remote data sources and does need privileges for all the tables in the remote DB that are used in calculation views.

The remote identity for _SYS_REPO (REPO_DB1_DB2) will need select access with grantable to others option:

GRANT SELECT ON “<SCHEMA>”.”T1″ TO REPO_DB1_DB2 WITH GRANT OPTION

You are now able to run select queries as well create a view in one tenant that needs access to table in a different tenant.

Some things to note:
  • Cross Database is currently limited to read-only access
  • Cross-database queries do currently not support multiple tenant database levels as part of a view hierarchy.
  • Hierarchy views with remote objects or calculation views with hierarchies on top of remote objects are not supported.

Dedicated SAP HANA XS Advanced Runtime host installation through resident hdblcm on HANA 2 SPS 01

$
0
0
In this blog I will cover installation of XS Advanced Runtime on a dedicated host on HANA 2 SPS 01. Even though my example covers HANA 2 SPS 01, the process is similar on HANA 1 SPS 11+.

Prerequisites:
  1. The SAP HANA system has been installed with its server software on a shared file system (export options rw, no_root_squash).
  2. The host has access to the installation directories <sapmnt> and <sapmnt>/<SID>.
  3. The SAP HANA system has been installed with the SAP HANA database lifecycle manager (HDBLCM).
  4. You are logged on as root user or as the system administrator user <sid>adm.
  5. The difference between the system time set on the installation host and the additional host is not greater than 180 seconds.
  6. The operating system administrator (<SID>adm) user may exist on the additional host. Make sure that you have the password of the existing <SID>adm user, and that the user attributes and group assignments are correct. The SAP HANA database lifecycle manager (HDBLCM) resident program will not modify the properties of any existing user or group.
  7. You have the latest XS Advanced Runtime from SMP
  8. Extract the EXTAPP…*.SAR file using SAPCAR : SAPCAR -xvf EXTAPP..*.SAR
Installation:

1. Resident hdblcm resides in the {mount_directory}/<SID> ( In my example: /media/shared/<SID>/hdblcm/)

2. Run hdblcm (In my example as root) with component_dirs option: ./hdblcm –component_dirs=/path_to_extracted_XSA_Runtime_Directory


sap hana mdc, sap-hana-xs, mdc, sap hana

3. Select update_component and then select xs (Install XS Advanced):

sap hana mdc, sap-hana-xs, mdc, sap hana

4. This is the step where this installation differs from installing XS Advanced on the same host. We will be adding a host to our system here. Give the FQDN of your new host here (make sure your hana host can resolve this):

sap hana mdc, sap-hana-xs, mdc, sap hana

5. Next steps including setting up XS Advanced installation including providing <sid>adm password, database user and password, XS advanced runtime master password, organization name, XSA Admin user and password. The most important step here is making the new host as xs_worker and not worker node.

In most of the options I used default but this can be configured as needed but might require extra steps.

sap hana mdc, sap-hana-xs, mdc, sap hana

6. In HANA Studio in the Landsacpe-> Services tab, you can see XS Advanced services (xscontroller, xsuaaserver and xsexecagent) running on a seperate host now. Additionally, in the Hosts tab you can see the new host is configured as xs_worker host:

sap hana mdc, sap-hana-xs, mdc, sap hana

sap hana mdc, sap-hana-xs, mdc, sap hana

Now your HANA is set up with XS Advanced on a dedicated server and can be utilized for XSA applications.

Bridging two worlds : Integration of SAP and Hadoop Ecosystems

$
0
0
Proliferation of web applications, social media and internet-of-things coupled with large-scale digitalization of business processes has lead to an explosive growth in generation of raw data. Enterprises across the industries are starting to recognize every form of data as a strategic asset and increasingly leveraging it for complex data driven business decisions. Big Data solutions are being used to realize enterprise ‘data lakes’, storing processed or raw data from all available sources and powering a variety of applications/use-cases.

Big Data solutions will also form a critical component of enterprise solutions for predictive analytics and IoT deployments in near future. SAP on-premise and on-demand solutions, especially HANA platform will need closer integration with the Hadoop ecosystem.

What exactly is ‘Big Data’ ?


Data sets can be characterized by their volume, velocity and variety. Big Data refers to the class of data with one or more of these attributes significantly higher than the traditional data sets.

SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

Big data presents unique challenges for all aspects of data processing solutions including acquisition, storage, processing, search, query, update, visualization, transfer and security of the data.

Hadoop Big Data Solution to the Rescue !


Hadoop is an open-source software framework for distributed storage and processing of Big Data using large clusters of machines.

Hadoop’s Modus Operandi –> Divide and Conquer : Break task in small chunks – store / process in parallel over multiple nodes – combine results

Hadoop is not the only available Big Data Solution. Several commercial distributions/variants of Hadoop and other potential alternatives exist which are architecturally very different.

Combining high speed in-memory processing capabilities of SAP HANA with Hadoop’s ability to cost-effectively store and process huge amounts of structured as well as unstructured data has limitless possibilities for business solutions.

Hadoop system can be an extremely versatile addition to any SAP business system landscape while acting as –
  • A simple database and/or an low cost archive to extend the storage capacity of SAP systems for retaining large volumes historical or infrequently used data
  • A flexible data store to enhance the capabilities of persistence layer of SAP systems and provide efficient storage for semi-structured and unstructured data like xml-json-text-images
  • A massive data processing / analytics engine to extend or replace analytical / transformational capabilities of SAP systems including SAP HANA
SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

Getting acquainted : Hadoop Ecosystem


Hadoop at its core is a Java based software library which provides utilities/modules for distributed storage and parallel data processing across a cluster of servers. However, in common parlance the term ‘Hadoop’ almost invariably refers to an entire ecosystem which includes a wide range of apache open-source and/or commercial tools based on the core software library.

Hadoop is currently available either as a set of open-source packages or via several enterprise grade commercial distributions. Hadoop solution are available as SaaS/PaaS cloud offerings from multiple vendors in addition to traditional offerings for on-premise deployments.

Snapshot of prominent distributions/services according to latest market guides from Gartner and Forrester Research :
  • Apache Hadoop [open source]
  • Cloudera Enterprise | Cloudera CDH [open source]
  • Hortonworks Data Platform HDP [open source]
  • MapR
  • IBM Big Insights ^^
  • Amazon Elastic MapReduce (EMR)
  • Microsoft Azure HDInsight
  • Google Cloud Dataproc
  • Oracle Big Data Cloud Services
  • SAP Cloud Platform Big Data Service (formerly SAP Altiscale Data Cloud)(^^ Potentially Discontinued Solution)
Hadoop core components serve as foundation for entire ecosystem of data access and processing solutions.
  • Hadoop HDFS is a scalable, fault-tolerant, distributed storage system which stores data using native operating system files over a large cluster of nodes. HDFS can support any type of data and provides high degree of fault-tolerance by replicating files across multiple nodes.
  • Hadoop YARN and Hadoop Common provide foundational framework and utilities for resource management across the cluster
Hadoop MapReduce is a framework for development and execution of distributed data processing applications. Spark and Tez which are alternate processing frameworks based on data-flow graphs are considered to be the next generation replacement of MapReduce as the underlying execution engine for distributed processing in Hadoop.

Map       : Split and distribute job
Reduce   : Collect and combine results

SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

Variety of data access / processing engines can run alongside Hadoop MapReduce engine to process HDFS datasets. Hadoop ecosystem is continuously evolving with components frequently having some complementing, overlapping and/or similar appearing capabilities but with vastly different underlying architectures or approach.

Popular components-applications-engines-tools within Hadoop ecosystem (NOT an exhaustive list; several more open source and vendor specific applications are used by enterprises for specific use-cases)
  • Pig — Platform for development and execution of high-level language (Pig Latin) scripts for complex ETL and data analysis jobs on Hadoop datasets
  • Hive — Read only relational database that runs on top of Hadoop core and enables SQL based querying to Hadoop datasets; Supported by Hive-Metastore
  • Impala — Massively Parallel Processing (MPP) analytical database and interactive SQL based query engine for real-time analytics
  • HBase — NoSQL (non-relational) db which provides real-time random read/write access to datasets in Hadoop; Supported by HCatalog
  • Spark — In-memory data processing engine which can run either over Hadoop or standalone as an alternative / successor to Hadoop itself
  • Solr — Search engine / platform enabling powerful full-text search and near real-time indexing
  • Storm — Streaming data processing engine for continuous computations & real-time analytics
  • Mahout — Library of statistical, analytical and machine learning software that runs on Hadoop and can be used for data mining and analysis
  • Giraph — Iterative graph processing engine based on MapReduce framework
  • Cassandra — Distributed NoSQL (non-relational) db with extreme high availability capabilities
  • Oozie — Scheduler engine to manage jobs and workflows
  • Sqoop — Extensible application for bulk transfer of data between Hadoop and structured data stores and relational databases
  • Flume — Distributed service enabling ingestion of high-volume streaming data into HDFS
  • Kafka — Stream processing and message brokering system
  • Ambari — Web based tool for provisioning, managing and monitoring Hadoop clusters and various native data access engines
  • Zookeeper – Centralized service maintaining Hadoop configuration information and enabling coordination among distributed Hadoop processes
  • Ranger – Centralized framework to define, administer and manage fine-grained access control and security policies consistently across Hadoop components
  • Knox – Application gateway which act as reverse proxy, provides perimeter security for Hadoop cluster and enables integration with SSO and IDM solutions

Bridging two worlds – Hadoop and SAP Ecosystems


SAP solutions, especially SAP HANA platform, can be ‘integrated’ with Hadoop ecosystem using a variety of solutions and approaches depending upon the specific requirements of any use case.

SAP solutions which should be considered for the integration include :
  • SAP BO Data Services
  • SAP BO BI Platform | Lumira
  • SAP HANA Smart Data Access
  • SAP HANA Enterprise Information Management
  • SAP HANA Data Warehousing Foundation – Data Lifecycle Manager
  • SAP HANA Spark Controller
  • SAP Near-Line Storage
  • SAP Vora
  • …. <possibly more !>
  • Apache Sqoop [Not officially supported by SAP for HANA]
SAP HANA can leverage HANA Smart Data Access to federate data from Hadoop (access Hadoop as a data source) without copying the remote data into HANA. SDA enables data federation (read/write) using virtual tables and supports Apache Hadoop/Hive and Apache Spark as remote data sources in addition to many other database systems including SAP IQ, SAP ASE, Teradata, MS SQL, IBM DB2, IBM Netezza and Oracle.

Hadoop can be used as a remote data source for virtual tables in SAP HANA using following adaptors (in-built within HANA):
  • Hadoop/Spark ODBC Adaptor — Require installation of Unix ODBC drivers + Apache Hive/Spark ODBC Drivers on HANA server
  • SPARK SQL Adaptor — Require installation of SAP HANA Spark Controller on Hadoop Cluster (Recommended Adaptor)
  • Hadoop Adaptor (WebHDFS)
  • Vora Adaptor
SAP HANA can also leverage HANA Smart Data Integration to replicate required data from Hadoop into HANA. SDI provides pre-built adaptors & adaptor SDK to connect to a variety of data sources including Hadoop. HANA SDI requires installation of Data Provisioning Agent (containing standard adaptors) and native drivers for the remote data source, on a standalone machine. SAP HANA XS engine based DWF-DLM can relocate data from HANA to/between HANA Dynamic Tiering, HANA Extension nodes, SAP IQ and Hadoop via Spark SQL adaptor / Spark Controller.

SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

SAP Vora is an in-memory query engine that runs on top of Apache Spark framework and provides enriched interactive analytics on the data in Hadoop. Data in Vora can be accessed in HANA either directly via Vora Adaptor or via SPARK SQL Adaptor (HANA Spark Controller). Supports Hortonworks, Cloudera and MapR.

SAP BO Data Services (BODS) is a comprehensive data replication/integration (ETL processing) solution. SAP BODS has capabilities (via the packaged drivers-connectors-adaptors) to access data in Hadoop, push data to Hadoop, process datasets in Hadoop and push ETL jobs to Hadoop using Hive/Spark queries, pig scripts, MapReduce jobs and direct interaction with HDFS or native OS files. SAP SLT does not have native capabilities to communicate with Hadoop.

SAP Business Objects (BI Platform, Lumira, Crystal Reports, …) can access and visualize data from Hadoop (HDFS – Hive – Spark – Impala – SAP Vora) with ability to combine it with data from SAP HANA and other non-SAP sources. SAP BO applications use their in-built ODBC/JDBC drivers or generic connectors to connect to Hadoop ecosystem. Apache Zeppelin can used for interactive analytics visualization from SAP Vora.

Apache Sqoop enables bulk transfer of data between unstructured, semi-structured and structured data stores. Sqoop can be used to transfer data between Apache Hadoop and relational databases including SAP HANA (although not officially supported by SAP).

Getting started : Hadoop Deployment Overview


Hadoop is a distributed storage and processing framework which would typically be deployed across a cluster consisting of upto several hundreds/thousands of independent machines, each participating in data storage as well as processing. Each cluster node could either be a bare-metal commodity server or a virtual machine. Some organizations prefer to have a small number of larger-sized clusters; others choose a greater number of smaller clusters based on workload profile and data volumes.

HDFS Data is replicated to multiple nodes for fault-tolerance. Hadoop clusters are typically deployed with a HDFS replication factor of three which means each data block has three replicas – the original plus two copies. Accordingly, storage requirements of a Hadoop cluster is more than four times the anticipated input/managed dataset size. Recommended storage option for Hadoop clusters is to have nodes with local (Direct Attached Storage – DAS) storage. SAN / NAS storage can be used but is not common (inefficient ?) since Hadoop cluster is inherently a ‘share nothing architecture’.

Nodes are distinguished by their type and role. Master nodes provide key central coordination services for the distributed storage and processing system while worker nodes are the actual storage and compute nodes.

Node roles represent set of services running as daemons or processes. Basic Hadoop cluster consists of NameNode (+ Standby), DataNode, ResourceManager (+ Standby) and NodeManager roles. NameNode coordinates data storage on DataNodes while ResourceManager node coordinates data processing on NodeManager nodes within the cluster. Majority of nodes within the cluster are workers which typically will perform both DataNode and NodeManager roles however there can be data-only or compute-only nodes as well.

Deployment of various other components/engines from Hadoop ecosystem brings more services and node types/roles in play which can be added to cluster nodes. Node assignment for various services of any application is specific to that application (refer to installation guides). Many such components also need their own database for operations which is a part of component services. Clusters can have dedicated nodes for application engines with very specific requirements like in-memory processing and streaming.

Master / Management nodes are deployed on enterprise class hardware with HA protection; while the worker can be deployed on commodity servers since the distributed processing paradigm itself and HDFS data replication provide the fault-tolerance. Hadoop has its own built-in failure-recovery algorithms to detect and repair Hadoop cluster components.

Typical specification of Hadoop Cluster nodes depending on whether workload profile is storage intensive or compute intensive. All nodes do not necessarily need to have identical specifications.
  • 2 quad-/hex-/octo-core CPUs   and   64-512 GB RAM
  • 1-1.5 disk per core; 12-24 1-4TB hard disks; JBOD (Just a Bunch Of Disks) configuration for Worker nodes and RAID protection for Master nodes
SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

Hadoop Applications (Data access / processing engines and tools like Hive, Hbase, Spark and Storm, SAP HANA Spark Controller and SAP Vora) can be deployed across the cluster nodes either using the provisioning tools like Ambari / Cloudera Manager or manually.

Deployment requirement could be : All nodes, at-least one node, one or more nodes, multiple nodes, specific type of node

SAP HANA Certifications, SAP HANA Guide, SAP HANA Materials, SAP Vora

Getting started : Deployment Overview of ‘bridging’ solutions from SAP


SAP Vora consists of a Vora Manager service and a number of core processing services which can be added to various compute nodes of the existing Hadoop deployment

— Install SAP Vora Manager on management node using SAP Vora installer
— Distribute SAP Vora RPMs to all cluster nodes and install using Hadoop Cluster Provisioning tools — Deploy Vora Manager on cluster nodes using Hadoop Cluster Provisioning tools
— Start Vora Manager and Deploy various Vora services across the cluster using Vora Manager

SAP HANA Spark Controller provides SQL interface to the underlying Hive/Vora tables using Spark SQL and needs to be added to at-least one of the master nodes of the existing Hadoop deployment

— Install Apache Spark assembly files (open source libraries); not provided by SAP installer
— Install SAP HANA Spark Controller on master node using Hadoop Cluster Provisioning tools

SAP HANA Smart Data Integration and Smart Data Access are native components of SAP HANA and do not require separate installation. Smart Data Integration do however require activation of Data Provisioning server and installation of Data Provisioning Agents.

— Enable Data Provisioning server on HANA system
— Deploy Data Provisioning Delivery Unit on HANA system
— Install and configure Data Provisioning Agents on remote datasource host or a standalone host

SAP HANA Data Warehousing Foundation – Data Lifecycle Manager is an XS engine based application and requires separate installation on HANA platform.

SAP Cloud Platform Big Data Service (formerly SAP Altiscale Data Cloud) is a fully managed cloud based Big Data platform which provides pre-installed, pre-configured, production ready Apache Hadoop platform.

— Service includes Hive, HCatalog, Tez and Oozie from Apache Hadoop ecosystem in addition to the core Hadoop MapReduce / Spark and HDFS layers
— Supports and provide runtime environment for Java, Pig, R, Ruby and Python languages
— Supports deployment of non-default third-party applications from the Hadoop ecosystem
— Service can be consumed via SSH and webservices
— Essentially a Platform-as-a-Service (PaaS) offering and comprises :
— Infrastructure provisioning
— Hadoop software stack deployment and configuration
— Operational support and availability monitoring
— Enables customers to focus on business priorities and their analytic / data-science aspects
by delegating technical setup and management of the Hadoop platform software stack and
the underlying infrastructure to SAP’s cloud platform support team

Final thoughts


Gartner’s market research and surveys assert that Hadoop adoption is steadily growing and also shifting from traditional the monolithic, on-premise deployments to ad hoc or on-demand cloud instances.

Get ready to accomplish big tasks with big data !!

SAP HANA – Detailed Memory Analysis Walkthrough

$
0
0

Introduction


In this exercise, I’ll give a walkthrough on how to perform a top down investigation on high & peak HANA memory usage and its associate top memory consumers with the aid of HANA SQL scripts attached to SAP Note 1969700 – SQL Statement Collection for SAP HANA and Statistics data (History / Current) collected by ESS / SSS (Mandatory. Recommended retention = 49 days)

I’m taking below example where I was asked to perform a memory analysis on a productive system where there was a spike of peak memory usage, around 5.9TB.



In-Memory Technology, sap-hana-admin, sap-hana-memory

Since the system is running on scale out environment, I will breakdown the analysis of each memory usage of each node:

Below we noticed that the node01 and node02 are one of those nodes with high peak used memory.

In-Memory Technology, sap-hana-admin, sap-hana-memory

Thus, in this content, I’ll show the memory analysis on node01 and node02 by using the 3 simple questions as quick start, where it provides us a clearer direction moving forward:
  1. When was the specific time that the peak usage happened?
  2. What were those top memory consumers during peak usage?
  3. What were the running processes/ threads during the peak usage?
In-Memory Technology, sap-hana-admin, sap-hana-memory

Node01:


When was the specific time that the peak usage happened?


First, we need to find out the specific day/ time the peak memory happened.

For my case, I knew the peak memory was occurred last week, and hence, I’m using the past week timestamp and below SQL script to figure the exact day/time of the peak memory.

select top 500 HOST, SERVER_TIMESTAMP,round(INSTANCE_TOTAL_MEMORY_USED_SIZE/1024/1024/1024, 2) as “Peak Used GB” from _SYS_STATISTICS.HOST_RESOURCE_UTILIZATION_STATISTICS where HOST = ‘node01’ and SERVER_TIMESTAMP between ‘03.07.2017 00:00:01’ and ‘10.07.2017 23:59:59’order by “Peak Used GB” desc

In-Memory Technology, sap-hana-admin, sap-hana-memory

In addition, you can also utilize script HANA_Resources_CPUAndMemory_History_1.00.74+_ESS attached to note 1969700 if you want to further break down the memory usage of the day: –

( SELECT     /* Modification section */
TO_TIMESTAMP(‘2017/07/07 08:00:01’, ‘YYYY/MM/DD HH24:MI:SS’) BEGIN_TIME,
TO_TIMESTAMP(‘2017/07/07 23:59:59’, ‘YYYY/MM/DD HH24:MI:SS’) END_TIME,
‘node01’ HOST,
‘ ‘ ONLY_SWAP_SPACE_USED,
-1 MIN_MEMORY_UTILIZATION_PCT,
-1 MIN_CPU_UTILIZATION_PCT,
‘X’ EXCLUDE_STANDBY,
‘TIME’ AGGREGATE_BY,      /* HOST, TIME or comma separated list, NONE for no aggregation */
‘TS30’ TIME_AGGREGATE_BY,   /* HOUR, DAY, HOUR_OF_DAY or database time pattern, TS<seconds> for time slice, NONE for no aggregation *
‘TIME’ ORDER_BY      /* TIME, HOST */
FROM
DUMMY

In-Memory Technology, sap-hana-admin, sap-hana-memory

What were the top memory consumers during peak usage

We’ve narrowed down the time where peak memory happened, now let’s investigate what were those top memory components used on node01 with below script:

HANA_Memory_TopConsumers_History_1.00.90+_ESS with the selection specific on time 09:00 – 10:35 on node01. You may want to adjust the modification section based on your situation.


( SELECT      /* Modification section */
TO_TIMESTAMP(‘2017/07/07 09:00:00’, ‘YYYY/MM/DD HH24:MI:SS’) BEGIN_TIME,
TO_TIMESTAMP(‘2017/07/07 10:35:00’, ‘YYYY/MM/DD HH24:MI:SS’) END_TIME,
‘node01’ HOST,
‘%’ PORT,
‘%’ SCHEMA_NAME,
‘%’ AREA,          /* ROW, COLUMN, TABLES, HEAP, % */
‘%’ SUBAREA,   /* ‘Row Store (Tables)’, ‘Row store (Indexes)’, ‘Row Store (Int. Fragmentation)’, ‘Row Store (Ext. Fragmentation)’, ‘Column Store (Main)’, ‘Column Store (Delta)’, ‘Column Store (Others)’ or ‘Heap (<component>)’ */
‘%’ DETAIL,                   /* Name of table or heap area */
‘ ‘ ONLY_SQL_DATA_AREAS,
‘ ‘ EXCLUDE_SQL_DATA_AREAS,
200 MIN_TOTAL_SIZE_GB,
‘USED’ KEY_FIGURE,        /* ALLOCATED, USED */
‘%’ OBJECT_LEVEL,         /* TABLE, PARTITION */
‘ ‘ INCLUDE_OVERLAPPING_HEAP_AREAS,
‘DETAIL’ AGGREGATE_BY,        /* SCHEMA, DETAIL, HOST, PORT, AREA, SUBAREA */
‘NONE’ TIME_AGGREGATE_BY     /* HOUR, DAY, HOUR_OF_DAY or database time pattern, TS<seconds> for time slice, NONE for no aggregation */

 *The output of the above script gives us an overview of top memory components used, but not the total memory used as allocated shared memory is not included and HOST_HEAP_ALLOCATORS only contains allocators with INCLUSIVE_SIZE_IN_USE > 1 GB, so the sum of used memory displayed is smaller than expected. Don’t be surprised if the total memory on top consumer returned doesn’t match the peak memory usage.

In-Memory Technology, sap-hana-admin, sap-hana-memory

From above, we’ve identified the top contributors were Pool/PersistenceManager/PersistentSpace/DefaultLPA/*, and we know the functionality of this heap allocator is to cache virtual file pages to enhance IO related performance (savepoint, merges, LOB, etc) and make better use of available memory. Thus, once used, they will remain in memory, be counted as “Used memory”, and HANA will evict them only if the memory is needed for other objects, such as table data or query intermediate results. This is the general design of HANA and not critical. Our system is not affected by any bugs since we are on rev122.09 and memory should be reclaimed when needed as we don’t see any OOM trace on the system.

You can always refer to 199997 – FAQ: SAP Memory to have better overview on the purpose of each memory components/ allocators.

With above walkthrough, it should give you a clearer picture on how to:
  1. Identify the day/ time when high memory usage occurred
  2. Identify what’s the top memory consumers during the high memory usage incident
Next, I’ll use the same approach to perform the peak memory RCA on node02

Node02


When was the specific time that the peak usage happened?

I’m using below SQL to find out the when’s the peak memory happened, time range from the last system restart until current: 18.06.2017 – 10.07.2017

select top 500 HOST, SERVER_TIMESTAMP,round(INSTANCE_TOTAL_MEMORY_USED_SIZE/1024/1024/1024, 2) as “Peak Used GB” from _SYS_STATISTICS.HOST_RESOURCE_UTILIZATION_STATISTICS
where HOST = ‘node02’ and SERVER_TIMESTAMP between ‘18.06.2017 00:00:01’ and ‘10.07.2017 23:59:59’
order by “Peak Used GB” desc

In-Memory Technology, sap-hana-admin, sap-hana-memory

As an example, I’ll focus on the top memory consumer happened on 1st July, between 10:00AM – 11:00AM

What were the top memory consumer during peak usage

Again, using script – HANA_Memory_TopConsumers_History_1.00.90+_ESS and adjusting the modification section:

( SELECT    /* Modification section */
TO_TIMESTAMP(‘2017/07/01 10:00:00’, ‘YYYY/MM/DD HH24:MI:SS’) BEGIN_TIME,
TO_TIMESTAMP(‘2017/07/01 11:00:00’, ‘YYYY/MM/DD HH24:MI:SS’) END_TIME,
‘node02’ HOST,
‘%’ PORT,
‘%’ SCHEMA_NAME,
‘%’ AREA,          /* ROW, COLUMN, TABLES, HEAP, % */
‘%’ SUBAREA,    /* ‘Row Store (Tables)’, ‘Row store (Indexes)’, ‘Row Store (Int. Fragmentation)’, ‘Row Store (Ext. Fragmentation)’, ‘Column Store (Main)’, ‘Column Store (Delta)’, ‘Column Store (Others)’ or ‘Heap (<component>)’ */
‘%’ DETAIL,                   /* Name of table or heap area */
‘ ‘ ONLY_SQL_DATA_AREAS,
‘ ‘ EXCLUDE_SQL_DATA_AREAS,
100 MIN_TOTAL_SIZE_GB,
‘USED’ KEY_FIGURE,            /* ALLOCATED, USED */
‘%’ OBJECT_LEVEL,         /* TABLE, PARTITION */
‘ ‘ INCLUDE_OVERLAPPING_HEAP_AREAS,
‘DETAIL’ AGGREGATE_BY,        /* SCHEMA, DETAIL, HOST, PORT, AREA, SUBAREA */
‘TS30’ TIME_AGGREGATE_BY     /* HOUR, DAY, HOUR_OF_DAY or database time pattern, TS<seconds> for time slice, NONE for no aggregation */
FROM
DUMMY

During this specific timeframe, Poo/itab was the top consumer followed by some large table/partition on the system. FYI, Poo/itab is a temporary SQL area to store column store intermediate search results.

In-Memory Technology, sap-hana-admin, sap-hana-memory

Since we’ve identify the top memory consumer, and now let’s find out what’s the thread/process running on that period.

What were the running processes/ threads during the peak usage?


Identify the running thread and process that possibly causing the high memory usage with script 
HANA_Threads_ThreadSamples_FilterAndAggregation_1.00.120+ 


SELECT                                                      /* Modification section */
TO_TIMESTAMP(‘2017/07/01 10:00:00’, ‘YYYY/MM/DD HH24:MI:SS’) BEGIN_TIME,
TO_TIMESTAMP(‘2017/07/01 11:00:00’, ‘YYYY/MM/DD HH24:MI:SS’) END_TIME,
‘node02’ HOST,
‘%’ PORT,
-1  THREAD_ID,
‘%’ THREAD_STATE,               /* e.g. ‘Running’, ‘Network Read’ or ‘Semaphore Wait’ */
‘%’ THREAD_TYPE,                /* e.g. ‘SqlExecutor’, ‘JobWorker’ or ‘MergedogMonitor’ */
‘%’ THREAD_METHOD,
‘%’ THREAD_DETAIL,
‘%’ STATEMENT_HASH,
‘%’ STATEMENT_ID,
‘%’ STATEMENT_EXECUTION_ID,
‘%’ DB_USER,
‘%’ APP_NAME,
‘%’ APP_USER,
‘%’ APP_SOURCE,
‘%’ LOCK_NAME,
‘%’ CLIENT_IP,
-1  CLIENT_PID,
-1 CONN_ID,
-1  MIN_DURATION_MS,
80  SQL_TEXT_LENGTH,
‘X’ EXCLUDE_SERVICE_THREAD_SAMPLER,
‘X’ EXCLUDE_NEGATIVE_THREAD_IDS,
‘X’ EXCLUDE_PHANTOM_THREADS,
‘NONE’ AGGREGATE_BY,                /* TIME, HOST, PORT, THREAD_ID, THREAD_TYPE, THREAD_METHOD, THREAD_STATE, THREAD_DETAIL, HASH, STATEMENT_ID, STAT_EXEC_ID, DB_USER, APP_NAME, APP_USER,APP_SOURCE, CLIENT_IP, CLIENT_PID, CONN_ID, LOCK_NAME or comma separated combinations, NONE for no aggregation */
‘MAX’ DURATION_AGGREGATION_TYPE,       /* MAX, AVG, SUM */
-1 RESULT_ROWS,
‘HISTORY’ DATA_SOURCE           /* CURRENT, HISTORY */

In-Memory Technology, sap-hana-admin, sap-hana-memory

From the result above we’ve identified that CHECK_TABLE_CONSISTENCY was the culprit of the high memory usage.

CHECK_TABLE_CONSISTENCY is an CPU and memory intensive job, especially for large column store tables the consistency check requires a lot of memory. This make sense as currently checking column tables (*MA01A2, *MP03A2, etc) are those of the largest tables and top consumers in the system.

From the investigation, I also learnt that Poo/itab was used to store the immediate result of the consistency check table and threads like “MakeHeapJob”, “ZipResultJob”, “search”, “ParallelForJob” were called to perform parallel sorting operation or parallel processing of input data during Table consistency check.

A practical example of ABAP on HANA optimization

$
0
0

A practical example of HANA optimization


In this blog, I’ll try to recap my journey through the land of SELECT’s and internal tables towards the realm of HANA. I wanted to take a slow-performing ABAP, not something from a tutorial or playground module from SAP, but something “real”, tangible, a proper “pain point”, and describe how it can be converted to using HANA view(s).

It became clear to me while writing up my experiences that this is not a task easily undertaken. First of all, ripping apart an existing ABAP – no matter how “simple” and “straightforward” it can appear at first sight, is not an easy job. Secondly, wrapping it all up in a comprehensible blog, complete with bullet-proof points and comprehensible screenshots, is aneven more difficult achievement. At times, I was not even sure if this was doable, or whether anyone could recap any learnings from what I had been through. Surely, there are more straightforward tutorial and examples out there?

But then again, why not? I finally decided that the only format I could give this document, is in the way of “key learnings” and general tips. I learned a lot, and hopefully I’ll be able to pass on some of the ideas I had along the way to others. I’m not going to discuss the original ABAP code in all its gory detail, this would possibly help you to understand the inner workings of the original program but would take ages. Instead, I hope to convey the general idea of the program to you, and show how you can translate it into HANA. I’ll explain the three view types, attribute, analytical, and calculation, and show you how I used them (and, more importantly, why I used them). I’ll also show some of the pitfalls and tips along the way, to help you avoid the same.

And, finally, this is my way of doing it. That’s a disclaimer, by the way, not a nod to Sinatra. There might be (and likely is!) better ways of solving the particular issue I had, and better ways of avoiding pitfalls. There might even be new issues arising from the way I implemented my solution  – feel free to comment. I’m here to learn as much as you are.

How to find a suitable candidate


Some people use speed dating, others register to various internet-based… oh, sorry. We were talking about HANA-optimization, not marriage prospects.

I found a nice program in dire need of being optimized by executing ST03N in a production system. ST03N is the workload monitor, sort of the “health display” of your Netweaver system. It has lots of cool features; among others is an option to look at the statistics for the most resource-demanding applications (or processes) running in that particular systems. Boy, was I happy when one of the top offenders turned out to be a custom-made ABAP! I quickly decided to rip that piece of underachieving code to pieces, replacing every sub-performing line with calls to some sleek & cool HANA views. I envisioned my re-worked ABAP/HANA hybrid development as a sleek Ferrari, blasting past the sluggish old ABAP original. And, of course, it would be a fun, easy-piecy job to do.

I would bitterly regret that decision.

Wow, it’s so…. BIG!!


Looking back, I can honestly say I made a huge mistake selecting this particular development for HANA optimization. Don’t get me wrong; I learned a lot along the way, but for a first-time project, something smaller and more manageable would have been far more suitable. I’ll repeat this in the key learnings session towards the end, but here goes: pick a development that spans 5-10 tables, not more. It’s far better to scale a small ridge first, before setting your sight on Everest.

And the problem with large ABAP developments (particularly old, somewhat chaotic ABAP developments) is that they’re unwieldy, erratic, chaotic, and obfuscated. Of course, this provides more opportunity for optimization, but you might just end up completely overwhelmed, and give up. Or have a heart attack. Or attempt to stick your laptop down the shredder (it won’t fit. Not even the newer, sleeker models with solid state drives. Trust me). A look at Mt. Everest from a distance of roughly 2.5 million milliseconds:





Get your SSH keys to HANA Express instance in Google Cloud Platform

$
0
0
You’ve been watching this SAP CodeTalk or following this tutorial on how to setup your SAP HANA, express edition, (or HXE) with the Google Cloud Launcher. Now you would like to connect to the instance from some SSH client, such as PuTTY or the console on your Mac or Linux computer.

You need the SSH keys, but where are they? If this is your first project and you have not generated your keys before, chances are that you won’t find them in your .ssh folder as this tutorial indicates:



SAP HANA Tutorials and Materials, SAP HANA Certifications

No need to panic. There’s just one more additional command before you can download your public and private keys.

Open a new gcloud console that is not logged in to the virtual machine using the “prompt” icon on the top-right corner:

SAP HANA Tutorials and Materials, SAP HANA Certifications

Take note of the name of your VM and use it to replace accordingly in the following command:

gcloud compute ssh <<YOUR VM NAME>>
It will ask for confirmation, issue a couple of warnings about not having any keys, ask for a passphrase (please use something you will remember) and finally confirm that your keys have been generated:

SAP HANA Tutorials and Materials, SAP HANA Certifications

Notice that it has logged you into the VM. In order to retrieve the keys, open a new gcloud console to fetch them from the project home folder:

SAP HANA Tutorials and Materials, SAP HANA Certifications

Go into your /home/<<google_user_id>>/.ssh and use the download icon to fetch your private and public SSH keys:

SAP HANA Tutorials and Materials, SAP HANA Certifications

Now you can connect from another console in your Mac or Linux computer using your key with:

ssh -i ~/.ssh/my-ssh-key your_gmail_user_name@external_vm_IP_address
or using PuTTY or WinSCP to transfer files from your Windows computer. More information on the latter or If you are getting an error like Unable to use key file “some path and private key” (OpenSSH SSH-2 private key) in this tutorial.

CDS view performance analysis using PlanViz in HANA studio

$
0
0
I am a newbie in CDS performance area and frequently I meet with some “weird” performance behavior which makes me really confused. After I consulted with performance experts, it turns out that all those “strange” behavior has their root cause and could be well explained – no strange at all but just works as designed! I will share with my finding and learning with you through this blog.

Note:


1. This blog is written based on the prerequisite that the Netweaver is connected to a HANA DB whose version listed below:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

2. The test CDS views I used in this blog are NOT part of SAP standard delivery, so you could NOT find them in any SAP system.

Test CDS views used in this blog


The hierarchy of test CDS views used in this blog is listed below.

1. CRMS4D_SALE_I and CRMS4D_SVPR_I are two item database tables which store the service order line item with type “Sales Item” and “Service Item”. The record numbers of these two tables are also listed in the picture.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

2. How the upper-most CDS view CRMS4V_C_ITEM_OPT2 is consumed:
According to different search criteria selected by end user, different SQL statement is populated dynamically:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

3. CRMS4V_I_ITEM_WO_STATUS_TEXT is just an union of two item database tables whose source code could be found from here.

4. CRMS4V_C_ITEM_OPT_TXT_DELAY: in search result UI, the status description is displayed:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

However in database table, the status value is modeled as checkbox,

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

on the other hand in CRM the status text is defined against status internal key with format I<four number digit>, so the CDS view CRMS4V_C_ITEM_OPT_TXT_DELAY is used to convert the boolean value to the expected internal I format, whose source code could be found from here.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

5. CRMS4V_C_ITEM_OPT2: consumed by ABAP code to serve the search request triggered from end UI.

How to do performance analysis using HANA studio planviz


1. switch on ST05 trace.

2. perform search against product id. I write a simple report to trigger the search from backend:

PARAMETERS: pid   TYPE comm_product-product_id OBLIGATORY DEFAULT 'AB0000000042',
            maxhit TYPE int4 OBLIGATORY DEFAULT 100.

DATA: lt_selection_parameter TYPE genilt_selection_parameter_tab,
      ls_query_parameters    TYPE genilt_query_parameters,
      ls_selection_parameter TYPE genilt_selection_parameter.
DATA(lo_core) = cl_crm_bol_core=>get_instance( ).
lo_core->load_component_set( 'ONEORDER' ).

ls_selection_parameter = VALUE #( attr_name = 'PRODUCT_ID' sign = 'I' option = 'EQ' low = pid ).
APPEND ls_selection_parameter TO lt_selection_parameter.

ls_query_parameters-max_hits = maxhit.

cl_crm_order_timer=>start( ).
TRY.
    DATA(lo_collection) = lo_core->dquery(
        iv_query_name               = 'BTQSrvOrd'
        it_selection_parameters            = lt_selection_parameter
        is_query_parameters                = ls_query_parameters ).
  CATCH cx_root INTO DATA(cx_root).
    WRITE:/ cx_root->get_text( ).
    RETURN.
ENDTRY.
cl_crm_order_timer=>stop( 'Search by Product ID name' ).
WRITE:/ |Number of Service Orders found: { lo_collection->size( ) }|.

3. Execute the search and deactivate the trace. Now the CDS view read operation could be found from the trace.

Edit->Display Execution Plan -> For Recorded Statement:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

4. In HANA studio, open this plv file:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

With plv file opened in HANA studio, all my previous doubt could be clarified.

All weird performance behaviors get clarified


With trace file available, all weird behavior could be well explained now.

1. different product ID leads to greatly different performance result

For product ID 3D0000000002, only 0.1 second is used to finish the query, while for product ID AB0000000042, 231 seconds is consumed.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials
SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

First open plv file for product ID 3D0000000002, the trace shows there are only 4123 records which fulfills the condition ORDERED_PROD = ‘3D0000000002’:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

This is consistent with what I have found in SE16:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

And for CRMS4D_SVPR_I, there are 20000 records whose product_id = ‘3D0000000002’. So after union, totally 4123 + 20000 = 24123 records are sent for upper process.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

finally these 24123 records are used to get 100 distinct records as result.
Now let’s turn to the trace file for product ID AB0000000042.
A huge number of records (182,272,424 ) are fetched from database table:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

So this search criteria does not make too much sense from business point of view – more search parameter is necessary to limit the records retrieved from database table for further process.

2. select vs select distinct

Execute below two statements in HANA studio seperately:
case 1: SELECT distinct “OBJECT_ID” FROM “SAPQGS”.”CRMS4VCITEMODL2″ WHERE “PRODUCT_ID” = ‘AB0000000042’ limit 100
case 2: SELECT “OBJECT_ID” FROM “SAPQGS”.”CRMS4VCITEMODL2″ WHERE “PRODUCT_ID” = ‘AB0000000042’ limit 100
Mark the SQL statement in HANA studio, “Visualize Plan->Execute”:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

The execution plan shows that the limit operation is done based on the huge number of records which leads to a comparatively poor performance – 1.1 seconds.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

In the case 2 where the distinct keyword is removed, due to the optimization of HANA execution engine, the limit operation is performed immediately during the time when the records are retrieved from database table, so only these 100 records are sent to subsequent process, which ends up with a comparatively good performance. This behavior is so called “limit push down”, which is not supported if distinct keyword is involved.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

3. Performance Gap between ST05 trace and the planViz opened in HANA Studio

I have once observed a “strange” behavior:

When I perform the query on my CDS view with two search parameters: product id and sold to party name, it will take on average 16~18 seconds to finish the query

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

When I download the trace file, open and execute it in HANA studio,

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

Only around 2 second is consumed in HANA studio.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

What makes this big difference when CDS view is consumed in ABAP and HANA layer with exactly the same SQL statement?
In fact the statement is not exactly the same at all.
In ABAP layer, the limit is specified dynamically – not fixed in the design time.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

However in HANA studio, the limit 100 is hard coded.

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

When I change the dynamically specified limit operation in ABAP with fixed limit, the performance in ABAP is now consistent with HANA studio:

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

SAP HANA Studio, SAP HANA Certifications, SAP HANA Tutorials and  Materials

The reason is, it is possible for HANA optimizer as a kind of cost-based optimizer to apply PRELIMIT_BEFORE_JOIN rule to constant LIMIT operator during execution plan generation for the non-prepared statement. However due to technical reasons, it is not possible for parameterized LIMIT operator, since it is impossible for HANA optimizer to estimate the cost of the statement containing parameterized LIMIT operator and decide the optimal plan based on the estimated cost – the optimization could not be applied unless we are well aware of how many records could be pruned with LIMIT operator.

A practical example of ABAP on HANA optimization

$
0
0

A practical example of HANA optimization


In this blog, I’ll try to recap my journey through the land of SELECT’s and internal tables towards the realm of HANA. I wanted to take a slow-performing ABAP, not something from a tutorial or playground module from SAP, but something “real”, tangible, a proper “pain point”, and describe how it can be converted to using HANA view(s).

It became clear to me while writing up my experiences that this is not a task easily undertaken. First of all, ripping apart an existing ABAP – no matter how “simple” and “straightforward” it can appear at first sight, is not an easy job. Secondly, wrapping it all up in a comprehensible blog, complete with bullet-proof points and comprehensible screenshots, is aneven more difficult achievement. At times, I was not even sure if this was doable, or whether anyone could recap any learnings from what I had been through. Surely, there are more straightforward tutorial and examples out there?

But then again, why not? I finally decided that the only format I could give this document, is in the way of “key learnings” and general tips. I learned a lot, and hopefully I’ll be able to pass on some of the ideas I had along the way to others. I’m not going to discuss the original ABAP code in all its gory detail, this would possibly help you to understand the inner workings of the original program but would take ages. Instead, I hope to convey the general idea of the program to you, and show how you can translate it into HANA. I’ll explain the three view types, attribute, analytical, and calculation, and show you how I used them (and, more importantly, why I used them). I’ll also show some of the pitfalls and tips along the way, to help you avoid the same.

And, finally, this is my way of doing it. That’s a disclaimer, by the way, not a nod to Sinatra. There might be (and likely is!) better ways of solving the particular issue I had, and better ways of avoiding pitfalls. There might even be new issues arising from the way I implemented my solution  – feel free to comment. I’m here to learn as much as you are.

How to find a suitable candidate


Some people use speed dating, others register to various internet-based… oh, sorry. We were talking about HANA-optimization, not marriage prospects.

I found a nice program in dire need of being optimized by executing ST03N in a production system. ST03N is the workload monitor, sort of the “health display” of your Netweaver system. It has lots of cool features; among others is an option to look at the statistics for the most resource-demanding applications (or processes) running in that particular systems. Boy, was I happy when one of the top offenders turned out to be a custom-made ABAP! I quickly decided to rip that piece of underachieving code to pieces, replacing every sub-performing line with calls to some sleek & cool HANA views. I envisioned my re-worked ABAP/HANA hybrid development as a sleek Ferrari, blasting past the sluggish old ABAP original. And, of course, it would be a fun, easy-piecy job to do.

I would bitterly regret that decision.

Wow, it’s so…. BIG!!


Looking back, I can honestly say I made a huge mistake selecting this particular development for HANA optimization. Don’t get me wrong; I learned a lot along the way, but for a first-time project, something smaller and more manageable would have been far more suitable. I’ll repeat this in the key learnings session towards the end, but here goes: pick a development that spans 5-10 tables, not more. It’s far better to scale a small ridge first, before setting your sight on Everest.

And the problem with large ABAP developments (particularly old, somewhat chaotic ABAP developments) is that they’re unwieldy, erratic, chaotic, and obfuscated. Of course, this provides more opportunity for optimization, but you might just end up completely overwhelmed, and give up. Or have a heart attack. Or attempt to stick your laptop down the shredder (it won’t fit. Not even the newer, sleeker models with solid state drives. Trust me). A look at Mt. Everest from a distance of roughly 2.5 million milliseconds:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

This, then, is the offending development; a seemingly inconspicuous report that calculates transportation costs. As can be seen, this is the third most charging transaction/program, measured in processing time as seen above. The screenshot is from a Productive environment.

I decided to look closer at the performance. Unable to do a proper runtime analysis in a production environment, I had to resort to a QA system, where data volumes were far lower. Still, I found some promising details:

When looking at the Hit list, we see that a lot of time is spent doing unit conversions:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Quite a bit of time is spent calling functions for unit conversions. This is actually one of HANA’s quick wins: you can avoid calling unit or currency conversions bozillions of times, simply by letting the HANA engine handle it.

So far, so good. I had my candidate.

How not to do things.
My first mistake was to go through the ABAP code from START-OF-SELECTION onwards, slashing out at each and every SELECT statement with my HANA machete, attempting to replace them all with HANA views, one by one. Well, not actually one by one, but more like merging related tables into attribute views, then collating these into analytical views, sort of hoping that I’d reach the summit before sunset. As by magic. This, of course, didn’t work. As any half-decent mountaineer knows, a bit of planning always pays off. Also, when it comes to old, grumpy ABAP’s, it’s not always obvious how a
development works.

As programs go, this one was actually quite well structured. Sort of. It consisted of roughly 15 FORM’s, where data was being selected into internal tables, followed by one final FORM merging everything together into the final display table used for the ALV. The problem was that the top-down approach which I initially started out with, failed to take into account everything that went on in the final FORM. Here, a LOOP at the “main” internal table (containing shipments, from tables VTTK/VTTP), contained READ statements for other internal tables (such as deliveries). By looking at these READ’s, I was shocked to see that in some cases, a missing corresponding entry in a secondary internal table would result in a CONTINUE statement.

Say what?

Consider the following:

LOOP at outer_table into work_area.

       READ inner_table WITH KEY so_and_so_from_outer_table BINARY SEARCH.

             IF sy-subrc <> 0.

                  CONTINUE.

             ENDIF.

             MOVE bunch_of_fields to final_internal_table.

ENDLOOP.

What happens here is that whenever there’s no corresponding entry in the inner table, the LOOP continues. What it really is, is an INNER JOIN on the two tables. Only if both contain corresponding entries, is the data collated into the resulting final table.

And this, of course, translates nicely into HANA. Problem is, you won’t “get it” until you arrive at that final piece of logic, and by then, you realize you have to re-structure your whole suite of HANA views. Or at least some of them.

But let’s not jump to conclusions. Let’s start with the beginning.

The original program structure


I mentioned the original program selected data from a range of tables, stored this in internal tables, then collating everything into a final table destined for ALV fame. This is a pattern followed by countless top-down reporting programs, and one which should – at least theoretically – be fairly simple to convert to HANA views, at least some portions of it.

I’m not going to go into lengthy (and boring) details about the inner workings of the program. I actually thought of doing so, but rejected the idea after a feeble attempt to recap all FORM’s and each and every SELECT. Instead, I’ll show you the main structure, since this is by far the easiest way to understand what’s going on. It also helps to comprehend why I structured my views the way I did. Here goes:
  • SELECT Shipments (headers and items) from VTTK and VTTP
  • SELECT Deliveries from LIKP for all the shipments
  • Do some unit conversions on a variety of fields
  • SELECT various other values from various lookup tables, for shipments and deliveries
  • SELECT some other aggregated values from a range of interrelated tables, based on already selected shipments or deliveries
  • Merge everything (which is currently stored in internal tables) into a final result table by LOOPing at the main shipment internal table and doing READ TABLE for everything else
  • Display the resulting internal table as an ALV
The above looks simple enough. The problem is we are talking roughly 15 different FORMs, each with selections from 1-2 specific DB tables. Some of the resulting internal tables are aggregations, not directly related to the shipments or deliveries, but based on other terms. This complicates a bit.

The full list of FORMS looks like this:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

As this is not a step-by-step blog outlining every single change made when transforming the above ABAP to HANA, I’ll instead focus on some key steps and learnings. The above should nevertheless give a decent impression of what the original development does, and how it selects its data.

Step 1: Start by looking at the final output.

What does the original program do? More often than not, the best way of understanding this is by looking at the final ALV table or output. It usually gives you a general idea on how to structure your main view, which will usually be an analytical or (more often) calculation view. In our case, the final output shows “shipments with related deliveries and some other related info”. The main LOOP in the final FORM that builds this internal table, is on an internal table containing data from VTTK and VTTP. This actually provides a nice tip on how your “data foundation” should look like. In general, the “main” tables of the final output are decent candidates for the “data foundation” of an analytical view. The “lookup” tables will be the JOINed tables of the same analytical view, or form part of their own analytical views if you need to aggregate anything.

By looking at the program structure above, we can already deduce that VTTK-VTTP will form a major part of the “data foundation”, likely of an analytical view. Likewise, since there is a direct relation to deliveries (table LIKP) we may be able to include LIKP in the data foundation of our main view.

And, before you ask: how do we know our main view will be an analytical view? Because we probably have to aggregate something. Usually, there’s no point outputting this kind of info without displaying some SUMmed quantities or amounts, like total delivery quantity per shipment, or total billed amount per customer. So, it’s fair to assume we’l be building at least one analytical view, somewhere along.

Of course, we’ll possibly end up needing a calculation view in the end. This is also a simple deduction: if we need more than one analytical view, we’ll need to JOIN these in a… calculation view, simply because you can’t JOIN an analytical view inside another analytical view. Too bad, really.

Step 2: create attribute views for simple hierarchies (including single tables)

Often, you will see simple structures like header-item, or a limited number of tables related to each other using referential joins, where we just need simple fields as output, without any measures (aggregations). These can easily go into attribute views. The same goes for single tables. The reason for creating attribute views for single tables is simple: this allows you to include then in analytical views, using JOINs. Other benefits of using attribute views are that you can set filters, and restrict the number of fields selected.

Example 1: attribute view for one table, filter set on SPRAS:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Example 2: attribute view including several tables

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Based on my program logic, I ended up with the following list of attribute views:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Most of these contain one single table. Also, most of them have a filter value on MANDT and SPRAS (I could also have defined input parameters – however, I came across an interesting “feature” regarding the client field – MANDT – more about that below).

Step 3: create analytical views for aggregations, even within one table

Now, the time has come for creating analytical views for anything that is, um, aggregated.

Example 1: analytical view on LIPS-LFIMG (delivery quantity):

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

This view simply aggregates the delivery item quantity LFIMG for all delivery items belonging to the same delivery number (VBELN). It contains one table. LFIMG is set to a measure in the Semantics section.

Example 2 : analytical view on VFKP-VFKN which aggregates NETWR on REBEL:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

This is a view involving two tables, VFKP and VFKN. Here, the measure NETWR is accumulated per REBEL (VFKP-REBEL, not VFKN-REBEL which is visible in the screenshot. We also need VFKN-REBEL since it is referenced in another view, hence showing).

Example 3: the “main” analytical view, combining all my attribute views with a multi-table “data foundation”:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Here, we see several tables in the analytical view’s Data Foundation. Do not make the misconception that a data foundation can only contain one table. In the above view, the data foundation comprises of four tables:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

This foundation is then JOINed with the attribute views to provide the backbone of the main result table.

We also see some calculated fields:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

This, by the way, is one of the staples of HANA – the ability to do currency and unit conversions “on the fly”.

Step 4: finish by creating calculation views – if necessary.

Calculation views are needed if you have created several analytical views, and need to join these, such as in my example. I already have a “main” analytical view (above), and also two others, since I needed to aggregate various quantities for deliveries etc.

This means I cannot simply stay with my analytical “main” view, since analytical views only accept attribute views in the JOIN section…

My resulting calculation view looks like this:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Here, I start with my main view, containing my “main loop” tables and the attribute views. I then JOIN a second analytical view, Z_AN_DEL_QUAN, to get delivery quantities aggregated by delivery number. So far, so good. In my second JOIN, I add another attribute view, showing a fairly complex constellation of tables:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

The reason for this is more for simplicity. The above collection of tables deserved their own attribute view, and the JOIN happens in the calculation view (instead of joining into the “main” analytical view) because it provides a better way of structuring the views. Less clutter, more readability.

The final JOIN (“Join_3”) sees the result of the two previous JOINs being joined to the last analytical view:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

As can be seen, the relationship is based on two fields, FKNUM and VBELN.

Step 5: Validate along the way.

This is something you should do with all views; look at the raw data output and try to spot any issues.

Step 6: create an ABAP external dictionary view

One thing to know about creating external views for ABAP, is that the mapping suggestions are not 100% correct for some fields, notably date, time, currency and quantity fields. These will have to be mapped manually (below screenshot shows some of them, but not all):

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Use the original DD tables in the ABAP dictionary when correcting the mapping proposed by the HANA Studio, which will usually be CHAR.

Another thing to beware of is that HANA has an ambiguous attitude towards reference fields (at least as of SP6):

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

It turns out we haven’t specified the reference fields for the amounts and quantities in our analytical view. However, there is no obvious way of doing this… a post elsewhere on SCN seems to suggest there is currently no way of handling this. Thus, we cannot use QUAN fields in external ABAP views for the time  being – all our quantity and currency fields have to be set as DECIMAL.

Wrapping up: calling the external view from ABAP


Finally, at the end of a lengthy process, we can swap all the FORM’s mentioned at the start with a simple SELECT on our newly defined
view:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

How immensely rewarding!

Performance analysis


For the performance analysis, transaction SAT was run for both the developments (original and HANA view-based version). A few pre-runs were done immediately before the measurements to fill buffers etc.

Result:

Original development:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

New HANA-based development:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

We see that the run time of our new development is roughly 13% of that of the original. Quite a nice improvement. Our system contains very limited data volumes, and higher-volume environments might show different savings – perhaps even greater.

Key learnings

And here, finally, I’ll list a few of the key learnings gathered during this exercise. This is not an exhaustive list, but hopefully useful…
  • Revise the original ABAP code first; there might be issues of bad design (refactoring needed), sub-optimal performance (bad SELECT’s, badly designed LOOP logic, wrong type of internal tables etc.), plus subsequent modifications may have left the ABAP code suffering in other ways (I came across an internal table that was no longer used – the code reading it was commented – but the original SELECT was still active… a second case involved exhaustive unit conversions which were then never displayed in the final output list. Somewhere down the line, they had been omitted – but the call to the conversion functions was still active…)
  • Simplify HANA view creation by starting with attribute views, then analytical views, then (if needed) calculation views. Do not attempt to make the views too complicated – it’s better to split logic up into several views. I don’t have performance data to back this up, but the application will be easier to maintain – also if you have to revise and re-test a specific view.
  • As for view types: you’ll need attribute views for single tables and simple relations, as long as there are no aggregations involved. Analytical views for aggregating data (think: sums). And calculation views if you need to JOIN one analytical view to at least one other.
  • The DATA Foundation of your analytical view can accommodate more than one table. In fact, it’s perfectly OK to JOIN tables to your “main” data foundation table, directly in the “data foundation” itself – instead of joining them in the “Logical join”.
  • Do iterative testing – use the data preview in HANA to verify the output of each view separately, instead of waiting until everything is assembled in a final calculation view.
  • Take note that MANDT is not a “forgettable” field, like in ABAP. Here is an example:
View Z2_AT_ROUTE_DESC without filter on MANDT:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Data preview:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

We see duplicates. This is regardless of the fact that the session client has been defined in the view’s Semantics:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Setting the default client to “103” resolves the issue:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide

Data preview:

SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guide
  • Make sure to distinguish about Attributes and Measures – quantities and amounts could come out wrongly  if this is erroneously set in the Semantics section. For instance, if you have a calculation view with more than one JOIN section, specifying quantities/amounts as “measures” everywhere can end up giving erroneous sums – based on your joins. If a quantity or amount is a measure in an originating view, it should be defined as an attribute if the purpose is to just keep the original figure through all your joins.
  • Don’t rely on being able to define reference fields for external (ABAP) views in HANA (using QUAN or CURR fields) – it doesn’t work, even though the error messages say you have to do it. You’ll have to resort to specifying these as DECIMAL instead.
  • Bring enough torches when entering a cave!

How To Create AMDP With Parameters In ABAP Modular Perspective In SAP HANA

$
0
0

What are AMDPs…


ABAP Managed Database Procedures are a new feature in AS ABAP allowing developers to write database procedures directly in ABAP. You can think of a Database Procedure as a function stored and executed in the database. The implementation language varies from one database system to another. In SAP HANA it is SQL Script. Using AMDP allows developers to create and execute those database procedures in the ABAP environment using ABAP methods and ABAP data types.

Advantage of AMDP Process


  • The main advantage of this method that only the AMDP class has to be transported wit ABAP transport mechanism.
  • No HANA delivery or HANA transport system is required in this process.
  • Developers only need ABAP development tools for building and managing the CDS view. There is no need for additional HANA development tools.

Features of ABAP Managed Database Procedure (AMDP)

  • Static check code and Syntax colouring are provided for embedded SQLScript
  • The user can set a Background colour for better visibility AMDP methods in the class.
  • The User can access other AMDP methods, ABAP dictionary view and ABAP tables in AMDP method.
  • AMDP method are called like other regular ABAP methods.
  • User can perform detailed analysis of various error during runtime in transaction ST22
  • Modification or Enhancement of regular ABAP classes can be done by Users.

Example of AMDP Class Definition


CLASS CL_AMBP_EXAMPLE DEFINITION.

PUBLIC SECTION.
INTERFACES IF_AMDP_MARKER_HDB. //Marker Interface for HANA DB//

METHODS process //Only ABAP code is possible//
IMPORTING it_param TYPE type1
EXPORTING et_param TYPE type2.

METHODS execute //SQLScript or ABAP code both are possible//
IMPORTING VALUE(it_param) TYPE type1
EXPORTING VALUE(et_param) TYPE type2. //specific parameter interface required//
CHANGING VALUE(ch_param) TYPE type3

ENDCLASS.

AMDP Class Implementation


CLASS CL_AMDP_EXAMPLE IMPLEMENTATION

METHODS process
// Write ABAP source code here//

ENDMETHOD

METHOD execute BY DATABASE PROCEDURE //AMDP method marker//
FOR HDB //Database platform//
LANGUAGE SQLScript //Database language//
[OPTIONS READ-ONLY] //Database-specific options//
USING name1 name2 etc… //List of used DDIC entities and AMDPs//

//Write here the SQLScript coding//
select * from dummy;

ENDMETHOD.
ENDCLASS.

Lets Take one Example: Creating AMDP with Input, Output Parameters


First Go to ABAP modeling perspective in HANA Studio

Windows->Perspective->Open Perspective->ABAP




Create ABAP class:


CLASS ztestk DEFINITION public.

PUBLIC SECTION.

types : tt_mara type table of mara.

interfaces : if_amdp_marker_hdb.

methods : my_method

importing value(im_matnr) type mara-matnr

exporting value(et_mara) type tt_mara.

ENDCLASS.

CLASS ztestk IMPLEMENTATION.

method my_method by database procedure for HDB

language sqlscript options read-only using MARA.

et_mara=SELECT * from MARA where MATNR= IM_MATNR;

endmethod.

ENDCLASS.

Save It(Control+S)

Open SAP GUI

Enter TCODE : SE24 (To see u r class is created or not)

We can see our methods and code also.


Click on Display


Enter TCODE : SE38 (To Create report )

In Report we are calling class (creating object to class)


Click on create



Type u r code here to call class

REPORT ZTESTK_REP.
PARAMETERS : p_matnr TYPE matnr DEFAULT ‘000000000000001109’.

DATA : r_amdp TYPE REF TO ztestk,
et_mara TYPE TABLE OF mara,
r_salv TYPE REF TO cl_salv_table.

CREATE OBJECT r_amdp.

r_amdp->my_method( EXPORTING im_matnr  = p_matnr
IMPORTING et_mara    = et_mara ).

TRY.
CALL METHOD cl_salv_table=>factory
IMPORTING
r_salv_table   = r_salv
CHANGING
t_table        = et_mara
.
CATCH cx_salv_msg .
ENDTRY.

r_salv->display( ).

——————————————
Finally press F8 Button to execute

Simplified security management in SAP HANA 2.0 SPS02

$
0
0
Managing secure data access and configuring systems securely are critical tasks in operating business systems today. SAP HANA has security built into its core, with a comprehensive framework and tooling for authentication and single sign-on, authorization and role management, user and identity management, audit logging, secure configuration and encryption.


With the new SAP HANA 2.0 SPS 02 release, new features were added that significantly simplify security configuration and management:
  • Easily manage different sets of users by assigning them to user groups with separate exclusive group administrators
  • Simplified encryption configuration, monitoring and key lifecycle management in SAP HANA cockpit

User Groups


With user groups you can manage related users together. Dedicated group administrators can be assigned to manage individual user groups exclusively and independently of each other. For example, you might want to group users by department, and assign dedicated group administrators to manage the users within their own department.

In the example scenario below, three user groups have been created for the Sales, Research, and Training departments. The user groups were set up for exclusive administration by dedicated group admins. So what are the group admins allowed to do? Let’s have a look at the group admins for the Sales user group
  • Only the group admins for the restricted user group Sales can create or delete users in this user group, and manage security properties of the Sales users.
  • The Sales group admins can only manage users of the Sales group, but not users from other user groups like Research or Training.

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

Encryption Management in SAP HANA Cockpit


The encryption configuration and monitoring capabilities of SAP HANA cockpit have been significantly enhanced.

As you know, SAP HANA offers comprehensive features for encrypting data at rest and in motion, including data volume encryption, log encryption, backup encryption, application encryption, and communication encryption (SSL/TLS). SAP HANA encryption features use SAP standard cryptographic library CommonCryptoLib, which is FIPS-certified.

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

Configuring encryption is now much easier: in SAP HANA cockpit, you can now enable data volume encryption, log encryption and backup encryption with the click of a single button:

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

You can drill down to more detailed information, for example to view when the configuration or the encryption keys were last changed:

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

Key management is the cornerstone of all encryption implementations. A whole new app has been added in SAP HANA cockpit for managing encryption keys, which gives you detailed information for example on the current key versions in use or the dates of the last key changes.

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

Tasks relevant for the life-cycle management of encryption keys are now also accessible from here, for example changing and securely backing up encryption keys.

SAP HANA Tutorials and Materials, SAP HANA 2.0, SAP HANA Security

What else?


These are just the highlights of new security features added with SAP HANA 2.0 SPS 02, but there is more!

Additional features have been added in the area of LDAP group authorization and security hardening of multi-container systems (multitenancy), and last but not least, in  secure application development where an SQLScript code scanner is now available.

SAP HANA 2.0 SPS 02: New Developer Features; Database Development

$
0
0
In this blog, I would like to introduce you to the new features for the database developer in SAP HANA 2.0 SPS 02.  We will focus on the database development topic including Core Data Services, as well as SQLScript.


Core Data Services(CDS)



Comments

The first new feature for CDS is the ability to use comments in your CDS source file.  This is done by using the @Comment annotation which can be placed anywhere in your source code.
Core Data Services(CDS), sap hana 2.0, sap hana database
Column level comments are transferred to the run-time objects in the catalog as well.  So if you were to open a table in Database Explorer, and look at the comments column, you would find the comment for your column.

Core Data Services(CDS), sap hana 2.0, sap hana database

Select from Table Functions

The next feature for CDS is the ability to select from a table function when defining a view in your CDS artifact.  As you can see in this example, we have defined a view which selects from the BusinessPartner table and does an inner join with a table function called getPartnerRoles

Core Data Services(CDS), sap hana 2.0, sap hana database


SQLScript


Procedure/Function Encryption

The first new feature for SQLScript is encryption.  Customers and Partners wanting to protect their IP have been asking for this for quite some time.  We use the WITH ENCRYPTION clause in the CREATE statement to flag the procedure or function as encrypted.  If a procedure or a function is created by using the WITH ENCRYPTION option, their definition is saved as an encrypted string that is not human readable. That definition is decrypted only when the procedure or function is compiled. The body in the CREATE statement is masked in various traces, monitoring views and the object definition in the catalog. Decryption of an encrypted procedure is not supported and not even be done by SAP.  Encrypting a procedure or function means that you will also lose supportability features.  For example, you will not be able to debug an encrypted procedure or function, nor will you be able to use PlanViz, traces, or any other supportability tool that can reveal the procedure or functions code.

Core Data Services(CDS), sap hana 2.0, sap hana database
Core Data Services(CDS), sap hana 2.0, sap hana database

CREATE OR REPLACE

The next new feature for SQLScript is CREATE OR REPLACE.   When creating a SQLScript procedure or function, you can use the OR REPLACE option to change the defined procedure or function if it already exits. There are some restrictions with this, for example, the signature of the procedure or function can not be changed.  Also, the security mode can not be changed.  Lastly, you can create a new header-only procedure of function, or you can define a body for an existing header-only procedure or function, but you can not replace an existing procedure or function with a header-only procedure of function.

Core Data Services(CDS), sap hana 2.0, sap hana database

System Variable for Line Number

The next feature is a system variable for line number.  In some cases, a developer requires to know the line number of the statement within a procedure/function which is currently being executed.  You can now use the system variable ::CURRENT_LINE_NUMBER to capture this.

Core Data Services(CDS), sap hana 2.0, sap hana database
Core Data Services(CDS), sap hana 2.0, sap hana database

Enhancements to INSERT Operator on Table Variables

In SAP HANA 2.0 SPS 01, we introduced the new table operators for table variables. This included operators such as INSERT, UPDATE, and DELETE.  At the time, when using the INSERT operator, you could only insert 1 row at a time.  In SPS 02, we’ve enhanced the INSERT operator to allow the possibility to insert many rows at one time.  As you can see in this example, we can now insert a set of rows from another table variable into a table variable or export parameter.

Core Data Services(CDS), sap hana 2.0, sap hana database

Parallel Execute Blocks for SELECT

We have supported parallel execution blocks for some time, but the statements allowed within these blocks were limited to INSERT, UPDATE, and DELETE DML statements.  SELECT was not allowed.  Now in SAP HANA 2.0 SPS 02, we now allow the use of SELECT statements in a parallel execution block.

Core Data Services(CDS), sap hana 2.0, sap hana database

SLEEP/WAKE Functions

Next, we have the SLEEP/WAKE functionality.  These new built-in procedures are part of the new SYS.SQLSCRIPT_SYNC library.  We are in the process of introducing a new library concept in general and this library is the first to be released for customer use. The plan for the next support packages is to add more and more built-in procedures and functions for customer use.  We also are looking to allow customers/partners to build their own reusable libraries as well.  But for now, we have the SLEEP_SECONDS procedure which allows you to put your current execution to sleep for a given number of seconds without burning CPU, and the WAKEUP_CONNECTION procedure which will wake the sleeping sessions.   In order to use these procedure, you must use the USING clause in your BEGIN statement block as well.

Core Data Services(CDS), sap hana 2.0, sap hana database

Core Data Services(CDS), sap hana 2.0, sap hana database


SQLScript Code Analyzer

Finally, we have the SQLScript Code Analyzer.   The SQLScript Code analyzer consists of two built in procedures that scan the course code of procedures and functions.   By applying a selected set of rules, they search for certain patterns that indicate problems regarding code quality, performance, or security.  The rules are defined in the SQLSCRIPT_ANALYZER_RULES table, and currently there are 4 rules.  We do plan to add more rules over time.  The SQLScript Code Analyzer is currently deactivated by default. It can be activated with an ini parameter or a session parameter:

alter system alter configuration ('indexserver.ini','SYSTEM') set

    ('sqlscript', 'enable_code_scanner') = 'true' with reconfigure;

set '__SQLSCRIPT_ENABLE_CODE_SCANNER' = 'true';

Again, there are two procedures which can be used to scan. The first procedure is the ANALYZE_SQLSCRIPT_DEFINTION which is used to analyze the source code of a single procedure or function which has yet to be created.  This procedure expects 2 input parameters, a string which contains the DDL of the procedure or function, and the list of rules to be applied.  It then returns 1 output parameter containing the findings.

Core Data Services(CDS), sap hana 2.0, sap hana database

Core Data Services(CDS), sap hana 2.0, sap hana database

The other procedure we can use is ANALYZE_SQLSCRIPT_OBJECTS which is used to analyze the source code of multiple procedures or functions that already exist in the catalog.  This procedure expects 2 input parameters, a list of objects which are to be scanned, and the list of rules.  It then returns 2 output parameters, a list of object which were scanned, and of course the findings.

Core Data Services(CDS), sap hana 2.0, sap hana database

As mentioned, there are currently 4 rules that are defined.  Below is the list of rules and their description.
  • UNNECESSARY_VARIABLE: Each variable is tested if it is used by any output parameter of the procedure or if it influences the outcome of the procedure. Relevant statements for the outcome could be DML statements, implicit result sets, conditions of control statements.
  • UNUSED_VARIABLE_VALUE: If a value assigned to a variable is not used in any other statement, the assignment can be removed. In case of default assignments in DECLARE statements, the default is never used.
  • UNCHECKED_SQL_INJECTION_SAFETY: Parameters of string type should always be checked for SQL injection safety if they are used in dynamic SQL. This rule checks if for any such parameter the function is_sql_injection_safe was called. For a simple conditional statement like IF is_sql_injection_safe(:var) = 0 THEN…, the control flow in the true branch is checked. The procedure should either end (by returning or by throwing an error) or the unsafe parameter value should be escaped with the functions escape_single_quotes or escape_double_quotes, depending on where the value is used. If the condition is more complex (e.g. more than one variable checked in one condition), a warning will be displayed, as it could only be checked if any execution of the dynamic sql has passed the sql injection check.
  • SINGLE_SPACE_LITERAL: This rule searches for string laterals consisting of only one space. If ABAP VARCHAR MODE is used, such string literals are treated as empty strings. In this case CHAR(32) can be used instead of ‘ ‘.

SAP HANA 2.0 SPS02 new feature: updated python driver

$
0
0

Introduction


As background: there have been two HANA python drivers available for some time:
  • The python driver included with the SAP HANA Client package (hdbcli). The client package included a distribution of python 2.7 and the driver works only with that distribution.
  • The python driver available on github (pyHDB). This driver is implemented in pure python, and worked with both python 2 and 3, but is not fully supported by SAP.
In SAP HANA SPS02, the hdbcli driver has been updated and now provides the following new features:
  • support for python 2.7 and python 3.4+.
  • installable into local python environments using the pip installer
  • the driver is built on the same C++ native library used by ODBC, ADO.NET, and other interfaces. As a result it provides all the protocol support built into that library, including support for Active/Active database configurations, network compression, and more.
  • improved support for calling stored procedures.
  • fully supported as part of the SAP HANA Client distribution.
This blog post walks you through the usual Getting Started tasks, including install, establishing a connection, executing SQL DDL commands, and handling result sets.

Installing the python driver


There are two steps to installing the python driver:
  1. Install the SAP HANA Client package on your computer. 
  2. Install the python driver from the HANA Client install into a local python environment.

Installing the python driver into a local python environment


You must have a python distribution installed on your local machine. Some python distributions allow you to create multiple independent virtual environments, each with its own version of python and its own set of packages. You can install the python driver into any local python environment.

From a command line:
  1. If you have multiple python environments, activate the python environment into which you wish to install the python driver. If you don’t have multiple python environments, just ignore this step.
  2. Confirm you have a supported python version (2.7, or 3.4+) by typing the following at a command prompt to show the python version:
  • python -V
  • Install the HANA Client package into your python environments using the pip installer.
    1. Here is a command for the default installation on Windows. The particular version of the zip file will probably be different:
    • pip install “C:\Program Files\SAP\hdbclient\hdbcli-2.N.N.zip”
  • Here is a command for the default installation on Linux.
    • pip install /full/path/to/file/hdbcli-2.N.N.tar.gz
    Now confirm you have the software properly installed. From a command prompt, type python to start a python interpreter. Then enter the following to confirm the package is properly installed:

    >>> from hdbcli import dbapi
    >>> dbapi.__name__
    'hdbcli.dbapi'

    All the code samples here are shown as if executed from a python interpreter.

    Connecting to a database


    Here is sample python code showing a database connection:

    >>>from hdbcli import dbapi
    >>>conn = dbapi.connect(
        address="<hostname>",
        port=<3NNMM>,
        user="<username>",
        password="<password>"
    )
    • For HANA tenant databases, you can use the port number 3NN13 (where NN is the SAP instance number).
    • For HANA system databases in a multitenant system, the port number is 3NN13.
    • For HANA single-tenant databases, the port number is 3NN15.
    For details of connection class methods, see <SAP Documentation Link>

    Executing SQL statements


    All SQL statements are executed using the cursor object. Here is how to execute a single statement that does not return a result set:

    >>> cursor = conn.cursor()
    >>> cursor.execute("CREATE TABLE T1 (ID INTEGER PRIMARY KEY, C2 VARCHAR(255))")
    >>> cursor.close()

    Binding parameters for SQL statements


    For statements that take a parameter, it is good practice to bind the parameter to a variable rather than to concatenate a string and execute it. The HANA client driver supports both the question mark and named styles of parameter binding as described in the DBAPI specification.

    Question mark parameter binding


    Here is an insert statement using the question mark parameter style. The statement inserts values into the T1 table created above. The parameter values are supplied as a python sequence, and may be literal values or variable names. This example uses literal values:

    >>> sql = 'INSERT INTO T1 (ID, C2) VALUES (?, ?)'
    >>> cursor = conn.cursor()
    >>> cursor.execute(sql, (1, 'hello'))
    True
    >>> cursor.execute(sql, (2, 'hello again'))
    True
    >>> cursor.close()

    Named parameter binding


    Here is another insert using the named parameter style. This time the values are supplied as a python dictionary, and in this case the example uses variable names.

    >>> sql = 'INSERT INTO T1 (ID, C2) VALUES (:id, :c2)'
    >>> cursor = conn.cursor()
    >>> id = 3
    >>> c2 = "goodbye"
    >>> cursor.execute(sql, {"id": id, "c2": c2})
    True
    >>> cursor.close()

    Querying data and handling result sets


    There are several ways of handling result sets, useful in different circumstances. Here is a simple way to loop over the rows of the result set.

    >>> sql = 'SELECT * FROM T1'
    >>> cursor = conn.cursor()
    >>> cursor.execute(sql)
    True
    >>> for row in cursor:
    ...    print(row)
    (1, 'hello')
    (2, 'hello again')

    If you know the result will have a single row, then cursor.fetchone() is useful

    >>> id = 2
    >>> sql = 'SELECT * FROM T1 WHERE ID = :id'
    >>> cursor = conn.cursor()
    >>> cursor.execute(sql, {"id": id})
    True
    >>> row = cursor.fetchone()
    >>> print(row)
    (2, 'hello again')

    If the result set is not too large, you may want to read it into a python variable in one step using cursor.fetchall().

    >>> sql = 'SELECT * FROM T1'
    >>> cursor = conn.cursor()
    >>> cursor.execute(sql)
    True
    >>> result_set = cursor.fetchall()
    >>> print(result_set)
    [(1, 'hello'), (2, 'hello again'), (3, 'goodbye')]

    Calling stored procedures


    You can get results from stored procedures in two ways: through output (or inout) parameters, or through returned results. In both cases you can use the cursor.callproc() method.

    Here is an example of how to return an output parameter.

    >>> cursor = conn.cursor()
    >>> # if the procedure exists, drop it
    >>> try:
    ...     cursor.execute("DROP PROCEDURE ADD2")
    ... except:
    ...     pass

    >>> # create the procedure
    >>> sql_proc = """
    ... create procedure ADD2 (in a int, in b int, out c int)
    ... language sqlscript
    ... reads sql data as
    ... begin
    ...     c := :a + :b;
    ... end
    ... """
    >>> cursor.execute(sql_proc)
    True
    >>> # call the procedure
    >>> params_in = (2, 5, None)
    >>> params_out = cursor.callproc("ADD2", params_in)
    >>> # close the cursor
    >>> cursor.close()
    >>> print(params_out)
    (2, 5, 7)

    … and here is an example of handling returned result sets. The cursor.callproc() method acts just like cursor.execute("SELECT...").

    >>> cursor = conn.cursor()
    >>> # if the procedure exists, drop it
    >>> try:
    ...     cursor.execute("DROP PROCEDURE PROC_T1")
    ... except:
    ...     pass

    >>> # create the procedure
    >>> sql_proc = """
    ... CREATE PROCEDURE PROC_T1(IN substring varchar(10))
    ... LANGUAGE SQLSCRIPT
    ... AS
    ... BEGIN
    ...     SELECT ID, C2 FROM T1
    ...     WHERE C2 LIKE '%' || :substring || '%';
    ... END
    ... """
    >>> cursor.execute(sql_proc)
    True
    >>> # call the procedure
    >>> cursor.callproc("PROC_T1", ("hello",))
    >>> for row in cursor:
    ...     print(row)
    (1, 'hello')
    (2, 'hello again')

    Streaming data


    There is a separate class for streaming data to or from SAP HANA. You only need to use this class if you wish to process data in a piecewise fashion. Typically this may be used for video or speech data, but here is a sample that uses an image.

    # get an image from the file system
    with open('image.png', 'rb') as f:
         img = f.read()
    # write to the database
    in_blob = dbapi.LOB()
    id = 2
    conn.setautocommit(False)
    cursor.execute("update T1 set PHOTO = :img WHERE id =:id", {"img": in_blob, "id": id})
    in_blob.write(data=img)
    in_blob.close()
    conn.commit()
    cursor.close()

    And here is an example of how to read the image back out of the database. The important part is the use of the read() method to fetch a portion of the blob at a time.

    # read it into a stream
    conn.setautocommit(True)
    cursor = conn.cursor()
    cursor.execute("select PHOTO from T1 where id = :id", {"id": id})
    out_blob = cursor.fetchone(True)[0]
    out_img = bytearray()
    while True:
        data = out_blob.read(size=200)
        if data is None:
            print("Done.")
            break
        else:
            print("Reading more data...")
            out_img.extend(data.tobytes())

    This completes the description of the new HANA python driver.

    SAP HANA 2.0 SPS02 new feature: Go language driver

    $
    0
    0

    Introduction


    Go is expressive, concise, clean, and efficient. Its concurrency mechanisms make it easy to write programs that get the most out of multicore and networked machines, while its novel type system enables flexible and modular program construction. Go compiles quickly to machine code yet has the convenience of garbage collection and the power of run-time reflection. It’s a fast, statically typed, compiled language that feels like a dynamically typed, interpreted language.


    SAP HANA 2.0 SPS02 includes a Go language driver. This blog post describes how to install it and how to execute simple database commands. It will describe the process for Linux, with Windows commands added where needed. I assume a very basic knowledge of the Go language and it assumes you have access to a HANA database on which you can create a table.

    Installing the Go language driver


    There are two steps to installing the Go driver:
    1. Install the SAP HANA Client package on your computer. 
    2. Install the Go driver from the HANA Client install into a local Go workspace.
    3. The Go driver has two components to it:
    • A Go language component
    • A C++ shared object (DLL on Windows) that implements the network protocol and provides a low-level interface.
    To install the Go driver you need to compile the Go language component (making sure the Go compiler can find the shared object) and then make both these components available to the executable at runtime.

    Installing the Go driver into a local Go workspace


    You must have Go installed on your local machine. From a command line:

    > cd ~/go/src

    > export CGO_LDFLAGS=/usr/sap/hdbclient/libdbcapiHDB.so

    > cp -r /usr/sap/hdbclient/SAP/ .

    > go install SAP/go-hdb/driver

    The driver is built in ~/go/pkg/linux_amd64/SAP (the directory identifying the CPU may have a different name if you are running in a different environment.

    At runtime, you will also need the HANA client shared object in the LD_LIBRARY_PATH environment variable (or PATH on Windows) or in the same directory as the compiled executable. Here we haveset the LD_LIBRARY_PATH variable, but you can also copy the file:

    > export LD_LIBRARY_PATH=:/usr/sap/hdbclient

    Connecting to a database


    Now create a directory to hold your first Go application:

    > mkdir hello

    > cd hello

    Using a text editor, create a file hello.go that contains this text, replacing the elements that make up connStr with connection parameters for a HANA instance to which you have access.

    package main

    import (
        "fmt"
        "log"
        "database/sql"
        _ "SAP/go-hdb/driver"
    )

    func main() {
        // define a connection string
        var connStr string
        connStr = "hdb://<user>:<password>@<host>:3NN13?DATABASENAME=<db>" 
        db, err := sql.Open("hdb", connStr)
        if err != nil {
            log.Fatal(err)
        } else {
            fmt.Println("Connected.")
        }
        defer db.Close()
    }

    Here are some comments on this code:

    package main

    This file is the entry point for the application.

    import (
            "fmt"
            "log"
            "database/sql"
            _ "SAP/go-hdb/driver"
    )

    Import some required packages. The “fmt” and “log” packages are for printing and logging output. The “database/sql” package defines the SQL interface for Go, and the final line imports the SAP HANA driver. The final import is an anonymous import (the underbar character) and is done so that none of the exported names are visible to the application code. Instead, all the application code does is interact with the database/sql package.

     var connStr string
     connStr = "hdb://<user>:<password>@<host>:3NN13?DATABASENAME=<db>"

    The connection string here is for a multi-tenant database container, and uses the 3NN13 port (NN being your HANA instance number) and tenant DATABASENAME to connect (the DATABASENAME parameter must be upper case). If you are connecting to a single-tenant database you would typically use a port of 3NN15 and no DATABASENAME.

    db, err := sql.Open("hdb", connStr)
    if err != nil {
        log.Fatal(err)
    } else {
        fmt.Println("Connected.")
    }
    defer db.Close()

    the sql.Open method establishes a connection using the “hdb” driver and the supplied connection string. If it is successful it returns db as a sql.DB object that is essentially a connection pool for accessing the database.

    If the Open method returns an error, that will be logged; on success the program prints “Connected” to the console.

    The defer db.Close() call is standard Go language practice: the defer ensure that as the function completes, the database session will be closed.

    Executing a query


    To execute a query, add a function and call it from immediately after the “defer db.Close()” line:

       ...
       defer db.Close()
       query(db)
    }

    Here is the function, which assumes a table T1 that has an integer column ID and a varchar column C2. Modify for your own table.

    func query(db *sql.DB) {
        rows, err := db.Query("SELECT ID, C2 FROM T1")
        if err != nil {
            log.Fatal(err)
        } else {
            var id int
            var c2 string
            for rows.Next() {
                err := rows.Scan(&id, &c2)
                if err != nil {
                    log.Fatal(err)
                } else {
                    fmt.Println(id, ", ", c2)
                }
            }
        }
        defer rows.Close()
    }

    The query is executed, and so long as there is not an error, the function loops over the rows of the query result (“for rows.Next()”) and prints out the values from each row.

    Executing prepared statements


    Here I will not go through the whole code, but describe the extra steps you have to make beyond what is in the query example above.

    First, add a couple of packages to your import statements:

    import (
            "context"
            "database/sql"
            "fmt"
            "log"
            "time"
            _ "SAP/go-hdb/driver"
    )

    Then, in the query() function, prepare the query using a question mark as a placeholder:

    queryStmt, err := db.Prepare("SELECT ID, C2 FROM T1 WHERE ID = ?")

    Create a query context, which defines a timeout:

    ctx, cancel := context.WithTimeout(context.Background(), 8*time.Second)
    defer cancel()
    rows, err := queryStmt.QueryContext(ctx, 1)

    The value “1” in the QueryContext sets the value of the statement parameter.

    You can then loop over the rows of the result set as before.

    Smart Data Streaming: Guaranteed Delivery Part 1

    $
    0
    0
    It’s common for streaming projects to use guaranteed delivery, or GD, which is a delivery mechanism that ensures all of the data going into a project, and all of the data processed by a project, reaches its destination – even if the client somehow becomes disconnected when the data is produced. Using log stores and checkpoints, GD works with streams, windows, as well as SDK publishers and subscribers, to prevent data loss.



    How does guaranteed delivery work?


    Enabling GD on a stream or window creates a GD session, which keeps all checkpointed output events in the assigned GD log store. The log store keeps the newest events up to the highest sequence number committed to the stream or window, removing any others and advancing the sequence number when it issues a GD commit – or essentially, once it has produced all necessary outputs and processing for a given input event.

    You can set GD support for a window in the HANA studio visual editor, or in CCL. Simply select the “Supports Guaranteed Delivery” checkbox in the Properties tab for the window and pick the appropriate log store from the dropdown, or add the corresponding CCL code:

    SAP HANA smart data streaming, SAP HANA Studio, sap hana, smart-data-streaming

    If you want to use GD, you must enable it and assign a GD log store to every stream and window in your project that won’t be recovered by a provider located upstream in the data flow of your project.

    Log stores save events through a process called a checkpoint. An event is checkpointed when it is successfully saved in the log store; it is committed when it reaches its destination successfully.

    With adapters, the GD mechanism is similar. The only difference is that before the adapter issues a commit for an event and its given sequence number, it ensures that the event and every event before it has been delivered and committed to the destination.

    When using GD with an SDK subscriber, the subscriber receives data and checkpoint messages from the streaming project, and sends commit messages back indicating the sequence number of the latest event that it has processed. Using GD with an SDK publisher is a similar process: the publisher sends data, then receives checkpoint and commit messages from the streaming project.

    Ensuring zero data loss


    Consistent recovery

    If you’re going to use guaranteed delivery in your streaming project, you should also enable consistent recovery in the project’s configuration (CCR) file.

    While guaranteed delivery ensures that all data is successfully transmitted from a project to its destinations, consistent recovery ensures that all data is successfully delivered to a project from its publishers.

    Essentially, guaranteed delivery protects your output data, while consistent recovery helps protect your input data. It also makes it easier to use multiple log stores by keeping every log store consistent with the others.

    With consistent recovery enabled, commits issued to the project will:
    • Pause publishers
    • Quiesce the project (meaning all queues between streams or windows are drained to ensure that all events are processed)
    • Checkpoint all log stores atomically (meaning all log stores are guaranteed to be at the same checkpoint number in case of a failure while checkpointing)
    Consistent recovery needs a checkpointing system so it can work properly. Either enable auto-checkpoint, or set a publisher to issue commit messages. Don’t use both!

    Auto checkpoint


    With auto checkpoint, you can choose how often log store checkpoints occur across all streams and windows in the project, rather than waiting for a publisher to issue a commit. With more frequent checkpoints, less data gets lost in the case of a crash. At a maximum frequency of 1, every record, transaction, or envelope published from a publisher will be protected – except for the very last item, which may be lost in a crash. If you have multiple publishers, the last transaction sent from each publisher could be lost.

    Be careful, though – more frequent checkpoints come at a cost of decreased performance and increased latency. When using auto checkpoint, choose a checkpoint interval that works with the performance needs of your project.

    You can make the changes in the CCR file through Studio by opening the file through the project explorer, or add the necessary code through a text editor in the <Options> tag.

    SAP HANA smart data streaming, SAP HANA Studio, sap hana, smart-data-streaming

    SAP HANA smart data streaming, SAP HANA Studio, sap hana, smart-data-streaming

    Implementing Graphical Hierarchy view for Profit Center in SAP HANA

    $
    0
    0
    This blog is about Standard hierarchy ,Standard hierarchy is a tree structure for grouping all profit centers which belong to a controlling area. When creating a Profit Center, you must assign it to a group (node) of the standard hierarchy.

    Below is the sample ProfitCenter Hierarchy how it looks in SAP ECC.



    SAP HANA, Calculation views

    There are two tables in SAP for groups in standard hierarchy.
    • SETNODE
    • SETLEAF
    To build a standard hierarchy, we have to read SETNODE for each level of the hierarchy and build the tree.

    In SAP, there are many different groups such as Cost Center Group, Cost Element Group, and Profit Center Group etc. Each and every group is identified with unique code in field SETCLASS. For Profit Center it is ‘0106’.

    To get the description texts, join the text tables SETHEADERT and CEPCT for Profit Center text to SETNODE and SETLEAF tables.

    All the nodes, with subsets are included in SETNAME field in SETNODE table. SETNAME in SETNODE becomes the parent and SETNAME (joined to SUBSET in SETNODE) in SETLEAF is the child.

    First create a base view to get Nodes, Sub Nodes and Profit Centers and there descriptions.

    This is the overview of the Base View.

    SAP HANA, Calculation views

    Join_1:Text join SETNODE table  to SETHEADERT table with Language column as LANGU from which we will get Nodes and related Descriptions.

    SAP HANA, Calculation views

    Join_2: Text join VALFROM to PRCTR with language column as SPRAS here we will get Profit Centers and related Descriptions.

    SAP HANA, Calculation views

    Union Join_1 and Join_2, create a target column for SubSet and map SUBSETNAME from Join_1,this will be used in main view.

    SAP HANA, Calculation views

    In SAP HANA there is no standard way to create this type of Hierarchy(In SAP HANA we have simple Level and Parent Child hierarchy)Create the Profit Center Hierarchy view and use the above created view as below,

    SAP HANA, Calculation views

    In Projection_1 we have to select the following columns and apply the filters mentioned below.

    SAP HANA, Calculation views

    There  are different groups in SAP like Cost Center, to identify the Profit Center group we have to filter SetClass=’0106‘

    Filters used in Projection_1 for Profit Center =’0106’,ControllingArea and Group Node.

    SAP HANA, Calculation views

    Join1: Left Join SUBSET to SUBSETNAME and also SetClass to SetClass  to get Level1,Level2 and Level3 with descriptions(Nodes and Leaves).

    SAP HANA, Calculation views

    Join_2: Inner Join SubSet to SetName to get Level4 and Level4 Description.

    The leaf node will either be a group node or Profit Center; Profit Center will always be a leaf node.

    SAP HANA, Calculation views

    Join_3: Left Join to get all the Level1,Level2,Level3 and Level4 with descriptions

    SAP HANA, Calculation views

    Proceed further for more levels depending on the requirement.

    Create a calculated column to get all Profit Centers into one column.This calculated column will contain leaf node of each field and can be used to join the hierarchy with the transaction data.

    SAP HANA, Calculation views

    Final Semantics is

    SAP HANA, Calculation views

    Save and Activate the view.

    Removing invalid and non-printable characters in HANA based BW transformation

    $
    0
    0
    Most customers know the problem that invalid characters can be found in data records. Data load or DSO activation for char type InfoObjects can fail due to permitted or non-printable characters in BW.

    There exist several solutions for this problem. Some customer uses function module calls in ABAP transformations for example RSKC_CHAVL_OF_IOBJ_CHECK or RSKC_ALLOWED_CHAR_GET. Other implement field routines with ABAP like

    REPLACE ALL OCCURRENCES OF REGEX '[^[:print:]]' IN RESULT
           WITH 'replace_char'
           IGNORING CASE.

    or…

    IF RESULT(1) = '!' OR return = '#'.
        RESULT(1) = 'replace_char'.
    ENDIF.

    Now BW/4HANA has the option to push down transformation logic to HANA database, to reduce runtime of data loads. For custom transformations to be pushed down to HANA database, start, end and expert routine type need to be created as AMDP script (ABAP Managed Database Procedure script) instead of an ABAP based routine.

    For the example we created a sample file with invalid characters ‘!’ and ‘#’ in record 2, 4, 5 and 6, and a non-printable character Tapstop in record 7.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    We load the data in a field based write optimized ADSO and want it to be loaded and activated in a standard ADSO with InfoObjects and changelog.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    To remove invalid and non-printable characters with an AMDP Script in a field routine, you can follow these steps.

    Step 1: Select rule type routine for the transformation rule, see (1). A popup dialog asks for processing type, see (2). Choose AMDP script to create an AMDP script based field routine.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    Step 2: an AMDP class will be generated with default method – PROCEDURE and default interface – IF_AMDP_MARKER_HDB. The BW Transformation framework opens the Eclipse based ABAP class editor.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    Step 3: Enter your code for the field routine in the body of the method PROCEDURE.

    You can use the following template. The code replaces the regular expression ([^[:print:]]|[#|!]) in source field “CHARACTERS” with replacement char ” and transfer the result in the target InfoObject “/BIC/ZSPECHAR”.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    If you want to remove invalid characters at the first position only, you can use this template.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    If necessary this can be extended with upper case conversion or similar.

    After removing the characters in transformation our example data will be loaded without errors and the data activation in ADSO works well.

    SAP BW/4HANA, BW SAP HANA Data Warehousing, HANA SQLScript

    Using Busines Data for Better IT Outcomes

    $
    0
    0
    SAP IT Operations Analytics (SAP ITOA) brings a lot of value to IT organizations:
    • Embedded Predictive Analytics makes it easy to anticipate and respond to issues before they happen
    • Realtime monitoring and alerting lets IT respond immediately when issues do occur
    • A holistic view helps simplify root-cause analysis and reduce meantime to resolution
    But while the stats associated with these benefits are impressive, the customer use-cases I find most compelling are bringing business context to IT data.

    SAP ITOA is a native HANA application—which means anything that can be written to HANA can be analyzed in SAP ITOA. This helps IT understand the impact of issues or outages. Here are some interesting use-cases for this…

    Topology Information


    Knowing what went down is good, but knowing what it impacts is better. If five servers fail at the same time, understanding what servers are running which applications can help IT prioritize a fix that will have the biggest impact to the business. For example, getting the Order Management System up would take priority over say Vacation Bookings.

    SAP HANA Tutorials and Materials, SAP HANA Certifications

    Prioritizing Productive Customers


    SAP’s IT is responsible for the datacenters that host out SaaS applications. We marry customer contract data with IT events as they stream in. So if there is an issue or an outage, our IT sees which customers are impacted and what their SLAs are. Of course fixing issues that impact productive customers would take priority over SAP employees, or customers with dev licensing.

    SAP HANA Tutorials and Materials, SAP HANA Certifications

    Understanding Physical Location


    We spoke with one educational institution who has different physical access depending on the building. Some buildings were locked from 10pm-6am. They were exploring implementing rules to suppress alerts between those hours if the known fix required manual intervention. No point waking someone up at 3am if they can’t access the building anyways.

    Another example of this is distribution centers…if you have issues impacting product shipment at two distribution centers you may prioritize the one that is closer to the end of business day (for example fix East Coast Issues before West Coast) since the West closes after the East.

    HR Data


    We’ve also seen customers bring in HR data, so they can understand impacted groups. Revenue-generating departments like sales may take priority over operational areas of the business.

    These are just a few examples of how enriching IT events with business data transforms SAP ITOA from a monitoring tool to a true analytics solution for IT. We also have a quick video on understanding the business impact of IT issues.

    How Analytics can enhance traditional IT Monitoring

    $
    0
    0
    According to EMA, the average IT organization has more than 10 monitoring tools. Often when speaking to IT departments about SAP IT Operations Analytics (SAP ITOA) they are concerned about adding yet another monitoring tool…

    But IT Operations differs from traditional monitoring and can enrich what most organizations are doing.

    SAP ITOA can connect direct to device via agent or streaming log files. Or it can leverage existing monitoring tools as a datasource, to avoid disrupting what’s already working in your IT department and to streamline deployment.

    Let’s explore some benefits of introducing analytics on top of IT monitoring tools:

    Get a Holistic View for Easier Root Cause Analysis & Reduced Mean Time to Resolution

    Since any given application relies on multiple infrastructure components (servers, virtual machines, network devices). Finding the root cause involves multiple teams and monitoring tools and invariably ends in finger-pointing. By giving all groups access to health and performance information across IT troubleshooting is faster and easier. Pinpointing the cause of an issue saves time and IT resources—if you know the issue is with a switch why have the applications team and server team work to solve the same issue.

    Leverage Predictive Analytics Across IT

    Most existing monitoring tools only look in the rearview mirror. Identifying and remedying IT issues as quickly as possible is important, but even better is to anticipate and respond to issues before they occur.

    SAP IT Operations Analytics has out-of-the-box predictive analytics. So you can forecast expected behavior and be alerted if something unanticipated happens.

    SAP HANA Tutorials and Materials, SAP HANA Certifications
    Static thresholds are a thing of the past with SAP ITOA. Outliers (in red) can trigger alerts when metrics are above or below the predicted range.

    Here’s a practical example—CPU load. Less load is inherently better than more load, so traditionally IT has relied on threshold alerting. For example, send an alert if the load is great than 80%. But what if you expect the load to be 70% and it is 20%. That is likely indicative of another issue (e.g. an application has crashed). Predictive would make an estimate of load based on historical information and if the deviation was too great it would trigger an alert.

    By using existing monitoring tools as a datasource for SAP ITOA you’ll be able to benefit from predictive analytics across IT. 

    Understand the Impact of Outages by Enriching IT events with Business Data

    IT events themselves aren’t really the issue—it’s the business impacts they cause. From lost revenue, to poor customer experience IT outages have a hard cost. SAP ITOA can help IT understand the business impact of outages so IT can respond and prioritize accordingly.

    SAP HANA Tutorials and Materials, SAP HANA Certifications

    Infrastructure can be assigned to business stakeholders in SAP ITOA. Looks like server02 is suffering from unscheduled downtime resulting in lost profit for the Sales organization.

    With the master data plugin you can marry IT events with non-IT data. Some examples are detailed in this blog, but include understanding impacted customers (and their SLAs), learning about the geographical location of outages, and understanding what related devices and applications may be impacted.

    We also have a quick video on enhancing existing monitoring tools with analytics for IT.

    Experience of my first SAP HANA project

    $
    0
    0

    INTRODUCTION


    This blog is related to few of  my experience working on  SAP HANA. It has the problems that I faced during a migration project (from Oracle PL/SQL to SAP HANA SQL Script) , in which I am working on and the possible solutions for those problems.

    The solutions that are explained in here may be helpful for developers who are facing similar kinds of problems. This blog will be useful for the beginners who are working on HANA development.


    Usage of RESULTVIEW


    Problem:

    Oracle materialized view has to be converted to HANA view, So I tried to create RESULTVIEW in a procedure. I found that creation of RESULTVIEW in HANA Studio Project could not be transported via the repository.

    Solution:
    • RESULTVIEW is useful while creating in console (to create a runtime view of data)  but it is not transportable via the repository.
    • So as an alternative I created a procedure with OUT parameter and used it to retrieve the result set.

    Changing the preferences in HANA studio


    Problem:

    Change the number of results shown after executing a query in SQLConsole.

    Solution:
    • To get more than 1000 rows of a table as result of a select query I navigated to
    • WINDOW->preferences->SAP HANA->Runtime->Result  and typed the max number of rows to be displayed.
    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    Problem: 

    To change the delimiter while exporting the result (data) of a query or changing the NULL representation character or any other settings related to the data export.

    Solution:

    To change the export settings such as DELIMIETER (Data separator) , NULL value representation , etc.
    Go to    WINDOW -> PREFERENCES -> SAP HANA ->RUNTIME -> COMMON

    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    Table types in project:


    Problem:

    I tried to create a table type in my project as I created in console but couldn’t be transported via repository.

    Solution:

    Creating table types in console is different from creating  them in a project .
    While creating a table type in project HANA expects a .hdbdd file to be created and appropriate artifacts to be mapped with each of the table type columns.

    Also the artifact is case sensitive. Here, I have mentioned the context as follows

    Context abcd{
    Type STR25 : String(25);
    };

    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    But , when creating the table type I created as

    TYPE TTT{
           “coLUM” : str25;
           };

    Which gives me the error (since str is in lower case):

    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    When I assign the column to the table type , I should use coLUM (case sensitive)

    Look at the following code snippet:

    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    create procedure sample(OUT abc “SRIVATSAN”.“SRI.DEV.DDL::ABCD.TTT“)
    as
    begin
    DECLARE colum varchar(25);
    colum := ‘hello’;
    abc = select colum from dummy;
    end;

    When I execute this , I get the error as follows :

    SAP DBTech JDBC: [1306] (at 138): return type mismatch: Attribute name “coLUM”  different from Attribute name: “COLUM” : line 7 col 1 (at pos 138).

    This shows that the column name should match exactly the way it is defined in .hdbdd file.

    create procedure sample(OUT abc “SRIVATSAN”.“SRI.DEV.DDL::ABCD.TTT“)
    as
    begin
    DECLARE colum varchar(25);
    colum := ‘hello’;
    abc = select colum as “coLUM”  from dummy;
    end;

    This code will work since the name we assign to the column (alias) is exactly the same as .hdbdd definition.

    Using Row store tables in modeling:

    Problem :

    When creating modeling views, I tried to use a table (of ROW type) which I couldn’t accomplish.

    Solution:
    • It is not possible to use ROW tables (Created as Row type) in modeling views. Only columnar tables can be used.
    • When I tried to drag and drop a ROW store table to the data foundation, it doesn’t allow.
    • Also when I right click the data foundation and choose Add Objects , It shows only the tables which are of COLUMN type.
    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    Exception handling:


    Problem:

    For few error codes in HANA, I was unable to retrieve the error message using ::SQL_ERROR_MESSAGE.

    Solution:

    For few error codes (::SQL_ERROR_CODE), HANA doesn’t return the  :: SQL_ERROR_MESSAGE  (returns a blank as error description),
    so to get the DESCRIPTION , I used the SYSTEM table M_ERROR_CODES from SYS schema (SYS.M_ERROR_CODES) by passing the error code to the column CODE and getting the DESCRIPTION column’s value.

    Here is a sample code ,

    PROCEDURE “SRI.DEV.LIB::P_LOG_ERROR”
                                            (IN IN_ERROR_CODE INTEGER,IN IN_OBJECT NVARCHAR(100),IN IN_ERROR_MSG NVARCHAR(2000))
           LANGUAGE SQLSCRIPT   SQL SECURITY INVOKER
           AS  BEGIN
           DECLARE V_NEXTVAL INTEGER;
           DECLARE V_ERROR_MSG NVARCHAR(2000);
           SELECT “SRIVATSAN”.“SRI.DEV.DDL::SEQ_ERRLOG”.NEXTVAL
           INTO   V_NEXTVAL
           FROM DUMMY;
                  V_ERROR_MSG := IN_ERROR_MSG;
           IF V_ERROR_MSG = ” THEN
                  SELECT DESCRIPTION INTO :V_ERROR_MSG
                   FROM SYS.M_ERROR_CODES WHERE CODE = :IN_ERROR_CODE;
           END IF;
           INSERT INTO “SRIVATSAN”.“SRI.DEV.DDL::SCP_ERRORLOG”
                   VALUES (:V_NEXTVAL,
                                CURRENT_UTCTIMESTAMP,
                                IN_ERROR_CODE,
                                IN_OBJECT,
                                :V_ERROR_MSG,
                                CURRENT_USER);
         END;

    SAP HANA Tutorials and Materials, SAP HANA Certifications, SAP HANA Guides

    This exception handling procedure will be called in all the main procedures with the 3 parameters
    (::SQL_ERROR_CODE , ‘<Name of the object that caused the exception>’,::SQL_ERROR_MESSAGE) so as to insert the error details in a table .

    Importing CSV file to SAP HNA table:


    Problem:

    I faced issues in importing CSV files to HANA tables. I was getting errors due to  formatting issues of date and time.

    Solution:
    • While importing a CSV file to HANA table, I ensure that the DATE data type is of the format YYYY-MM-DD and the Timestamp has YYYY-MM-DD HH:MM:SS.
    • To do this I opened the CSV file in MS Excel, selected all the columns with data/time, right clicked and chose Format Cell and chose Custom option to type the exact format of the column (Date / time / Date Time).
    • Saved  the excel file and then I started importing the file to table which worked fine.

    Altering the table columns:


    Problem:

    I tried to change the data type of a column (ROW TYPE table) , which was not supported.

    Solution:

    Only tables created as Column store allows altering the data type of columns (Eg., When a column with DATE is being altered to TIMESTAMP , that is not allowed if the table is created as ROWstore)

    Look at the following row table,
    create row table rowtype (col1 integer,col2 date);

    Here, I have created a row store table called “rowtype” with two columns namely col1 as integer type and col2 as date type.

    Now, I try to alter the table to change the data type of col2 from date to Timestamp ,
    Alter table rowtype alter (col2 timestamp)

    I got the error as follows:

    Could not execute ‘Alter table rowtype alter (col2 timestamp)’ in 220 ms 781 µs .
    SAP DBTech JDBC: [7] (at 27): feature not supported: cannot modify data type of column: COL2: line 1 col 28 (at pos 27)

    Now , the same is tried with the column type table ,
    create column table coltype(col1 integer,col2 date);

    Altering the column data type :
    Alter table coltype alter (col2 timestamp)

    Statement ‘Alter table coltype alter (col2 timestamp)’
    successfully executed in 236 ms 633 µs  (server processing time: 15 ms 169 µs) – Rows Affected: 0

    Whereas increasing the length of a column (same datatype) is possible in both row and column types.

    create  row table rowtype(col1 varchar2(20),col2 date);

    Alter table rowtype alter (col1 varchar(40))

    Statement ‘Alter table rowtype alter (col1 varchar(40))’
    successfully executed in 278 ms 609 µs  (server processing time: 4 ms 254 µs) – Rows Affected: 0

    Streaming Analytics: Developing in SAP Web IDE, 2.0 SP 02 Edition

    $
    0
    0
    Since its introduction in 1.0 SP 12, the streaming plugin for SAP Web IDE has been improving release to release. With 2.0 SP 02 come more new features and enhancements for both the plugin and the streaming runtime tool, which should make your development, administration, and monitoring through these tools just a little bit easier.

    Here’s a quick summary of the changes to streaming modules in SAP Web IDE:


    • The streaming plugin isn’t enabled by default anymore – you need to enable it in the Web IDE preferences before creating streaming modules.
    • CCL and CCR files inherit the streaming module name.
    • You can view the preferences for any streaming project element directly in the graphical viewer.
    • You can build streaming modules to any custom streaming workspace.
    And here’s what’s new in the streaming runtime tool:
    • You can register custom workspaces – that is, add any existing streaming workspace to the tool.
    • You can monitor statistics for streams and windows through the monitor.
    • You can trace data flow through the event tracer.

    Enabling the plugin


    From SP 02 onward, the plugin isn’t automatically enabled once you install everything and set up the streaming service broker. So, if you log into the Web IDE and can’t find streaming modules anywhere, not to worry: you can easily enable the plugin, and the link to the streaming analytics runtime tool, through the Web IDE preferences.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    CCL and CCR file names


    You also don’t need to worry about confusing your multiple “model.ccl” and “model.ccr” files anymore – they now inherit the name of the streaming module.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    Properties in the graphical viewer


    If you open up the graphical viewer, you’ll notice that there’s a new icon in the toolbar. Clicking this icon opens up the option to turn on the properties pane. Then, when you select any element in the graphical viewer, the properties for that element show up in the pane.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    Building streaming modules to custom workspaces


    Here’s arguably the most important feature for streaming modules in this release: building streaming modules to any streaming workspace, on any streaming server (as long as you have access to it – make sure you have the right streaming permissions before publishing). This means you’re no longer limited to building streaming projects to the default XSA space, and don’t have to use the default generated workspace names.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    Once you enter your connection details once, they’re saved for the session. Next time you publish, you just need to enter your password again.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    If this workspace isn’t on the streaming runtime tool yet, it won’t publish to the tool automatically. However, you can also register any streaming workspace (from any streaming server) on the streaming runtime tool. Then, it functions as any other streaming runtime tool workspace, where you can view projects, data services, and PAL models.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    If you’ve used the streaming runtime tool and the SAP Web IDE for streaming development before this release, you probably also know the struggle of getting data services and PAL models set up. Before, you had to publish an incomplete (but working) project to create a runtime tool workspace, then add your data services and models, and finally go back to the Web IDE to continue developing the project. With this feature, the first step of that process is gone – there’s now a “Streaming Workspaces” folder in the streaming runtime tool by default, where you can register a workspace and add whatever you need before even starting development.

    Using the monitor and event tracer


    If you’ve developed and tested streaming projects in the HANA studio in the past, you might be familiar with the monitor and event tracer. In the streaming runtime tool, you can open either of them by right-clicking on a project, then selecting the relevant view.

    The monitor gives you a breakdown of what’s going on in every stream and window in your project. Each element is represented by a rectangle with a list of statistics: rows and transactions per second, the amount of CPU usage, and a bunch of other useful information. You can set thresholds for acceptable values, for either queue depth or rows per second, so that the relevant stream or window changes color whenever it passes the thresholds – a quick visual indicator if something is going wrong.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions

    The event tracer is a simpler tool that gives you a high-level overview of the project. Here, you can see data flow, again represented by different colored shapes, though this time by event type.

    SAP HANA smart data streaming, SAP HANA Tutorials and Materials, SAP HANA Certificarions
    Viewing all 711 articles
    Browse latest View live


    <script src="https://jsc.adskeeper.com/r/s/rssing.com.1596347.js" async> </script>