SAP Integration

SAP Integration – Part 2: Microsoft OLAP together with SAP BW

3 Comments

In the first part of this series I described the integration of the Microsoft reporting layer directly on SAP BW. The second part shows how to integrate with Microsoft OLAP and SAP BW. The following pictures highlights the different layers.

image

Ok, so let’s have a look into the details.

The data extraction barrier

Connecting a Analysis Services Cube directly on a SAP BW as data source is not a good idea. We would suffer much on performance. So the data has to be extracted out of SAP BW to a SQL Server first, but how? There are different possibilities we have with SQL Server Integration Services:

  • SQL Server Feature Pack: details
  • 3rd party provider like Theobald Software: XtractIS
    In general SAP doesn’t make our life easy if we want to extract data from BW. There is no good extraction support in general for other destinations than SAP and Customer need an Open Hub License in order to store SAP BW data outside of SAP not matter how it is extracted.
      SQL Server Feature Pack and Theobald Software

    So let’s have a closer look on the extraction possibilities. No matter if you choose the SQL Server Feature Pack functionality or another 3rd party software they all have the same problem and this is the mechanism how SAP supports data extraction. The following picture show an overview about the possibilities you have with Theobald Software:

image

I want to focus on the Open Hub Service (OHS) functionality this is also the way how the SQL Server Feature Pack supports data extraction. Open Hub Services in SAP are a way to model extraction tables (OHS tables) and fill them with data for relevant for extraction. All BW data objects are supported for extraction and the extraction process supports initial and delta loads. There is a very good whitepaper on how to use OHS written by Hermann Däubler which everybody should read before to start working with OHS.

If you are now using the SQL Server Feature Pack connectors or Theobald Software to connect to the OHS tables SAP data is always extracted via the application layer of SAP. What does it mean? When you start the extraction process via OHS the data is read from the SAP BW database, loaded into the application memory of SAP and is sent out in small data packages (like 50.000 rows per package). Then the next package is read from the database, loaded into memory and sent out to Integration Service until the whole data is extracted from the OHS table. I started a test case on a SAP test system and want to give you some numbers on performance:

  • extraction of 23 million records (22 GB of data) took about 30 hours!!!
  • 50% of overall execution time was waiting for data providing by SAP

image

This approach doesn’t really scale for large data extractions like for a Data Warehouse, where you would really suffer on performance. So what are the options?

Direct SAP BW database access

The use of the Open Hub Services within SAP to provide SAP data in extraction tables is fine and works well. The only problem is how to connect to the OHS tables in a faster way? It is allowed by SAP to directly connect to the underlying SAP database (like Oracle or SQL Server) via a normal OleDB provider and access the OHS tables directly. Accessing internal SAP BW tables is not allowed.

So we use Integration Services connecting to the OHS table via an OleDB provider and loading data directly from the database which gives us a really great performance. It is also possible to secure the access only to the OHS tables for the database user which access the SAP BW database. Here are the numbers for OleDB access of my test case:

  • extraction of 23 million records (22 GB of data) took about 2 hours
    The overall throughput on my test system is still not the best but I want you to compare the numbers instead (30 hours compared to 2 hours).
    There is one important thing to add when using this approach. Because there is no coordination mechanism by default that tells you, that the SAP extract is finished into the OHS table and that the data can be used for extraction. So you need a communication service on top. This service can be implemented in SAP BW tracking some metadata like which export is available in which OHS tables, how many rows are provided by that export, is it a full or delta export, what is the RequestID to identify the extract … This service can be implemented via a BW ABAP function exposed as a web service. It can be also used to call SAP back and mark data ready for deletion after loading it into the Microsoft BI environment.

    Feeding the Data Warehouse

    After the data extraction barrier has been covered we can now start modeling the Data Warehouse (as I described here), implement it in Analysis Services and feed the cube with data from SAP BW.

What are your experiences with extracting data out of SAP BW? Are there any other solutions out there? If yes, please let me know.

SAP Integration – Part 1: Microsoft BI on top of SAP BW

5 Comments

The next series I decided to write about is SAP Integration with Microsoft BI. Why? Especially in the German speaking area  SAP is used very often in case of ERP (SAP ERP), CRM (SAP CRM) or Business Intelligence (SAP Business Warehouse) software. We see a lot of SAP customers changing their Data Warehouse platform to Microsoft BI for different reasons, e.g. costs, performance, usability, … . So how can I integrate with SAP and which options are possible? I will also provide some pros and cons for every scenario.

Let’s start with the easiest way which is directly connect the Microsoft BI frontend tools like Excel or Reporting Services to SAP BW (SAP Business Warehouse).

The following picture shows an architecture layer how this could look like.

SAP Part 1

All layers beside the presentation layer are implemented with SAP. Data from different SAP source systems (SAP ERP, SAP CRM) are integrated into a Enterprise Data Warehouse (EDW) and then populated to SAP BW Queries, InfoCubes or Multiprovider.

This a typical hub & spoke architecture where the EDW Layer represents the Hub, implemented with SAP BW, and the Data Warehouse layer represents a couple of spokes, also implemented with SAP BW.

On the Microsoft side you can use Excel (Pivot table or PowerPivot) and SQL Server Reporting Services to directly connect to a SAP BW and use it as a report data source.

 

Excel & PowerPivot

Excel Pivot

In order to use Excel Pivot tables to connect to SAP BW a version of the SAP BW OleDB provider for OLAP has to be installed on the client. You can run an installation of the SAP-BI-Frontend-Setup to install the OleDB provider for SAP. After the provider has been installed you can use Excel Pivot table or Excel Pivot table charts to visualize SAP data. A detailed step by step guide can be found here: Connecting to SAP BW with Excel PivotTables and ODBO.

PowerPivot

In order to use PowerPivot (Microsoft’s in memory reporting engine) to connect to SAP BW you have some more options:

  • Importing data into PowerPivot via an Excel PivotTable connected to SAP BW
  • Importing data into PowerPivot via SQL Server Reporting Services connected to SAP BW
  • Importing data into PowerPivot from a text file generated using SAP BW Open Hub Service
  • Importing data into PowerPivot from a SQL Server Analysis Services cube connected to SAP BW created using ERP-Link iNet.BI
  • Importing data into PowerPivot using Xtract PPV connected to SAP BW

I can highly recommend the whitepaper from the SQL Server Customer Advisory Team (SQL CAT) where all those scenarios are described in more detail: PowerPivot for Excel to analyze SAP data.

SQL Server Reporting Services

Prerequisites

1.) Enable the SAP BW provider within Reporting Services

Reporting Services already ships with a SAP BW connector. In Reporting Services 2005 this connector has to be activated (for whatever reason) within the RSReportDesigner.config first, located in <drive>:\Program Files\Microsoft Visual Studio 8\Common7\IDE\PrivateAssemblies. Within the data section the line with the extension name SAPBW just has to be uncommented like in the following picture.

image

Since SQL Server 2008 this provider is activated by standard.

2.) Start the XML/A service in SAP BW

The Microsoft .NET Data Provider 1.0 for SAP NetWeaver BI communicates with SAP BW by using XML for Analysis (XML/A), which makes some InfoProviders available directly. XML/A provides direct access to QueryCubes and also to InfoCubes and MultiProviders. InfoCubes are the native multidimensional data structure in SAP BW. MultiProviders are InfoProviders that consolidate data from multiple InfoProviders. ODS Objects are not accessible directly from XML/A.

Before you can connect to a SAP BW via the XML/A interface the service has to be activated first. In order to do this go to your SAP BW and do the following steps:

1. Open “Maintain service”.

2. Right click the xmla service and select “Test Service”.

clip_image002

3. A browser window opens and displays the XML SOAP schema. The URL should look like <server>:<port>/sap/bw/xml/soap/xmla

3.) Enable the QueryCube for external access

For every QueryCube that needs to be accessed outside SAP BW the remote access options has to be enabled first. In order to do this do the following steps:

1. Open the BEx Query Designer.

2. In the Query Properties dialog box, click the Extended tab.

3. Select the Allow External Access to this Query check box, and then click OK.

clip_image001 

Create a report

After all prerequisites have been setup correctly we can now create a Reporting Services report. Open a new Reporting Services project within the Microsoft Business Intelligence Development Studio and create a new data source. Pic SAP NetWeaver BI as the data source type of the connection as show in the picture below.

image

After you clicked OK you have to specify the connection parameters which should look like this:

image

And that’s it. Now you can test your connection and then create your first report.

There is a very good whitepaper published by Microsoft with some more details on how to use Reporting Services with SAP BW which I recommend for further details: Using Reporting Services with SAP NetWeaverBI

Summary

Pros

As you can see it’s not that hard to connect the Microsoft BI frontend tools to SAP BW. I would recommend this solution for Proof of Concepts, Demos or if users want to concentrate on the frontend functionality first and reuse the backend (SAP BW) in the first step.

Cons

I often get the feedback that users like the Microsoft platform because they have a better ability of modeling the Data Warehouse. Compared to SAP BW which has the concept of predefined content models which are hard to adopt Microsoft BI has the full flexibility to integrate business users into the modeling process. This integration scenario doesn’t provide from this flexibility because we reuse whatever is there in SAP BW.

Additionally there is also no performance benefit with this solution and bad performance is often a point SAP users complain about. Also not all SAP BW functionalities are supported on the Microsoft BI frontend side and to setup a security integration can be sometimes very hard.

What other integration scenarios are possible? Stay tuned, I will explain them in the following posts.