OBIEE Data Lineage Solution

.

Tracking OBIEE reporting data

BI implementations may use hundreds of data warehouse tables and thousands of reports, and one of the most common requirements in a BI project is the ability to track back the data used in reports. The possibility to quickly identify the set of columns and tables used in a given report or dashboard, or to check which reports may be affected by a change of physical column in the data warehouse, is also crucial for development. This calls for a data lineage solution which is accessible to users and developers,  and allows checking such data on an ad hoc basis.

Various vendors offer data lineage solutions, but these can be expensive and vendor-specific. With our simple solution, we combine Catalog Manager and Administration Tool export sources to create an easily accessible solution for tracking data lineage in OBIEE.

By implementing the OBIEE data lineage solution, we can check the following:

  1. Which physical tables and columns are used in a given report or dashboard
  2. Which reports use given physical columns or tables; this is especially important when modifying an existing table, as any change in the table’s structure must take existing reports into consideration
  3. Which are the most commonly used columns in reports in a given subject area; identifying the most commonly used columns in a report can hint at creating indexes to improve the overall performance of the OBIEE implementation even further.

Solution

The ClearPeaks OBIEE Data Lineage Solution gathers all the required data lineage information in one place and uses it as a source for OBIEE reports within Data Lineage Subject Area. Two sources are combined to achieve this:

  1. List of OBIEE reports and presentation columns used

Catalog Manager provides an option to export the list of reports and columns used. The export can be done either with the Catalog Manager UI tool or through the command line utility. We use the latter option, as it allows automation of the whole process later.

  1. Mappings of presentation columns back to the physical columns

Such mappings can be obtained manually by creating Repository Documentation from Administration Tool’s Repository Documentation utility. The output will be a repository documentation in the form of comma separated values file (.csv). This will contain column mapping from presentation through logical to physical layers, including column formula expressions. Another way to obtain column mappings is by extracting the OBIEE repository in .xml file format through the command line utility. Our solution uses the second option, as the repository file will be used in our automated script.

OBIEE Data Lineage Solution

Once we have obtained both files, we need to populate the data lineage tables.

The data can be transformed and manually inserted into the tables, but in our solution we use a script (which can run on the OBIEE server) that parses the data and inserts it into the tables.

Data Lineage Subject Area

Once we have populated the data lineage tables and their data model has been mapped in Administration Tool, we can create and run reports in OBIEE using Data Lineage Subject Area and filter the results according to our requirements.

OBIEE Data Lineage Solution

Let us look at a few of the use cases for the Data Lineage Subject Area:

Use case 1. Which data warehouse tables and columns are used in a given report?

We would like to know which data warehouse tables and columns are used in a particular report or dashboard. We can create a report with a list of the columns used by a given OBIEE report and their data lineage:

OBIEE Data Lineage Solution


Use case 2.
Which reports use given physical tables?

We want to know how many and which reports or dashboards are using given physical tables or particular columns; this could be very useful when assessing the potential impact of column formula or table structure changes on reporting. Using Data Lineage Subject Area we can fetch up the list of OBIEE reports used by a given physical table:

OBIEE Data Lineage Solution


Use case 3.
Which reports use given subject areas?

We need to know which reports and dashboards are accessing data from given subject areas. This may be particularly useful when revising users’ access permissions.

OBIEE Data Lineage Solution


Future Improvements

OBIEE Data Lineage Subject Area can also serve as a backbone for further developments, providing complex and complete solutions for managing existing OBIEE project implementations. Here are some examples and potential benefits of merging additional information into the solution:

Usage Tracking data – allows analysis of the physical tables used in the most accessed reports, the tables and columns in reports not used by anyone, removing non-used reports or underlying tables.

Data warehouse database metadata – such as table size, indexes on columns. This allows for performance analysis of the most heavily used tables and columns by report usage.

ETL Data Lineage – additional layer of data lineage tracking – allowing the tracking of data back to the very source – can be achieved by adding ETL transformation data obtained from ETL systems. For example, it is possible to track all the ETL transformations on a given presentation column down to the source system.

Catalog metadata – it is possible to extract additional information regarding catalog objects such as user permissions, report owners, last modified date etc., to further enhance the solution usability.

Adding all the above components creates a strong core for the OBIEE management dashboard, allowing continuous improvement through:

* tracking data from the report to the source database column
* constant analysis of the most used tables in order to improve performance
* checking which users have permissions for given OBIEE reports and dashboards

All of the above is accessible from the OBIEE front-end, providing a convenient and quick way to facilitate many daily business-as-usual tasks in BI deployments.

Conclusion

The ClearPeaks OBIEE Data Lineage Solution can be easily deployed in any project using Oracle Business Intelligence Enterprise Edition. The solution can be run from the command line tools, which makes it possible to create automated jobs to extract and update data on a regular basis.

If you think your OBIEE project could benefit from our data lineage solution, please contact us for more information via our web contact form or by leaving your comments below!

Blog Article Author: Rafal Ostaszewski

Pop up effect in OBIEE using jQuery

.

In this article of our OBIEE Customization Series I will show you how to introduce the Pop up effect in OBIEE using jQuery. The tooltip can provide additional details (like an explanation of a KPI) or load content such as target report.

Motivation

For our example let’s imagine we have a summary level report with purchase orders listed. We can navigate to order details level, but that would require leaving the current page or opening the content of detail report in a new window. What if we could have a summary level report and peek into details on the same page? Implementing the dynamic navigation to a report inside the tooltip will allow us to achieve that.

Pop up effect in OBIEE using jQuery

Because OBIEE Answers and Dashboards allow using web technologies such as HTML5, JavaScript, CSS3, and AJAX we are able to customize the look and user interface of any report just like we can do with websites. To achieve that, we will use the most popular JavaScript library called jQuery. It’s an optimized .js file containing Document Object Model (DOM), events, effects and Ajax functions that can be called from the report. JQuery is widely used in website development and allows implementing complex components using pre-defined methods defined in library. For the tooltip we will need also a plugin, an additional library that defines the effect. There are many available plugins for tooltip functionality both free and requiring paid license. Many UI libraries such as jQuery UI or Kendo UI offer, among others, a tooltip functionality. For the example in this article we will use the Tipped tooltip library, which offers a lot of customization options but requires a licence for commercial use.
As an example in this article we will invoke a tooltip that loads the contents of target report when hovered on a column’s value. To create this effect we will need to use AJAX technology. Such method is already included in the jQuery library. We will need to specify the target page or report from which the external content will be loaded.

Implementing the tooltip

Let’s start by creating a simple report showing Order Number, Number of Order Lines and Order Cost.

graph1

Our objective here is to provide additional information of the orders displayed in the first report but without navigating. Therefore let’s create another simple report that will contain more details on a given Order and its Order Lines. We need to add a filter - Order Number is prompted to enable navigation.

graph2

Let’s go to the results tab and edit the views. The contents of this report will be displayed in a tooltip. We can add a pie chart with breakdown of the line cost and save the report as Order Lines Report.

graph3

Let’s get back to the main report. We will need to add a static text view with code to load libraries. Those files could be referred pointing to the file on the internet (we can do it by loading jQuery library from google developers and for example jQuery UI library which contains tooltip functionality). However we should have those files on our OBIEE server. Often internal network security settings disallow loading content (especially script files with .js extension) from external sources. A one-domain policy is used to prevent security breaches. Because of that we will need to save those library files on OBIEE server. To be able to link a file on a OBIEE server we will need to create a virtual directory on OBIEE server in weblogic settings.
When we have the virtual directory created we can refer to the libraries using such link: http://localhost:9704/userfiles/jquery.1.9.0.min.js

graph4

 

First, we need to load the jQuery library:

<script type="text/javascript" src="http://localhost1:9704/user_files/jquery-1.9.0.min.js"></script>

Tipped libraries:  

<script type="text/javascript" src="http://localhost:9704/user_files/Tipped/tipped-3.1.8/js/tipped/tipped.js"></script>
<script type="text/javascript" src="http://localhost:9704/user_files/Tipped/tipped-3.1.8/js/excanvas/excanvas.js"></script>
 

Tipped CSS stylesheet, which contains the styles used for tooltip customization:

<link rel="stylesheet" type="text/css" href="http://localhost:9704/user_files/Tipped/tipped-3.1.8/css/tipped/tipped.css"/>

Then let’s edit the column formula to dynamically generate a tooltip with navigation to Order Line Details.

graph5

We need a couple of elements here. Instead of just "Order"."Order Number" column value we need to embed some more elements in the formula code. The code inside will contain HTML tags so let’s mark Constains HTML Markup option.

graph6

For our example we will need to use the following column formula: '

'<div id="popup' ||"Order"."Order Number"|| '">
' ||"Order"."Order Number"|| '
</div>'
||'<script type="text/javascript"> Tipped.create("#popup' ||"Order"."Order Number"||'", "saw.dll?Go&path=%2Fshared%2FReports%2FOrder%20Lines%20Report&Action=extract&p0=1&p1=eq&p2=%22Order%22.%22Order%20Number%22&p3='||"Order"."Order Number"||'",
{ajax: true,
 skin:"white",
 hook: "bottomright",
 border: { size:1},
 afterUpdate: function()
{}
} );
</script>'

Explanation:
To separate HTML formatting from the OBIEE column value ("Order"."Order Number") we will need to use single quote (') and to merge HTML formatting with the column value we need to use concatenate (||).

<divmarkup is a placeholder for our tooltip. Its id is generated dynamically by concatenation with value of the "Order"."Order Number" column.
Here we initialize Tipped script and call plugin’s method to create a tooltip. The syntax of the script to create a Tipped tooltip is as follows:
<script type="text/javascript">
Tipped.create("#target", "url", { ajax: true });
</script>

# is a jQuery selector that points to a single element with the given id attribute, in our case #popup div concatenated with Order Number. Then there is a url with the external content to be loaded inside ("url"). The important part here is the ajax: true option, which allows loading of external content.

<script type="text/javascript"> Tipped.create("#popup' ||"Order"."Order Number"||'

So basically, the popup will display whatever contents are displayed in the url, therefore we can pass a GOURL link to the detail report passing the order as a parameter. Note the Extract parameter which displays just the results of target report in a format without the paging control, hot links and other elements.

"saw.dll?Go&path=%2Fshared%Reports%2FOrder%20Lines%20Report &Action=extract&p0=1&p1=eq&p2=%22Order%22.%22Order%20Number%22&p3='||"Order"."Order Number"||'"

Here we specify the formatting options of the tooltip. For Tipped tooltip to load external content we will need to use  ajax: true option. Tipped offers richness of customization options for skins, positioning and effects.

{ajax: true,
 skin:"white",
 hook: "bottomright",
 border: { size:1},
 afterUpdate: function()
{}
}

Because the column formula does not include the whole script (it references to the scripts loaded in the Narrative View) we will get a “Formula syntax is invalid” error:

graph7

We can ignore it, as it will work correctly with the whole script at the report execution. We also need to change the Data Format Column Properties:

graph8

We need to override Default Data Format in order to treat text as HTML:

graph9

Let’s make sure that report has the Static Text view added to the Compound Layout.

graph10

Now we can try out our development. The table with Orders should display the tooltip with navigation to target report with the Order details.

graph11

On hover, the order details from the target report are displayed in a tooltip:

graph12

The contents of the tooltip are displayed dynamically based on the navigation condition (in our case Order Number)

Maintenance and other considerations

We need to take into consideration that by embedding scripts to the OBIEE reports we are adding another layer of complexity.  This is especially true for the maintenance reasons as it requires more time and skills for the end users and developers to edit and create reports that contain scripts. Another concern should be that adding too many scripts to the report may negatively affect the performance of the report. We also need to take into consideration the web browsers used by users in the company. Some of the plugins and the latest versions of jQuery library are fully compatible only with the latest versions of browsers.

Conclusions

Thanks to the possibility of using many web technologies such as JavaScript we are able to customize OBIEE reports and dashboards. Tooltip can be a valuable tool to enhance the functionality of OBIEE reports, allowing users to get detailed information when hovering over a link.

Do you have another method to achieve the pop-up effect? Please leave us a comment below, we would be happy to answer it! If you would like to know more about visual enhancements in OBIEE, read our blog posts series about enhanced visualization:

Rich Visualizations in OBIEE with Javascript

Setting Custom Data Format in OBIEE Answers

.

Sometimes it is necessary to display numeric values in specified format or replace the null values in table or pivot table with zeros or custom text. OBIEE Answers and Dashboards give us a possibility to customize the data masks for presentation. This could be useful when we want to change the display of data for the purpose of a given report. Using Custom Data Format feature we can change the masks for numeric values, change the display of null values or show dates in custom format.  In this article I will provide some examples on how to deal with custom data formatting for numeric values, dates and null values.

Continue reading this post >

privacy policy - Copyright © 2000-2010 ClearPeaks

topnav