OBIEE Data Lineage Solution

.

Tracking OBIEE reporting data

BI implementations may use hundreds of data warehouse tables and thousands of reports, and one of the most common requirements in a BI project is the ability to track back the data used in reports. The possibility to quickly identify the set of columns and tables used in a given report or dashboard, or to check which reports may be affected by a change of physical column in the data warehouse, is also crucial for development. This calls for a data lineage solution which is accessible to users and developers,  and allows checking such data on an ad hoc basis.

Various vendors offer data lineage solutions, but these can be expensive and vendor-specific. With our simple solution, we combine Catalog Manager and Administration Tool export sources to create an easily accessible solution for tracking data lineage in OBIEE.

By implementing the OBIEE data lineage solution, we can check the following:

  1. Which physical tables and columns are used in a given report or dashboard
  2. Which reports use given physical columns or tables; this is especially important when modifying an existing table, as any change in the table’s structure must take existing reports into consideration
  3. Which are the most commonly used columns in reports in a given subject area; identifying the most commonly used columns in a report can hint at creating indexes to improve the overall performance of the OBIEE implementation even further.

Solution

The ClearPeaks OBIEE Data Lineage Solution gathers all the required data lineage information in one place and uses it as a source for OBIEE reports within Data Lineage Subject Area. Two sources are combined to achieve this:

  1. List of OBIEE reports and presentation columns used

Catalog Manager provides an option to export the list of reports and columns used. The export can be done either with the Catalog Manager UI tool or through the command line utility. We use the latter option, as it allows automation of the whole process later.

  1. Mappings of presentation columns back to the physical columns

Such mappings can be obtained manually by creating Repository Documentation from Administration Tool’s Repository Documentation utility. The output will be a repository documentation in the form of comma separated values file (.csv). This will contain column mapping from presentation through logical to physical layers, including column formula expressions. Another way to obtain column mappings is by extracting the OBIEE repository in .xml file format through the command line utility. Our solution uses the second option, as the repository file will be used in our automated script.

OBIEE Data Lineage Solution

Once we have obtained both files, we need to populate the data lineage tables.

The data can be transformed and manually inserted into the tables, but in our solution we use a script (which can run on the OBIEE server) that parses the data and inserts it into the tables.

Data Lineage Subject Area

Once we have populated the data lineage tables and their data model has been mapped in Administration Tool, we can create and run reports in OBIEE using Data Lineage Subject Area and filter the results according to our requirements.

OBIEE Data Lineage Solution

Let us look at a few of the use cases for the Data Lineage Subject Area:

Use case 1. Which data warehouse tables and columns are used in a given report?

We would like to know which data warehouse tables and columns are used in a particular report or dashboard. We can create a report with a list of the columns used by a given OBIEE report and their data lineage:

OBIEE Data Lineage Solution


Use case 2.
Which reports use given physical tables?

We want to know how many and which reports or dashboards are using given physical tables or particular columns; this could be very useful when assessing the potential impact of column formula or table structure changes on reporting. Using Data Lineage Subject Area we can fetch up the list of OBIEE reports used by a given physical table:

OBIEE Data Lineage Solution


Use case 3.
Which reports use given subject areas?

We need to know which reports and dashboards are accessing data from given subject areas. This may be particularly useful when revising users’ access permissions.

OBIEE Data Lineage Solution


Future Improvements

OBIEE Data Lineage Subject Area can also serve as a backbone for further developments, providing complex and complete solutions for managing existing OBIEE project implementations. Here are some examples and potential benefits of merging additional information into the solution:

Usage Tracking data – allows analysis of the physical tables used in the most accessed reports, the tables and columns in reports not used by anyone, removing non-used reports or underlying tables.

Data warehouse database metadata – such as table size, indexes on columns. This allows for performance analysis of the most heavily used tables and columns by report usage.

ETL Data Lineage – additional layer of data lineage tracking – allowing the tracking of data back to the very source – can be achieved by adding ETL transformation data obtained from ETL systems. For example, it is possible to track all the ETL transformations on a given presentation column down to the source system.

Catalog metadata – it is possible to extract additional information regarding catalog objects such as user permissions, report owners, last modified date etc., to further enhance the solution usability.

Adding all the above components creates a strong core for the OBIEE management dashboard, allowing continuous improvement through:

* tracking data from the report to the source database column
* constant analysis of the most used tables in order to improve performance
* checking which users have permissions for given OBIEE reports and dashboards

All of the above is accessible from the OBIEE front-end, providing a convenient and quick way to facilitate many daily business-as-usual tasks in BI deployments.

Conclusion

The ClearPeaks OBIEE Data Lineage Solution can be easily deployed in any project using Oracle Business Intelligence Enterprise Edition. The solution can be run from the command line tools, which makes it possible to create automated jobs to extract and update data on a regular basis.

If you think your OBIEE project could benefit from our data lineage solution, please contact us for more information via our web contact form or by leaving your comments below!

Blog Article Author: Rafal Ostaszewski

Write Back Functionality in OBIEE

.

About Write Back Functionality:

One of the interesting attributes that OBIEE provides is the facility to enable users to add/update data back to the database. The user can have a column for which values can be entered in the user interface (UI) section on their platform and this can be updated in database. This could have multiple benefits as end users may want to rank their customers or rate their regional business based on performance, and be able to use this data from time to time. This converts OBIEE into a useful reporting tool and mini application for modifying business data.

Requirements for implementing the functionality:

Implementing write back requires the configuration of multiple objects within the architecture i.e. Database, Connection Pool, Presentation, BMM and Physical Layers, UI privileges, Column/Table properties etc.

Example on implementing the Write back functionality:

Here I am going to demonstrate how to make the Attribute2 column in the Product table (Sample apps) to be a writeable column.

  • Edit instanceconfig.xml

This is the initial step to enabling Write Back in OBIEE. Open the instance config file from the location – <Middleware>/instances/instance1/config/OracleBIPresentationServicesComponent/coreapplication_obipsn

Under <DSN>, Add <LightWriteBack >true</LightWriteBack >

  • Enable Write Back in the Repository tables

Open the RPD in Offline mode. Then expand the Logical table Product in the BMM layer. Double click on the column Attribute2 and in the general tab enable ‘Writeable’.

image 1

In the presentation layer expand the table Product, double click on the column Attribute2, and in permissions change this column as Read/Write for BI author.

image 2

  • Setting direct database request permission

In the RPD, goto manage > Identity > application roles > BI Author > Permission> select execute Direct DB request> select Allow

image 3

  • Disable cache for physical tables

Select the SAMP_PRODUCTS_D table in the physical layer and disable cacheable option.

Double click on D2 customer > unselect override source table and cacheable.

image 4

Deploy the modified RPD and restart the BI Presentation services.

  • Grant write back privilege to users

Log on to OBIEE presentation services > Administration > manage privileges > Write Back property and click on denied: authenticated user > granted: to authenticated user

  • Create Analysis for Write Back

Create a new analysis with columns P1 Product and P6 Attribute2. Open the column property of Attribute2, select the Write Back tab and enable it. Save the analysis.

image 5

  • Create write back XML template

Goto <Middleware>/instances/instance1/bifoundation/OracleBIPresentationServicesComponent/coreapplication_obips1/analyticsRes/customMessages

Append the attached tags to the Write Back template.xml file (attached Write Back template.xml for reference)

<WebMessage name="wb_prod_attribute"> -- This web message is the reference for this block in the presentation

<XML>

<writeBack connectionPool="Sample Relational Connection"> -- Set the name as in the RPD file

<insert></insert>

<update>

UPDATE SAMP_PRODUCTS_D SET ATTRIBUTE_2='@2' WHERE PROD_DSC='@1' –- define the update query and refer the columns with their position in the answers

</update>

</writeBack>

</XML>

</WebMessage>

image 7 image 6

  • Enable Write Back in table view

Open the saved analysis > table view > edit view > Table view property > Write Back tab > Select enable Write Back and provide the name as wb_prod_attribute (Saved WebMessage name in the xml). Save the Analysis.

image 8

With this step, we have completed the configuration of Write Back in OBIEE. Now this should be tested in order to validate the Write Back configuration.

  • Testing the Write Back Option

Open the saved report > Click on Update.

This changes the column attribute2 to writeable. Change the value and click apply

image 9

Edit the column to the desired value.

image 10

Click Apply and Done

Now open the SQL developer and check the Product in the edited row.

SELECT PROD_DSC,ATTRIBUTE_2 FROM SAMP_PRODUCTS_D where prod_dsc = ‘7 Megapixel Digital Camera’

image 11

Now we can see that the changes made in the answers are reflected in the DB.
By using this simple this technique OBIEE can act as a front end form for updating data in the database.

Pop up effect in OBIEE using jQuery

.

In this article of our OBIEE Customization Series I will show you how to introduce the Pop up effect in OBIEE using jQuery. The tooltip can provide additional details (like an explanation of a KPI) or load content such as target report.

Motivation

For our example let’s imagine we have a summary level report with purchase orders listed. We can navigate to order details level, but that would require leaving the current page or opening the content of detail report in a new window. What if we could have a summary level report and peek into details on the same page? Implementing the dynamic navigation to a report inside the tooltip will allow us to achieve that.

Pop up effect in OBIEE using jQuery

Because OBIEE Answers and Dashboards allow using web technologies such as HTML5, JavaScript, CSS3, and AJAX we are able to customize the look and user interface of any report just like we can do with websites. To achieve that, we will use the most popular JavaScript library called jQuery. It’s an optimized .js file containing Document Object Model (DOM), events, effects and Ajax functions that can be called from the report. JQuery is widely used in website development and allows implementing complex components using pre-defined methods defined in library. For the tooltip we will need also a plugin, an additional library that defines the effect. There are many available plugins for tooltip functionality both free and requiring paid license. Many UI libraries such as jQuery UI or Kendo UI offer, among others, a tooltip functionality. For the example in this article we will use the Tipped tooltip library, which offers a lot of customization options but requires a licence for commercial use.
As an example in this article we will invoke a tooltip that loads the contents of target report when hovered on a column’s value. To create this effect we will need to use AJAX technology. Such method is already included in the jQuery library. We will need to specify the target page or report from which the external content will be loaded.

Implementing the tooltip

Let’s start by creating a simple report showing Order Number, Number of Order Lines and Order Cost.

graph1

Our objective here is to provide additional information of the orders displayed in the first report but without navigating. Therefore let’s create another simple report that will contain more details on a given Order and its Order Lines. We need to add a filter - Order Number is prompted to enable navigation.

graph2

Let’s go to the results tab and edit the views. The contents of this report will be displayed in a tooltip. We can add a pie chart with breakdown of the line cost and save the report as Order Lines Report.

graph3

Let’s get back to the main report. We will need to add a static text view with code to load libraries. Those files could be referred pointing to the file on the internet (we can do it by loading jQuery library from google developers and for example jQuery UI library which contains tooltip functionality). However we should have those files on our OBIEE server. Often internal network security settings disallow loading content (especially script files with .js extension) from external sources. A one-domain policy is used to prevent security breaches. Because of that we will need to save those library files on OBIEE server. To be able to link a file on a OBIEE server we will need to create a virtual directory on OBIEE server in weblogic settings.
When we have the virtual directory created we can refer to the libraries using such link: http://localhost:9704/userfiles/jquery.1.9.0.min.js

graph4

 

First, we need to load the jQuery library:

<script type="text/javascript" src="http://localhost1:9704/user_files/jquery-1.9.0.min.js"></script>

Tipped libraries:  

<script type="text/javascript" src="http://localhost:9704/user_files/Tipped/tipped-3.1.8/js/tipped/tipped.js"></script>
<script type="text/javascript" src="http://localhost:9704/user_files/Tipped/tipped-3.1.8/js/excanvas/excanvas.js"></script>
 

Tipped CSS stylesheet, which contains the styles used for tooltip customization:

<link rel="stylesheet" type="text/css" href="http://localhost:9704/user_files/Tipped/tipped-3.1.8/css/tipped/tipped.css"/>

Then let’s edit the column formula to dynamically generate a tooltip with navigation to Order Line Details.

graph5

We need a couple of elements here. Instead of just "Order"."Order Number" column value we need to embed some more elements in the formula code. The code inside will contain HTML tags so let’s mark Constains HTML Markup option.

graph6

For our example we will need to use the following column formula: '

'<div id="popup' ||"Order"."Order Number"|| '">
' ||"Order"."Order Number"|| '
</div>'
||'<script type="text/javascript"> Tipped.create("#popup' ||"Order"."Order Number"||'", "saw.dll?Go&path=%2Fshared%2FReports%2FOrder%20Lines%20Report&Action=extract&p0=1&p1=eq&p2=%22Order%22.%22Order%20Number%22&p3='||"Order"."Order Number"||'",
{ajax: true,
 skin:"white",
 hook: "bottomright",
 border: { size:1},
 afterUpdate: function()
{}
} );
</script>'

Explanation:
To separate HTML formatting from the OBIEE column value ("Order"."Order Number") we will need to use single quote (') and to merge HTML formatting with the column value we need to use concatenate (||).

<divmarkup is a placeholder for our tooltip. Its id is generated dynamically by concatenation with value of the "Order"."Order Number" column.
Here we initialize Tipped script and call plugin’s method to create a tooltip. The syntax of the script to create a Tipped tooltip is as follows:
<script type="text/javascript">
Tipped.create("#target", "url", { ajax: true });
</script>

# is a jQuery selector that points to a single element with the given id attribute, in our case #popup div concatenated with Order Number. Then there is a url with the external content to be loaded inside ("url"). The important part here is the ajax: true option, which allows loading of external content.

<script type="text/javascript"> Tipped.create("#popup' ||"Order"."Order Number"||'

So basically, the popup will display whatever contents are displayed in the url, therefore we can pass a GOURL link to the detail report passing the order as a parameter. Note the Extract parameter which displays just the results of target report in a format without the paging control, hot links and other elements.

"saw.dll?Go&path=%2Fshared%Reports%2FOrder%20Lines%20Report &Action=extract&p0=1&p1=eq&p2=%22Order%22.%22Order%20Number%22&p3='||"Order"."Order Number"||'"

Here we specify the formatting options of the tooltip. For Tipped tooltip to load external content we will need to use  ajax: true option. Tipped offers richness of customization options for skins, positioning and effects.

{ajax: true,
 skin:"white",
 hook: "bottomright",
 border: { size:1},
 afterUpdate: function()
{}
}

Because the column formula does not include the whole script (it references to the scripts loaded in the Narrative View) we will get a “Formula syntax is invalid” error:

graph7

We can ignore it, as it will work correctly with the whole script at the report execution. We also need to change the Data Format Column Properties:

graph8

We need to override Default Data Format in order to treat text as HTML:

graph9

Let’s make sure that report has the Static Text view added to the Compound Layout.

graph10

Now we can try out our development. The table with Orders should display the tooltip with navigation to target report with the Order details.

graph11

On hover, the order details from the target report are displayed in a tooltip:

graph12

The contents of the tooltip are displayed dynamically based on the navigation condition (in our case Order Number)

Maintenance and other considerations

We need to take into consideration that by embedding scripts to the OBIEE reports we are adding another layer of complexity.  This is especially true for the maintenance reasons as it requires more time and skills for the end users and developers to edit and create reports that contain scripts. Another concern should be that adding too many scripts to the report may negatively affect the performance of the report. We also need to take into consideration the web browsers used by users in the company. Some of the plugins and the latest versions of jQuery library are fully compatible only with the latest versions of browsers.

Conclusions

Thanks to the possibility of using many web technologies such as JavaScript we are able to customize OBIEE reports and dashboards. Tooltip can be a valuable tool to enhance the functionality of OBIEE reports, allowing users to get detailed information when hovering over a link.

Do you have another method to achieve the pop-up effect? Please leave us a comment below, we would be happy to answer it! If you would like to know more about visual enhancements in OBIEE, read our blog posts series about enhanced visualization:

Rich Visualizations in OBIEE with Javascript

Oracle Discoverer De-Support: how do you visualise your future BI and reporting?

.

If you are still using Oracle Discoverer for some or all of your reporting and BI requirements, you are probably struggling to deliver on user demands for better mobile/tablet access and richer information visualisation to improve interpretation, understanding and informed decision-making.

Also, it’s not new news that Oracle Discoverer is on its way out.  It ceased to be Oracle’s strategic BI toolset as long ago as 2006, when OBIEE came to the fore as Oracle’s new flagship BI platform.  Premier Support ceases in June 2014, and Oracle recommends all customers to migrate away from Discoverer by June 2015.

Many customers invested considerably in Discoverer prior to the release of OBIEE, and through challenging economic times have sweated the asset rather than undertake the transition to a more state-of-the-art platform.

Now that Discoverer is moving into its final stages of life, it is time to revisit the strategy for current Discoverer customers.

Why should I be migrating away from Discoverer?

From a business viewpoint, you are already well behind the curve in terms of the BI capability offered to you by Discoverer – and that could easily mean that you are falling behind your competitors who have gained the edge of working with newer, more able technology.

The releases of Discoverer around 2006, including Discoverer for OLAP, were actually pretty good at the time and brought it to a comparable level with other products of the day.  But just look around at what else has changed since then in the IT world – mobile devices, social media, advanced visualisation, big data – so that today, even if Discoverer is doing what it says on the tin, it is not equipped to move forward in any way.  Many end-user capabilities that are considered “must have” today are just not possible with Discoverer; indeed, even in its heyday, whilst Discoverer was a good workhorse, it never did score particularly good ratings for usability.

The last “features” release of Discoverer was 11gR1 in June 2009, and the terminal release of bug fixes and minor features was 11.1.1.7.0.  According to Oracle’s latest Statement of Direction (March 2014), there will be no further releases.  Premier Support ceases in June 2014, and Oracle recommends all customers to migrate away from Discoverer by June 2015.

A common objection from some customers is that they have a large estate of Discoverer reports and it would be far too major, complex and costly a task to replace them all.  Some customers have many hundreds of Discoverer reports, so at first glance it may indeed look like a huge task and not worth the cost.

However, a closer look may reveal that:

  • Actually, a large percentage of the “hundreds of reports” is redundant now – many of them are old versions or multiple temporary re-hashes of the same report.  So it may be that only tens of reports are still fit for active purpose.
  • Many of the reports are being used solely as a staging post to extract data to Excel for wrestling into shape for the final output – this is clearly an inefficient, error-prone and costly (in human time) process
  • There is a now cottage-industry of Excel (or other tools) producing newer report requirements that were not built into the Discoverer suite because it was not meeting the business need.  Again this could be costly, slow and error-prone.  Maybe the IT department thinks that Discoverer is still delivering what is required, but the business has moved on and may even have purchased other BI tools
  • Every time the underlying systems are upgraded, the whole Discoverer suite has to be re-tested and modified – Discoverer does not have the same metadata capabilities as OBIEE.  Many customers implemented Discoverer direct against transactional systems or eBusiness Suite without a data warehouse, meaning that every change to an underlying database could mean major rewrites of the Discoverer reporting suite.  The cost of a few iterations of this could outweigh the cost of replacement.

What do the changes to available Support mean?

When Premier Support ends (June 2014), the main change is that Certification with most new third party products/versions and new Oracle products ceases.  You can choose to keep the rest of the available Support features by paying an additional fee for Extended Support (free to June 2015).

If you choose not to opt for Extended Support, or in any case from June 2017, then you are on Sustaining Support.  This provides updates, fixes, security alerts, critical patch updates and upgrade tools/scripts for pre-existing issues only – not for anything new that occurs.  Also, there is no further certification against other Oracle or third-party products.  Basically you could maintain the software as-is, but if you were to change any of your source systems, database versions, etc, then you would be running the risk that if it broke, you would have no Support redress.

You are maybe planning on moving forward in other areas – e.g. an R12 Upgrade to eBusiness Suite, or migrating your database to 12c.  If you have any such major changes planned, it makes even more sense to reconsider whether Discoverer remains fit for purpose or whether it is now time to look at its replacement.  If you plan to move to the Fusion edition of eBusiness Suite, BI Foundation Suite becomes a pre-requisite.

Oracle specifically recommends that you should migrate by June 2015.

What should I replace it with?

Conventional wisdom, and Oracle recommendation, says:

  • Migrate the licenses to Oracle BI Foundation Suite or OBIEE.  There may be a partial rebate on the costs if you have a current Support contract, but this depends on versions and needs to be checked with your Oracle Account Manager (ClearPeaks can facilitate and advise on this)
  • For each Discoverer report, consider whether it is more appropriate to replace with Answers (the OBIEE analytic tool) or BI Publisher (the OBIEE tool for more structured reporting)
  • If you are an eBusiness Suite customer with packaged Discoverer content, replace with Oracle BI Applications

However, there are further considerations in making your decisions:

  • If you run direct against source databases – e.g. eBusiness Suite – would you benefit from changing to a datamart/warehouse approach to both future-proof your reporting and open up your data for analysis by making it available in a more digestible form for business users?
  • If your Discoverer “reports” are in reality being used for data extraction to another BI toolset, should you replace this manually intensive process with a proper ETL tool such as Oracle Data Integrator?
  • If your user base is small, could you migrate to the less costly BI Standard Edition One? (there are restrictions on this product so you would need advice)
  • If the business users have already moved on and adopted – in part – an alternative BI tool, does it make sense to enhance it to an Enterprise approach and incorporate the Discoverer content into the chosen BI toolset?
  • Would the use of a discovery-based BI tool fit your requirements better, or complement your more structured reporting?

ClearPeaks specialises not only in OBIEE/Foundation Suite, Oracle BI Applications and Endeca, but also in leading third-party BI products, so we are well-placed to provide advice on your best approach.

How should I go about it?

It makes sense to consult the expertise of a specialist BI implementer since, having waited this long before migrating, you will want to get it right!

ClearPeaks can help you to review your current estate of Discoverer reports with a thorough Health-Check and analysis of usage, together with capturing some of today’s aspirational requirements of the business rather than necessarily just doing a like-for-like replacement.  We can assist with licensing considerations, reviewing alternative replacement strategies, and making sure that you get the best fit at the most acceptable cost, with a solution that will hopefully live for as long as your Discoverer solution did!

We can help you to plan the migration, train appropriate IT staff / power users in the new technologies, run and test the migration scripts if appropriate (although in many cases it is more effective to use the old report as a design template and rebuild from scratch), and generally assist you through the implementation.  Or if you prefer –and are all busy with the day-job – we can take the problem away and build your new solution on one of our servers then commission it when ready to switch over.

Just give us a call and we will be delighted to help assess what is right for you!

OBIEE Customization Series – How to impress managers

.

When it comes to BI tools comparison one of the points in discussion is the Data Visualization or also referred as Data Discovery Capabilities.

Why data visualization matters? The human brain is not able to extrapolate data and map it correctly on its context. We are not capable to look at a set of data and at first sight understand the trending and its criticality.

With the use of visual aid like graphs, charts, maps we provide to the audience the capability to understand the data, its trend, the critical parts, and to have a clear picture of the overall situation.

The dashboard designer has to think about how visual aids can help the user to consume the dashboard effectively. The goal here is that the user must be provided with a clear idea of a particular business situation at a glance. Giving clear understanding and keeping things simple will be one of our great assets for user adoption as users will see the benefits from the first time.

You are starting to understand the importance of visualizations on the BI projects and then it is time to know how to implement it, and probably time to add a talented designer into your BI team structure.

On this new visualization series we will cover from the very basic graph creation to the principals of dashboard design and how to present it on a professional fashion, in a nutshell:

privacy policy - Copyright © 2000-2010 ClearPeaks

topnav