Experimenting with the new Oracle BI Baseline Validation Tool

The Oracle BI Baseline Validation Tool is a long overdue new tool introduced in 12c that allows BI administrators  to perform regression testing after a BI migration or a BI upgrade. The goal of this post is to document my experimentation with this new tool in Windows 7. As an example I’ve got an environment with OBIEE 11.1.1.7 installed and another equivalent environment with version 11.1.1.9. The BVT is run from the command with a configuration file and set of arguments. Following are the high-level steps for using the Oracle BI BVT:

1. Establish a baseline (set of dashboards, reports), in this example we compare all items  
2. Configure the configuration file to specify the location of the target catalog, Oracle
   BI Presentation Services login information, and the plug-ins to activate. A sample configuration testconfig.xml is provided out the box, one can create a new using the BVT utility
3. Run the Oracle BVT tool three times:
• First run to generate the baseline
• Second run to generate the same data from the patched environment
• Third run to compare results between the before and after web catalog

The BVT allows a fine grain level comparison allowing even the ability to set a maximum tolerance for floating point fluctuations between reports.

Oracle BI Validation Tool Installation

Requirements

• Java 1.6 and higher
• Windows: Windows 7 and higher, Windows Server 2012 and higher
• Linux: Oracle Linux U5 or Oracle Linux 7
• Mozilla Firefox or IE

It  is shipped with the Oracle Business Intelligence Developer Client Tool, after installation, the tool is located in

[Installation Folder]\bi\components\oracle.bi.bvt

Once the zip file is uncompressed

Configuration File

Looking at the sample configuration file, there are 2 sections

Deployment

Description of the target environment that the tests will run against, the deployment name can be updated in order to describe your specific environment, in our case we will define the following deployment names; “OBIEE11117base” to designate our baseline OBIEE 11.1.1.7 environment and ““OBIEE11119upgrade” to designate the upgraded environment. The deployment names will be fed to the BVT as command line arguments

Test Plug-ins

This section describes the different types of comparisons (5) that are available for the web catalog; each type of comparison is developed as a plug-in. Following are the plug-ns available out of the box:

• Catalog Plug-in Compares the metadata of objects in catalog folders,
  
• UI Plug-in [com.oracle.biee.bvt.plugin.ui] : Compares screen captures of individual analyses and dashboards.
  
• Reports Plug-in [com.oracle.biee.bvt.plugin.report]: Exports the results of the analyses to CSV and generates a comparison report.
  
• Logical Query Plug-in [com.oracle.biee.bvt.plugin.logicalquery] : Downloads and saves the logical SQL generated for analyses
  
• Dashboard Plug-in [com.oracle.biee.bvt.plugin.dashboard]: Exports the results of the analyses to XML and generates a comparison
  report.

The plug-ins can be enabled by setting the “enabled” tag to true.

For more details about the plug-ins, consult the official documentation: Comparing Oracle Business Intelligence Deployments Using the Oracle Business Intelligence Baseline
Validation Tool.

Let’s update the sample configuration file and capture the baseline for our OBIEE 11.1.1.7 web catalog. We enter the URL for the BI environment and the username/password. Note that you can chose to enter the password as a command line argument with the “-password” flag followed by the actual password; if no value is passed with the flag, you will be prompted to enter it.

We run the following command in order to capture our 11.1.1.7 baseline, the syntax to capture the configuration for our BI 11.1.1.7 is:

bin\obibvt.bat -config TestConfig.xml -deployment BI11117base

Where testconfig.xml is the name of the configuration file and BI111117base is the deployment name to run. As an example, we’ve enabled all the tests and run the BVT to capture our OBIEE 11.1.1.7 baseline. A folder named after the deployment name is created under the results folder, a progress status is displayed on the screen while the processing is underway.

Note that you can set timeout for loading each report, by default, the timeout to download a single report is 15000 milliseconds (15 seconds). Once the script is done executing, sub folders are created in the main results folder

We run the script again in order to capture the 11.1.1.9 environment, this will enable us to run comparisons between environments

bin\obibvt.bat -config TestConfig.xml -deployment BI11119

Once the script is done running, a second folder is created in the main Results folder

We can now run the same script with the –compareresults flag in order to perform comparisons between the 11.1.1.7 and the 11.1.1.9 environments.

Now navigate to the Comparisons folders and examine the differences between 11.1.1.7 and 11.1.1.9; for each test there is an html file with the color coded test results, relevant details and hyperlinks to items of concern.

Web Catalog comparisons

Details are provided such as :

• number of of BI objects in the base environment
  
• number of BI objects in the target environment
  
• items only present in the base environment
  
• items only present in the target environment
  
• items that are different

Drill down links provide more details where there are exceptions

 

Dashboard comparisons

Differences between dashboard details are provided.

Logical Query comparisons

Differences in logical SQLs between both environments are documented.

Reports Comparisons:

An export to CSV of the results for each of the analyses is performed for each environment and a comparison report is performed.

UI comparisons

UI plugin compares screen captures pixel by pixel of individual analyses and dashboards and displays a comparison report side by side. A score is assigned to each comparison based on the match: a score of 1 indicates a perfect match while 0 indicates a complete mismatch. The threshold for a pass of fail score can be set in the configuration file, it is 0.95 by default. The side by side comparisons is very handy and help you see the discrepancies in look and feel.

Error

p.s: Note that we were only to have all tests run successfully using Firefox version 34.0; tests will fail while running with Internet 11.0 and Firefox 42.0 we got the following error in the log:

NS_ERROR_XPC_CI_RETURNED_FAILURE: Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE)

Apparently this error is caused by the Selenium web driver in later version of Firefox, you can read more about it here.

It is worth noting that there is a flag in the configuration file that allows you (1) to set the default browser for running the tests and (2) another flag where you can specify the number of browser instances to run concurrently while performing the tests.

Conclusion

The goal of this post was to document a preliminary experiment with the new regression testing tool that shipped with obiee 12c; setting up the configuration and the running comparisons is fairly straightforward; the results are detailed and handy. There is no doubt that this tool will be useful for OBIEE regression testing. In conclusion, the Oracle BI development team is making big strides in addressing pain points that have made working with OBIEE and migrating and maintaining different Oracle BI environments a challenge. One shortcoming of the tool though is that BI-Publisher reports are not included in the comparisons, hopefully future releases of the BVT will include this type of comparisons.