The Oracle BI Baseline Validation Tool is a long overdue new tool introduced in 12c that allows BI administrators to perform regression testing after a BI migration or a BI upgrade. The goal of this post is to document my experimentation with this new tool in Windows 7. As an example I’ve got an environment with OBIEE 11.1.1.7 installed and another equivalent environment with version 11.1.1.9. The BVT is run from the command with a configuration file and set of arguments. Following are the high-level steps for using the Oracle BI BVT:
- Establish a baseline (set of dashboards, reports), in this example we compare all items
- Configure the configuration file to specify the location of the target catalog, Oracle
BI Presentation Services login information, and the plug-ins to activate. A sample configuration testconfig.xml is provided out the box, one can create a new using the BVT utility - Run the Oracle BVT tool three times:
- First run to generate the baseline
- Second run to generate the same data from the patched environment
- Third run to compare results between the before and after web catalog
The BVT allows a fine grain level comparison allowing even the ability to set a maximum tolerance for floating point fluctuations between reports.
Oracle BI Validation Tool Installation
Requirements
- Java 1.6 and higher
- Windows: Windows 7 and higher, Windows Server 2012 and higher
- Linux: Oracle Linux U5 or Oracle Linux 7
- Mozilla Firefox or IE
It is shipped with the Oracle Business Intelligence Developer Client Tool, after installation, the tool is located in
[Installation Folder]\bi\components\oracle.bi.bvt
Once the zip file is uncompressed
Configuration File
Looking at the sample configuration file, there are 2 sections
Deployment
Description of the target environment that the tests will run against, the deployment name can be updated in order to describe your specific environment, in our case we will define the following deployment names; “OBIEE11117base” to designate our baseline OBIEE 11.1.1.7 environment and ““OBIEE11119upgrade” to designate the upgraded environment. The deployment names will be fed to the BVT as command line arguments
Test Plug-ins
This section describes the different types of comparisons (5) that are available for the web catalog; each type of comparison is developed as a plug-in. Following are the plug-ns available out of the box:
- Catalog Plug-in Compares the metadata of objects in catalog folders,
- UI Plug-in [com.oracle.biee.bvt.plugin.ui] : Compares screen captures of individual analyses and dashboards.
- Reports Plug-in [com.oracle.biee.bvt.plugin.report]: Exports the results of the analyses to CSV and generates a comparison report.
- Logical Query Plug-in [com.oracle.biee.bvt.plugin.logicalquery] : Downloads and saves the logical SQL generated for analyses
- Dashboard Plug-in [com.oracle.biee.bvt.plugin.dashboard]: Exports the results of the analyses to XML and generates a comparison
report.
The plug-ins can be enabled by setting the “enabled” tag to true.
For more details about the plug-ins, consult the official documentation: Comparing Oracle Business Intelligence Deployments Using the Oracle Business Intelligence Baseline
Validation Tool.
Let’s update the sample configuration file and capture the baseline for our OBIEE 11.1.1.7 web catalog. We enter the URL for the BI environment and the username/password. Note that you can chose to enter the password as a command line argument with the “-password” flag followed by the actual password; if no value is passed with the flag, you will be prompted to enter it.
We run the following command in order to capture our 11.1.1.7 baseline, the syntax to capture the configuration for our BI 11.1.1.7 is:
bin\obibvt.bat -config TestConfig.xml -deployment BI11117base
Where testconfig.xml is the name of the configuration file and BI111117base is the deployment name to run. As an example, we’ve enabled all the tests and run the BVT to capture our OBIEE 11.1.1.7 baseline. A folder named after the deployment name is created under the results folder, a progress status is displayed on the screen while the processing is underway.
Note that you can set timeout for loading each report, by default, the timeout to download a single report is 15000 milliseconds (15 seconds). Once the script is done executing, sub folders are created in the main results folder
We run the script again in order to capture the 11.1.1.9 environment, this will enable us to run comparisons between environments
bin\obibvt.bat -config TestConfig.xml -deployment BI11119
Once the script is done running, a second folder is created in the main Results folder
We can now run the same script with the –compareresults flag in order to perform comparisons between the 11.1.1.7 and the 11.1.1.9 environments.
Now navigate to the Comparisons folders and examine the differences between 11.1.1.7 and 11.1.1.9; for each test there is an html file with the color coded test results, relevant details and hyperlinks to items of concern.
Web Catalog comparisons
Details are provided such as :
- number of of BI objects in the base environment
- number of BI objects in the target environment
- items only present in the base environment
- items only present in the target environment
- items that are different
Drill down links provide more details where there are exceptions
Dashboard comparisons
Differences between dashboard details are provided.
Logical Query comparisons
Differences in logical SQLs between both environments are documented.
Reports Comparisons:
An export to CSV of the results for each of the analyses is performed for each environment and a comparison report is performed.
UI comparisons
UI plugin compares screen captures pixel by pixel of individual analyses and dashboards and displays a comparison report side by side. A score is assigned to each comparison based on the match: a score of 1 indicates a perfect match while 0 indicates a complete mismatch. The threshold for a pass of fail score can be set in the configuration file, it is 0.95 by default. The side by side comparisons is very handy and help you see the discrepancies in look and feel.
Error
p.s: Note that we were only to have all tests run successfully using Firefox version 34.0; tests will fail while running with Internet 11.0 and Firefox 42.0 we got the following error in the log:
NS_ERROR_XPC_CI_RETURNED_FAILURE: Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE)
Apparently this error is caused by the Selenium web driver in later version of Firefox, you can read more about it here.
It is worth noting that there is a flag in the configuration file that allows you (1) to set the default browser for running the tests and (2) another flag where you can specify the number of browser instances to run concurrently while performing the tests.
Conclusion
The goal of this post was to document a preliminary experiment with the new regression testing tool that shipped with obiee 12c; setting up the configuration and the running comparisons is fairly straightforward; the results are detailed and handy. There is no doubt that this tool will be useful for OBIEE regression testing. In conclusion, the Oracle BI development team is making big strides in addressing pain points that have made working with OBIEE and migrating and maintaining different Oracle BI environments a challenge. One shortcoming of the tool though is that BI-Publisher reports are not included in the comparisons, hopefully future releases of the BVT will include this type of comparisons.
