Google Analytics Alternative All Things Oracle Business Intelligence and more: Experimenting with the new Oracle BI Baseline Validation Tool

Sunday, November 15, 2015

Experimenting with the new Oracle BI Baseline Validation Tool

The Oracle BI Baseline Validation Tool is a long overdue new tool introduced in 12c that allows BI administrators  to perform regression testing after a BI migration or a BI upgrade. The goal of this post is to document my experimentation with this new tool in Windows 7. As an example I’ve got an environment with OBIEE 11.1.1.7 installed and another equivalent environment with version 11.1.1.9. The BVT is run from the command with a configuration file and set of arguments. Following are the high-level steps for using the Oracle BI BVT:

  1. Establish a baseline (set of dashboards, reports), in this example we compare all items  
  2. Configure the configuration file to specify the location of the target catalog, Oracle
    BI Presentation Services login information, and the plug-ins to activate. A sample configuration testconfig.xml is provided out the box, one can create a new using the BVT utility
  3. Run the Oracle BVT tool three times:
    • First run to generate the baseline
    • Second run to generate the same data from the patched environment
    • Third run to compare results between the before and after web catalog

The BVT allows a fine grain level comparison allowing even the ability to set a maximum tolerance for floating point fluctuations between reports.

Oracle BI Validation Tool Installation

Requirements
    • Java 1.6 and higher
    • Windows: Windows 7 and higher, Windows Server 2012 and higher
    • Linux: Oracle Linux U5 or Oracle Linux 7
    • Mozilla Firefox or IE

It  is shipped with the Oracle Business Intelligence Developer Client Tool, after installation, the tool is located in

[Installation Folder]\bi\components\oracle.bi.bvt

path


Once the zip file is uncompressed


uncompressed


Configuration File


config2


Looking at the sample configuration file, there are 2 sections


Deployment


Description of the target environment that the tests will run against, the deployment name can be updated in order to describe your specific environment, in our case we will define the following deployment names; “OBIEE11117base” to designate our baseline OBIEE 11.1.1.7 environment and ““OBIEE11119upgrade” to designate the upgraded environment. The deployment names will be fed to the BVT as command line arguments


Test Plug-ins


last1


This section describes the different types of comparisons (5) that are available for the web catalog; each type of comparison is developed as a plug-in. Following are the plug-ns available out of the box:



  • Catalog Plug-in Compares the metadata of objects in catalog folders,
  • UI Plug-in [com.oracle.biee.bvt.plugin.ui] : Compares screen captures of individual analyses and dashboards.
  • Reports Plug-in [com.oracle.biee.bvt.plugin.report]: Exports the results of the analyses to CSV and generates a comparison report.
  • Logical Query Plug-in [com.oracle.biee.bvt.plugin.logicalquery] : Downloads and saves the logical SQL generated for analyses
  • Dashboard Plug-in [com.oracle.biee.bvt.plugin.dashboard]: Exports the results of the analyses to XML and generates a comparison
    report.

The plug-ins can be enabled by setting the “enabled” tag to true.


For more details about the plug-ins, consult the official documentation: Comparing Oracle Business Intelligence Deployments Using the Oracle Business Intelligence Baseline
Validation Tool.


Let’s update the sample configuration file and capture the baseline for our OBIEE 11.1.1.7 web catalog. We enter the URL for the BI environment and the username/password. Note that you can chose to enter the password as a command line argument with the “-password” flag followed by the actual password; if no value is passed with the flag, you will be prompted to enter it.


config3


We run the following command in order to capture our 11.1.1.7 baseline, the syntax to capture the configuration for our BI 11.1.1.7 is:


1



bin\obibvt.bat -config TestConfig.xml -deployment BI11117base


Where testconfig.xml is the name of the configuration file and BI111117base is the deployment name to run. As an example, we’ve enabled all the tests and run the BVT to capture our OBIEE 11.1.1.7 baseline. A folder named after the deployment name is created under the results folder, a progress status is displayed on the screen while the processing is underway.


config4


Note that you can set timeout for loading each report, by default, the timeout to download a single report is 15000 milliseconds (15 seconds). Once the script is done executing, sub folders are created in the main results folder


results1


We run the script again in order to capture the 11.1.1.9 environment, this will enable us to run comparisons between environments


bin\obibvt.bat -config TestConfig.xml -deployment BI11119

Once the script is done running, a second folder is created in the main Results folder


config6


We can now run the same script with the –compareresults flag in order to perform comparisons between the 11.1.1.7 and the 11.1.1.9 environments.


config7b


Now navigate to the Comparisons folders and examine the differences between 11.1.1.7 and 11.1.1.9; for each test there is an html file with the color coded test results, relevant details and hyperlinks to items of concern.


Web Catalog comparisons


Details are provided such as :



  • number of of BI objects in the base environment
  • number of BI objects in the target environment
  • items only present in the base environment
  • items only present in the target environment
  • items that are different

Drill down links provide more details where there are exceptions


comp1 


Dashboard comparisons


Differences between dashboard details are provided.


comp2


Logical Query comparisons


Differences in logical SQLs between both environments are documented.


comp3


Reports Comparisons:


An export to CSV of the results for each of the analyses is performed for each environment and a comparison report is performed.


comp4


UI comparisons


UI plugin compares screen captures pixel by pixel of individual analyses and dashboards and displays a comparison report side by side. A score is assigned to each comparison based on the match: a score of 1 indicates a perfect match while 0 indicates a complete mismatch. The threshold for a pass of fail score can be set in the configuration file, it is 0.95 by default. The side by side comparisons is very handy and help you see the discrepancies in look and feel.


comp5


Error


p.s: Note that we were only to have all tests run successfully using Firefox version 34.0; tests will fail while running with Internet 11.0 and Firefox 42.0 we got the following error in the log:


NS_ERROR_XPC_CI_RETURNED_FAILURE: Component returned failure code: 0x80570015 (NS_ERROR_XPC_CI_RETURNED_FAILURE)

Apparently this error is caused by the Selenium web driver in later version of Firefox, you can read more about it here.


It is worth noting that there is a flag in the configuration file that allows you (1) to set the default browser for running the tests and (2) another flag where you can specify the number of browser instances to run concurrently while performing the tests.


last


Conclusion


The goal of this post was to document a preliminary experiment with the new regression testing tool that shipped with obiee 12c; setting up the configuration and the running comparisons is fairly straightforward; the results are detailed and handy. There is no doubt that this tool will be useful for OBIEE regression testing. In conclusion, the Oracle BI development team is making big strides in addressing pain points that have made working with OBIEE and migrating and maintaining different Oracle BI environments a challenge. One shortcoming of the tool though is that BI-Publisher reports are not included in the comparisons, hopefully future releases of the BVT will include this type of comparisons.

12 comments:

  1. This comment has been removed by the author.

    ReplyDelete
  2. Hi,
    The information is very much useful,Thank you.
    I am trying to explore this tool. I have installed OBIEE12C client in my machine, I found this tool under path: ORACLE_BI_HOME\Middleware\Oracle_Home\bi\components\oracle.bi.bvt.
    As you said above, I could not able to see any "Results" folder in the folder structure.
    Could you please help me on this.
    Do I need to install any thing else??

    Thank you,
    Ram.

    ReplyDelete
  3. Results folder will be created automatically once you run the BVT first time.

    ReplyDelete
  4. I get following error while using this tool. Any help
    D:\ObieeBVT>bin\obibvt.bat -config testconfig.xml -deployment PreUpgrade -password
    Enter Password for user:
    INFO: Result folder: Results\PreUpgrade
    INFO: CatalogPlugin: Initialized
    INFO: CatalogPlugin: Set Up for Test Run
    INFO: Starting [com.oracle.biee.bvt.plugin.catalog.CatalogPlugin]:[GetCatalogTest]
    INFO: CatalogPlugin: GetCatalogTest()
    SEVERE: Failed to connect to server. Check if server is accessible: http://dev-obireporting.mckinsey.com/analytics/saw.dll/wsdl/v6/private
    SEVERE: null
    java.lang.reflect.InvocationTargetException
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
    at java.lang.reflect.Method.invoke(Unknown Source)
    at com.oracle.biee.bvt.UpgradeTool.runPluginTests(UpgradeTool.java:675)
    at com.oracle.biee.bvt.UpgradeTool.run(UpgradeTool.java:812)
    at com.oracle.biee.bvt.UpgradeTool.main(UpgradeTool.java:999)
    Caused by: com.sun.xml.internal.ws.wsdl.parser.InaccessibleWSDLException: 2 counts of InaccessibleWSDLException.

    ReplyDelete
    Replies
    1. Any Luck here, I m getting same error..

      Delete
    2. We are also getting the same error. Can anyone help me please?

      Delete
  5. The above issue generally happens when the server URL isn't provided correctly in the xml file. Please check the analysis URL and make sure that the URL doesn't end with "/" character.

    ReplyDelete
  6. Hi,

    Great information, thanks a bunch.

    I am however running into one last issue. To perform a proper regression test, I need to put prompted URL's in the dashboardprompts.txt file. I created these URL's (Catalog -> Report -> Set prompts -> Create prompted URL) and entered them in the corresponding file.

    When I then fire off the tool it starts processing, but when it starts with the prompted URL's, all of them fail, returning the following error: "SEVERE:Failed to decode provided link. Reason: null"

    Anyone got a clue on how to fix this?

    ReplyDelete
  7. Great stuff provided by the Admin here… look into this for Obiee online training

    ReplyDelete
  8. Hey, I want to compare same OBIEE 11g reports in two different environment(basically the XML comparison of the same report in different environment) without using any tools and i need to automate the comparison, is there a way for it? If so Please suggest it. Thanks in advance.

    ReplyDelete