Google Analytics Alternative All Things Oracle Business Intelligence and more: October 2015

Friday, October 30, 2015

OBIEE 12c new command line utilities

Oracle Business intelligence 12c introduced a new set of command line utilities that will make it easier for BI System administrator to manage the OBIEE service instance ; release 12c has shifted the process control operations that used to be performed  from the web applications such as Enterprise Manager Fusion Manager to command line scripting utilities. Operations such as uploading an RPD can no longer be accomplished from EM. The new command utilities are the following:

  • lisconnectionpool
  • updateconnectionpool
  • listrpdvariables
  • updaterpdvariables
  • downloadrpd
  • uploadrpd
  • renameapproles
  • deletapproles
  • renameusers
  • deleteusers

All these commands invoke under the covers a new REST API that enables to collect information and perform certain operations on your instance.

- Examples of information that can be collected  from the API

  • health of BI System instance
  • Service instance status
  • list of services instances

- Examples of operations that can be performed

  • create/delete new service instance
  • Check BI service instance overall health, components health, dependencies health
  • Export BAR file
  • perform operations on the RPD such as  those we listed under the new commands (upload/download RPD etc.) , getrepositories (list of RPD from the BI Server),

To obtain an exhaustive list of services, one can use a REST client such as RESTClient or curl. The screen shot below shows an example using a RESTClient plugin for Firefox:

RESTClientsi

The REST API can be invoked from the following endpoints (PORT is the managed server port, 9502 by default)

http://[hostname]:[PORT]/bi-lcm/v1/s1 : for all listing service instances, creating a new service instance

http://[hostname]:[PORT]/bi-lcm/v1/s1/ssi : for information and operations on our single service instance (ssi)

status

    • /bi-lcm/v1/s1/ssi/health/checks: health check for SSI

checks

    • /bi-lcm/v1/s1/ssi/health/components: health check for internal components

components

    • /bi-lcm/v1/s1/ssi/health/components: health check for dependencies
  • components
  • http://[hostname]:[PORT]/bi-lcm/v1/s1/ssi/rpd: Operations on the RPD for this SI
    • /bi-lcm/v1/s1/ssi/download/downloadrpd:
    • /bi-lcm/v1/s1/ssi/rpd/uploadrpd:
    • /bi-lcm/v1/s1/ssi/rpd/listconnectionpool:
    • /bi-lcm/v1/s1/ssi/rpd/updateconnectionpool:
    • /bi-lcm/v1/s1/ssi/rpd/users/rename:
    • /bi-lcm/v1/s1/ssi/rpd/users/delete:
    • /bi-lcm/v1/s1/ssi/rpd/approles/delete
    • /bi-lcm/v1/s1/ssi/rpd/approles/rename

Utilities

Coming back to the new commands listed in the documentation, there is a main script called data-model-cmd.sh to run that takes each of the new command line utility as an argument.

[ORACLE_HOME]/user_projects/domains/bi/bitools/bin/data-model-cmd.sh

data-modelsh

downloadrpd

To get the correct syntax  for each of the new command line utilities, run the script data-model-cmd.sh with the –h flag; note that the flag needs to be lowercase; for example to show the correct syntax for the downloadrpd command, run the following:

./data-model-cmd.sh downloadrpd–h

Usage

downloadRPD

Note that the port is 9502 on my default installation

downloadRPDexample

you will be prompted to enter the RPD password, then a status message will be displayed.

 downloadRPDresults

Note that the logging level was set to FINEST in order to get more debug output, this is done by altering the logging.properties file located in:

[Oracle_Home]/bi/modules/oracle.bi.commandline.tools/scripts/

log

uploadrpd

Usage

./data-model-cmd.sh uploadrpd  -h
usage: downloadrpd -I <RPD filename> -W <RPD password> -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-SSL] [-Y]
-SSL - use a secure connection to the server

uploadRPD


listconnectionpool


Usage


 ./data-model-cmd.sh listconnectionpool -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-V <true/false>] [-O <outputFile.json>] [-SSL]

File outputFile.json content:
{
"Title":"List Connection Pools",
"Conn-Pool-Info":[
{
"uid":"<uid1>",
"connPool":"<conn pool name1>",
"parentName":"<parent name1>",
"user":"<username1>",
"password":"<password1>",
"dataSource":"<data Source value1>",
"appServerName":"<app server name1>"
},
{
"uid":"<uid2>",
"connPool":"<conn pool name2>",
"parentName":"<parent name2>",
"user":"<username2>",
"password":"<password2>",
"dataSource":"<data Source value2>",
"appServerName":"<app server name2>"
}
],

Output: JSON file containing the list of connection pools and details


lisconnectionpool


updateconnectionpool


Usage


 ./data-model-cmd.sh updateconnectionpool -C <connPoolList.json> -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-SSL]
File connPoolList.json content:
{
"Title":"List Connection Pools",
"Conn-Pool-Info":[
{
"uid":"<uid1>",
"connPool":"<conn pool name1>",
"parentName":"<parent name1>",
"user":"<username1>",
"password":"<password1>",
"dataSource":"<data Source value1>",
"appServerName":"<app server name1>"
},
{
"uid":"<uid2>",
"connPool":"<conn pool name2>",
"parentName":"<parent name2>",
"user":"<username2>",
"password":"<password2>",
"dataSource":"<data Source value2>",
"appServerName":"<app server name2>"
}
],
"Variables-In-Conn-Pool":[
{
"uid":"<uid1>",
"variable":"<VARNAME1>",
"value":"'<VALUE1>'"
},
{
"uid":"<uid2>",
"variable":"<VARNAME2>",
"value":"'<VALUE2>'"
}
]
}

Input file: JSON file containing the changes to be made to the connections pool (1 or many). Example is shown below


updaCPinputfile


updateconnectionpool


listrpdvariables


Usage


./data-model-cmd.sh listrpdvariables -U <cred_username> [-P <cred_password>] [-SI <service instance>] [-S <hostname>] [-N <port_number>] [-V <comma or new line separated FILE containing selected variables names>] [-O <outputFile.json>] [-SSL] [-H]

File outputFile.json content:
{
"Title":"List Rpd Variables",
"Rpd-Variables":[
{
"uid":"<uid1>",
"variable":"<VARNAME1>",
"value":"'<STRING_VALUE>'"
},
{
"uid":"<uid2>",
"variable":"<VARNAME2>",
"value":"<NUMBER_VALUE>"
},
{
"uid":"<uid3>",
"variable":"<VARNAME3>",
"value":"'<EXPRESSION>'"
},
{
"uid":"<uid4>",
"variable":"<VARNAME4>",
"value":"<FUNCTION>"
}
]
}

Output: JSON file containing the list of RPD variable


listrpdVariable


Rather than displaying all the RPD variables, you can create a CSV file containing the list of variables you’d like to output and feed it to the script with the –V flag.


updaterpdvariables


Usage


/data-model-cmd.sh updaterpdvariables -C <rpdVariablesList.json> -U <cred_username> [-P <cred_password>] [-SI <service instance>] [-S <hostname>] [-N <port_number>] [-SSL] [-H]

File rpdVariablesList.json content:
{
"Title":"List Rpd Variables",
"Rpd-Variables":[
{
"uid":"<uid1>",
"variable":"<VARNAME1>",
"value":"'<STRING_VALUE>'"
},
{
"uid":"<uid2>",
"variable":"<VARNAME2>",
"value":"<NUMBER_VALUE>"
},
{
"uid":"<uid3>",
"variable":"<VARNAME3>",
"value":"'<EXPRESSION>'"
},
{
"uid":"<uid4>",
"variable":"<VARNAME4>",
"value":"<FUNCTION>"
}
]
}

updateRPD


renameapproles


Usage


[oracle@demo bin]$ ./data-model-cmd.sh  renameapproles -T <approlenamelist.json> [-L <plugin list>] -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-SSL]

File approlenamelist.json content:
{
"Title":"Target Application Roles",
"App-Roles":[
{ "oldname":"<current_approle1>", "newname":"<new_approle1>" },
{ "oldname":"<current_approle2>", "newname":"<new_approle2>" },
{ "oldname":"<current_approle3>", "newname":"<new_approle3>"}
]
}

deletapproles


Usage




renameusers


Usage


 ./data-model-cmd.sh -T <usernamelist.json> [-L <plugin list>] -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-SSL]

File usernamelist.json content:
{
"Title":"Target Users",
"Users":[
{ "oldname":"<current_user1>", "newname":"<new_user1>" },
{ "oldname":"<current_user2>", "newname":"<new_user2>" },
{ "oldname":"<current_user3>", "newname":"<new_user3>" }
]
}

 


deleteusers


Usage


 ./data-model-cmd.sh  deleteusers -T <usernamelist.json> [-L <plugin list>] -U <cred user> [-P <cred password>] [-SI <service instance>] [-S <host>] [-N <port>] [-SSL]

File usernamelist.json content:
{
"Title":"Target Users",
"Users":[
{ "name":"<user1>" },
{ "name":"<user2>" },
{ "name":"<user3>" }
]
}

The purpose of this post was to experiment with the new command line tools and to list all the commands with the correct syntax for easy future reference. The new commands are part of the first version v1 of the BI Lifecycle Management REST API; there no doubt that more commands will be added to the API in future releases, this is a big step forward for automating operations and monitoring the BI instance without having to interact with the UI.


N.B.


I was unsuccessful at running the user and role management commands (last 4 command listed) : the error message is “Operation failed”, using curl to perform the same operation by invoking the REST API directly failed as well. As soon as we find resolve this issue, this post will be updated.

Tuesday, October 27, 2015

Migrating from OBIEE 11g to 12c: Sample App v406 as an example

The intent of this post is to document the migration steps from OBIEE 11g Linux based Sample Application v406 ( version 11.1.1.7.140415) to 12c and experiment with a new migration utility that was introduced in 12c.     The migration method as indicated in the documentation is “Out-of-place” where a new OBIEE 12c installation is performed  followed by a migration of relevant files to the new installation using the BI migration tool. This new tool generates a single cohesive artifact containing the RPD, web cat and security settings that can be imported into the BI 12c environment; “In-place” (OBIEE binaries upgraded) is no longer available. Note that only OBIEE release 11.1.1.7 or later can be migration to 12c. The migration process used to be a major pain point in the OBIEE lifecycle management.

Install OBIEE 12c

The installation steps were covered in a previous post; even though those steps were for Windows 7; they are  very similar to those in Linux. Note that Sample App v406 include JDK1.7; OBIEE 12c requires JDK1.8_051. You can download the the Java rpm from the Oracle website and install it by issuing the following command

java1a

then configure Java to use the latest JDK, details on updating Java are covered in this blog.

commandLine

install1

config1

One thing to note is that the start|stop|status services take quite a long time to run; even longer in Linux than in Windows from my preliminary testing.According to Oracle Support the potential root cause may be the lack of “entropy” on the Linux operation system not this kind of  entropy, rather the following type of entropy.

Generate the Migration Tool

The tool creates a bundle OBIEE 11g archive which contains security configuration information, OBIEE RPD and web catalog. The migration process is a 2 steps process: first we create the bundle and then we import the bundle into the newly created 12c system. The BI migration tool is a jar file made available after OBIEE 12c installation; it is located in

[ORACLE_HOME]/user_projects/domains/bi/bitools/bin/migration-tool.sh

where bi is the domain name

BAR1a

We run the following command to generate the BI Migration Tool:

./migration-tool.sh package bi-migration-tool.jar

Note that when your run

./migration-tool.sh

at the command line, it will echo the following;

BAr2a

This may mislead you into thinking that the command expect arguments such as “out” etc… That is not the case! The correct syntax is indeed the one that takes “package” and the output jar file as an argument. The syntax above will be used after creating the jar file to generate the export bundle (next section).

  • “package” tells the BI Migration Script to perform a packaging operation.
  • “bi-migration-tool.jar” is the filename of the migration tool jar file where the output is written.

BAR4

BAR5

Here is the jar file that was created

BAR8

Now that we’ve generated the bi-migration-tool.jar, we can create a bundle that contains an export of our metadata information from the 11g Oracle home.

Create the export bundle

The export bundle is created by running the following command at the shell prompt

java -jar bi-migration-tool.jar out <oracle 11g home> <domain home> <output export bundle path>

Where

  • out tells the migration tool to perform and export operation
  • <oracle 11g home> is Oracle BI home directory (Oracle_BI1)
  • <domain home> is the BI11g domain directory which is with out of the box installation  user_projects/domains/bifoundation_domain directory
  • <output export bundle path> is the export bundle where the output is written, note that the output file has to have .jar extension

BAR10

  • /app/oracle/biee/Oracle_BI1 is the OBIEE Sample Application v406 Oracle BI Home
  • /app/oracle/biee/user_projects/domains/bifoundation_domain is OBIEE Sample Application  v406 Oracle BI Home
  • /home/oracle/11gexport.jar is the export bundle

When it’s done and it is successful, the last message is “Migration action succeeded” .BAr11

The export bundle  has been created in /home/oracle/

BAr12

Importing the Export Bundle

 

There are 2  approaches for importing the previously created export bundle:

1) Using the the BI 12 configuration assistant; this is done when you’re performing the BI 12c configuration for the first time. This is the method recommended by Oracle.

2) Using the BI Migration script to import the export bundle , this is for cases when the configuration has already done (domain and BI service instance have been created)

We shall use the second approach given that we’ve already configured our BI 12c environment.

The same script migration-tool.sh script will be used with a different flag “in”

migration-tool.sh in <export bundle> <service instance name>

Where:

  • in tells the migration tool to perform an import operation
  • <export bundle> is the previously created export bundle
  • <service instance name>  is the BI service instance. (ssi)

According to the Oracle documentation, “<service instance name> specifies the name of the service instance, which is service1”.  We observed that the script failed when the argument “service1” is included; the error message is following:

Configuration error: MSI not supported. Service instance must be: ssi

BAr15

10-28-2015 12-04-29 PM

After a while, you’re prompted to enter the RPD password.

One can monitor all the configuration changes by looking at the log file.

BAr16

If the migration is successful; Migration action succeeded is displayed.

BAr17

Next steps after migration

Once the migration is done, the documentation recommends making updates to the repository in order to avoid authentication issues; this is applicable when migrating from OBIEE 11.1.1.7 to 12.2.1. Note that OBIEE Sample App v406 is version 11.1.1.7.140415 therefore this step is not applicable; however I will go through the steps in order to get accustomed with the new command line tools such as  downloadrpd.

Open the OBIEE repository in offline mode:

Navigate to [OBIEE DOMAIN HOME]/bitools/bin

and run the command below:

./data-model-cmd.sh downloadrpd –u uname –p pwd –o downloaded.rpd –w rpd_password

Where:

  • downloadrpd is a new command line introduced in 12c, it will downloads the OBIEE repository
  • uname is username
  • pwd is the password
  • -o is the output RPD
  • -w is the repository password

To get help on the expected argument, run the following command line

./data-model-cmd.sh downloadrpd –h

downloadRPD

We run the command, if successful the message RPD download completed successfully is displayed.

One thing to note is that instead of service1 the OBIEE 12c instance is named ssi , the Oracle documentation does not include the –SI flag, however without it you’ll run into an error message:

Operation failed. Service Instance key “service1” does not exist, probably because service1 is added implicitly as a command line argument when omitted.

 downloadRPD1

Note that OBIEE 12c is more command line centric for deployment, it is no longer possible to upload the OBIEE repository from Web Logic enterprise manager.

We then open the OBIEE 12c repository using the Oracle BI Client Tool which unfortunately still runs only in Windows.

rpd1

The DYNAMIC_OLAP_LOGIN is not present in the list of variables.

Top upload the “updated” repository, we use a new command line utility uploadrp that will load the OBIEE repository ONLINE!!!

No need (no longer possible) to log into Weblogic enterprise manager, navigate to the correct domain, lock the configuration, upload, apply, restart etc…

./data-model-cmd.sh uploadrpd –u uname –p pwd –i downloaded.rpd –w rpd_password –SI ssi

here again, note that the flag –SI is missing from the documentation.

 

uploadrpd

If upload is successful, you get the following

uploadrpd1

Now the moment of truth; log into the new OBIEE 12c application and…success, we can clearly see the list of dashboards from the v406 Sample App.

obiee2

As a sanity check, let’s display the 2.10 Vanilla Charts dashboard to confirm that the analyses are running correctly.

obiee3

Sure enough, the charts/analyses are rendering correctly!

Post Migration steps

There are Post Migration steps to be done after migration not covered here, for more details, refer to  the Oracle documentation, the high level steps are listed below:

The documentation makes reference to a new tool : the Oracle BI Baseline Validation Tool which enables you to identify differences during life cycle operations such as migration, it doesn’t look like the new tool is available for download yet.

In this blog post, we went over the migration steps from OBIEE 11g to OBIEE 12c using the OBIEE Sample Application v406 (version 11.1.1.7.14); there are several new command line utilities and tools introduced in 12c that will simplify the OBIEE lifecycle management; however there are still several steps that will have to be implemented manually.

Epilogue: (to quote the new BI 12c config): It turns out that the Oracle BI Validation Tool is bundled with the BI12c client download; it is located in [BI Client Install Directory]\bi\components\oracle.bi.bvt

Friday, October 23, 2015

OBIEE 12c installation steps

As you probably already know from the blogosphere and twitter, Oracle released OBIEE12c today right before OOW15. This is a quick post to document the software installation in Windows 7 . The software is available on OTN:

http://www.oracle.com/technetwork/middleware/bi-enterprise-edition/downloads/business-intelligence-2717951.html

As indicated on the OTN download page, the installation order is following:

A. Download and install JDK 1.8 from this link, the current release is JDK-8u65

B. Download and install the Web Logic server; installation will be covered in this post

  • fmw_12.2.1.0.0_infrastructure_Disk1_1of1.zip

C. Download and install Oracle Business Intelligence 12c (12.2.1.0.0) consisting of 2 zip file

  • fmw_12.2.1.0.0_bi_windows64_Disk1_1of2.zip
  • fmw_12.2.1.0.0_bi_windows64_Disk1_2of2.zip

D. Download and Install Oracle Business Intelligence Developer Client Tool (12.2.1.0.0) for LINUX installations.

Install Fusion Middleware 12c Infrastructure

this can be accomplished in Windows from the command prompt by issuing:

a. Java –jar fmw_12.2.1.0.0_infrastructure.jar

infra1

  infra2

infra3

Search for Fusion Middleware patches

infra5

infra7

infra8

infra9

infra10

infra11

infra12

infra13

infra14

 

2. Step 2: unzip the two zip file, you should end up with one executable and a folder named Disk2

clip_image001

Install OBIEE 12c by running the setup executable 

Following are few screen shots, one thing to note is that the installation has been simplified; in this release, one type of installation is available: Enterprise installation. Another major departure from 11g is the deprecation of the OPMN service.

screen1

Specify the previously created Oracle_Home

install1

install3

install4

install5

install6

nstall7

install8

install9

nstall10

Configure OBIEE 12c

Navigate to Oracle_Home/bi/bin

and run the configuration script: config.cmd in Windows.

config2

config3

config4

Create the BIPLATFORM schemas, the simplest option is to create new schemas on the fly; one can also using existing schemas created with the RCU.In 12c, the RCU is located in [ORACLE_HOME]\oracle_common\bin

config5

 

Specify the applications port range

config7

Specify that to install: sample application, existing BI application or a clean slate.

config8

Confirmation window

config9

The configuration process starts; there are fewer steps than in 11g and the process runs much faster. One thing to note is that the “Retry” and “Continue” buttons are not available; in OBIEE 12c, the configuration steps cannot be retried. In the event of a failure, one will have to go through the process again (run RCU, clean up domain directory [ORACLE_HOME]/user_projects/domains/bi similar to the 11g Software Only installation and run the configuration steps.

 

config10

config11

As mentioned previously, the configuration steps run much faster; once it reaches 100%, there is particular step (Start All Servers)  that take a relatively long time;

config12

if all goes well

config13

config14

One thing to note at least on 2 similar Windows installation is that the “Oracle Business Intelligence Enterprise Edition” folder that we are accustomed to is not created in the “Program Folder”.

Here is the new exciting OBIEE 12c home page (technically the Oracle Visual Analyzer is the new home page), cleaner user interface based on Oracle Alta UI and dare I say Tableau-like!

homepage

 

With the changes introduced by the new OBIEE 12c version, certain paths have changed; following are new path details:

OBIEE new path to executables and scripts

Start|Stop|Status scripts

[ORACLE_HOME]/user_projects/domains/bi/bitools/bin - stop.sh | start.cmd | start.cmd |status.cmd

startstop

 

[ORACLE_HOME]/bi/lib/bi-bar.jar (CreateBIArchive java executable)

[ORACLE_HOME]/bi/bifoundation/server/bin/

[ORACLE_HOME]/bi/migration-tool/jlib/bi-migration-tool.jar  OBI11g to OBIEE12c migration tool.

OBI Server, Scheduler , ClusterController executables

executables

Catalog manager

[ORACLE_HOME]/user_projects/domains/bi/bitools/bin/runcat.sh

catalog

Javahost

[ORACLE_HOME]/bi/modules/oracle.bi.cam.obijh/startOBIJH.cmd

javahost

Java applications (xmlpserver.ear, analytics.ear etc…)

[ORACLE_HOME]/bi/bifoundation/jee/

BI presentation service executable

[ORACLE_HOME]/bi/bifoundation/web/bin/ sawserver

Repository file and web catalog

[ORACLE_HOME]/user_projects/domains/bi/bidata/service_instances/ssi/metadata

rpd and webcat

Log files

  • [ORACLE_HOME]/user_projects/domains/bi/servers/obis1/logs/

logfiles

Weblogic

  • Administration Server: [ORACLE_HOME]/user_projects/domains/bi/servers
  • Managed Server: [ORACLE_HOME]/user_projects/domains/bi/servers

System Components

  • OBI Server: [ORACLE_HOME]/user_projects/domains/bi/servers/obis1/logs Presentation Server: [ORACLE_HOME]/user_projects/domains/bi/servers/obips1/logs
  • Catalog Manager: /ORACLE_HOME]/user_projects/domains/bi/servers/obips1/logs/catalogmanager
    • Scheduler: [ORACLE_HOME]/user_projects/domains/bi/servers/obisch1/logs
    • JavaHost: [ORACLE_HOME]/user_projects/domains/bi/servers/obijh1/logs
    • ClusterController: [ORACLE_HOME]/user_projects/domains/bi/servers/obiccs1/logs