Monitoring the Planning JVM in EPM Environment

Hyperion Planning - Version 11.1.2.3.000 and later
Information in this document applies to any platform.
8.31.15

Purpose

The purpose of this article is to provide information on tools that are used for monitoring the Planning JVM. At times when you encounter slow performance or JVM crash the monitoring information is requested by support for troubleshooting and this KM will help you in gathering providing it to Support. The same principals can be applied to any JVM in the EPM System.

Scope

 The target audience for this KM is the Planning Server adminstrators who configure and maintain the Planning environment.
For discussion here we have Planning Deployed on JRockit.
We will use the Jrocket Mission Control (JRMC) and JRCMD and Weblogic Administration console to monitor Planning JVM.
There are many other tools to monitor the JVM that are not covered here.

Details

 

Java Thread Dump

A java thread dump is a snapshot that shows what every thread in the JVM process is
doing at a particular point in time
Important to note:
Thread dumps do not give solutions. They help identify problems and hotspots in running java applications

• The server does not respond to new requests
• Requests time out
• Requests take longer and longer to process
• The server is no longer reported as running

Refer to the Article Note 1098691.1Different ways to take thread dumps in WebLogic Server
Below are the  screen shots of taking thread dump using the Weblogic Console and Jrcmd tool.

Weblogic Console

Start the Weblogic Admin Server.
Logon to the WebLogic Admin Server console at http://server_name.example.com:7001/console
Click Domain Structure -> Environment -> Servers .
Click the name of the managed server, e.g., "Planning0"



Under "Settings for Planning0", select "Monitoring" and then  sub tab "Threads".

Start the test that demonstrates the performance problem.  While the test is running click the "Dump Thread Stacks" button.


Copy and save the page to a new file " Planning0-ThreadDump.log ", starting with
===== FULL THREAD DUMP ===============
Collect the files from <DOMAIN_HOME>\servers\AdminServer\logs\AdminServer.log
<DOMAIN_HOME>/servers/Planning0/logs/Planning0.log.
<DOMAIN_HOME> is usually <EPM_ORACLE_HOME>\user_projects\domains\EPMSystem .
 Attach these files and Planning0-ThreadDump.log to the SR.
Take at least 3 thread dumps, around 10 seconds apart, collected when the server hang issue is observed

Jrcmd

A command line tool that sends the commands to a given JRockit JVM process.
 JAVA_HOME\bin\jrcmd.exe (Windows)
JAVA_HOME/bin/jrcmd.sh (Linux)
 jrcmd <pid> print_threads

Reading the Thread Dump

Reading the logs
Different JVM Vendors display the data in different formats
• Markers for start/end of thread dumps
• Reporting of locks
• Thread states and method signatures
• However, the underlying data (stack) exposed remains the same
across vendors
• Thread dumps can contain lots of data
• Trying to read them in a text editor can be very tricky

Here we are using the  

http://java.net/projects/threadlogic/

Thread Logic to open the same log saved earlier planning

VM  Metrics

You can benefit from the recording’s code profiling that identifies heavily utilized classes or from the event profiling which quantifies time spent; for example, by garbage collection, blocked threads, code compilation.
The flight recorder can be used using the JRocket Mission Control and Jrocket Command  Here are the settings that need to be done for the Flight Recorder to work

Flight Recording With The JRockit Mission Control tool

This provides support for in-depth monitoring of the JRockit Java virtual machine’s environment and its performance as well as providing the ability for product support and developers to analyze runtime behaviors and diagnose failures. It provides out of the box ability to record metrics over a period of time and to preserve those metrics in a file which can be forwarded to support and then to development.
The JRockit Mission Control Client executable is located in JROCKIT_HOME/bin.
Otherwise, you have to type the full path to the executable file, as shown below:

JAVA_HOME\bin\jrmc.exe (Windows)
JAVA_HOME/bin/jrmc (Linux)


The following changes to the startup script setCustomParamsPlanning.sh (on unix or Linux) and the service registry to use the JRMC

-Djava.rmi.server.hostname=dxxx(hostname)
-Xmanagement:ssl=false,authenticate=false,autodiscovery=true,port=8888

Once the Registry has been configured you can use the JRMC to connect to the Planning JVM


Once you are connected to the Planning JVM you can use the JRMC to start the Flight Recorder.
Select the pertinent WebLogic server node, right-click and select Start Flight Recording.


Once the recording completes the flightRecording*.jfr file can be viewed.


From this information in the recording, memory usage, garbage collection, threading, etc. you can use the JRockit documentation found here:

http://download.oracle.com/docs/cd/E15289_01/index.htm

Memory and CPU usage

Monitoring the JVM  to see how memory, CPU usage, threads, and methods are being used during normal operation and to see how the JVM is stressed during specific testing is helpful to get a good idea about the health of the system

Console in The JRockit Mission Control tool

To view real-time behavior of your application and of Oracle JRockit JVM, you can connect to an instance of the JRockit JVM and view real-time information through the JRockit Management Console. Typical data that you can view is thread usage, CPU usage, and memory usage

 Jrcmd commands

 Refer Oracle JRockit Mission Control Use Cases
 http://docs.oracle.com/cd/E13150_01/jrockit_jvm/jrockit/geninfo/diagnos/appMCUseCases.html

Jave Heap Dump

Collect a heap dump at the time of the out of memory condition and provide the dump file to support for review.

On JRockit you can produce an out of memory diagnostic file by passing the following parameters to the setCustomParamPlanning.sh or the Windows registry

-Djrockit.oomdiagnostics=true -Djrockit.oomdiagnostics.filename=/home/oracle

Note that the heap dump file will be a little larger than the maximum defined heap size so make sure that the  defined HeapDumpPath has enough disk space to accommodate the file(s).

In JRockit Mission Control go to the "Advanced" icon then the "Diagnostic Commands" tab and select and execute the "hprofdump" command.
The file is produced in
\Oracle\Middleware\user_projects\domains\EPMSystem\heapdump_Mon_Mar_24_08_41_16_2014.hprof

 

Reading the Heap Dump

A user friendly tool that can be used to open heap dump files is Eclipse Memory Analyzer which is also known as MAT. MAT can be downloaded free from the Web and works out of the box for analysis of JRockit and HotSpot heap dumps. Heap dump files from JRockit will have the name format jrockit_<process id>.hprof

Here  are some screen shots from the heap dump analysis done in Eclipse Memory Analyzer (MAT). Overview Tab of the Biggest Java Objects Retained by Size and the List of Actions and Reports: