Hyperion Planning JVM OutOfMemory

Hyperion Planning
Oracle WebLogic Server - Version 11.1.1.6.0 to 11.1.1.6.0 [Release 11g]
Oracle Exalytics Software - Version 2.0.0.0.0 to 2.0.0.0.0 [Release 2.0]
Information in this document applies to any platform.

Purpose

 The purpose of this KM to look at some of of the common reasons for getting the OutOfMemory error messages.

Scope

We will discuss setting the heap for the Planning environment and resolving the OutOfMemory error messages that are commonly encountered.

Details

Setting the heap for the Planning JVM.

System performance is greatly influenced by the size of the Java heap available to the JVM. The server heap settings depends on number of Planning applications, number of users on system, typical operations. Optimal JVM settings can be determined only by close monitoring of application server performance under peak realistic load. Newly deployed web application server instances use default memory heap settings, which are often too small to accommodate EPM System requirements.
Main points to consider when setting the heap:

• Max heap size should not exceed the available RAM. The heap sizes should be set to values such that the maximum amount of memory used by the JVM does not exceed the amount of available physical RAM. If this value is exceeded, the OS starts paging and performance degrades significantly. The JVM always uses more memory than the heap size. The memory required for internal JVM functionality, native libraries outside of the JVM, and permanent generation memory (for the Sun JVM only: memory required to store classes and methods) is allocated in addition to the heap size settings.
  
• Use 64 bit JVM for allocation of a larger heap. You cannot use 32bit JVM in order to allocate a heap of more than 1024 MB, you need to use a 64bit JVM in order to allocate more than 1024 MB of maximum heap. Development recommends for servers running Web applications in the standard deployment model, servers with physical memory of 12 GB and heap sizes of 8 GB. This recommendation is based on performance testing, which has shown that the number of concurrent users can increase almost linearly up to heap sizes of 8 GB. Having servers with larger memory allows greater flexibility with larger applications on these servers.
  Reference link: http://docs.oracle.com/cd/E17236_01/epm.1112/epm_deploy_guide_1112200.pdf.
  
• Garbage Collection. By default, JRockit JVM uses the dynamic garbage collection strategy to optimize for throughput. For EPM component, it is not recommended to change to other available dynamic strategies. By setting minimum heap size = maximum heap size you get a controlled environment where you get a good heap size right from the start and also to minimize minor/major garbage collections.

OutOfMemory Errors and Possible Solutions

At anytime Planning runs OutOfMemory, which  crashes the Planning environment will require a restart of the Planning service to restore functionality. There are many causes for out of memory exceptions, too small of a heap defined, a memory leak, not enough Perm Gen space defined, not enough physical memory being available on the machine where the JVM runs or poorly designed Planning forms, or a memory leak in the Application Code.
Normally when a Panning  runs out of memory it will be displayed in the server logfiles. Below are few errors that you may encounter:

• java.lang.OutOfMemoryError: getNewTla
  Each thread allocates TLA (Thread Local Area) to store short living objects, and benefits GC to reclaim space. The respective thread can then allocate objects in its TLA without synchronizing with other threads. If particular TLA size become full, then new TLA will be allotted. When you encounter the error related to getNewTla you can try the tuning parameter below:
  -XXtlasize:min=32k,preferred=256k,wasteLimit=32k 
  This switch controls the size of memory blocks that are provided to processing threads as they request memory for the creation of objects local to the thread. Increasing the size of the provided blocks reduces memory fragmentation and allows larger objects to be stored in thread-local memory which results in better performance.
  min=size 
  Sets the minimum size of a TLA.
  preferred=size 
  Sets the preferred size of a TLA. The system will try to get TLAs of this size if possible, but will accept TLAs down to the minimum size, if that's what's available. Occasionally, a TLA can get larger than the preferred size, too. The preferred size must not be lower than the minimum size.
  wasteLimit=size 
  Sets the waste limit for TLAs. This is the maximum amount of free memory that a TLA is allowed to have when a thread requires a new TLA.
  
• java.lang.OutOfMemoryError: PermGen space
  The message indicates that the permanent generation space is full. The permanent generation is the area of the heap where class and method objects are stored for HotSpot JDK . If an application loads a very large number of classes, then the size of the permanent generation might need to be increased using the -XX:MaxPermSize option.
  -XX:PermSize and -XX:MaxPermSize
   
• java.lang.StackOverflowError
  If the computation in a thread requires a larger Java Virtual Machine stack than is permitted, the Java  Virtual Machine throws a StackOverflowError. If Java Virtual Machine stacks can be dynamically expanded, and expansion is attempted but insufficient memory  can be made available to effect the expansion, or if insufficient memory can be made available to create the  initial Java Virtual Machine stack for a new thread, the Java Virtual Machine throws an OutOfMemoryError.
  Change the thread stack size by using the -Xss option at JVM startup
  -Xss2048K
    
• java.lang.OutOfMemoryError: Java heap space
  This indicates that the maximum defined heap space in -Xmx has been exceeded. There are two possibilities: the maximum JVM memory setting, Xmx, is too low, or in some cases, the operating system is unable to allocate the requested memory. If the value specified for the Xmx option is already high 4gb or more, then it is possible that the operating system is unable to provide the requested memory, particularly when multiple JRockit virtual machines are running on the same server. JRockit enables a performance optimization that requires part of the JVM to run in the first 2GB of memory. When multiple JVMs are running it is possible for them to run out of space in that 2GB region.
  In this case there are two solutions, either increase the Xmx setting to beyond 4gb  or disable the optimization by adding the JVM option -XXcompressedRefs=false
  -XXcompressedRefs=false
  -Xmx=8Gb
    
• java.lang.OutOfMemoryError: class allocation
  There is insufficient native memory for the Java  Runtime Environment to continue.  This usually related to the Swap space. Increase the swap space on the server and use the parameter to reserve the space for class allocation -XX:MaxClassBlockMemory=75M
  This switch instructs JRockit how much memory to reserve for application code that has been loaded and compiled. The default size of the memory pool is 50M and it is recommended that this size is increased to 75M, especially in the case of a “compact deployment”. Over time JRockit compiles and optimizes more of the application and platform code and this result in an increase of the memory requirement. With the default value of 50M it is possible to encounter an Out-of-Memory error after sufficient up-time and heavy application usage.
  -XX:MaxClassBlockMemory=75M

Useful parameters for enhancing the performance of the JRocket JVM

The Following Parameters can be added to the Planning Service in the registry on Windows or to the setCustomParams.sh|bat  script when not using the service to start the Planning Process.

• -XXaggressive
  This switch instructs JRockit to more assertively optimize application code and utilize memory. By default JRockit balances the cost of performing optimization and the estimated performance improvements that they create. This switch is used to indicate that the server has sufficient resources to perform more optimization up-front without affecting performance.
  
• -XXgcThreads=4
  This switch instructs JRockit how many processing threads to create for memory reclamation processes. Some server hardware may make it difficult for JRockit to determine the appropriate number of threads to create. To support a large number of concurrent users, the value specified should equal the number of physical processors
  
• -XX:JITThreads=2This switch instructs JRockit to use multiple threads for code compilation, rather than the default of a single thread. Values greater than 2 may be specified for more powerful servers, and useful in the case of “compact deployments”, but a value greater than 4 is probably excessive.
  
• -XX:OptThreads=2
  This switch instructs JRockit to use multiple threads for code optimization, rather than the default of a single thread. Values greater than 2 may be specified but 2 threads is probably adequate except in  the case of “compact deployments” in which case a value of 3 or 4 may be preferred. The OptThreads value does not need to be as high as the JITThread value as optimization occurs less often than compilation.
  
• -XX:MaxClassBlockMemory=75M
  This switch instructs JRockit how much memory to reserve for application code that has been loaded and compiled. The default size of the memory pool is 50M and it is recommended that this size is increased to 75M, especially in the case of a “compact deployment”. Over time JRockit compiles and optimizes more of the application and platform code and this result in an increase of the memory requirement. With the default value of 50M it is possible to encounter an Out-of-Memory error after sufficient up-time and heavy application usage.
  
• -XXcompressedRefs:size=32GBWhen multiple WebLogic application servers are running on a server, significant performance gains can be introduced by including the startup switch. It also eliminates the potential for out of memory errors When you run the JRockit JVM on a 64-bit system with a heap size less than 4 GB, if native. This enables heap to grow up to 32 GB heap (practical limit of 25 GB). This will fix the issue native OutOfMemory errors. Moreover heap size still be restricted till the limits defined using -Xmx only benefit would be your heap will get the capability to grow beyond 4 GB up to 25 GB due to increase in size of pointer to 32 bit address.

Steps to make changes to Planning JRocket JVM

The optimizations are enabled via the setCustomParamsPlanning.sh|bat startup command script if the Planning process is on Unix/Linux environment or via the Windows registry if the Planning JVM is started as a service.

Startup Script

In the setCustomParamsPlanning script found at \Oracle\Middleware\user_projects\epmsystem1\bin\deploymentScripts\
The options can be appended to the following line:

SET JAVA_OPTIONS = %JAVA_OPTIONS% -XXaggressive 

Windows Registry

1. To update the Windows registry, the JVMOptionCount value must first be increased to the appropriate decimal value:
   
2. Next a new JVMOption DWORD value is created:
   
3. The value name must be JVMOption# where # is the next number in the options sequence:
   

Planning Application Use and Design to avoid the OutOfMemory Errors

• Turn off process management for Version and Scenario members for which it is not needed
  By default, new Scenario and Version members are enabled for process management. Each Entity-Scenario-Version combination (each Planning unit) costs resources because Planning must check who currently owns the Planning unit and check the security settings. If you have many Scenarios and Versions the number of combinations increases rapidly.
  Decide what Versions and Scenarios need to be available for process management, and then disable the others. If you are not using process management at all then you can disable process management support for all Version and Scenario members. To disable process management support, edit a member of the Scenario or Version dimension. In the member properties, uncheck the "Enable for process management" checkbox. Repeat for each member you want to disable process management for. When finished, do a database refresh using the Administration> Manage Database page and restart the Planning service.
  
  
• Optimize the design of web forms
  Large web forms impose by far the heaviest load on the Planning JVM. Optimizing the design of forms can make a big difference to how fast a form opens and how many users can open it concurrently. Web form design is a complex area. Please see Document 779502.1 for more information on best practice for web form design.