Diagnosing out of memory problems in ICS on AIX&windows

from <http://www-1.ibm.com/support/docview.wss?rs=693&context=SSBTEG&dc=DA490&dc=DA4A10&dc=DB520&dc=DA4A30&dc=DB530&dc=DA480&dc=D700&dc=DA4A20&dc=DB560&dc=DA440&dc=DA460&dc=DA430&dc=DA470&dc=DB550&dc=D600&dc=DB600&dc=DA420&dc=DA500&dc=DA410&dc=DB540&dc=DA400&dc=DB700&dc=DB510&uid=swg21250293&loc=en_US&cs=utf-8&lang=en>

Problem

This document is designed to help you collect data for problems related to running out of memory. Gathering specific information before calling IBM Support will help you communicate the problem and save time analyzing the data.

Solution

Garbage collector output (verbose:gc), javacore files, and heapdumps may all be helpful in diagnosing out of memory issues with WebSphere® InterChange Server. If you have already contacted IBM Support, continue reading to begin collecting WebSphere InterChange Server MustGather data. Otherwise, please consult WebSphere Interchange Server MustGather: read first. Collecting JVM specific information Enabling and collecting verbose:gc Verbose:gc logs provide statistics on the state of memory during every garbage collection. To enable verbose:gc, add the verbose:gc flag to the Java™ executable file in the startup script. Then restart InterChange Server to activate verbose:gc logging. To enable verbose:gc logging, refer to the instructions that follow for your platform. . AIX Modify the ics_manager file so that the Java executable command line reads as follows: exec $JAVA_ICS -verbose:gc -classpath "${CWCLASSES}"... Verify that verbose:gc messages are being logged to the ics_manager.log file. Windows Modify the start_server.bat file so that the Java executable command line reads as follows: NaV -verbose:gc -Djava.....%2 %3 %4 %5 2>verbosegc.log Garbage collector output in the previous example is redirected to the verbosegc.log file written to in NaV\jre\ directory. Collecting heapdumps and javacore files Heapdumps contain a list of objects in the Java heap. They are usually created when an out of memory situation occurs in WebSphere InterChange Server. A heapdump can also be forced when a user sends a particular signal to the Java process. The heapdump file can be very large, so ensure that adequate disk space is available. To enable heapdumps, refer to the instructions that follow for your platform. AIX Enable heapdumps by adding the following environment variables. export IBM_HEAP_DUMP=true export IBM_HEAPDUMP=true export IBM_HEAPDUMP_OUTOFMEMORY=true export IBM_JAVADUMP_OUTOFMEMORY=true export IBM_JAVA_HEAPDUMP_TEXT=true optional: export IBM_HEAPDUMPDIR=alternate_directory/ export IBM_JAVADUMPDIR=alternate_directory You can add the variables to the ics_manager script or to the profile. Restart InterChange Server to activate heapdumps. Check that heapdumps has been activated by sending a kill -3 while InterChange Server is running. The WebSphere InterChange Server process will be momentarily interrupted to write the heapdump, but will not be killed. A message will be logged to the ics_manager.log file indicating the exact location of the heapdump file. Windows Enable heapdumps by adding the following environment variables. set IBM_HEAP_DUMP=true set IBM_HEAPDUMP=true set IBM_HEAPDUMP_OUTOFMEMORY=true set IBM_JAVADUMP_OUTOFMEMORY=true set IBM_JAVA_HEAPDUMP_TEXT=true optional: set IBM_HEAPDUMPDIR=alternate_directory set IBM_JAVADUMPDIR=alternate_directory You can add the variables to the start_server.bat script or to the Windows environment. Restart InterChange Server to activate heapdumps. Check that heapdumps have been activated by sending a Ctrl-Break signal in the InterChange Server console window. The InterChange Server process will be momentarily interrupted to write the heapdump, but will not be killed. A message will be logged indicating the exact location of the heapdump file. Sending diagnostic information to IBM Technical Support To send diagnostic information to IBM Technical Support, follow the instructions in the document Exchanging information with IBM Technical Support for problem determination. For a list of all technotes, downloads, and educational materials specific to WebSphere Interchange Server, search the WebSphere Interchange Server support site.