[Db-derby Wiki] Update of "DebuggingDerbyJITIssues" by MyrnavanLunteren

[Apache Wiki <wi...@apache.org>]

Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Db-derby Wiki" for change notification.

The "DebuggingDerbyJITIssues" page has been changed by MyrnavanLunteren:
https://wiki.apache.org/db-derby/DebuggingDerbyJITIssues

New page:
= DebuggingJITIssues =
JIT problems manifest themselves in many ways. Often the problem is an absurd incorrect return value or expression evaluation, which can lead to a wide variety of symptoms.  Intermittent issues are good JIT suspects. A JIT issue is always a JVM problem.

However,  intermittent issues can be caused by other factors, such as race conditions or timing issues. The first step for analyzing intermittent issues is to run the program multiple times, to get a feeling for the frequency. Then the first step for debugging an issue where JIT involvement is suspected is to create a (small) test case, and run it with JIT forced on and with off. If JIT is implicated, it will fail with JIT on, and pass with JIT off.

Waiting for a JVM's JIT fix can be impractical, and switching JIT off completely may cause a severe reduction in performance. But it may be possible to analyze the problem and work around it, either in the source code, or by using tools of the JVM.

For instance, the IBM JVM provides tools for diagnosing JIT issues, described in the section "Diagnosing a JIT or AOT problem" here: http://pic.dhe.ibm.com/infocenter/java7sdk/v7r0/topic/com.ibm.java.zos.70.doc/diag/tools/jitpd_diagnose.html

== Example ==
Below is an example of analysis as it might be applied to a problem in Derby. The details (file numbers, failing methods) have been invented.

0 ) Confirm it's JIT

 . Confirm that the JIT is implicated by running the program showing intermittent wrong behavior with JIT off and ensure it passes. This can be done by running with -Xint or -Xnojit. You may need to run multiple times to be sure, depending on how frequent the failure is. If the suspected problem is in a 'old harness' test, you can pass the properties on with -Djvmparams, but it may be easier to set the properties using the JVM's environment variable IBM_JAVA_OPTIONS.

1) Get a reliable small test case.

 . This is especially important with JIT issues, because  the process of fully analyzing a JIT issue requires many iterations. Ensure that the test case does indeed reproduce the problem. It can be frustrating to get a small test case for a JIT issue because seemingly minor code changes (such as System.out.println lines) may make the problem disappear.

2) Iterate to get the problem to occur consistently.

 . -Xjit:count=0 will force JIT compilation from the first iteration. This will often make an intermittent JIT issue fail reliably.  If this does not work try the various optimization levels:
  . -Xjit:count=0,optLevel=scorching
  . -Xjit:count=0,optLevel=veryHot
  . -Xjit:count=0,optLevel=hot
  . -Xjit:count=0,optLevel=warm
  . -Xjit:count=0,optLevel=cold
  . -Xjit:count=0,optLevel=noOpt
 e.g.
  . java -Xjit:count=0,optLevel=scorching org.apache.derby.tools.ij repro.sql
 . or
  . java -Xjit:count=0 org.apache.derby.tools.ij repro.sql

3) Locate the failing method.

 . Once you have a reliably failing test case you can locate the failing method with a binary search through the JIT's log file.

 . a) Obtain the method compilation levels.
  . Run the small test case by adding  the 'verbose' and 'vlog' options ('-Xjit:count=0.verbose.vlog=<filename>') to the command line. This will list the compiled methods in a file named  <filename>.<date>.<time>.<pid> e.g.
   . java -Xjit:count=0,verbose,vlog=vlog.out org.apache.derby.tools.ij repro.sql
  . will create a file called vlog.out.20131015.104401.6840.

 . b) Iterate through the vlog file.
  . First identify how many lines (methods) there are in the file resulting from the run with 'verbose.vlog' (For instance, open it in an editor that has a line count, or use command line options like shell's 'wc' command). Then run the program again with the JIT parameter 'limitFile=(<filename>,<m>,<n>)', where <filename> is the path to the vlog result file, and <m> and <n> are line numbers indicating the first and the last methods in the 'limit' file that should be compiled. Using this approach, you can perform a binary search to find the line identifying the failing method. For example, for a 1000 line file you might run with
   . java -Xjit:count=0,limitfile=\(vlog.out,1,500\) org.apache.derby.tools.ij repro.sql
  Repeat, depending on whether this reproduces the problem, switch to the other half, or narrow down further. The iterations might look something like this:
   . java -Xjit:count=0,limitfile=\(vlog.out,1,500\) org.apache.derby.tools.ij repro.sql  -> reproduces
   . java -Xjit:count=0,limitfile=\(vlog.out,1,250\) org.apache.derby.tools.ij repro.sql  -> not
   . java -Xjit:count=0,limitfile=\(vlog.out,250,500\) org.apache.derby.tools.ij repro.sql  -> reproduces
   . ..
   java -Xjit:count=0,limitfile=\(vlog.out,250,258\) org.apache.derby.tools.ij repro.sql  -> reproduces
   . java -Xjit:count=0,limitfile=\(vlog.out,250,257\) org.apache.derby.tools.ij repro.sql  -> not
   . java -Xjit:count=0,limitfile=\(vlog.out,258,258\) org.apache.derby.tools.ij repro.sql  -> reproduces

  . Open the file to find the problematic method, for instance:
   . :258: (warm) java/lang/Math.max(II)I @ 067FE280-067FE2A5 JavaOrdinaryMethod Q_SZ=0 Q_SZI=0 QW=6 bcsz=11 sync compThread=0 OSR=0 CpuLoad=0%(0%avg) JvmCpu=0%

4) report the issue.

 . You should report the problem to the JVM vendor using your support channels. Assuming it is a problem that shows up using Derby, you can additionally log an issue in JIRA, identifying the full details of the JVM and Derby version. This will help other users running into the same issue. However, the Derby community will not report a JVM issue for you, and the issue will be closed as 'Invalid' in JIRA.

5) Work around the issue.

 . Once you have located the failing method, you can work around the issue. One way is to use the JVM's tooling by either telling the JIT to compile the method at a lower optimization level:
  . java -Xjit:{java/lang/Math.max(II)I}(optLevel=cold,count=0) org.apache.derby.tools.ij repro.sql
 or excluding just this method from JIT compilation, e.g.
  . java -Xjit:exclude={java/lang/Math.max(II)I} org.apache.derby.tools.ij repro.sql