[impala] 01/02: IMPALA-5392: Display full Java stacktraces in /jvm-threadz

[jo...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git

commit 6570c05fd449582c5a0094dd0f3ae0b3f23faf83
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Sun Mar 19 13:56:00 2023 -0700

    IMPALA-5392: Display full Java stacktraces in /jvm-threadz
    
    jvm-threadz provides information about all the JVM
    threads running in an Impalad/Catalogd. Currently, this
    relies on ThreadInfo.toString() for a text representation
    of the thread information and stack. This output only contains
    the first few frames, which can be an issue for deeper stacks
    (e.g. the Catalog talking to HMS via Thrift). The output does
    contain useful non-stacktrace information like what lock the
    thread is waiting on.
    
    This keeps the first line of the ThreadInfo.toString() output
    that contains non-stacktrace information. Then it walks through
    the StackTraceElements from ThreadInfo.getStackTrace() and
    adds the full stack string. This is intended to mimic
    the jstack output (with indentation and "at").
    
    To avoid licensing issues, I have not looked at any of the
    OpenJDK code (which is GPL 2.0) or any of the previous attempts
    at solving this issue (which consulted OpenJDK code).
    
    To make the stacks display correctly in the WebUI page,
    this puts the thread information in a <span> with
    "white-space: pre-wrap", which respects the newlines in
    the original output.
    
    Testing:
     - gdb attached to the HMS and then issued a select in
       impala-shell to force a metadata load. Then, looked at
       the catalogd /jvm-threadz page to see the longer stacks.
    
    Change-Id: I6730336600a8130e6452c682bcf249cac528ceee
    Reviewed-on: http://gerrit.cloudera.org:8080/19632
    Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
    Tested-by: Impala Public Jenkins <im...@cloudera.com>
    Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
 fe/src/main/java/org/apache/impala/common/JniUtil.java | 18 +++++++++++++++++-
 www/jvm-threadz.tmpl                                   |  2 +-
 2 files changed, 18 insertions(+), 2 deletions(-)

diff --git a/fe/src/main/java/org/apache/impala/common/JniUtil.java b/fe/src/main/java/org/apache/impala/common/JniUtil.java
index 6835a1d75..d447156c3 100644
--- a/fe/src/main/java/org/apache/impala/common/JniUtil.java
+++ b/fe/src/main/java/org/apache/impala/common/JniUtil.java
@@ -20,6 +20,7 @@ package org.apache.impala.common;
 import java.io.PrintWriter;
 import java.io.StringWriter;
 import java.io.Writer;
+import java.lang.StackTraceElement;
 import java.lang.management.GarbageCollectorMXBean;
 import java.lang.management.ManagementFactory;
 import java.lang.management.MemoryMXBean;
@@ -328,7 +329,22 @@ public class JniUtil {
       for (ThreadInfo threadInfo: threadBean.dumpAllThreads(true, true)) {
         TJvmThreadInfo tThreadInfo = new TJvmThreadInfo();
         long id = threadInfo.getThreadId();
-        tThreadInfo.setSummary(threadInfo.toString());
+        // The regular ThreadInfo.toString() method limits the depth of the stacktrace.
+        // To get around this, we use the first line of the toString() output (which
+        // contains non-stacktrace information) and then construct our own stacktrace
+        // based on ThreadInfo.getStackTrace() information.
+        StringBuffer customSummary = new StringBuffer();
+        String regularSummary = threadInfo.toString();
+        int firstNewlineIndex = regularSummary.indexOf("\n");
+        // Keep only the first line from the regular summary
+        customSummary.append(regularSummary.substring(0, firstNewlineIndex));
+        customSummary.append("\n");
+        // Append a full stack trace that mimics how jstack displays the stack
+        // (with indentation and "at")
+        for (StackTraceElement ste : threadInfo.getStackTrace()) {
+          customSummary.append("\tat " + ste.toString() + "\n");
+        }
+        tThreadInfo.setSummary(customSummary.toString());
         tThreadInfo.setCpu_time_in_ns(threadBean.getThreadCpuTime(id));
         tThreadInfo.setUser_time_in_ns(threadBean.getThreadUserTime(id));
         tThreadInfo.setBlocked_count(threadInfo.getBlockedCount());
diff --git a/www/jvm-threadz.tmpl b/www/jvm-threadz.tmpl
index 71bde47cf..6c9fc9dc4 100644
--- a/www/jvm-threadz.tmpl
+++ b/www/jvm-threadz.tmpl
@@ -40,7 +40,7 @@ under the License.
   <tbody>
     {{#jvm-threads}}
     <tr>
-      <td>{{summary}}</td>
+      <td><span style="white-space: pre-wrap;">{{summary}}</span></td>
       <td>{{cpu_time_sec}}</td>
       <td>{{user_time_sec}}</td>
       <td>{{blocked_time_ms}}</td>