You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@impala.apache.org by jo...@apache.org on 2023/03/20 19:10:32 UTC
[impala] branch master updated (a6333aed6 -> 9ccb5f7bf)
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
from a6333aed6 IMPALA-10983: Wait more in wait_for_event_processing if there is progress
new 6570c05fd IMPALA-5392: Display full Java stacktraces in /jvm-threadz
new 9ccb5f7bf IMPALA-11937: Fix wrong GROUP BY ordinal in PERF_AGG-Q10
The 2 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
fe/src/main/java/org/apache/impala/common/JniUtil.java | 18 +++++++++++++++++-
.../workloads/targeted-perf/queries/aggregation.test | 4 ++--
www/jvm-threadz.tmpl | 2 +-
3 files changed, 20 insertions(+), 4 deletions(-)
[impala] 01/02: IMPALA-5392: Display full Java stacktraces in /jvm-threadz
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 6570c05fd449582c5a0094dd0f3ae0b3f23faf83
Author: Joe McDonnell <jo...@cloudera.com>
AuthorDate: Sun Mar 19 13:56:00 2023 -0700
IMPALA-5392: Display full Java stacktraces in /jvm-threadz
jvm-threadz provides information about all the JVM
threads running in an Impalad/Catalogd. Currently, this
relies on ThreadInfo.toString() for a text representation
of the thread information and stack. This output only contains
the first few frames, which can be an issue for deeper stacks
(e.g. the Catalog talking to HMS via Thrift). The output does
contain useful non-stacktrace information like what lock the
thread is waiting on.
This keeps the first line of the ThreadInfo.toString() output
that contains non-stacktrace information. Then it walks through
the StackTraceElements from ThreadInfo.getStackTrace() and
adds the full stack string. This is intended to mimic
the jstack output (with indentation and "at").
To avoid licensing issues, I have not looked at any of the
OpenJDK code (which is GPL 2.0) or any of the previous attempts
at solving this issue (which consulted OpenJDK code).
To make the stacks display correctly in the WebUI page,
this puts the thread information in a <span> with
"white-space: pre-wrap", which respects the newlines in
the original output.
Testing:
- gdb attached to the HMS and then issued a select in
impala-shell to force a metadata load. Then, looked at
the catalogd /jvm-threadz page to see the longer stacks.
Change-Id: I6730336600a8130e6452c682bcf249cac528ceee
Reviewed-on: http://gerrit.cloudera.org:8080/19632
Reviewed-by: Wenzhe Zhou <wz...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Quanlong Huang <hu...@gmail.com>
---
fe/src/main/java/org/apache/impala/common/JniUtil.java | 18 +++++++++++++++++-
www/jvm-threadz.tmpl | 2 +-
2 files changed, 18 insertions(+), 2 deletions(-)
diff --git a/fe/src/main/java/org/apache/impala/common/JniUtil.java b/fe/src/main/java/org/apache/impala/common/JniUtil.java
index 6835a1d75..d447156c3 100644
--- a/fe/src/main/java/org/apache/impala/common/JniUtil.java
+++ b/fe/src/main/java/org/apache/impala/common/JniUtil.java
@@ -20,6 +20,7 @@ package org.apache.impala.common;
import java.io.PrintWriter;
import java.io.StringWriter;
import java.io.Writer;
+import java.lang.StackTraceElement;
import java.lang.management.GarbageCollectorMXBean;
import java.lang.management.ManagementFactory;
import java.lang.management.MemoryMXBean;
@@ -328,7 +329,22 @@ public class JniUtil {
for (ThreadInfo threadInfo: threadBean.dumpAllThreads(true, true)) {
TJvmThreadInfo tThreadInfo = new TJvmThreadInfo();
long id = threadInfo.getThreadId();
- tThreadInfo.setSummary(threadInfo.toString());
+ // The regular ThreadInfo.toString() method limits the depth of the stacktrace.
+ // To get around this, we use the first line of the toString() output (which
+ // contains non-stacktrace information) and then construct our own stacktrace
+ // based on ThreadInfo.getStackTrace() information.
+ StringBuffer customSummary = new StringBuffer();
+ String regularSummary = threadInfo.toString();
+ int firstNewlineIndex = regularSummary.indexOf("\n");
+ // Keep only the first line from the regular summary
+ customSummary.append(regularSummary.substring(0, firstNewlineIndex));
+ customSummary.append("\n");
+ // Append a full stack trace that mimics how jstack displays the stack
+ // (with indentation and "at")
+ for (StackTraceElement ste : threadInfo.getStackTrace()) {
+ customSummary.append("\tat " + ste.toString() + "\n");
+ }
+ tThreadInfo.setSummary(customSummary.toString());
tThreadInfo.setCpu_time_in_ns(threadBean.getThreadCpuTime(id));
tThreadInfo.setUser_time_in_ns(threadBean.getThreadUserTime(id));
tThreadInfo.setBlocked_count(threadInfo.getBlockedCount());
diff --git a/www/jvm-threadz.tmpl b/www/jvm-threadz.tmpl
index 71bde47cf..6c9fc9dc4 100644
--- a/www/jvm-threadz.tmpl
+++ b/www/jvm-threadz.tmpl
@@ -40,7 +40,7 @@ under the License.
<tbody>
{{#jvm-threads}}
<tr>
- <td>{{summary}}</td>
+ <td><span style="white-space: pre-wrap;">{{summary}}</span></td>
<td>{{cpu_time_sec}}</td>
<td>{{user_time_sec}}</td>
<td>{{blocked_time_ms}}</td>
[impala] 02/02: IMPALA-11937: Fix wrong GROUP BY ordinal in PERF_AGG-Q10
Posted by jo...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
joemcdonnell pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/impala.git
commit 9ccb5f7bfc5ffef0001f116abe032231feef8efd
Author: stiga-huang <hu...@gmail.com>
AuthorDate: Fri Feb 24 07:50:02 2023 +0800
IMPALA-11937: Fix wrong GROUP BY ordinal in PERF_AGG-Q10
This fixes the wrong GROUP BY ordinal used in PERF_AGG-Q10 so the query
can actually run.
Verified by
bin/run-workload.py --workloads=targeted-perf \
--table_formats=parquet/none \
--query_names=PERF_AGG-Q10
Change-Id: I285f74a12bb16c60d30ffe958337b054baf86479
Reviewed-on: http://gerrit.cloudera.org:8080/19531
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Daniel Becker <da...@cloudera.com>
Tested-by: Daniel Becker <da...@cloudera.com>
---
testdata/workloads/targeted-perf/queries/aggregation.test | 4 ++--
1 file changed, 2 insertions(+), 2 deletions(-)
diff --git a/testdata/workloads/targeted-perf/queries/aggregation.test b/testdata/workloads/targeted-perf/queries/aggregation.test
index 68b9b07db..9723a47b1 100644
--- a/testdata/workloads/targeted-perf/queries/aggregation.test
+++ b/testdata/workloads/targeted-perf/queries/aggregation.test
@@ -2714,7 +2714,7 @@ SELECT
'grouping value' g
FROM
lineitem
-GROUP BY 2
+GROUP BY 3
---- LABELS
AGG1,AGG2,G
---- RESULTS
@@ -2732,4 +2732,4 @@ row_regex: .*FastLimitCheckExceededRows: [1-9][0-9]*
select l_orderkey from lineitem group by 1 limit 10;
---- RUNTIME_PROFILE
row_regex: .*FastLimitCheckExceededRows: [1-9][0-9]*
-====
\ No newline at end of file
+====