You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2024/09/12 16:43:39 UTC
[PR] MINOR Add a thread dump on build timeout [kafka]
mumrah opened a new pull request, #17181:
URL: https://github.com/apache/kafka/pull/17181
To help with debugging build timeouts, this patch adds a thread dump script to run in parallel with the JUnit tests. 5 minutes prior to the build timeout of 3 hours, this script will iteratively send `kill -3` to the Gradle worker processes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758023169
##########
.github/scripts/thread-dump.sh:
##########
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SLEEP_MINUTES=$(($TIMEOUT_MINUTES-5))
+echo "Dumping threads in $SLEEP_MINUTES minutes"
+sleep $(($SLEEP_MINUTES*60));
+
+echo "Dumping threads now..."
+sleep 5;
+
+for GRADLE_WORKER_PID in `jps | grep GradleWorkerMain | awk -F" " '{print $1}'`;
+do
+ echo "Thread Dump for GradleWorkerMain pid $GRADLE_WORKER_PID";
+ kill -3 $GRADLE_WORKER_PID;
Review Comment:
To my knowledge, the JUnit tests are all run on the GradleWorkerMain java processes. Here's a `jps` during a local test run
```
66818 GradleWorkerMain
66884 GradleWorkerMain
18725 GradleDaemon
66631 GradleWorkerMain
66886 FindBugs2
66889 FindBugs2
65804 GradleWrapperMain
66924 Jps
66862 GradleWorkerMain
66798 GradleWorkerMain
66864 GradleWorkerMain
66899 Main
66841 GradleWorkerMain
13976 Main
66811 GradleWorkerMain
65754 GradleDaemon
65917 GradleWorkerMain
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #17181:
URL: https://github.com/apache/kafka/pull/17181#issuecomment-2346812178
From the raw logs of https://github.com/apache/kafka/actions/runs/10835121701?pr=17181
```
2024-09-12T16:46:09.6343038Z Dumping threads in 5 minutes
2024-09-12T16:46:10.6064558Z Starting a Gradle Daemon (subsequent builds will be faster)
2024-09-12T16:46:22.9075240Z
2024-09-12T16:46:22.9076166Z > Configure project :
2024-09-12T16:46:22.9078028Z Starting build with version 4.0.0-SNAPSHOT (commit id xxxxxxxx) using Gradle 8.10, Java 17 and Scala 2.13.14
2024-09-12T16:46:22.9079901Z Build properties: ignoreFailures=false, maxParallelForks=2, maxScalacThreads=4, maxTestRetries=1
2024-09-12T16:46:28.9054435Z
...
2024-09-12T16:51:14.5056001Z Gradle Test Run :core:test > Gradle Test Executor 9 > AuthorizerIntegrationTest > testOffsetFetchAllTopicPartitionsAuthorization(String) > "testOffsetFetchAllTopicPartitionsAuthorization(String).quorum=zk" PASSED
2024-09-12T16:51:14.5057941Z
2024-09-12T16:51:14.5059650Z Gradle Test Run :core:test > Gradle Test Executor 9 > AuthorizerIntegrationTest > testOffsetFetchAllTopicPartitionsAuthorization(String) > "testOffsetFetchAllTopicPartitionsAuthorization(String).quorum=kraft" STARTED
2024-09-12T16:51:15.0199756Z Thread Dump for GradleWorkerMain pid 2224
2024-09-12T16:51:15.1075100Z
2024-09-12T16:51:15.1076319Z 2024-09-12 16:51:15
2024-09-12T16:51:15.1077590Z Full thread dump OpenJDK 64-Bit Server VM (17.0.12+7 mixed mode, sharing):
2024-09-12T16:51:15.1078565Z
2024-09-12T16:51:15.1080327Z Threads class SMR info:
2024-09-12T16:51:15.1081076Z _java_thread_list=0x00007f35b41089c0, length=16, elements={
2024-09-12T16:51:15.1082118Z 0x00007f365c02c160, 0x00007f365c140d40, 0x00007f365c142130, 0x00007f365c14a340,
2024-09-12T16:51:15.1083426Z 0x00007f365c14b700, 0x00007f365c14cb20, 0x00007f365c14e4e0, 0x00007f365c14fa20,
2024-09-12T16:51:15.1084603Z 0x00007f365c150ea0, 0x00007f365c185930, 0x00007f365c188020, 0x00007f365c8fa100,
2024-09-12T16:51:15.1085898Z 0x00007f365c8fd030, 0x00007f365ca3f8b0, 0x00007f365ca46220, 0x00007f35994fae10
2024-09-12T16:51:15.1087095Z }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758072782
##########
.github/scripts/thread-dump.sh:
##########
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SLEEP_MINUTES=$(($TIMEOUT_MINUTES-5))
+echo "Dumping threads in $SLEEP_MINUTES minutes"
+sleep $(($SLEEP_MINUTES*60));
+
+echo "Dumping threads now..."
+sleep 5;
+
+for GRADLE_WORKER_PID in `jps | grep GradleWorkerMain | awk -F" " '{print $1}'`;
+do
+ echo "Thread Dump for GradleWorkerMain pid $GRADLE_WORKER_PID";
+ kill -3 $GRADLE_WORKER_PID;
Review Comment:
> How can we do that? grep for "kafka" in the thread dump after the fact? I'm not sure we could figure this out prior to generating the dump
"grep" seems to be a acceptable way. we could call `jstack` to check it before archiving it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah merged PR #17181:
URL: https://github.com/apache/kafka/pull/17181
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1757888591
##########
.github/workflows/build.yml:
##########
@@ -110,9 +110,12 @@ jobs:
# --continue: Keep running even if a test fails
# -PcommitId Prevent the Git SHA being written into the jar files (which breaks caching)
id: junit-test
+ env:
+ TIMEOUT_MINUTES: 180
run: |
set +e
- timeout 180m ./gradlew --build-cache --continue \
+ ./.github/scripts/thread-dump.sh &
+ timeout ${TIMEOUT_MINUTES}m ./gradlew --build-cache --continue \
Review Comment:
That aside, should we add `timeout` to the summary? Maybe we can save the thread dump to a file and then archive it to be able download directly.
##########
.github/scripts/thread-dump.sh:
##########
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SLEEP_MINUTES=$(($TIMEOUT_MINUTES-5))
+echo "Dumping threads in $SLEEP_MINUTES minutes"
+sleep $(($SLEEP_MINUTES*60));
+
+echo "Dumping threads now..."
+sleep 5;
+
+for GRADLE_WORKER_PID in `jps | grep GradleWorkerMain | awk -F" " '{print $1}'`;
+do
+ echo "Thread Dump for GradleWorkerMain pid $GRADLE_WORKER_PID";
+ kill -3 $GRADLE_WORKER_PID;
Review Comment:
Should we dump the processes only if they have keyword "kafka"? That can avoid unnecessary dump I'd say
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758044662
##########
.github/scripts/thread-dump.sh:
##########
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SLEEP_MINUTES=$(($TIMEOUT_MINUTES-5))
+echo "Dumping threads in $SLEEP_MINUTES minutes"
+sleep $(($SLEEP_MINUTES*60));
+
+echo "Dumping threads now..."
+sleep 5;
+
+for GRADLE_WORKER_PID in `jps | grep GradleWorkerMain | awk -F" " '{print $1}'`;
+do
+ echo "Thread Dump for GradleWorkerMain pid $GRADLE_WORKER_PID";
+ kill -3 $GRADLE_WORKER_PID;
Review Comment:
Sorry for unclear comment :(
My point was we should print the thread dump only if they have kafka-related stack. The thread dump of this PR have two unrelated processes (GradleWorkerMain).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #17181:
URL: https://github.com/apache/kafka/pull/17181#issuecomment-2346781421
@mjsax I think this can help us debug StreamThreadTest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758066637
##########
.github/scripts/thread-dump.sh:
##########
@@ -0,0 +1,29 @@
+#!/bin/bash
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+SLEEP_MINUTES=$(($TIMEOUT_MINUTES-5))
+echo "Dumping threads in $SLEEP_MINUTES minutes"
+sleep $(($SLEEP_MINUTES*60));
+
+echo "Dumping threads now..."
+sleep 5;
+
+for GRADLE_WORKER_PID in `jps | grep GradleWorkerMain | awk -F" " '{print $1}'`;
+do
+ echo "Thread Dump for GradleWorkerMain pid $GRADLE_WORKER_PID";
+ kill -3 $GRADLE_WORKER_PID;
Review Comment:
How can we do that? grep for "kafka" in the thread dump after the fact? I'm not sure we could figure this out prior to generating the dump
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1759031592
##########
.github/scripts/junit.py:
##########
@@ -259,9 +259,15 @@ def pretty_time_duration(seconds: float) -> str:
# Print special message if there was a timeout
exit_code = get_env("GRADLE_EXIT_CODE", int)
if exit_code == 124:
+ thread_dump_url = get_env("THREAD_DUMP_URL")
logger.debug(f"Gradle command timed out. These are partial results!")
logger.debug(summary)
- logger.debug("Failing this step because the tests timed out.")
+ if thread_dump_url:
+ print("\nThe JUnit tests were cancelled due to a timeout. Thread dumps were generated before the job was cancelled. "
+ "Download [thread dumps]({thread_dump_url}).\n")
Review Comment:
I guess it needs to be `F-Strings`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758115374
##########
.github/workflows/build.yml:
##########
@@ -141,3 +144,11 @@ jobs:
with:
name: build-scan-test-${{ matrix.java }}
path: ~/.gradle/build-scan-data
Review Comment:
(this comment is unrelated to this PR)
Should we add `compression-level: 9` to this archive as all of them are plaintext.
##########
.github/workflows/build.yml:
##########
@@ -141,3 +144,11 @@ jobs:
with:
name: build-scan-test-${{ matrix.java }}
path: ~/.gradle/build-scan-data
+ - name: Archive Thread Dumps
Review Comment:
Maybe the artifact url generated by this task should be passed to "Parse JUnit tests". Than can enable "Parse JUnit tests" to add url to summary (like junit reports)
##########
.github/workflows/build.yml:
##########
@@ -141,3 +144,11 @@ jobs:
with:
name: build-scan-test-${{ matrix.java }}
path: ~/.gradle/build-scan-data
Review Comment:
(this comment is unrelated to this PR)
Should we add `compression-level: 9` to this archive as all of them are plaintext.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on code in PR #17181:
URL: https://github.com/apache/kafka/pull/17181#discussion_r1758033092
##########
.github/workflows/build.yml:
##########
@@ -110,9 +110,12 @@ jobs:
# --continue: Keep running even if a test fails
# -PcommitId Prevent the Git SHA being written into the jar files (which breaks caching)
id: junit-test
+ env:
+ TIMEOUT_MINUTES: 180
run: |
set +e
- timeout 180m ./gradlew --build-cache --continue \
+ ./.github/scripts/thread-dump.sh &
+ timeout ${TIMEOUT_MINUTES}m ./gradlew --build-cache --continue \
Review Comment:
> That aside, should we add timeout to the summary?
Good idea. Will add
> Maybe we can save the thread dump to a file and then archive it to be able download directly.
This is a great idea :) I'll see what I can do here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Add a thread dump on build timeout [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #17181:
URL: https://github.com/apache/kafka/pull/17181#issuecomment-2349045901
@chia7712 here's a run with the latest changes https://github.com/apache/kafka/actions/runs/10850348751?pr=17181
The "Parse JUnit Tests" step produces the following stdout:
```
Gradle command timed out. These are partial results!
1026 tests cases run in 41s. 1026 PASSED ✅, 0 FAILED ❌, 0 FLAKY ⚠️ , 0 SKIPPED 🙈, and 0 errors.
Failing this step because the tests timed out. Thread dumps were taken and archived here: https://github.com/apache/kafka/actions/runs/10850348751/artifacts/1930070177
Error: Process completed with exit code 1.
```
The thread dump archive is also in the job summary https://github.com/apache/kafka/actions/runs/10850348751?pr=17181#summary-30111636589
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org