You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2025/01/15 16:17:01 UTC
[PR] (Testing) Increase maxParallelForks to 4 [kafka]
mumrah opened a new pull request, #18560:
URL: https://github.com/apache/kafka/pull/18560
(no comment)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698712779
With one extra Gradle worker than there are cores, and 3 max forks
<img width="1259" alt="image" src="https://github.com/user-attachments/assets/69da1112-c011-42d3-8dbd-59eee8fc8874" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2697456721
maxParallelForks 2
https://develocity.apache.org/s/x65mt52zf2o3o/timeline
<img width="1265" alt="image" src="https://github.com/user-attachments/assets/c879cec6-8d90-494a-85d7-3e4caa26db0d" />
Better than no forks, but still around 25m of only a single core doing work.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2697926936
maxParallelForks 3
https://develocity.apache.org/s/j5rg72fru7uky/timeline
<img width="1259" alt="image" src="https://github.com/user-attachments/assets/024c4477-8803-4c9f-a897-1c9a4a5b3160" />
Good utilization of all cores 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Increase JUnit max forks and memory for CI [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah merged PR #18560:
URL: https://github.com/apache/kafka/pull/18560
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593613352
Past (failed) attempt at this - https://github.com/apache/kafka/pull/15099
At that time, quite a lot of our tests used to fail because they expected sequential execution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593752028
Looks like increasing to 4 parallel forks does speed things up a bit, so that's good. I'd also like to see how things behave when using JUnit 5's parallelism. For now I'm trying these settings which should allow classes to run in parallel, but run test methods on the same thread.
```
junit.jupiter.execution.parallel.enabled = true
junit.jupiter.execution.parallel.mode.default = same_thread
junit.jupiter.execution.parallel.mode.classes.default = concurrent
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593479028
I wish this can speedup our CI :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2596003931
Many tests continue to fail after turning off the thread checker. Lots of Mockito errors. I guess a lot of our tests assume a single thread.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593997346
Seems our thread checker is causing issues
```
org.opentest4j.AssertionFailedError: Found 16 unexpected threads during @BeforeAll: `kafka-1000-snapshot-generator-event-handler,quorum-controller-1000-event-handler,quorum-controller-3000-event-handler,kafka-1000-metadata-loader-event-handler,executor-raft-expiration-executor,kafka-producer-network-thread | producer-3,kafka-0-metadata-loader-event-handler,controller-1000-registration-manager-event-handler,broker-0-directory-assignments-manager-event-handler,kafka-admin-client-thread | adminclient-30,controller-3000-registration-manager-event-handler,client-metrics-reaper,kafka-3000-snapshot-generator-event-handler,kafka-3000-metadata-loader-event-handler,broker-0-lifecycle-manager-event-handler,kafka-0-snapshot-generator-event-handler` ==> expected: <true> but was: <false>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698591377
maxParallelForks 4
<img width="1255" alt="image" src="https://github.com/user-attachments/assets/8cb255c6-7b1a-40c2-9d3b-c3bdbf54941c" />
Interestingly, for this run, `:core:test` was a good bit faster, but other tasks (like `:clients:test`) ran slower. This suggests that the core tests are not as CPU bound as some other modules.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2696161481
With no JUnit parallelism:
https://develocity.apache.org/s/gvzcoeky63jac/timeline
<img width="1251" alt="image" src="https://github.com/user-attachments/assets/5e4746e8-6c5a-42b2-be5b-ba153e3f19fa" />
With only Gradle task parallelism, we do not have enough task granularity to keep all 4 cores busy for the whole build.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698997750
With 6 workers and 4 max forks
https://develocity.apache.org/s/zwzovvnv5zjaw/timeline?sort=longest
<img width="1260" alt="image" src="https://github.com/user-attachments/assets/c57e7e75-0bb5-4359-9b64-9ce6e445fac8" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2699332120
The aggressive over allocation of workers seems to be unstable. The Java 23 job has been killed twice by GHA.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] MINOR Tweak Gradle parallelism for CI [kafka]
Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2699631239
As a data point, just running `:core:test` by itself takes 37m. In most other cases it takes around an hour.
https://develocity.apache.org/s/mxqpzhcnynana/timeline?sort=longest
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org