You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "mumrah (via GitHub)" <gi...@apache.org> on 2025/01/15 16:17:01 UTC

[PR] (Testing) Increase maxParallelForks to 4 [kafka]

mumrah opened a new pull request, #18560:
URL: https://github.com/apache/kafka/pull/18560

   (no comment)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698712779

   With one extra Gradle worker than there are cores, and 3 max forks
   
   <img width="1259" alt="image" src="https://github.com/user-attachments/assets/69da1112-c011-42d3-8dbd-59eee8fc8874" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2697456721

   maxParallelForks 2
   
   https://develocity.apache.org/s/x65mt52zf2o3o/timeline
   
   <img width="1265" alt="image" src="https://github.com/user-attachments/assets/c879cec6-8d90-494a-85d7-3e4caa26db0d" />
   
   Better than no forks, but still around 25m of only a single core doing work.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2697926936

   maxParallelForks 3
   
   https://develocity.apache.org/s/j5rg72fru7uky/timeline
   
   <img width="1259" alt="image" src="https://github.com/user-attachments/assets/024c4477-8803-4c9f-a897-1c9a4a5b3160" />
   
   Good utilization of all cores 👍 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR Increase JUnit max forks and memory for CI [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah merged PR #18560:
URL: https://github.com/apache/kafka/pull/18560


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "divijvaidya (via GitHub)" <gi...@apache.org>.
divijvaidya commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593613352

   Past (failed) attempt at this - https://github.com/apache/kafka/pull/15099
   
   At that time, quite a lot of our tests used to fail because they expected sequential execution. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593752028

   Looks like increasing to 4 parallel forks does speed things up a bit, so that's good. I'd also like to see how things behave when using JUnit 5's parallelism. For now I'm trying these settings which should allow classes to run in parallel, but run test methods on the same thread.
   
   ```
   junit.jupiter.execution.parallel.enabled = true
   junit.jupiter.execution.parallel.mode.default = same_thread
   junit.jupiter.execution.parallel.mode.classes.default = concurrent
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593479028

   I wish this can speedup our CI :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2596003931

   Many tests continue to fail after turning off the thread checker. Lots of Mockito errors. I guess a lot of our tests assume a single thread.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2593997346

   Seems our thread checker is causing issues
   ```
   org.opentest4j.AssertionFailedError: Found 16 unexpected threads during @BeforeAll: `kafka-1000-snapshot-generator-event-handler,quorum-controller-1000-event-handler,quorum-controller-3000-event-handler,kafka-1000-metadata-loader-event-handler,executor-raft-expiration-executor,kafka-producer-network-thread | producer-3,kafka-0-metadata-loader-event-handler,controller-1000-registration-manager-event-handler,broker-0-directory-assignments-manager-event-handler,kafka-admin-client-thread | adminclient-30,controller-3000-registration-manager-event-handler,client-metrics-reaper,kafka-3000-snapshot-generator-event-handler,kafka-3000-metadata-loader-event-handler,broker-0-lifecycle-manager-event-handler,kafka-0-snapshot-generator-event-handler` ==> expected: <true> but was: <false>
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698591377

   maxParallelForks 4
   
   <img width="1255" alt="image" src="https://github.com/user-attachments/assets/8cb255c6-7b1a-40c2-9d3b-c3bdbf54941c" />
   
   Interestingly, for this run, `:core:test` was a good bit faster, but other tasks (like `:clients:test`) ran slower. This suggests that the core tests are not as CPU bound as some other modules.  


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2696161481

   With no JUnit parallelism: 
   
   https://develocity.apache.org/s/gvzcoeky63jac/timeline
   
   <img width="1251" alt="image" src="https://github.com/user-attachments/assets/5e4746e8-6c5a-42b2-be5b-ba153e3f19fa" />
   
   With only Gradle task parallelism, we do not have enough task granularity to keep all 4 cores busy for the whole build.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2698997750

   With 6 workers and 4 max forks
   
   https://develocity.apache.org/s/zwzovvnv5zjaw/timeline?sort=longest
   
   <img width="1260" alt="image" src="https://github.com/user-attachments/assets/c57e7e75-0bb5-4359-9b64-9ce6e445fac8" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] (Testing) Increase maxParallelForks to 4 [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2699332120

   The aggressive over allocation of workers seems to be unstable. The Java 23 job has been killed twice by GHA. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] MINOR Tweak Gradle parallelism for CI [kafka]

Posted by "mumrah (via GitHub)" <gi...@apache.org>.
mumrah commented on PR #18560:
URL: https://github.com/apache/kafka/pull/18560#issuecomment-2699631239

   As a data point, just running `:core:test` by itself takes 37m. In most other cases it takes around an hour.
   
   https://develocity.apache.org/s/mxqpzhcnynana/timeline?sort=longest


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org