[jira] [Commented] (KAFKA-18921) Low CPU utilization during streams integration tests

["A. Sophie Blee-Goldman (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-18921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17932770#comment-17932770 ] 

A. Sophie Blee-Goldman commented on KAFKA-18921:
------------------------------------------------

[~davidarthur] is there a way to figure out how many tests are like this? Also, I assume this was a successful test? It wouldn't be too surprising for failed tests to have long pauses while they wait for output that never comes.

Anyways I ask because the example test you pulled was recently added so there might be something odd going on with it (cc [~mjsax] who added this). I'm also slightly suspicious of the reset tests in particular which might be waiting for things to....reset

> Low CPU utilization during streams integration tests
> ----------------------------------------------------
>
>                 Key: KAFKA-18921
>                 URL: https://issues.apache.org/jira/browse/KAFKA-18921
>             Project: Kafka
>          Issue Type: Improvement
>          Components: build, streams
>            Reporter: David Arthur
>            Priority: Major
>         Attachments: image-2025-03-04-17-05-48-768.png
>
>
> While analyzing some build performance, I noticed that our CPU is not well utilized during the streams integration tests. We should investigate to make sure we're not needlessly waiting during the tests.
>  
> Here is an example from this build scan https://develocity.apache.org/s/rqe4e2vh5jr72/timeline?page=1&sort=longest
> !image-2025-03-04-17-05-48-768.png!
> Notice that the ":streams:integration-tests:test" worker is running longer than the other works and we can see during that time that the CPU is not very utilized. This suggests that the tests are in some idle/waiting state.
>  
> I tried to reproduce this locally and found at 40 second pause in a random test.
>  
> [2025-03-04 16:56:11,606] INFO [Consumer clientId=consumer-f2eaa005-b1e1-406f-8c52-e50c71b4d807-58, groupId=f2eaa005-b1e1-406f-8c52-e50c71b4d807] Resetting offset for partition outputTopic_3-0 to position FetchPosition\{offset=0, offsetEpoch=Optional.empty, currentLeader=LeaderAndEpoch{leader=Optional[localhost:59515 (id: 0 rack: null isFenced: false)], epoch=0}}. (org.apache.kafka.clients.consumer.internals.SubscriptionState:447)
>     [2025-03-04 16:56:55,214] INFO [Consumer clientId=testAutoOffsetId-a426e1fb-61ad-4ce8-92ca-f0fe337c2565-StreamThread-1-consumer, groupId=testAutoOffsetId] Successfully joined group with generation Generation\{generationId=4, memberId='testAutoOffsetId-a426e1fb-61ad-4ce8-92ca-f0fe337c2565-StreamThread-1-consumer-2f798603-d538-4041-a90d-2e1b922dd5af', protocol='stream'} (org.apache.kafka.clients.consumer.internals.ConsumerCoordinator:666)
>  
> I grabbed a thread dump:
>  
> at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readRecords(IntegrationTestUtils.java:1177)
>         at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.readKeyValues(IntegrationTestUtils.java:1136)
>         at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.lambda$waitUntilMinKeyValueRecordsReceived$4(IntegrationTestUtils.java:710)
>         at org.apache.kafka.streams.integration.utils.IntegrationTestUtils$$Lambda$3338/0x00000070018f4fb0.call(Unknown Source)
>         at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:483)
>         at org.apache.kafka.test.TestUtils.retryOnExceptionWithTimeout(TestUtils.java:451)
>         at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:708)
>         at org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived(IntegrationTestUtils.java:681)
>         at org.apache.kafka.streams.integration.FineGrainedAutoResetIntegrationTest.shouldResetByDuration(FineGrainedAutoResetIntegrationTest.java:334)
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)