[jira] [Updated] (KAFKA-21262) Share consumer benchmarks are flaky: the run is cut off at --timeout

["Eric Chang (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-21262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Chang updated KAFKA-21262:
-------------------------------
    Summary: Share consumer benchmarks are flaky: the run is cut off at --timeout  (was: Share consumer benchmark system tests are flaky)

> Share consumer benchmarks are flaky: the run is cut off at --timeout
> --------------------------------------------------------------------
>
>                 Key: KAFKA-21262
>                 URL: https://issues.apache.org/jira/browse/KAFKA-21262
>             Project: Kafka
>          Issue Type: Bug
>          Components: system tests, tools
>            Reporter: Eric Chang
>            Assignee: Eric Chang
>            Priority: Major
>         Attachments: build-968-no-fix-ValueError.txt, build-969-tool-fix-only-zero-records.txt, build-970-all-fixes.txt
>
>
> Benchmark.test_share_consumer_throughput and Benchmark.test_producer_and_share_consumer sometimes fail with:
> {code}
> ValueError: could not convert string to float: ' data.consumed.in.MB'
> {code}
> There are three problems.
> h3. 1. ShareConsumerPerformance uses --timeout as a limit for the whole run
> The option is documented as "The maximum allowed time in milliseconds between returned records", and each consumer thread already follows that. But consume() waits for the threads with awaitTermination(timeout + 100ms) and then calls shutdownNow(). So the whole run can take at most timeout + 100ms, even when records are still arriving.
> When the run is longer, poll() is interrupted and throws InterruptException. Since KAFKA-19450 the exception is rethrown, so the tool exits with code 1 and prints only the header line. The Python service then tries to parse the header line as numbers and fails with the ValueError above.
> The tests use --timeout 20000. In a recent run, 9 of 10 measured runs took longer than 20 seconds.
> h3. 2. The first share group in a cluster has to wait for __share_group_state
> The __share_group_state topic (50 partitions) is created when the first share group is used. On a busy test machine this took more than 10 seconds. The consumer was already a stable member of the group, but it got no records until the topic was ready. With --timeout 20000 the consumer can give up before it reads anything.
> h3. 3. The tests do not check how many records were read
> The service only parses the MB and rate columns. A run that reads 0 records still passes.
> With only problem 1 fixed, 5 of 10 runs passed with 0 records read.
> h3. Proposed fix
> * Remove the total time limit in ShareConsumerPerformance.consume() and wait for each thread. Add a unit test.
> * In the two benchmark tests, read and release one record with kafka-console-share-consumer.sh before the measured run, so the share group is ready.
> * In the two benchmark tests, fail if the consumers read fewer records than expected.
> h3. Test results
> Both tests were repeated 5 times on the kafka-e2e Jenkins job (jenkins.opensource4you.tw).
> * Only the tool fix: 10 of 10 passed, but 5 runs read 0 records.
> * All three fixes: 10 of 10 passed, and every run read all records.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)