You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Apurva Mehta (JIRA)" <ji...@apache.org> on 2017/06/06 04:54:18 UTC
[jira] [Commented] (KAFKA-5375) Transactions: Concurrent
transactional consumer loses messages when there are broker bounces
[ https://issues.apache.org/jira/browse/KAFKA-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16038146#comment-16038146 ]
Apurva Mehta commented on KAFKA-5375:
-------------------------------------
This is a red herring. The problem was with the consumer timeout setting in the python consumer wrapper. It was set to 5 seconds. When there are broker bounces, the transactions can't commit for a while. This often crosses the 5 second threshold, at which point the consumer closes, causing this error.
However, there are still failures of this test due to other reasons. I will open a separate ticket for that.
> Transactions: Concurrent transactional consumer loses messages when there are broker bounces
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-5375
> URL: https://issues.apache.org/jira/browse/KAFKA-5375
> Project: Kafka
> Issue Type: Bug
> Reporter: Apurva Mehta
> Assignee: Apurva Mehta
> Priority: Blocker
> Labels: exactly-once
> Fix For: 0.11.0.0
>
>
> With the system test modifications in KAFKA-5366, the concurrent reader almost always consumes a fraction of the expected messages when there are broker bounces. A consumer running without concurrent writes consumes all the messages from the topic in question in the same test.
> {noformat}
> ================================================================================
> test_id: kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=brokers
> status: FAIL
> run time: 1 minute 59.169 seconds
> Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0
> Traceback (most recent call last):
> File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 123, in run
> data = self.run_test()
> File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 176, in run_test
> return self.test_context.function(self.test)
> File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line 321, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
> File "/opt/kafka-dev/tests/kafkatest/tests/core/transactions_test.py", line 238, in test_transactions
> (len(input_message_set), len(concurrently_consumed_message_set))
> AssertionError: Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)