[jira] [Resolved] (KAFKA-5375) Transactions: Concurrent transactional consumer loses messages when there are broker bounces

["Apurva Mehta (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-5375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apurva Mehta resolved KAFKA-5375.
---------------------------------
    Resolution: Fixed
      Assignee: Apurva Mehta  (was: Jason Gustafson)

This turned out to be due to the test code. In particular, the consumer would timeout due to the lack of data because the transactions could not commit when there were broker bounces. 

I fixed the test so that the consumers wait until they read the number of expected messages instead. This is more robust. With those changes, this problem disappears.

> Transactions: Concurrent transactional consumer loses messages when there are broker bounces
> --------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5375
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5375
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Apurva Mehta
>            Assignee: Apurva Mehta
>            Priority: Blocker
>              Labels: exactly-once
>             Fix For: 0.11.0.0
>
>         Attachments: KAFKA-5375.tar.gz
>
>
> With the system test modifications in KAFKA-5366, the concurrent reader almost always consumes a fraction of the expected messages when there are broker bounces. A consumer running without concurrent writes consumes all the messages from the topic in question in the same test.
> {noformat}
> ================================================================================
> test_id:    kafkatest.tests.core.transactions_test.TransactionsTest.test_transactions.failure_mode=clean_bounce.bounce_target=brokers
> status:     FAIL
> run time:   1 minute 59.169 seconds
>     Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0
> Traceback (most recent call last):
>   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 123, in run
>     data = self.run_test()
>   File "/usr/local/lib/python2.7/dist-packages/ducktape/tests/runner_client.py", line 176, in run_test
>     return self.test_context.function(self.test)
>   File "/usr/local/lib/python2.7/dist-packages/ducktape/mark/_mark.py", line 321, in wrapper
>     return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File "/opt/kafka-dev/tests/kafkatest/tests/core/transactions_test.py", line 238, in test_transactions
>     (len(input_message_set), len(concurrently_consumed_message_set))
> AssertionError: Input and concurrently consumed output message sets are not equal. Num input messages: 20000. Num concurrently_consumed_messages: 0
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)