[jira] [Commented] (KAFKA-5364) Producer attempts to send transactional messages before adding partitions to transaction

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5364?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16033956#comment-16033956 ] 

ASF GitHub Bot commented on KAFKA-5364:
---------------------------------------

GitHub user apurvam opened a pull request:

    https://github.com/apache/kafka/pull/3202

    KAFKA-5364: Don't fail producer if drained partition is not yet in transaction

    Due to the async nature of the producer, it is possible to attempt to drain a messages whose partition hasn't been added to the transaction yet. Before this patch, we considered this a fatal error. However, it is only in error if the partition isn't in the queue to be sent to the coordinator. 
    
    This patch updates the logic so that we only fail the producer if the partition would never be added to the transaction. If the partition of the batch is yet to be added, we will simply wait for the partition to be added to the transaction before sending the batch to the broker.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/apurvam/kafka KAFKA-5364-ensure-partitions-added-to-txn-before-send

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/kafka/pull/3202.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #3202
    
----
commit 7dd2f7f9c3c59c80ed0cac4965df819a8c7ddcbb
Author: Apurva Mehta <ap...@confluent.io>
Date:   2017-06-02T00:18:08Z

    Fix logic in ensurePartitionAddedToTransaction to account for partitions which would be added to the transaction.

----


> Producer attempts to send transactional messages before adding partitions to transaction
> ----------------------------------------------------------------------------------------
>
>                 Key: KAFKA-5364
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5364
>             Project: Kafka
>          Issue Type: Sub-task
>          Components: clients, core, producer 
>    Affects Versions: 0.11.0.0
>            Reporter: Apurva Mehta
>            Assignee: Apurva Mehta
>            Priority: Blocker
>              Labels: exactly-once
>             Fix For: 0.11.0.0
>
>
> Due to a race condition between the sender thread and the producer.send(), the following is possible: 
> # In KakfaProducer.doSend(), we add partitions to the transaction and then do accumulator.append. 
> # In Sender.run(), we check whether there are transactional request. If there are, we send them and wait for the response. 
> # If there aren't we drain the accumulator queue and send the produce requests.
> # The problem is that the sequence step 2, 1, 3 is entire possible. This means that we won't send the 'AddPartitions' request but yet try to send the produce data. Which results in a fatal error and requires the producer to close. 
> The solution is that in the accumulator.drain, we should check again if there are pending add partitions requests, and if so, don't drain anything.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)