[jira] [Commented] (KAFKA-5756) Synchronization issue on flush

["Chris Egerton (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-5756?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16999546#comment-16999546 ] 

Chris Egerton commented on KAFKA-5756:
--------------------------------------

I believe a very similar issue may still be possible and that the synchronization added in https://github.com/apache/kafka/pull/3702, while an improvement, still doesn't prevent all possible errors caused by concurrent calls to {{WorkerSourceTask::commitOffsets}} (which, as noted earlier in the ticket, can from from both the periodic offset commit from the {{SourceTaskOffsetCommitter}} class and the end-of-life offset commit from the {{WorkerSourceTask}} itself).

The {{WorkerSourceTask}} class takes care to ensure that {{OffsetStorageWriter::beginFlush}} isn't invoked concurrently, which was implemented as part of https://github.com/apache/kafka/pull/3702. However, there doesn't appear to be anything in place to prevent that method from being invoked before a flush has completed (either via a call to {{OffsetStorageWriter::cancelFlush}} or to {{OffsetStorageWriter::doFlush::get}}). If this occurs, [an exception is thrown|https://github.com/apache/kafka/blob/c6e25bb362899f4e6335ac5578b1cae31b7f2575/connect/runtime/src/main/java/org/apache/kafka/connect/storage/OffsetStorageWriter.java#L108-L112] stating that "the framework should not allow this".

Reopening this issue until the above scenario has been addressed.

> Synchronization issue on flush
> ------------------------------
>
>                 Key: KAFKA-5756
>                 URL: https://issues.apache.org/jira/browse/KAFKA-5756
>             Project: Kafka
>          Issue Type: Bug
>          Components: KafkaConnect
>    Affects Versions: 0.10.2.0
>            Reporter: Oleg Kuznetsov
>            Priority: Major
>             Fix For: 0.11.0.1, 1.0.0
>
>
> Access to *OffsetStorageWriter#toFlush* is not synchronized in *doFlush()* method, whereas this collection can be accessed from 2 different threads:
> - *WorkerSourceTask.execute()*, finally block
> - *SourceTaskOffsetCommitter*, from periodic flush task



--
This message was sent by Atlassian Jira
(v8.3.4#803005)