You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2011/08/23 09:56:29 UTC
[jira] [Created] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
AsyncProducer shutdown logic causes data loss
---------------------------------------------
Key: KAFKA-116
URL: https://issues.apache.org/jira/browse/KAFKA-116
Project: Kafka
Issue Type: Bug
Affects Versions: 0.7
Reporter: Neha Narkhede
The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13090490#comment-13090490 ]
Neha Narkhede commented on KAFKA-116:
-------------------------------------
thanks for pointing that out. I checked it in now.
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-116:
--------------------------------
Attachment: KAFKA-116.patch
1. Fixed the shutdown bug by setting the closed queue flag before adding the shutdown command to the async producer queue
2. Added more logging at the trace level
3. Refactored the blocking async producer logic to use case match instead of nested if else
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-116:
--------------------------------
Status: Patch Available (was: Open)
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089888#comment-13089888 ]
Jun Rao commented on KAFKA-116:
-------------------------------
+1
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Chris Burroughs (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089965#comment-13089965 ]
Chris Burroughs commented on KAFKA-116:
---------------------------------------
IllegalQueueStateException was missed in the commit.
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-116) AsyncProducer shutdown logic causes
data loss
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-116?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-116:
--------------------------
Resolution: Fixed
Fix Version/s: 0.7
Status: Resolved (was: Patch Available)
> AsyncProducer shutdown logic causes data loss
> ---------------------------------------------
>
> Key: KAFKA-116
> URL: https://issues.apache.org/jira/browse/KAFKA-116
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Fix For: 0.7
>
> Attachments: KAFKA-116.patch
>
>
> The current shutdown logic of the AsyncProducer allows adding events to the queue, after adding the shutdown command to it. The ProducerSendThread drains all the data in the queue until it hits the shutdown command. Hence, all data added after the shutdown command is lost.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira