You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Neha Narkhede (JIRA)" <ji...@apache.org> on 2011/08/17 05:02:27 UTC
[jira] [Created] (KAFKA-107) Bug in serialize and collate logic in
the DefaultEventHandler
Bug in serialize and collate logic in the DefaultEventHandler
-------------------------------------------------------------
Key: KAFKA-107
URL: https://issues.apache.org/jira/browse/KAFKA-107
Project: Kafka
Issue Type: Bug
Affects Versions: 0.7
Reporter: Neha Narkhede
There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-107) Bug in serialize and collate logic
in the DefaultEventHandler
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086522#comment-13086522 ]
Neha Narkhede commented on KAFKA-107:
-------------------------------------
Jun, the code is not zipping a map with a list. That won't even compile. The code is just zipping a sequence of tuples with a sequence of ByteBufferMessageSet. The problem is that one sequence is generated using the map API on the original HashMap and another sequence is generated using the map API on another HashMap. So ordering of data across those sequences is not guaranteed. Hence, a zip on those sequences will not associate the keys with the correct values.
I will add a unit test to cover this, and remove the unreferenced package imports.
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-107) Bug in serialize and collate logic in
the DefaultEventHandler
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-107:
--------------------------------
Attachment: KAFKA-107.patch
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-107) Bug in serialize and collate logic
in the DefaultEventHandler
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086512#comment-13086512 ]
Jun Rao commented on KAFKA-107:
-------------------------------
Another thing, please remove unreferenced package imports.
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-107) Bug in serialize and collate logic
in the DefaultEventHandler
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13086422#comment-13086422 ]
Jun Rao commented on KAFKA-107:
-------------------------------
Ok, so the problem was that the code was trying to zip a map (topicsAndPartitions) and a list (messages). Even though they are derived from the same map, there is no guarantee that the map is going to be iterated in the same order as the list. So, in the future, we need to be careful about using zip. We need to make sure that the two collections to be zipped have well defined ordering (e.g., list) for iteration and the ordering is what we want. Could you double check other usage of zip introduced in the compression patch?
The patch looks fine. However, could you also add a unit test (probably just at the serialize() level) that exposes this problem?
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (KAFKA-107) Bug in serialize and collate logic
in the DefaultEventHandler
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087317#comment-13087317 ]
Jun Rao commented on KAFKA-107:
-------------------------------
+1
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107-unit-test.patch, KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-107) Bug in serialize and collate logic in
the DefaultEventHandler
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-107:
--------------------------
Resolution: Fixed
Fix Version/s: 0.7
Status: Resolved (was: Patch Available)
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Fix For: 0.7
>
> Attachments: KAFKA-107-unit-test.patch, KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-107) Bug in serialize and collate logic in
the DefaultEventHandler
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-107:
--------------------------------
Status: Patch Available (was: Open)
This patch refactors the code in the DefaultEventHandler serialize API to fix the reordering bug
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-107) Bug in serialize and collate logic in
the DefaultEventHandler
Posted by "Neha Narkhede (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Neha Narkhede updated KAFKA-107:
--------------------------------
Attachment: KAFKA-107-unit-test.patch
Adding a unit test to the test suite for AsyncProducer that catches the reordering bug in serialize() API of DefaultEventHandler
> Bug in serialize and collate logic in the DefaultEventHandler
> -------------------------------------------------------------
>
> Key: KAFKA-107
> URL: https://issues.apache.org/jira/browse/KAFKA-107
> Project: Kafka
> Issue Type: Bug
> Affects Versions: 0.7
> Reporter: Neha Narkhede
> Attachments: KAFKA-107-unit-test.patch, KAFKA-107.patch
>
>
> There is a bug in the serialize and collate in the DefaultEventHandler, that uses the map() API on a hashmap to convert a sequence of messages to a ByteBufferMessageSet, based on the compression configs. The usage of the zip() API after the map() API on a hashmap, has the side effect of reordering the mapping between the keys and the values.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira