You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jay Kreps (JIRA)" <ji...@apache.org> on 2012/09/14 18:24:09 UTC
[jira] [Created] (KAFKA-512) Remove checksum from
ByteBufferMessageSet
Jay Kreps created KAFKA-512:
-------------------------------
Summary: Remove checksum from ByteBufferMessageSet
Key: KAFKA-512
URL: https://issues.apache.org/jira/browse/KAFKA-512
Project: Kafka
Issue Type: Bug
Reporter: Jay Kreps
Fix For: 0.8
Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.
The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().
If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-512) Remove checksum from
ByteBufferMessageSet.iterator
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao updated KAFKA-512:
--------------------------
Priority: Blocker (was: Major)
Labels: bugs (was: )
> Remove checksum from ByteBufferMessageSet.iterator
> --------------------------------------------------
>
> Key: KAFKA-512
> URL: https://issues.apache.org/jira/browse/KAFKA-512
> Project: Kafka
> Issue Type: Bug
> Reporter: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
>
> Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.
> The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().
> If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (KAFKA-512) Remove checksum from
ByteBufferMessageSet.iterator
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao resolved KAFKA-512.
---------------------------
Resolution: Fixed
Fixed in KAFKA-506.
> Remove checksum from ByteBufferMessageSet.iterator
> --------------------------------------------------
>
> Key: KAFKA-512
> URL: https://issues.apache.org/jira/browse/KAFKA-512
> Project: Kafka
> Issue Type: Bug
> Reporter: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
>
> Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.
> The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().
> If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (KAFKA-512) Remove checksum from
ByteBufferMessageSet.iterator
Posted by "Jay Kreps (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jay Kreps updated KAFKA-512:
----------------------------
Summary: Remove checksum from ByteBufferMessageSet.iterator (was: Remove checksum from ByteBufferMessageSet)
> Remove checksum from ByteBufferMessageSet.iterator
> --------------------------------------------------
>
> Key: KAFKA-512
> URL: https://issues.apache.org/jira/browse/KAFKA-512
> Project: Kafka
> Issue Type: Bug
> Reporter: Jay Kreps
> Fix For: 0.8
>
>
> Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.
> The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().
> If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (KAFKA-512) Remove checksum from
ByteBufferMessageSet.iterator
Posted by "Jun Rao (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/KAFKA-512?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jun Rao closed KAFKA-512.
-------------------------
Assignee: Jay Kreps
> Remove checksum from ByteBufferMessageSet.iterator
> --------------------------------------------------
>
> Key: KAFKA-512
> URL: https://issues.apache.org/jira/browse/KAFKA-512
> Project: Kafka
> Issue Type: Bug
> Reporter: Jay Kreps
> Assignee: Jay Kreps
> Priority: Blocker
> Labels: bugs
> Fix For: 0.8
>
>
> Messages are explicitly checksumed in Log.append. But there is also a checksum computed and checked automatically in ByteBufferMessageSet.iterator as we iterate. This iterator is used quite a lot and as a result we compute this checksum 39 times on a single message produce. It turns out the default crc32 implementation in java is quite expensive so this is not good.
> The proposed fix is to remove the automatic checksum from the iterator and add explicit isValid() checks in the consumer as well as retaining the existing check in Log.append().
> If folks are in agreement I will probably include this in the KAFKA-506 patch as that already contains a lot of ByteBufferMessageSet changes.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira