[jira] [Updated] (KAFKA-739) Handle null values in Message payload

["Jay Kreps (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Jay Kreps updated KAFKA-739:
----------------------------

    Attachment: KAFKA-739-v1.patch

This patch is more extensive than I expected because I found a hole in the logic when handling deletes in the log compactor. The changes are as follows:

1. Handle null properly in Message.scala and miscellaneous other places.
2. Fix the logic for handling deletes. Previously we guaranteed that we would retain delete records only in the dirty section of the log. This is not sufficient, because a bootstrapping consumer might see a message, but the subsequent delete message might be gc'd before the consumer sees it.
3. OffsetMap.scala: make the map exact using a probing scheme. This means that the tail of the log is actually now fully deduplicated. The motivation for this is making delete-handling easier since to remove a delete tombstone you need to ensure that there are no prior occurrences of that message. Also added a counter on the number of collisions, just to help with any debugging.
4. Added a new configuration log.cleaner.delete.retention.ms that controls the length of time for which delete records are retained. This is implicitly a limit on the amount of time the consumer can spend bootstrapping and still get a consistent bootstrap. Once the topic-level config patch goes in, this will be made available at the topic level and can be set with the create topic tool
5. Added a peek() method to iterator template. Didn't end up using it, but it is a useful feature
6. Changed the integration test tool to issue deletes and changed the verification to handle delete records properly. Redid testing now with deletes included.
7. Added a variety of unit tests for null messages

                
> Handle null values in Message payload
> -------------------------------------
>
>                 Key: KAFKA-739
>                 URL: https://issues.apache.org/jira/browse/KAFKA-739
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Jay Kreps
>            Assignee: Jay Kreps
>             Fix For: 0.8.1
>
>         Attachments: KAFKA-739-v1.patch
>
>
> Add tests for null message payloads in producer, server, and consumer.
> Ensure log cleaner treats these as deletes.
> Test that null keys are rejected on dedupe logs.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira