You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "Sebastian Klemke (JIRA)" <ji...@apache.org> on 2016/01/04 11:15:40 UTC
[jira] [Updated] (FLINK-3188) Deletes in Kafka source should be
passed on to KeyedDeserializationSchema
[ https://issues.apache.org/jira/browse/FLINK-3188?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sebastian Klemke updated FLINK-3188:
------------------------------------
Attachment: testcase.patch
Test case for the patch
> Deletes in Kafka source should be passed on to KeyedDeserializationSchema
> -------------------------------------------------------------------------
>
> Key: FLINK-3188
> URL: https://issues.apache.org/jira/browse/FLINK-3188
> Project: Flink
> Issue Type: Bug
> Components: Kafka Connector
> Affects Versions: 1.0.0
> Reporter: Sebastian Klemke
> Assignee: Robert Metzger
> Attachments: kafka-deletions.patch, testcase.patch
>
>
> When keys are deleted in the kafka queue, they show up as keys with null payload. Currently in Flink 1.0-SNAPSHOT, these deletions are silently skipped, without increasing current offset.
> This leads to two problems:
> 1. When a fetch window contains only deletions, LegacyFetcher gets stuck
> 2. For KeyedDeserializationSchemas, it would make sense to pass deletions to the deserializer, so that it can decide to wrap deleted keys as a deletion command. This is also more consistent with the semantics of keys in Kafka queues: When compaction is activated, only the latest message with the same key needs to be kept by Kafka.
> We propose the attached patch as a workaround for both issues.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)