[jira] [Commented] (BEAM-12008) KafkaIO does not handle null keys

["Beam JIRA Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-12008?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17424050#comment-17424050 ] 

Beam JIRA Bot commented on BEAM-12008:
--------------------------------------

This issue is assigned but has not received an update in 30 days so it has been labeled "stale-assigned". If you are still working on the issue, please give an update and remove the label. If you are no longer working on the issue, please unassign so someone else may work on it. In 7 days the issue will be automatically unassigned.

> KafkaIO does not handle null keys
> ---------------------------------
>
>                 Key: BEAM-12008
>                 URL: https://issues.apache.org/jira/browse/BEAM-12008
>             Project: Beam
>          Issue Type: Bug
>          Components: io-java-kafka
>            Reporter: Daniel Collins
>            Assignee: Weiwen Xu
>            Priority: P2
>              Labels: stale-P2, stale-assigned
>          Time Spent: 9h 20m
>  Remaining Estimate: 0h
>
> Kafka [ConsumerRecord|https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRecord.html#key--] and [ProducerRecord|https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#key--] 'key' fields are explicitly allowed to be null. In addition, on the producer side, setting a null key is the way that the user indicates that they want a [random partition for their message|[https://github.com/apache/kafka/blob/9adfac280392da0837cfd8d582bc540951e94087/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L67].]
>  
> Beam KafkaIO does not support null keys in byte[] mode (read side: [https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L727|https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L727)]
> write side: [https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java#L58])
>  
> since it would defer to ByteArrayCoder which does not support null arrays.
>  
> BeamKafkaTable suffers the same issue https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java#L144



--
This message was sent by Atlassian Jira
(v8.3.4#803005)