You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/04 20:43:02 UTC
[GitHub] [beam] damccorm opened a new issue, #20941: KafkaIO does not handle null keys
damccorm opened a new issue, #20941:
URL: https://github.com/apache/beam/issues/20941
Kafka [ConsumerRecord](https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/consumer/ConsumerRecord.html#key--) and [ProducerRecord](https://kafka.apache.org/27/javadoc/org/apache/kafka/clients/producer/ProducerRecord.html#key--) 'key' fields are explicitly allowed to be null. In addition, on the producer side, setting a null key is the way that the user indicates that they want a [random partition for their message]([https://github.com/apache/kafka/blob/9adfac280392da0837cfd8d582bc540951e94087/clients/src/main/java/org/apache/kafka/clients/producer/internals/DefaultPartitioner.java#L67).]
Beam KafkaIO does not support null keys in byte[] mode (read side: [https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L727](https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaIO.java#L727))
write side: [https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java#L58](https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaRecordCoder.java#L58))
since it would defer to ByteArrayCoder which does not support null arrays.
BeamKafkaTable suffers the same issue https://github.com/apache/beam/blob/9e0997760cf3320f1a1d0c4342d3dff559a25775/sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaTable.java#L144
Imported from Jira [BEAM-12008](https://issues.apache.org/jira/browse/BEAM-12008). Original Jira may contain additional context.
Reported by: dpcollins-google.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] gabrywu commented on issue #20941: KafkaIO does not handle null keys
Posted by "gabrywu (via GitHub)" <gi...@apache.org>.
gabrywu commented on issue #20941:
URL: https://github.com/apache/beam/issues/20941#issuecomment-1625060660
anyone here? anyone updtes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org