You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "svetakvsundhar (via GitHub)" <gi...@apache.org> on 2025/03/25 19:59:16 UTC
[PR] Passing in key metadata when Reading From Kafka [beam]
svetakvsundhar opened a new pull request, #34426:
URL: https://github.com/apache/beam/pull/34426
Fixes https://github.com/apache/beam/issues/34391
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #34426: Passing in key metadata when Reading From Kafka
URL: https://github.com/apache/beam/pull/34426
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "fozzie15 (via GitHub)" <gi...@apache.org>.
fozzie15 commented on code in PR #34426:
URL: https://github.com/apache/beam/pull/34426#discussion_r2017773287
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java:
##########
@@ -340,4 +346,61 @@ public void testManagedMappings() {
assertTrue(configSchemaFieldNames.contains(paramName));
}
}
+
+ @Test
+ public void testGetRawBytesKvToRowFunction() {
+ Schema testSchema =
+ Schema.builder()
+ .addField("key", Schema.FieldType.BYTES)
+ .addField("value", Schema.FieldType.BYTES)
+ .build();
+
+ SerializableFunction<KV<byte[], byte[]>, Row> kvToRow =
+ KafkaReadSchemaTransformProvider.getRawBytesKvToRowFunction(testSchema);
+
+ KV<byte[], byte[]> inputKv =
+ KV.of(
+ "testKey".getBytes(StandardCharsets.UTF_8),
+ "testValue".getBytes(StandardCharsets.UTF_8));
+ Row outputRow = kvToRow.apply(inputKv);
+
+ assertEquals("testKey", new String(outputRow.getBytes("key"), StandardCharsets.UTF_8));
+ assertEquals("testValue", new String(outputRow.getBytes("value"), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ public void testGetRawBytesKvToRowFunctionEmptyKey() {
Review Comment:
Can we also add an integration test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34426:
URL: https://github.com/apache/beam/pull/34426#issuecomment-2767651179
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "svetakvsundhar (via GitHub)" <gi...@apache.org>.
svetakvsundhar commented on PR #34426:
URL: https://github.com/apache/beam/pull/34426#issuecomment-2752383523
Hi @ahmedabu98 ,
I think this PR is in a good place for some initial feedback. There are a couple of things/open notes I had:
(1) Need to still implement this with the `PROTO` and `JSON` format (right now it's only supported with `RAW`.
(2) It seems like the `ReadSchemaTransformProvider` doesn't have it's own file for integration tests. What are your thoughts on me creating a new file for integration tests, to test this change?
Any other comments on the general design/flow would be good as well!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34426:
URL: https://github.com/apache/beam/pull/34426#issuecomment-2930529150
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "fozzie15 (via GitHub)" <gi...@apache.org>.
fozzie15 commented on code in PR #34426:
URL: https://github.com/apache/beam/pull/34426#discussion_r2018693841
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java:
##########
@@ -340,4 +346,61 @@ public void testManagedMappings() {
assertTrue(configSchemaFieldNames.contains(paramName));
}
}
+
+ @Test
+ public void testGetRawBytesKvToRowFunction() {
+ Schema testSchema =
+ Schema.builder()
+ .addField("key", Schema.FieldType.BYTES)
+ .addField("value", Schema.FieldType.BYTES)
+ .build();
+
+ SerializableFunction<KV<byte[], byte[]>, Row> kvToRow =
+ KafkaReadSchemaTransformProvider.getRawBytesKvToRowFunction(testSchema);
+
+ KV<byte[], byte[]> inputKv =
+ KV.of(
+ "testKey".getBytes(StandardCharsets.UTF_8),
+ "testValue".getBytes(StandardCharsets.UTF_8));
+ Row outputRow = kvToRow.apply(inputKv);
+
+ assertEquals("testKey", new String(outputRow.getBytes("key"), StandardCharsets.UTF_8));
+ assertEquals("testValue", new String(outputRow.getBytes("value"), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ public void testGetRawBytesKvToRowFunctionEmptyKey() {
Review Comment:
I just saw above comment about ITs. You can add them to KafkaIOIT, where you can write a pipeline which triggers your code path as if it were a user's pipeline.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34426:
URL: https://github.com/apache/beam/pull/34426#issuecomment-2959069239
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "fozzie15 (via GitHub)" <gi...@apache.org>.
fozzie15 commented on code in PR #34426:
URL: https://github.com/apache/beam/pull/34426#discussion_r2017754519
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -191,8 +203,64 @@ public PCollectionRowTuple expand(PCollectionRowTuple input) {
}
if ("RAW".equals(format)) {
- beamSchema = Schema.builder().addField("payload", Schema.FieldType.BYTES).build();
- valueMapper = getRawBytesToRowFunction(beamSchema);
+ boolean withKeyMetadata = Boolean.TRUE.equals(configuration.getWithKeyMetadata());
+ if (withKeyMetadata) {
Review Comment:
This if block is really long. Can we put the logic here into its own function? Something like
```
if (withKeyMetadata) {
return someFnHere(beamSchema);
} else {
beamSchema = Schema.builder().addField("payload", Schema.FieldType.BYTES).build();
valueMapper = getRawBytesToRowFunction(beamSchema);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #34426:
URL: https://github.com/apache/beam/pull/34426#discussion_r2025229196
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProviderTest.java:
##########
@@ -340,4 +346,61 @@ public void testManagedMappings() {
assertTrue(configSchemaFieldNames.contains(paramName));
}
}
+
+ @Test
+ public void testGetRawBytesKvToRowFunction() {
+ Schema testSchema =
+ Schema.builder()
+ .addField("key", Schema.FieldType.BYTES)
+ .addField("value", Schema.FieldType.BYTES)
+ .build();
+
+ SerializableFunction<KV<byte[], byte[]>, Row> kvToRow =
+ KafkaReadSchemaTransformProvider.getRawBytesKvToRowFunction(testSchema);
+
+ KV<byte[], byte[]> inputKv =
+ KV.of(
+ "testKey".getBytes(StandardCharsets.UTF_8),
+ "testValue".getBytes(StandardCharsets.UTF_8));
+ Row outputRow = kvToRow.apply(inputKv);
+
+ assertEquals("testKey", new String(outputRow.getBytes("key"), StandardCharsets.UTF_8));
+ assertEquals("testValue", new String(outputRow.getBytes("value"), StandardCharsets.UTF_8));
+ }
+
+ @Test
+ public void testGetRawBytesKvToRowFunctionEmptyKey() {
Review Comment:
+1 to keep using KafkaIOIT. It should already have some integration tests for KafkaIO as a ManagedIO (look for `Managed.write/read`), which uses the SchemaTransform under the hood
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Passing in key metadata when Reading From Kafka [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #34426:
URL: https://github.com/apache/beam/pull/34426#discussion_r2025281767
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -103,6 +105,16 @@ public Row apply(byte[] input) {
};
}
+ public static SerializableFunction<KV<byte[], byte[]>, Row> getRawBytesKvToRowFunction(
+ Schema rawSchema) {
+ return new SimpleFunction<KV<byte[], byte[]>, Row>() {
+ @Override
+ public Row apply(KV<byte[], byte[]> input) {
+ return Row.withSchema(rawSchema).addValues(input.getKey(), input.getValue()).build();
+ }
+ };
+ }
+
Review Comment:
This should probably be a fixed function with a fixed output Schema (i.e. not a method)
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -139,6 +151,59 @@ Row getConfigurationRow() {
}
}
+ private PCollectionRowTuple expandRawWithKeyMetadata(
Review Comment:
There's a lot of duplicate code from `expand()` here :(
I'm a fan of making small code changes if possible. Does it make sense to consolidate this with the existing `expand()` code block?
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -155,6 +160,8 @@ public class KafkaIOIT {
private static final int BUFFER_MEMORY_CONFIG = 100554432;
private static final int RETRY_BACKOFF_MS_CONFIG = 5000;
+ private static String bootstrapServers;
Review Comment:
Does this ever get initialized?
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -274,6 +283,77 @@ public void testKafkaIOFailsFastWithInvalidTopics() throws IOException {
cancelIfTimeouted(readResult, readState);
}
+ @Test
+ public void testKafkaReadWithKeyMetadata() {
+ // 1. Produce test data to Kafka
+ Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, bootstrapServers);
+ producerProps.put(
+ ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
+ producerProps.put(
+ ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, ByteArraySerializer.class.getName());
+
+ try (KafkaProducer<byte[], byte[]> producer = new KafkaProducer<>(producerProps)) {
+ producer.send(
+ new ProducerRecord<>(
+ topicName,
+ "key1".getBytes(StandardCharsets.UTF_8),
+ "value1".getBytes(StandardCharsets.UTF_8)));
+ producer.send(
+ new ProducerRecord<>(
+ topicName,
+ "key2".getBytes(Charset.defaultCharset()),
+ "value2".getBytes(Charset.defaultCharset())));
+ }
+
+ // 2. Define the expected schema for the output Rows
+ Schema expectedSchema =
+ Schema.builder()
+ .addField("key", Schema.FieldType.BYTES)
+ .addField("payload", Schema.FieldType.BYTES)
+ .build();
+
+ // 3. Read from Kafka using KafkaIO
Review Comment:
we wanna test with the SchemaTransform right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org