You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/12/02 18:39:51 UTC
[GitHub] [beam] pabloem opened a new pull request, #24495: Implement Kafka Write Schema Transform
pabloem opened a new pull request, #24495:
URL: https://github.com/apache/beam/pull/24495
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
codecov[bot] commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1341480081
# [Codecov](https://codecov.io/gh/apache/beam/pull/24495?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#24495](https://codecov.io/gh/apache/beam/pull/24495?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6c24637) into [master](https://codecov.io/gh/apache/beam/commit/3e4051cd005fac32e9f12871461cf7566092615a?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3e4051c) will **decrease** coverage by `0.20%`.
> The diff coverage is `47.82%`.
> :exclamation: Current head 6c24637 differs from pull request most recent head 115fdbe. Consider uploading reports for the commit 115fdbe to get more accurate results
```diff
@@ Coverage Diff @@
## master #24495 +/- ##
==========================================
- Coverage 73.57% 73.36% -0.21%
==========================================
Files 723 719 -4
Lines 96304 97125 +821
==========================================
+ Hits 70853 71257 +404
- Misses 24104 24521 +417
Partials 1347 1347
```
| Flag | Coverage Δ | |
|---|---|---|
| go | `51.64% <39.90%> (ø)` | |
| python | `82.98% <97.14%> (-0.42%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/24495?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/funcx/fn.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L2ZuLmdv) | `57.30% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/funcx/output.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L291dHB1dC5nbw==) | `85.71% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/funcx/sideinput.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2Z1bmN4L3NpZGVpbnB1dC5nbw==) | `96.34% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/graph/coder/iterable.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL2NvZGVyL2l0ZXJhYmxlLmdv) | `42.71% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/graph/xlang.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL2dyYXBoL3hsYW5nLmdv) | `100.00% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/graphx/xlang.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvZ3JhcGh4L3hsYW5nLmdv) | `30.72% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/harness/session.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9zZXNzaW9uLmdv) | `19.51% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/symbols.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvc3ltYm9scy5nbw==) | `10.00% <0.00%> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/xlangx/registry.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3JlZ2lzdHJ5Lmdv) | `42.46% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/xlangx/resolve.go](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L3Jlc29sdmUuZ28=) | `25.60% <ø> (ø)` | |
| ... and [220 more](https://codecov.io/gh/apache/beam/pull/24495/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1335721894
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a diff in pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #24495:
URL: https://github.com/apache/beam/pull/24495#discussion_r1041497842
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -68,15 +81,22 @@ public List<String> inputCollectionNames() {
@Override
public List<String> outputCollectionNames() {
- return Lists.newArrayList("OUTPUT");
+ return Lists.newArrayList("output");
Review Comment:
correct. Trying to standardize on lowercase
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -502,6 +511,103 @@ public void testKafkaWithDelayedStopReadingFunction() {
readElementMetric(readResult, NAMESPACE, READ_ELEMENT_METRIC_NAME));
}
+ public static final Schema KAFKA_TOPIC_SCHEMA =
+ Schema.builder()
+ .addStringField("name")
+ .addInt64Field("userId")
+ .addInt32Field("age")
+ .addBooleanField("ageIsEven")
+ .addDoubleField("temperature")
+ .addArrayField("childrenNames", Schema.FieldType.STRING)
+ .build();
+
+ private static final int FIVE_MINUTES_IN_MS = 5 * 60 * 1000;
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformJson() {}
Review Comment:
ahhh good catch. we don't have automatic conversion, but I've hardcoded the json schema
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kafka;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.schemas.utils.JsonUtils;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.checkerframework.checker.initialization.qual.Initialized;
+import org.checkerframework.checker.nullness.qual.NonNull;
+import org.checkerframework.checker.nullness.qual.UnknownKeyFor;
+
+public class KafkaWriteSchemaTransformProvider
+ extends TypedSchemaTransformProvider<
+ KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration> {
+
+ public static final Set<String> SUPPORTED_FORMATS = Sets.newHashSet("JSON", "AVRO");
+
+ @Override
+ protected @UnknownKeyFor @NonNull @Initialized Class<KafkaWriteSchemaTransformConfiguration>
+ configurationClass() {
+ return KafkaWriteSchemaTransformConfiguration.class;
+ }
+
+ @Override
+ protected @UnknownKeyFor @NonNull @Initialized SchemaTransform from(
+ KafkaWriteSchemaTransformConfiguration configuration) {
+ if (!SUPPORTED_FORMATS.contains(configuration.getFormat())) {
+ throw new IllegalArgumentException(
+ "Format "
+ + configuration.getFormat()
+ + " is not supported. "
+ + "Supported formats are: "
+ + String.join(", ", SUPPORTED_FORMATS));
+ }
+ return new KafkaWriteSchemaTransform(configuration);
+ }
+
+ static final class KafkaWriteSchemaTransform implements SchemaTransform, Serializable {
+ final KafkaWriteSchemaTransformConfiguration configuration;
+
+ KafkaWriteSchemaTransform(KafkaWriteSchemaTransformConfiguration configuration) {
+ this.configuration = configuration;
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized PTransform<
+ @UnknownKeyFor @NonNull @Initialized PCollectionRowTuple,
+ @UnknownKeyFor @NonNull @Initialized PCollectionRowTuple>
+ buildTransform() {
+ return new PTransform<PCollectionRowTuple, PCollectionRowTuple>() {
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+ Schema inputSchema = input.get("input").getSchema();
+ final SerializableFunction<Row, byte[]> toBytesFn =
+ configuration.getFormat().equals("JSON")
+ ? JsonUtils.getRowToJsonBytesFunction(inputSchema)
+ : AvroUtils.getRowToAvroBytesFunction(inputSchema);
+
+ final Map<String, String> configOverrides = configuration.getProducerConfigUpdates();
+ input
+ .get("input")
+ .apply(
+ "Map Rows to Kafka Messages",
+ MapElements.via(
+ new SimpleFunction<Row, KV<byte[], byte[]>>(
+ row -> KV.of(new byte[1], toBytesFn.apply(row))) {}))
+ .apply(
+ KafkaIO.<byte[], byte[]>write()
+ .withTopic(configuration.getTopic())
+ .withBootstrapServers(configuration.getBootstrapServers())
+ .withProducerConfigUpdates(
+ configOverrides == null
+ ? new HashMap<>()
+ : new HashMap<String, Object>(configOverrides))
+ .withKeySerializer(ByteArraySerializer.class)
+ .withValueSerializer(ByteArraySerializer.class));
+ return PCollectionRowTuple.empty(input.getPipeline());
+ }
+ };
+ }
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized String identifier() {
+ return "beam:schematransform:org.apache.beam:kafka_write:v1";
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized List<@UnknownKeyFor @NonNull @Initialized String>
+ inputCollectionNames() {
+ return Collections.singletonList("input");
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized List<@UnknownKeyFor @NonNull @Initialized String>
+ outputCollectionNames() {
+ return Collections.emptyList();
+ }
+
+ @AutoValue
+ @DefaultSchema(AutoValueSchema.class)
+ public abstract static class KafkaWriteSchemaTransformConfiguration implements Serializable {
Review Comment:
changed it in the design doc.
we've decided to do [datastore][operation]schematransform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1341434036
Looks like need to add the topic `kafka-schema-transform` here: https://github.com/apache/beam/blob/6c24637b451d762ba39f54a56a8a335283173307/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml#L35
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1337929984
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1341397896
Run Java_Kafka_IO_Direct PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on a diff in pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
Abacn commented on code in PR #24495:
URL: https://github.com/apache/beam/pull/24495#discussion_r1041291087
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -502,6 +511,103 @@ public void testKafkaWithDelayedStopReadingFunction() {
readElementMetric(readResult, NAMESPACE, READ_ELEMENT_METRIC_NAME));
}
+ public static final Schema KAFKA_TOPIC_SCHEMA =
+ Schema.builder()
+ .addStringField("name")
+ .addInt64Field("userId")
+ .addInt32Field("age")
+ .addBooleanField("ageIsEven")
+ .addDoubleField("temperature")
+ .addArrayField("childrenNames", Schema.FieldType.STRING)
+ .build();
+
+ private static final int FIVE_MINUTES_IN_MS = 5 * 60 * 1000;
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformJson() {}
Review Comment:
Is this test supposed to be implemented in this PR? Or a "TODO" mark for empty test
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,166 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.kafka;
+
+import com.google.auto.value.AutoValue;
+import java.io.Serializable;
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import javax.annotation.Nullable;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.schemas.utils.AvroUtils;
+import org.apache.beam.sdk.schemas.utils.JsonUtils;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.transforms.SerializableFunction;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Sets;
+import org.apache.kafka.common.serialization.ByteArraySerializer;
+import org.checkerframework.checker.initialization.qual.Initialized;
+import org.checkerframework.checker.nullness.qual.NonNull;
+import org.checkerframework.checker.nullness.qual.UnknownKeyFor;
+
+public class KafkaWriteSchemaTransformProvider
+ extends TypedSchemaTransformProvider<
+ KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration> {
+
+ public static final Set<String> SUPPORTED_FORMATS = Sets.newHashSet("JSON", "AVRO");
+
+ @Override
+ protected @UnknownKeyFor @NonNull @Initialized Class<KafkaWriteSchemaTransformConfiguration>
+ configurationClass() {
+ return KafkaWriteSchemaTransformConfiguration.class;
+ }
+
+ @Override
+ protected @UnknownKeyFor @NonNull @Initialized SchemaTransform from(
+ KafkaWriteSchemaTransformConfiguration configuration) {
+ if (!SUPPORTED_FORMATS.contains(configuration.getFormat())) {
+ throw new IllegalArgumentException(
+ "Format "
+ + configuration.getFormat()
+ + " is not supported. "
+ + "Supported formats are: "
+ + String.join(", ", SUPPORTED_FORMATS));
+ }
+ return new KafkaWriteSchemaTransform(configuration);
+ }
+
+ static final class KafkaWriteSchemaTransform implements SchemaTransform, Serializable {
+ final KafkaWriteSchemaTransformConfiguration configuration;
+
+ KafkaWriteSchemaTransform(KafkaWriteSchemaTransformConfiguration configuration) {
+ this.configuration = configuration;
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized PTransform<
+ @UnknownKeyFor @NonNull @Initialized PCollectionRowTuple,
+ @UnknownKeyFor @NonNull @Initialized PCollectionRowTuple>
+ buildTransform() {
+ return new PTransform<PCollectionRowTuple, PCollectionRowTuple>() {
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+ Schema inputSchema = input.get("input").getSchema();
+ final SerializableFunction<Row, byte[]> toBytesFn =
+ configuration.getFormat().equals("JSON")
+ ? JsonUtils.getRowToJsonBytesFunction(inputSchema)
+ : AvroUtils.getRowToAvroBytesFunction(inputSchema);
+
+ final Map<String, String> configOverrides = configuration.getProducerConfigUpdates();
+ input
+ .get("input")
+ .apply(
+ "Map Rows to Kafka Messages",
+ MapElements.via(
+ new SimpleFunction<Row, KV<byte[], byte[]>>(
+ row -> KV.of(new byte[1], toBytesFn.apply(row))) {}))
+ .apply(
+ KafkaIO.<byte[], byte[]>write()
+ .withTopic(configuration.getTopic())
+ .withBootstrapServers(configuration.getBootstrapServers())
+ .withProducerConfigUpdates(
+ configOverrides == null
+ ? new HashMap<>()
+ : new HashMap<String, Object>(configOverrides))
+ .withKeySerializer(ByteArraySerializer.class)
+ .withValueSerializer(ByteArraySerializer.class));
+ return PCollectionRowTuple.empty(input.getPipeline());
+ }
+ };
+ }
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized String identifier() {
+ return "beam:schematransform:org.apache.beam:kafka_write:v1";
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized List<@UnknownKeyFor @NonNull @Initialized String>
+ inputCollectionNames() {
+ return Collections.singletonList("input");
+ }
+
+ @Override
+ public @UnknownKeyFor @NonNull @Initialized List<@UnknownKeyFor @NonNull @Initialized String>
+ outputCollectionNames() {
+ return Collections.emptyList();
+ }
+
+ @AutoValue
+ @DefaultSchema(AutoValueSchema.class)
+ public abstract static class KafkaWriteSchemaTransformConfiguration implements Serializable {
Review Comment:
Have we decided the naming pattern? the design doc https://docs.google.com/document/d/1qW9O3VxdGxUM887TdwhD1iH9AdNbpu0_wXbCGvFP0OM/edit# suggested `KafkaSchemaTransformReadConfiguration` possibly need update on that side
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -41,19 +41,32 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.joda.time.Duration;
@AutoService(SchemaTransformProvider.class)
public class KafkaReadSchemaTransformProvider
extends TypedSchemaTransformProvider<KafkaReadSchemaTransformConfiguration> {
+ final Boolean isTest;
+ final Integer testTimeoutSecs;
+
+ public KafkaReadSchemaTransformProvider() {
+ this(false, 0);
+ }
+
+ KafkaReadSchemaTransformProvider(Boolean isTest, Integer testTimeoutSecs) {
Review Comment:
Consider add a `@VisibleForTesting` annotation as this constructor is supposed to be used in test only?
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -502,6 +511,103 @@ public void testKafkaWithDelayedStopReadingFunction() {
readElementMetric(readResult, NAMESPACE, READ_ELEMENT_METRIC_NAME));
}
+ public static final Schema KAFKA_TOPIC_SCHEMA =
+ Schema.builder()
+ .addStringField("name")
+ .addInt64Field("userId")
+ .addInt32Field("age")
+ .addBooleanField("ageIsEven")
+ .addDoubleField("temperature")
+ .addArrayField("childrenNames", Schema.FieldType.STRING)
+ .build();
+
+ private static final int FIVE_MINUTES_IN_MS = 5 * 60 * 1000;
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformJson() {}
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformAvro() {
+ runReadWriteKafkaViaSchemaTransforms(
+ "AVRO", AvroUtils.toAvroSchema(KAFKA_TOPIC_SCHEMA).toString());
+ }
+
+ public void runReadWriteKafkaViaSchemaTransforms(String format, String schemaDefinition) {
+ PCollectionRowTuple.of(
+ "input",
+ writePipeline
+ .apply("Generate records", GenerateSequence.from(0).to(1000))
+ .apply(
+ "Transform to Beam Rows",
+ MapElements.into(TypeDescriptors.rows())
+ .via(
+ numb ->
+ Row.withSchema(KAFKA_TOPIC_SCHEMA)
+ .addValue(numb.toString()) // Name
+ .addValue(Long.valueOf(numb.hashCode())) // User ID
+ .addValue(numb.intValue()) // Age
+ .addValue(numb % 2 == 0) // ageIsEven
+ .addValue(new Random(numb).nextDouble())
+ .addValue(
+ Lists.newArrayList(
+ Long.toString(numb + 1),
+ Long.toString(numb + 2))) // childrenNames
+ .build()))
+ .setRowSchema(KAFKA_TOPIC_SCHEMA))
+ .apply(
+ "Write to Kafka",
+ new KafkaWriteSchemaTransformProvider()
+ .from(
+ KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration
+ .builder()
+ .setTopic(options.getKafkaTopic() + "-schema-transform")
+ .setBootstrapServers(options.getKafkaBootstrapServerAddresses())
+ .setFormat(format)
+ .build())
+ .buildTransform());
+
+ PAssert.that(
+ PCollectionRowTuple.empty(readPipeline)
+ .apply(
+ "Read from unbounded Kafka",
+ // A timeout of 30s for local, container-based tests, and 2 minutes for
+ // real-kafka tests.
+ new KafkaReadSchemaTransformProvider(
+ true, options.isWithTestcontainers() ? 30 : 120)
+ .from(
+ KafkaReadSchemaTransformConfiguration.builder()
+ .setDataFormat(format)
+ .setAutoOffsetResetConfig("earliest")
+ .setSchema(schemaDefinition)
+ .setTopic(options.getKafkaTopic() + "-schema-transform")
+ .setBootstrapServers(options.getKafkaBootstrapServerAddresses())
+ .build())
+ .buildTransform())
+ .get("output"))
+ .containsInAnyOrder(
+ LongStream.range(0L, 1000L)
+ .<Row>mapToObj(
+ numb ->
+ Row.withSchema(KAFKA_TOPIC_SCHEMA)
+ .addValue(Long.toString(numb)) // Name
+ .addValue(Long.valueOf(Long.valueOf(numb).hashCode())) // User ID
+ .addValue(Long.valueOf(numb).intValue()) // Age
+ .addValue(numb % 2 == 0) // ageIsEven
+ .addValue(new Random(numb).nextDouble())
+ .addValue(
+ Lists.newArrayList(
+ Long.toString(numb + 1),
+ Long.toString(numb + 2))) // childrenNames
+ .build())
+ .collect(Collectors.toList()));
+
+ PipelineResult writeResult = writePipeline.run();
+ writeResult.waitUntilFinish();
Review Comment:
Shall we assert pipeline status (not fail)? Same for the read pipeline below.
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -68,15 +81,22 @@ public List<String> inputCollectionNames() {
@Override
public List<String> outputCollectionNames() {
- return Lists.newArrayList("OUTPUT");
+ return Lists.newArrayList("output");
Review Comment:
Just wondering the purpose of UPPER to lower case change. Is it some cleanup?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem merged PR #24495:
URL: https://github.com/apache/beam/pull/24495
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1335979343
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @apilloud for label java.
R: @johnjcasey for label io.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1337926414
r: @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1340150842
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1343348245
the same kafka instance was being used for all tests - what happened is that the JSON tests were running against the same topic, and thus they were re-reading data that previous tests had written. For some reason the data was corrupted.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1339547191
Thanks! Looking into it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1341507148
tests passed : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1340076479
@Abacn ready for review : )
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1338428838
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
Abacn commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1341484943
> Looks like need to add the topic `kafka-schema-transform` here:
>
> https://github.com/apache/beam/blob/6c24637b451d762ba39f54a56a8a335283173307/.test-infra/kubernetes/kafka-cluster/05-kafka/configmap-config.yaml#L35
never mind, that line is for performance test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a diff in pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on code in PR #24495:
URL: https://github.com/apache/beam/pull/24495#discussion_r1041497667
##########
sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java:
##########
@@ -502,6 +511,103 @@ public void testKafkaWithDelayedStopReadingFunction() {
readElementMetric(readResult, NAMESPACE, READ_ELEMENT_METRIC_NAME));
}
+ public static final Schema KAFKA_TOPIC_SCHEMA =
+ Schema.builder()
+ .addStringField("name")
+ .addInt64Field("userId")
+ .addInt32Field("age")
+ .addBooleanField("ageIsEven")
+ .addDoubleField("temperature")
+ .addArrayField("childrenNames", Schema.FieldType.STRING)
+ .build();
+
+ private static final int FIVE_MINUTES_IN_MS = 5 * 60 * 1000;
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformJson() {}
+
+ @Test(timeout = FIVE_MINUTES_IN_MS)
+ public void testKafkaViaSchemaTransformAvro() {
+ runReadWriteKafkaViaSchemaTransforms(
+ "AVRO", AvroUtils.toAvroSchema(KAFKA_TOPIC_SCHEMA).toString());
+ }
+
+ public void runReadWriteKafkaViaSchemaTransforms(String format, String schemaDefinition) {
+ PCollectionRowTuple.of(
+ "input",
+ writePipeline
+ .apply("Generate records", GenerateSequence.from(0).to(1000))
+ .apply(
+ "Transform to Beam Rows",
+ MapElements.into(TypeDescriptors.rows())
+ .via(
+ numb ->
+ Row.withSchema(KAFKA_TOPIC_SCHEMA)
+ .addValue(numb.toString()) // Name
+ .addValue(Long.valueOf(numb.hashCode())) // User ID
+ .addValue(numb.intValue()) // Age
+ .addValue(numb % 2 == 0) // ageIsEven
+ .addValue(new Random(numb).nextDouble())
+ .addValue(
+ Lists.newArrayList(
+ Long.toString(numb + 1),
+ Long.toString(numb + 2))) // childrenNames
+ .build()))
+ .setRowSchema(KAFKA_TOPIC_SCHEMA))
+ .apply(
+ "Write to Kafka",
+ new KafkaWriteSchemaTransformProvider()
+ .from(
+ KafkaWriteSchemaTransformProvider.KafkaWriteSchemaTransformConfiguration
+ .builder()
+ .setTopic(options.getKafkaTopic() + "-schema-transform")
+ .setBootstrapServers(options.getKafkaBootstrapServerAddresses())
+ .setFormat(format)
+ .build())
+ .buildTransform());
+
+ PAssert.that(
+ PCollectionRowTuple.empty(readPipeline)
+ .apply(
+ "Read from unbounded Kafka",
+ // A timeout of 30s for local, container-based tests, and 2 minutes for
+ // real-kafka tests.
+ new KafkaReadSchemaTransformProvider(
+ true, options.isWithTestcontainers() ? 30 : 120)
+ .from(
+ KafkaReadSchemaTransformConfiguration.builder()
+ .setDataFormat(format)
+ .setAutoOffsetResetConfig("earliest")
+ .setSchema(schemaDefinition)
+ .setTopic(options.getKafkaTopic() + "-schema-transform")
+ .setBootstrapServers(options.getKafkaBootstrapServerAddresses())
+ .build())
+ .buildTransform())
+ .get("output"))
+ .containsInAnyOrder(
+ LongStream.range(0L, 1000L)
+ .<Row>mapToObj(
+ numb ->
+ Row.withSchema(KAFKA_TOPIC_SCHEMA)
+ .addValue(Long.toString(numb)) // Name
+ .addValue(Long.valueOf(Long.valueOf(numb).hashCode())) // User ID
+ .addValue(Long.valueOf(numb).intValue()) // Age
+ .addValue(numb % 2 == 0) // ageIsEven
+ .addValue(new Random(numb).nextDouble())
+ .addValue(
+ Lists.newArrayList(
+ Long.toString(numb + 1),
+ Long.toString(numb + 2))) // childrenNames
+ .build())
+ .collect(Collectors.toList()));
+
+ PipelineResult writeResult = writePipeline.run();
+ writeResult.waitUntilFinish();
Review Comment:
good idea. done!
##########
sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaReadSchemaTransformProvider.java:
##########
@@ -41,19 +41,32 @@
import org.apache.beam.vendor.guava.v26_0_jre.com.google.common.collect.Lists;
import org.apache.kafka.clients.consumer.ConsumerConfig;
import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.joda.time.Duration;
@AutoService(SchemaTransformProvider.class)
public class KafkaReadSchemaTransformProvider
extends TypedSchemaTransformProvider<KafkaReadSchemaTransformConfiguration> {
+ final Boolean isTest;
+ final Integer testTimeoutSecs;
+
+ public KafkaReadSchemaTransformProvider() {
+ this(false, 0);
+ }
+
+ KafkaReadSchemaTransformProvider(Boolean isTest, Integer testTimeoutSecs) {
Review Comment:
great point. done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #24495: Implement Kafka Write Schema Transform
Posted by GitBox <gi...@apache.org>.
pabloem commented on PR #24495:
URL: https://github.com/apache/beam/pull/24495#issuecomment-1340150924
oops I'll look at kafka issues
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org