You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2024/04/09 20:57:55 UTC
[PR] Iceberg Write SchemaTransform [beam]
ahmedabu98 opened a new pull request, #30910:
URL: https://github.com/apache/beam/pull/30910
Creating a SchemaTransform for the recently added Iceberg sink (#30797)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1573060734
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedSchemaTransformPayload.newBuilder()
+ .setUnderlyingTransformIdentifier(managedConfig.getTransformIdentifier())
+ .setYamlConfig(managedConfig.resolveUnderlyingConfig())
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ManagedSchemaTransform transform) {
Review Comment:
Added `toSnakeCase` and `toCamelCase` utils to both Schema and Row. Using these now in Managed and Iceberg translation to keep consistent with our snake_case convention
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1575301305
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Reordered to prioritize registered transforms
Hopefully @kennknowles will have time afterwards to run end-to-end tests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574617763
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Should we add a second conditional here? If we see a `SCHEMA_TRANSFORM` URN, we can instead fetch the schema-aware identifier from the payload
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L166-L174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571190348
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -87,15 +88,14 @@ public class Managed {
.build();
/**
- * Instantiates a {@link Managed.Read} transform for the specified source. The supported managed
- * sources are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified source. The
Review Comment:
Alternatively, we could pull identifiers directly from the generated proto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571585343
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
Review Comment:
I think the idea is to store the config row and schema information in the PTransform spec, so that the runner can inspect and use it to build an upgraded transform
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571769272
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
The runner should be able to extract this information from schema + serialized row just fine. Best to not be redundant (no two sources of truth, as well as being more efficient, though the latter is a lesser concern). For things exclusively for display, there's display data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572420189
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
> Let's see if we can remove this
Do you mean we should remove the Snapshot part? e.g. only output the table string identifier instead of a KV<table identifier, Snapshot> ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571175210
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
I don't think Java only types are completely avoidable (at least without a significant refactoring). I think the main concern is failing to deserialize objects when upgrading the Java transform in the future. So we implemented `Externalizable` and added customer serializations that should be future proof for Java only types used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571435515
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Although adding dynamic destinations in a Beam Schema compatible way is a future goal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571765273
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedSchemaTransformPayload.newBuilder()
+ .setUnderlyingTransformIdentifier(managedConfig.getTransformIdentifier())
+ .setYamlConfig(managedConfig.resolveUnderlyingConfig())
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ManagedSchemaTransform transform) {
Review Comment:
If we want to settle on snake_case naming conventions, let's add explicit schema-mutating utilities to do so. (The serialized bytes should be compatible.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572550142
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
We should probably output a Row (or, rather schema'd POJO) with table identifier and any relevant information from snapshot (which can be safely expanded in the future if we use schema'd data).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569421542
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
e.g. use "beam:schematransform:org.apache.beam:managed:v1" for both proto payload URN and ManagedSchemaTransform identifier?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574759110
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
This sounds important to include in this release too, otherwise users would have to upgrade their SDK again to take advantage of what we have in this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574617763
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Should we add a second conditional here? If we see a `SCHEMA_TRANSFORM` URN, we can instead fetch the schema-aware identifier from the payload
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L166-L174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg Write SchemaTransform [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1560237022
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
Review Comment:
We have to figure out how to generate the spec for schema transforms. I think we can generate something generic.
URN: this can be the same as the identifier or a deravative of it
payload: We can introduce a generic SchemaTransformPayload that includes the Schema and serialized row bytes (using RowCoder).
cc: @kennknowles @robertwb in case they thought about this before.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
Review Comment:
Note that we cannot use this when constructive the transform from the service. So we will need to use the underlying `Row` based constructor. May be this will suffice but we should at least test it since this relies on `AutoValueSchema` etc.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class Config {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_Config.Builder();
+ }
+
+ public abstract String getTable();
+
+ public abstract CatalogConfig getCatalogConfig();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setTable(String tables);
+
+ public abstract Builder setCatalogConfig(CatalogConfig catalogConfig);
+
+ public abstract Config build();
+ }
+
+ public void validate() {
+ getCatalogConfig().validate();
+ }
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class CatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_CatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract CatalogConfig build();
+ }
+
+ Set<String> validTypes =
+ Sets.newHashSet(
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HADOOP,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HIVE,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_REST);
+
+ public void validate() {
+ if (Strings.isNullOrEmpty(getCatalogType())) {
+ checkArgument(
+ validTypes.contains(Preconditions.checkArgumentNotNull(getCatalogType())),
+ "Invalid catalog type. Please pick one of %s",
+ validTypes);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ static class IcebergWriteSchemaTransform extends SchemaTransform {
+ private final Config configuration;
+
+ IcebergWriteSchemaTransform(Config configuration) {
+ this.configuration = configuration;
+ }
+
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+
+ PCollection<Row> rows = input.get(INPUT_TAG);
+
+ CatalogConfig catalogConfig = configuration.getCatalogConfig();
+
+ IcebergCatalogConfig.Builder catalogBuilder =
+ IcebergCatalogConfig.builder()
+ .setName(catalogConfig.getCatalogName())
+ .setIcebergCatalogType(catalogConfig.getCatalogType())
+ .setWarehouseLocation(catalogConfig.getWarehouseLocation());
+
+ if (!Strings.isNullOrEmpty(catalogConfig.getCatalogType())) {
+ catalogBuilder = catalogBuilder.setIcebergCatalogType(catalogConfig.getCatalogType());
+ }
+ if (!Strings.isNullOrEmpty(catalogConfig.getWarehouseLocation())) {
+ catalogBuilder = catalogBuilder.setWarehouseLocation(catalogConfig.getWarehouseLocation());
+ }
+
+ // TODO: support dynamic destinations
+ DynamicDestinations dynamicDestinations =
+ DynamicDestinations.singleTable(TableIdentifier.parse(configuration.getTable()));
+
+ IcebergWriteResult result =
+ rows.apply(
+ IcebergIO.writeToDynamicDestinations(catalogBuilder.build(), dynamicDestinations));
+
+ PCollection<Row> snapshots =
+ result
+ .getSnapshots()
+ .apply(MapElements.via(new SnapshotToRow()))
+ .setRowSchema(SnapshotToRow.SNAPSHOT_SCHEMA);
+
+ return PCollectionRowTuple.of(OUTPUT_TAG, snapshots);
+ }
+
+ @VisibleForTesting
+ static class SnapshotToRow extends SimpleFunction<KV<String, Snapshot>, Row> {
+ static final Schema SNAPSHOT_SCHEMA =
Review Comment:
We should document this at the top level.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572459197
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574759110
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
This sounds important to include in this release too, otherwise users would have to upgrade their SDK again to take advantage of Managed transforms?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571575064
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Proto will make it easy to do upgrading since we can just pass that in. YAML will force us to build the Row when upgrading. I don't mind having both which will allow us to go either way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568193051
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
Review Comment:
That conversation was in regards to ManagedSchemaTransformTranslation, because over there we are pulling the schema from the provider.
But after working that one a little longer, I realized it's not that big of a deal and figured it's better to keep it consistent with the other Translation behavior (they all use snake_case names).
It may be unavoidable however if we start looking into a generic SchemaTransform translation (#30943)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572844067
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
Review Comment:
As discussed offline, we'll be removing the new protos and instead using the existing SchemaTransformPayload
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1575060121
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
For this to work, you also have to change the order below so that we look for providers before trying to process expansion requests using the SCHEMA_TRANSFORM urn.
https://github.com/apache/beam/blob/1a5dc1c7fab798a2ee8f526935432397648028f5/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L593
Please run an end-to-end test using Iceberg to confirm that this works. Also, we have to make sure that this doesn't break regular schema-transform expansion used by YAML etc (hopefully existing tests will be suffice for this).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572846871
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Will remove this payload, runner can deserialize the payload to find the config fields.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571531368
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
Review Comment:
This is almost entirely identical to https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto#L114
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
Review Comment:
This seems highly redundant with https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/external_transforms.proto#L114
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedSchemaTransformPayload.newBuilder()
+ .setUnderlyingTransformIdentifier(managedConfig.getTransformIdentifier())
+ .setYamlConfig(managedConfig.resolveUnderlyingConfig())
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ManagedSchemaTransform transform) {
Review Comment:
ManagedConfig is already @DefaultSchema; its toRow (and, below fromRow) methods should be used rather than needing to be manually implemented. (This is how it works as a SchemaTransform.)
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
What is the PCoder used for the `PCollection<KV<String, Snapshot>>` here? I'm concerned there may be update compatibility issues with it.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,183 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.Collections;
+import java.util.List;
+import org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.managed.ManagedTransformConstants;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaFieldDescription;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+
+/**
+ * SchemaTransform implementation for {@link IcebergIO#writeRows}. Writes Beam Rows to Iceberg and
+ * outputs a {@code PCollection<Row>} representing snapshots created in the process.
+ */
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ static final Schema OUTPUT_SCHEMA =
+ Schema.builder()
+ .addStringField("table")
+ .addStringField("operation")
+ .addMapField("summary", Schema.FieldType.STRING, Schema.FieldType.STRING)
+ .addStringField("manifestListLocation")
+ .build();
+
+ @Override
+ public String description() {
+ return "Writes Beam Rows to Iceberg.\n"
+ + "Returns a PCollection representing the snapshots produced in the process, with the following schema:\n"
+ + "{\"table\" (str), \"operation\" (str), \"summary\" (map[str, str]), \"manifestListLocation\" (str)}";
+ }
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration, configurationSchema());
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return ManagedTransformConstants.ICEBERG_WRITE;
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class Config {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_Config.Builder();
+ }
+
+ @SchemaFieldDescription("Identifier of the Iceberg table to write to.")
+ public abstract String getTable();
+
+ @SchemaFieldDescription("Configuration parameters used to set up the Iceberg catalog.")
+ public abstract IcebergSchemaTransformCatalogConfig getCatalogConfig();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setTable(String tables);
+
+ public abstract Builder setCatalogConfig(IcebergSchemaTransformCatalogConfig catalogConfig);
+
+ public abstract Config build();
+ }
+
+ public void validate() {
+ getCatalogConfig().validate();
+ }
+ }
+
+ static class IcebergWriteSchemaTransform extends SchemaTransform {
+ private final Config configuration;
+ private final Row configurationRow;
+
+ IcebergWriteSchemaTransform(Config configuration, Schema configSchema) {
+ this.configuration = configuration;
+
+ configurationRow =
+ Row.withSchema(configSchema)
+ .withFieldValue("table", configuration.getTable())
+ .withFieldValue("catalogConfig", configuration.getCatalogConfig().toRow())
+ .build();
+ }
+
+ Row getConfigurationRow() {
+ return configurationRow;
+ }
+
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+
+ PCollection<Row> rows = input.get(INPUT_TAG);
+
+ IcebergSchemaTransformCatalogConfig catalogConfig = configuration.getCatalogConfig();
+
+ IcebergCatalogConfig.Builder catalogBuilder =
+ IcebergCatalogConfig.builder().setName(catalogConfig.getCatalogName());
+
+ if (!Strings.isNullOrEmpty(catalogConfig.getCatalogType())) {
+ catalogBuilder = catalogBuilder.setIcebergCatalogType(catalogConfig.getCatalogType());
+ }
+ if (!Strings.isNullOrEmpty(catalogConfig.getWarehouseLocation())) {
+ catalogBuilder = catalogBuilder.setWarehouseLocation(catalogConfig.getWarehouseLocation());
+ }
+
+ // TODO: support dynamic destinations
+ DynamicDestinations dynamicDestinations =
+ DynamicDestinations.singleTable(TableIdentifier.parse(configuration.getTable()));
+
+ IcebergWriteResult result =
+ rows.apply(IcebergIO.writeRows(catalogBuilder.build()).to(dynamicDestinations));
+
+ PCollection<Row> snapshots =
+ result
+ .getSnapshots()
+ .apply(MapElements.via(new SnapshotToRow()))
Review Comment:
FYI, it may not be enough to just map to Rows, as all the intermediate data choices will be sticky for pipeline update. (We may be stuck with this for some existing sources, but we should think long and hard about what this means for anything new.)
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java:
##########
@@ -53,19 +53,19 @@ public class YamlUtils {
.put(Schema.TypeName.BYTES, str -> BaseEncoding.base64().decode(str))
.build();
- public static Row toBeamRow(@Nullable String yamlString, Schema schema) {
+ public static @Nullable Row toBeamRow(@Nullable String yamlString, Schema schema) {
return toBeamRow(yamlString, schema, false);
}
- public static Row toBeamRow(
+ public static @Nullable Row toBeamRow(
@Nullable String yamlString, Schema schema, boolean convertNamesToCamelCase) {
if (yamlString == null || yamlString.isEmpty()) {
List<Field> requiredFields =
schema.getFields().stream()
.filter(field -> !field.getType().getNullable())
.collect(Collectors.toList());
if (requiredFields.isEmpty()) {
- return Row.nullRow(schema);
+ return null;
Review Comment:
Why this change? It seems like non-nullable values are preferable. (Similarly below.)
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
This class looks like it's used for actual data which may cause backwards compatibility headaches in the future. Rather than a custom FileWriteResult.FileWriteResultCoder.class, could we declare it as a AutoSchema and use the SchemaCoder?
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Are we not supporting a config url as well as an inlined config?
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
It seems we should provide either the yaml or the schema + encoded row, not both. (They'll both have the same information, right?) I think we were leaning towards just the yaml.
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
Review Comment:
I don't think we need to set the payload at all, everything we need is already in the configRow, right?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,157 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+
+ @Override
+ public String getUrn() {
+ return READ_PROVIDER.identifier();
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema =
+ SchemaTranslation.schemaToProto(READ_PROVIDER.configurationSchema(), true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_PROVIDER.configurationSchema()).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
Review Comment:
As elsewhere, we don't need to store the config in both the payload and the config row annotations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571618296
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Both inline config (YAML string) and config URL are part of `expansion_payload`
I believe the rationale for having a separate `yaml_config` was to make it easier to extract and display config values from the runner side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572849067
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
We decided for now to encode a "version number" along with the data so that in the future we can fork if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1575064090
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformCatalogConfig.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.util.Set;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaFieldDescription;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class IcebergSchemaTransformCatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergSchemaTransformCatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ @SchemaFieldDescription("Valid types are: {hadoop, hive, rest}")
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract IcebergSchemaTransformCatalogConfig build();
+ }
+
+ public static final Schema SCHEMA;
+
+ static {
+ try {
+ SCHEMA =
+ SchemaRegistry.createDefault()
+ .getSchema(IcebergSchemaTransformCatalogConfig.class)
+ .sorted()
+ .toSnakeCase();
Review Comment:
Probably it will be sufficient to mention this in Java docs of `TypedSchemaTransformProvider`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571664164
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
Review Comment:
Ah, probably you meant that we can get the same information from transform annotations. I think it's cleaner to have this information in the payload than annotations (and we should get rid of annotations in the future).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571579105
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
Review Comment:
Not sure I understand. The idea here is to use the `Row` available in the transform payload from the job being submitted/updated to upgrade the transform via an expansion service.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on PR #30910:
URL: https://github.com/apache/beam/pull/30910#issuecomment-2071122460
Thank you all for the valuable feedback. Merging this now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568192121
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The transform's configuration schema
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
+// This can be used by runners that wish to override an underlying transform
+// with a different implementation.
+message ManagedTransformPayload {
+ // The underlying transform's URN.
+ string underlying_transform_urn = 1;
+ // The underlying transform's configuration Schema.
Review Comment:
Thanks, nice catch. will adjust the names and descriptions. Let me know if this works
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformTranslation.java:
##########
@@ -276,7 +279,7 @@ static RunnerApi.PTransform toProto(
*
* <p>Does not register the {@code appliedPTransform} within the provided {@link SdkComponents}.
*/
- static RunnerApi.PTransform toProto(
+ public static RunnerApi.PTransform toProto(
Review Comment:
Thanks, cleaning up
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571175210
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
I don't think serializing Java only types are completely avoidable (at least without a significant refactoring). I think the main concern is failing to deserialize objects when upgrading the Java transform in the future. So we implemented `Externalizable` and added customer serializations that should be future proof for Java only types used here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571623838
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Keep in mind the encoded row here is the top-level configuration for Managed transform (which includes the configuration for the underlying transform, in the form of a yaml string or URL).
Replied the same to the comment below, but I believe the `yaml_config` field here is not for functional use. Just to more easily extract config values from the runner side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571613769
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java:
##########
@@ -53,19 +53,19 @@ public class YamlUtils {
.put(Schema.TypeName.BYTES, str -> BaseEncoding.base64().decode(str))
.build();
- public static Row toBeamRow(@Nullable String yamlString, Schema schema) {
+ public static @Nullable Row toBeamRow(@Nullable String yamlString, Schema schema) {
return toBeamRow(yamlString, schema, false);
}
- public static Row toBeamRow(
+ public static @Nullable Row toBeamRow(
@Nullable String yamlString, Schema schema, boolean convertNamesToCamelCase) {
if (yamlString == null || yamlString.isEmpty()) {
List<Field> requiredFields =
schema.getFields().stream()
.filter(field -> !field.getType().getNullable())
.collect(Collectors.toList());
if (requiredFields.isEmpty()) {
- return Row.nullRow(schema);
+ return null;
Review Comment:
reverted back to null row
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569419609
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
We use "beam:transform:managed:v1" to identify a ManagedTransformPayload. Should these URNs be resolved to one thing?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571076923
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -87,15 +88,14 @@ public class Managed {
.build();
/**
- * Instantiates a {@link Managed.Read} transform for the specified source. The supported managed
- * sources are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified source. The
Review Comment:
Created a new ManagedTransformConstants class to use as a source for supported identifiers. Figured this information doesn't need to be exposed in Managed API.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1575062508
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class IcebergIO {
+
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
+ return new AutoValue_IcebergIO_WriteRows.Builder().setCatalogConfig(catalog).build();
Review Comment:
No I'm fine if you want to leave it but generally good to get code in if we don't need it :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574655711
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class IcebergIO {
+
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
+ return new AutoValue_IcebergIO_WriteRows.Builder().setCatalogConfig(catalog).build();
Review Comment:
Probably not, I can revert them if you like
Out of curiosity, is there an inclination against builders?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571753065
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
Review Comment:
Should this be Managed rather than Standard?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1573140471
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -51,38 +53,47 @@ static class IcebergReadSchemaTransformTranslator
@Override
public String getUrn() {
- return READ_PROVIDER.identifier();
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
}
@Override
public @Nullable FunctionSpec translate(
AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
throws IOException {
- SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Schema snakeCaseSchema = READ_SCHEMA.toSnakeCase();
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(snakeCaseSchema, true);
Row configRow = toConfigRow(application.getTransform());
ByteArrayOutputStream os = new ByteArrayOutputStream();
- RowCoder.of(READ_SCHEMA).encode(configRow, os);
+ RowCoder.of(snakeCaseSchema).encode(configRow, os);
return FunctionSpec.newBuilder()
.setUrn(getUrn())
.setPayload(
SchemaTransformPayload.newBuilder()
+ .setIdentifier(READ_PROVIDER.identifier())
.setConfigurationSchema(expansionSchema)
.setConfigurationRow(ByteString.copyFrom(os.toByteArray()))
- .setIdentifier(getUrn())
.build()
.toByteString())
.build();
}
@Override
public Row toConfigRow(IcebergReadSchemaTransform transform) {
- return transform.getConfigurationRow();
+ // Will retrieve a Row with snake_case naming convention.
+ // Transform expects camelCase convention, so convert back
+ // TODO(https://github.com/apache/beam/issues/31061): Remove conversion when
+ // TypedSchemaTransformProvider starts generating with snake_case convention
+ return transform.getConfigurationRow().toSnakeCase();
}
@Override
public IcebergReadSchemaTransform fromConfigRow(Row configRow, PipelineOptions options) {
Review Comment:
Added some minor changes to TypedSchemaTransformProvider.
- It now expects a snake_case Row, which it then converts to camelCase to generate a ConfigT object (AutoValueSchema expects camelCase field names).
- When getting configurationSchema(), it generates a camelCase schema from ConfigT, converts it to snake_case, then outputs.
Effectively, all TypedSchemaTransformProvider implementations will now have snake_case configuration schemas.
I updated the translation logic to reflect this, and also updated existing Python wrapper logic to work with snake_case parameters.
@robertwb I wonder if we need to update things in Beam YAML too. For example the renaming happening in this file: https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/python/apache_beam/yaml/standard_io.yaml#L25-L40
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572845746
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
We'll be using `SchemaTransformPayload`, which means the yaml config will be found inside the encoded `configuration_row`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574920631
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Took a stab at this in the latest commit ([7130e56](https://github.com/apache/beam/pull/30910/commits/7130e569d5962b9cc029aa4a5f2995e51c9c529d)). Let me know if this is sufficient
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574641508
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformCatalogConfig.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.util.Set;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaFieldDescription;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class IcebergSchemaTransformCatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergSchemaTransformCatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ @SchemaFieldDescription("Valid types are: {hadoop, hive, rest}")
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract IcebergSchemaTransformCatalogConfig build();
+ }
+
+ public static final Schema SCHEMA;
+
+ static {
+ try {
+ SCHEMA =
+ SchemaRegistry.createDefault()
+ .getSchema(IcebergSchemaTransformCatalogConfig.class)
+ .sorted()
+ .toSnakeCase();
Review Comment:
Done.
Wondering if there's a way to make it more known that we are setting this convention (ie. lexicographically sorted field names, snake_case)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571019914
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedTransformPayload.newBuilder()
+ .setUnderlyingTransformUrn(managedConfig.getTransformIdentifier())
Review Comment:
As discussed offline, the underlying transform is the SchemaTransform.
Making name changes to the proto to reflect this (underlying_transform_urn --> underlying_transform_identifier)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571630170
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Looks like it's SerializableCoder:
https://github.com/apache/beam/blob/d05196dea8a393ef2afec8e95c49a309375a42d9/sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/AppendFilesToTables.java#L63-L66
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571166638
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Can we simply omit the dynamic destinations parameter for now? I'd really like to avoid schemas with java-only types.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568192121
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The transform's configuration schema
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
+// This can be used by runners that wish to override an underlying transform
+// with a different implementation.
+message ManagedTransformPayload {
+ // The underlying transform's URN.
+ string underlying_transform_urn = 1;
+ // The underlying transform's configuration Schema.
Review Comment:
Thanks, nice catch. will adjust the names and descriptions
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The transform's configuration schema
Review Comment:
How does this sound:
`"The schema of the configuration row used to upgrade the transform"`.
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java:
##########
@@ -147,14 +147,27 @@ public static Row toBeamRow(
}
@SuppressWarnings("nullness")
- public static Row toBeamRow(Map<String, Object> yamlMap, Schema rowSchema, boolean toCamelCase) {
+ public static @Nullable Row toBeamRow(
+ @Nullable Map<String, Object> map, Schema rowSchema, boolean toCamelCase) {
+ if (map == null || map.isEmpty()) {
+ List<Field> requiredFields =
+ rowSchema.getFields().stream()
+ .filter(field -> !field.getType().getNullable())
+ .collect(Collectors.toList());
+ if (requiredFields.isEmpty()) {
+ return null;
+ } else {
Review Comment:
What count are you referring to?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568193214
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
Review Comment:
This translation is for the IO itself, not for its SchemaTransform implementation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1570923877
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571623838
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Keep in mind the encoded row here is the configuration for Managed transform (which includes the configuration for the underlying transform, in the form of a yaml string or URL).
Replied the same to the comment below, but I believe the `yaml_config` field here is not for functional use. Just to more easily extract config values from the runner side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1570932919
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
+
+ byte[] catalogBytes = configRow.getBytes("catalog_config");
+ if (catalogBytes != null) {
+ builder = builder.setCatalogConfig((IcebergCatalogConfig) fromByteArray(catalogBytes));
+ }
+ String tableIdentifier = configRow.getString("table_identifier");
+ if (tableIdentifier != null) {
+ builder = builder.setTableIdentifier(TableIdentifier.parse(tableIdentifier));
+ }
+ return builder.build();
+ } catch (InvalidClassException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class ReadRegistrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ @SuppressWarnings({
+ "rawtypes",
+ })
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(AutoValue_IcebergIO_ReadRows.class, new IcebergIOReadTranslator())
+ .build();
+ }
+ }
+
+ static class IcebergIOWriteTranslator implements TransformPayloadTranslator<WriteRows> {
+
+ static final Schema WRITE_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .addNullableByteArrayField("dynamic_destinations")
+ .build();
+
+ public static final String ICEBERG_WRITE_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_write:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_WRITE_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, WriteRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(WRITE_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(WRITE_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(WriteRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+ if (transform.getDynamicDestinations() != null) {
+ fieldValues.put("dynamic_destinations", toByteArray(transform.getDynamicDestinations()));
Review Comment:
It's not as clear to me how DynamicDestinations should get serialized
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571772809
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
Review Comment:
I think we should keep `ManagedSchemaTransformPayload` separate since this has additional fields (and `underlying_transform_identifier` and `identifier` are different semantically).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569449523
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
I'm fine with making both use the URN format (and documenting that).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571595848
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedSchemaTransformPayload.newBuilder()
+ .setUnderlyingTransformIdentifier(managedConfig.getTransformIdentifier())
+ .setYamlConfig(managedConfig.resolveUnderlyingConfig())
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ManagedSchemaTransform transform) {
Review Comment:
sounds good
fyi this was done initially to have a nice snake_case naming for the fields (toRow will generate camel case names from the getters)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572459530
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg Write SchemaTransform [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30910:
URL: https://github.com/apache/beam/pull/30910#issuecomment-2046087851
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574038797
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Actually, we use the urn when registering `TransformProvider`s at the `ExpansionService`.
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L188
Hence' this will require us to do future modifications where we keep track of `TransformProvider`s using the schema-transform identifier. I think this is OK but wanted to call this out.
The other option will be go to back to using different URNs for different schema-aware transforms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574617763
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Should we add a second conditional here? If we see a `SCHEMA_TRANSFORM` URN, we instead fetch the schema-aware identifier from the payload
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L166-L174
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 merged PR #30910:
URL: https://github.com/apache/beam/pull/30910
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568192720
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java:
##########
@@ -19,73 +19,116 @@
import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
import org.apache.beam.sdk.io.Read;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
import org.apache.iceberg.Table;
import org.apache.iceberg.catalog.TableIdentifier;
import org.checkerframework.checker.nullness.qual.Nullable;
public class IcebergIO {
- public static WriteRows writeToDynamicDestinations(
- IcebergCatalogConfig catalog, DynamicDestinations dynamicDestinations) {
- return new WriteRows(catalog, dynamicDestinations);
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
Review Comment:
I felt it was more clear, but it's not needed
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java:
##########
@@ -19,73 +19,116 @@
import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
import org.apache.beam.sdk.io.Read;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
import org.apache.iceberg.Table;
import org.apache.iceberg.catalog.TableIdentifier;
import org.checkerframework.checker.nullness.qual.Nullable;
public class IcebergIO {
- public static WriteRows writeToDynamicDestinations(
- IcebergCatalogConfig catalog, DynamicDestinations dynamicDestinations) {
- return new WriteRows(catalog, dynamicDestinations);
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
+ return new AutoValue_IcebergIO_WriteRows.Builder().setCatalogConfig(catalog).build();
Review Comment:
WriteRows is abstract so we can't instantiate it with `WriteRows.Builder()...`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569225112
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
+// This can be used by runners that wish to override an underlying transform
Review Comment:
// The information available in the payload can be used by runners to override the `ManagedSchemaTransform `.
##########
.github/workflows/beam_PreCommit_Xlang_Generated_Transforms.yml:
##########
@@ -111,4 +111,4 @@ jobs:
- name: run Cross-Language Wrapper Validation script
uses: ./.github/actions/gradle-command-self-hosted-action
with:
- gradle-command: :sdks:python:test-suites:direct:crossLanguageWrapperValidationPreCommit
\ No newline at end of file
+ gradle-command: :sdks:python:test-suites:direct:crossLanguageWrapperValidationPreCommit --info
Review Comment:
Nit: missing newline
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Update to use Externalizable ?
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedTransformPayload.newBuilder()
+ .setUnderlyingTransformUrn(managedConfig.getTransformIdentifier())
Review Comment:
We want this to be the URN of the underlying transform not the schema-transform identifier. Maintain a mapping somewhere and add unit test for consistency ?
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -108,8 +107,8 @@ public static ManagedTransform read(String source) {
}
/**
- * Instantiates a {@link Managed.Write} transform for the specified sink. The supported managed
- * sinks are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified sink. The supported
Review Comment:
We updated `ManagedTransform` to directly implement `PTransform` to avoid confusion.
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
Review Comment:
nit: s/build/upgrade
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
Review Comment:
for a `ManagedSchemaTransform`.
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
+// This can be used by runners that wish to override an underlying transform
+// with a different implementation.
+message ManagedTransformPayload {
+ // The underlying transform's URN.
+ string underlying_transform_urn = 1;
+ // The managed transform configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to build the managed transform.
Review Comment:
s/build/override
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
Change to "beam:schematransform:org.apache.beam:managed:v1" to conform to the convention ?
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
Review Comment:
Add:
// The information available in the payload can be used by runners to override the corresponding schema-aware transform .
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -67,19 +70,21 @@ public ManagedSchemaTransformProvider() {}
"Found multiple SchemaTransformProvider implementations with the same identifier "
+ schemaTransformProvider.identifier());
}
- schemaTransformProviders.put(schemaTransformProvider.identifier(), schemaTransformProvider);
+ if (supportedIdentifiers == null
+ || supportedIdentifiers.contains(schemaTransformProvider.identifier())) {
+ schemaTransformProviders.put(
+ schemaTransformProvider.identifier(), schemaTransformProvider);
+ }
}
} catch (Exception e) {
throw new RuntimeException(e.getMessage());
}
-
- schemaTransformProviders.entrySet().removeIf(e -> !supportedIdentifiers.contains(e.getKey()));
}
@DefaultSchema(AutoValueSchema.class)
@AutoValue
@VisibleForTesting
- abstract static class ManagedConfig {
+ public abstract static class ManagedConfig {
Review Comment:
Can this and other classes defined in this file be package private ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergCatalogConfig.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
Review Comment:
Update to use Externalizable ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.sdk.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
+
+ byte[] catalogBytes = configRow.getBytes("catalog_config");
+ if (catalogBytes != null) {
+ builder = builder.setCatalogConfig((IcebergCatalogConfig) fromByteArray(catalogBytes));
+ }
+ String tableIdentifier = configRow.getString("table_identifier");
+ if (tableIdentifier != null) {
+ builder = builder.setTableIdentifier(TableIdentifier.parse(tableIdentifier));
+ }
+ return builder.build();
+ } catch (InvalidClassException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class ReadRegistrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ @SuppressWarnings({
+ "rawtypes",
+ })
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(AutoValue_IcebergIO_ReadRows.class, new IcebergIOReadTranslator())
+ .build();
+ }
+ }
+
+ static class IcebergIOWriteTranslator implements TransformPayloadTranslator<WriteRows> {
+
+ static final Schema WRITE_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .addNullableByteArrayField("dynamic_destinations")
+ .build();
+
+ public static final String ICEBERG_WRITE_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_write:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_WRITE_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, WriteRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(WRITE_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(WRITE_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(WriteRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+ if (transform.getDynamicDestinations() != null) {
+ fieldValues.put("dynamic_destinations", toByteArray(transform.getDynamicDestinations()));
Review Comment:
As discussed offline, we should consider implementing `Externalizable` for the complex objects serialized here (`CatalogConfig` and `DynamicDestinations`).
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -87,15 +88,14 @@ public class Managed {
.build();
/**
- * Instantiates a {@link Managed.Read} transform for the specified source. The supported managed
- * sources are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified source. The
Review Comment:
Let's move schema-transforms identifiers for specific I/Os to constants and refer to them from both here and identifier() methods of corresponding `SchemaTransformProvider` implementations for consistency.
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -122,29 +145,28 @@ protected SchemaTransform from(ManagedConfig managedConfig) {
"Could not find transform with identifier %s, or it may not be supported",
managedConfig.getTransformIdentifier());
- // parse config before expansion to check if it matches underlying transform's config schema
- Schema transformConfigSchema = schemaTransformProvider.configurationSchema();
- Row transformConfig;
- try {
- transformConfig = getRowConfig(managedConfig, transformConfigSchema);
- } catch (Exception e) {
- throw new IllegalArgumentException(
- String.format(
- "Specified configuration does not align with the underlying transform's configuration schema [%s].",
- transformConfigSchema),
- e);
- }
-
- return new ManagedSchemaTransform(transformConfig, schemaTransformProvider);
+ return new ManagedSchemaTransform(managedConfig, schemaTransformProvider);
}
- private static class ManagedSchemaTransform extends SchemaTransform {
+ public static class ManagedSchemaTransform extends SchemaTransform {
Review Comment:
Ditto.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1570932919
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
+
+ byte[] catalogBytes = configRow.getBytes("catalog_config");
+ if (catalogBytes != null) {
+ builder = builder.setCatalogConfig((IcebergCatalogConfig) fromByteArray(catalogBytes));
+ }
+ String tableIdentifier = configRow.getString("table_identifier");
+ if (tableIdentifier != null) {
+ builder = builder.setTableIdentifier(TableIdentifier.parse(tableIdentifier));
+ }
+ return builder.build();
+ } catch (InvalidClassException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class ReadRegistrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ @SuppressWarnings({
+ "rawtypes",
+ })
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(AutoValue_IcebergIO_ReadRows.class, new IcebergIOReadTranslator())
+ .build();
+ }
+ }
+
+ static class IcebergIOWriteTranslator implements TransformPayloadTranslator<WriteRows> {
+
+ static final Schema WRITE_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .addNullableByteArrayField("dynamic_destinations")
+ .build();
+
+ public static final String ICEBERG_WRITE_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_write:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_WRITE_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, WriteRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(WRITE_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(WRITE_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(WriteRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+ if (transform.getDynamicDestinations() != null) {
+ fieldValues.put("dynamic_destinations", toByteArray(transform.getDynamicDestinations()));
Review Comment:
It's not as clear to me how DynamicDestinations should get serialized
update: Talked offline and for now will only implement custom serialization for OneTableDynamicDestination
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571190348
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -87,15 +88,14 @@ public class Managed {
.build();
/**
- * Instantiates a {@link Managed.Read} transform for the specified source. The supported managed
- * sources are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified source. The
Review Comment:
Alternatively, we could pull identifiers directly from the proto
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571580305
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformTranslation.java:
##########
@@ -0,0 +1,143 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.managed;
+
+import static org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedConfig;
+import static org.apache.beam.sdk.managed.ManagedSchemaTransformProvider.ManagedSchemaTransform;
+import static org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.HashMap;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.ManagedSchemaTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class ManagedSchemaTransformTranslation {
+ static class ManagedSchemaTransformTranslator
+ implements TransformPayloadTranslator<ManagedSchemaTransform> {
+ private final ManagedSchemaTransformProvider provider;
+ static final Schema SCHEMA =
+ Schema.builder()
+ .addStringField("transform_identifier")
+ .addNullableStringField("config")
+ .addNullableStringField("config_url")
+ .build();
+
+ public ManagedSchemaTransformTranslator() {
+ provider = new ManagedSchemaTransformProvider(null);
+ }
+
+ @Override
+ public String getUrn() {
+ return provider.identifier();
+ }
+
+ @Override
+ @SuppressWarnings("argument")
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ManagedSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(SCHEMA, true);
+ ManagedConfig managedConfig = application.getTransform().getManagedConfig();
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ ManagedSchemaTransformPayload.newBuilder()
+ .setUnderlyingTransformIdentifier(managedConfig.getTransformIdentifier())
+ .setYamlConfig(managedConfig.resolveUnderlyingConfig())
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ManagedSchemaTransform transform) {
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg Write SchemaTransform [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1561302426
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
Review Comment:
Right, under the hood TypedSchemaTransformProvider creates a schema for the AutoValue class and uses it for the actual `from(Row config)` method.
Adding a sanity test anyways to check we can build the transform using a Row
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class Config {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_Config.Builder();
+ }
+
+ public abstract String getTable();
+
+ public abstract CatalogConfig getCatalogConfig();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setTable(String tables);
+
+ public abstract Builder setCatalogConfig(CatalogConfig catalogConfig);
+
+ public abstract Config build();
+ }
+
+ public void validate() {
+ getCatalogConfig().validate();
+ }
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class CatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_CatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract CatalogConfig build();
+ }
+
+ Set<String> validTypes =
+ Sets.newHashSet(
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HADOOP,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HIVE,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_REST);
+
+ public void validate() {
+ if (Strings.isNullOrEmpty(getCatalogType())) {
+ checkArgument(
+ validTypes.contains(Preconditions.checkArgumentNotNull(getCatalogType())),
+ "Invalid catalog type. Please pick one of %s",
+ validTypes);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ static class IcebergWriteSchemaTransform extends SchemaTransform {
+ private final Config configuration;
+
+ IcebergWriteSchemaTransform(Config configuration) {
+ this.configuration = configuration;
+ }
+
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+
+ PCollection<Row> rows = input.get(INPUT_TAG);
+
+ CatalogConfig catalogConfig = configuration.getCatalogConfig();
+
+ IcebergCatalogConfig.Builder catalogBuilder =
+ IcebergCatalogConfig.builder()
+ .setName(catalogConfig.getCatalogName())
+ .setIcebergCatalogType(catalogConfig.getCatalogType())
+ .setWarehouseLocation(catalogConfig.getWarehouseLocation());
+
+ if (!Strings.isNullOrEmpty(catalogConfig.getCatalogType())) {
+ catalogBuilder = catalogBuilder.setIcebergCatalogType(catalogConfig.getCatalogType());
+ }
+ if (!Strings.isNullOrEmpty(catalogConfig.getWarehouseLocation())) {
+ catalogBuilder = catalogBuilder.setWarehouseLocation(catalogConfig.getWarehouseLocation());
+ }
+
+ // TODO: support dynamic destinations
+ DynamicDestinations dynamicDestinations =
+ DynamicDestinations.singleTable(TableIdentifier.parse(configuration.getTable()));
+
+ IcebergWriteResult result =
+ rows.apply(
+ IcebergIO.writeToDynamicDestinations(catalogBuilder.build(), dynamicDestinations));
+
+ PCollection<Row> snapshots =
+ result
+ .getSnapshots()
+ .apply(MapElements.via(new SnapshotToRow()))
+ .setRowSchema(SnapshotToRow.SNAPSHOT_SCHEMA);
+
+ return PCollectionRowTuple.of(OUTPUT_TAG, snapshots);
+ }
+
+ @VisibleForTesting
+ static class SnapshotToRow extends SimpleFunction<KV<String, Snapshot>, Row> {
+ static final Schema SNAPSHOT_SCHEMA =
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1573067555
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -51,38 +53,47 @@ static class IcebergReadSchemaTransformTranslator
@Override
public String getUrn() {
- return READ_PROVIDER.identifier();
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
}
@Override
public @Nullable FunctionSpec translate(
AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
throws IOException {
- SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Schema snakeCaseSchema = READ_SCHEMA.toSnakeCase();
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(snakeCaseSchema, true);
Row configRow = toConfigRow(application.getTransform());
ByteArrayOutputStream os = new ByteArrayOutputStream();
- RowCoder.of(READ_SCHEMA).encode(configRow, os);
+ RowCoder.of(snakeCaseSchema).encode(configRow, os);
return FunctionSpec.newBuilder()
.setUrn(getUrn())
.setPayload(
SchemaTransformPayload.newBuilder()
+ .setIdentifier(READ_PROVIDER.identifier())
.setConfigurationSchema(expansionSchema)
.setConfigurationRow(ByteString.copyFrom(os.toByteArray()))
- .setIdentifier(getUrn())
.build()
.toByteString())
.build();
}
@Override
public Row toConfigRow(IcebergReadSchemaTransform transform) {
- return transform.getConfigurationRow();
+ // Will retrieve a Row with snake_case naming convention.
+ // Transform expects camelCase convention, so convert back
+ // TODO(https://github.com/apache/beam/issues/31061): Remove conversion when
+ // TypedSchemaTransformProvider starts generating with snake_case convention
+ return transform.getConfigurationRow().toSnakeCase();
}
@Override
public IcebergReadSchemaTransform fromConfigRow(Row configRow, PipelineOptions options) {
Review Comment:
We also want to make sure that the SchemaTransformProvider.configuration() method for schema-aware transforms we develop returns a schema consistent with what we provide here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568193684
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -108,8 +107,8 @@ public static ManagedTransform read(String source) {
}
/**
- * Instantiates a {@link Managed.Write} transform for the specified sink. The supported managed
- * sinks are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified sink. The supported
Review Comment:
Can `ManagedTransform` take on that role? It extends SchemaTransform (ie. PTransform<PCollectionRowTuple, PCollectionRowTuple>) but we can change that to `PCollectionTuple` to include generic types in the future. Within, it can do whatever conversions necessary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg SchemaTransforms and Translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1568031375
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The transform's configuration schema
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
+// This can be used by runners that wish to override an underlying transform
+// with a different implementation.
+message ManagedTransformPayload {
+ // The underlying transform's URN.
+ string underlying_transform_urn = 1;
+ // The underlying transform's configuration Schema.
Review Comment:
This should be Managed Schema Transform's schema and payload not the underlying transform's.
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The transform's configuration schema
Review Comment:
Because the configuration schema is same as the schema of the row used for upgrading ?
(what we really want here is the latter)
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,229 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.Collections;
+import java.util.List;
+import java.util.Set;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+/**
+ * SchemaTransform implementation for {@link IcebergIO#writeToDynamicDestinations}. Writes Beam Rows
+ * to Iceberg and outputs a {@code PCollection<Row>} representing snapshots created in the process.
+ */
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ static final Schema OUTPUT_SCHEMA =
+ Schema.builder()
+ .addStringField("table")
+ .addStringField("operation")
+ .addMapField("summary", Schema.FieldType.STRING, Schema.FieldType.STRING)
+ .addStringField("manifestListLocation")
+ .build();
+
+ @Override
+ public String description() {
+ return "Writes Beam Rows to Iceberg.\n"
+ + "Returns a PCollection representing the snapshots produced in the process, with the following schema:\n"
+ + "{\"table\" (str), \"operation\" (str), \"summary\" (map[str, str]), \"manifestListLocation\" (str)}";
+ }
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class Config {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_Config.Builder();
+ }
+
+ public abstract String getTable();
+
+ public abstract CatalogConfig getCatalogConfig();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+ public abstract Builder setTable(String tables);
+
+ public abstract Builder setCatalogConfig(CatalogConfig catalogConfig);
+
+ public abstract Config build();
+ }
+
+ public void validate() {
+ getCatalogConfig().validate();
+ }
+ }
+
+ @DefaultSchema(AutoValueSchema.class)
+ @AutoValue
+ public abstract static class CatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergWriteSchemaTransformProvider_CatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract CatalogConfig build();
+ }
+
+ Set<String> validTypes =
+ Sets.newHashSet(
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HADOOP,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_HIVE,
+ CatalogUtil.ICEBERG_CATALOG_TYPE_REST);
+
+ public void validate() {
+ if (Strings.isNullOrEmpty(getCatalogType())) {
+ checkArgument(
+ validTypes.contains(Preconditions.checkArgumentNotNull(getCatalogType())),
+ "Invalid catalog type. Please pick one of %s",
+ validTypes);
+ }
+ }
+ }
+
+ @VisibleForTesting
+ static class IcebergWriteSchemaTransform extends SchemaTransform {
+ private final Config configuration;
+
+ IcebergWriteSchemaTransform(Config configuration) {
+ this.configuration = configuration;
+ }
+
+ @Override
+ public PCollectionRowTuple expand(PCollectionRowTuple input) {
+
+ PCollection<Row> rows = input.get(INPUT_TAG);
+
+ CatalogConfig catalogConfig = configuration.getCatalogConfig();
+
+ IcebergCatalogConfig.Builder catalogBuilder =
+ IcebergCatalogConfig.builder()
+ .setName(catalogConfig.getCatalogName())
+ .setIcebergCatalogType(catalogConfig.getCatalogType())
+ .setWarehouseLocation(catalogConfig.getWarehouseLocation());
+
+ if (!Strings.isNullOrEmpty(catalogConfig.getCatalogType())) {
+ catalogBuilder = catalogBuilder.setIcebergCatalogType(catalogConfig.getCatalogType());
+ }
+ if (!Strings.isNullOrEmpty(catalogConfig.getWarehouseLocation())) {
+ catalogBuilder = catalogBuilder.setWarehouseLocation(catalogConfig.getWarehouseLocation());
+ }
+
+ // TODO: support dynamic destinations
+ DynamicDestinations dynamicDestinations =
+ DynamicDestinations.singleTable(TableIdentifier.parse(configuration.getTable()));
+
+ IcebergWriteResult result =
+ rows.apply(
+ IcebergIO.writeToDynamicDestinations(catalogBuilder.build(), dynamicDestinations));
+
+ PCollection<Row> snapshots =
+ result
+ .getSnapshots()
+ .apply(MapElements.via(new SnapshotToRow()))
+ .setRowSchema(OUTPUT_SCHEMA);
+
+ return PCollectionRowTuple.of(OUTPUT_TAG, snapshots);
+ }
+
+ @VisibleForTesting
+ static class SnapshotToRow extends SimpleFunction<KV<String, Snapshot>, Row> {
+ @Override
+ public Row apply(KV<String, Snapshot> input) {
+ Snapshot snapshot = input.getValue();
+ Row row =
+ Row.withSchema(OUTPUT_SCHEMA)
+ .addValues(
+ input.getKey(),
+ snapshot.operation(),
+ snapshot.summary(),
+ snapshot.manifestListLocation())
+ .build();
+ System.out.println("SNAPSHOT: " + snapshot);
Review Comment:
Remove prints.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
Review Comment:
We discussed offline and this PR includes updated URNs/payloads for the new schema-aware transforms
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
+
+ byte[] catalogBytes = configRow.getBytes("catalog_config");
+ if (catalogBytes != null) {
+ builder = builder.setCatalogConfig((IcebergCatalogConfig) fromByteArray(catalogBytes));
+ }
+ String tableIdentifier = configRow.getString("table_identifier");
+ if (tableIdentifier != null) {
+ builder = builder.setTableIdentifier(TableIdentifier.parse(tableIdentifier));
+ }
+ return builder.build();
+ } catch (InvalidClassException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class ReadRegistrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ @SuppressWarnings({
+ "rawtypes",
+ })
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(AutoValue_IcebergIO_ReadRows.class, new IcebergIOReadTranslator())
+ .build();
+ }
+ }
+
+ static class IcebergIOWriteTranslator implements TransformPayloadTranslator<WriteRows> {
+
+ static final Schema WRITE_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .addNullableByteArrayField("dynamic_destinations")
+ .build();
+
+ public static final String ICEBERG_WRITE_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_write:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_WRITE_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, WriteRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(WRITE_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(WRITE_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(WriteRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+ if (transform.getDynamicDestinations() != null) {
+ fieldValues.put("dynamic_destinations", toByteArray(transform.getDynamicDestinations()));
Review Comment:
Ditto regarding Java serialization of complex objects.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java:
##########
@@ -19,73 +19,116 @@
import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
import org.apache.beam.sdk.io.Read;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
import org.apache.iceberg.Table;
import org.apache.iceberg.catalog.TableIdentifier;
import org.checkerframework.checker.nullness.qual.Nullable;
public class IcebergIO {
- public static WriteRows writeToDynamicDestinations(
- IcebergCatalogConfig catalog, DynamicDestinations dynamicDestinations) {
- return new WriteRows(catalog, dynamicDestinations);
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
+ return new AutoValue_IcebergIO_WriteRows.Builder().setCatalogConfig(catalog).build();
Review Comment:
Note that we don't necessarily have to use AutoValue here. If we just keep the existing "WriteRows" and register a TransformTranslation for that, it should still work (and probably simpler).
(and similarly for ReadRows)
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/YamlUtils.java:
##########
@@ -147,14 +147,27 @@ public static Row toBeamRow(
}
@SuppressWarnings("nullness")
- public static Row toBeamRow(Map<String, Object> yamlMap, Schema rowSchema, boolean toCamelCase) {
+ public static @Nullable Row toBeamRow(
+ @Nullable Map<String, Object> map, Schema rowSchema, boolean toCamelCase) {
+ if (map == null || map.isEmpty()) {
+ List<Field> requiredFields =
+ rowSchema.getFields().stream()
+ .filter(field -> !field.getType().getNullable())
+ .collect(Collectors.toList());
+ if (requiredFields.isEmpty()) {
+ return null;
+ } else {
Review Comment:
Also assert for the count ?
##########
sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformTranslation.java:
##########
@@ -276,7 +279,7 @@ static RunnerApi.PTransform toProto(
*
* <p>Does not register the {@code appliedPTransform} within the provided {@link SdkComponents}.
*/
- static RunnerApi.PTransform toProto(
+ public static RunnerApi.PTransform toProto(
Review Comment:
This access modifier change should not be needed.
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/Managed.java:
##########
@@ -108,8 +107,8 @@ public static ManagedTransform read(String source) {
}
/**
- * Instantiates a {@link Managed.Write} transform for the specified sink. The supported managed
- * sinks are:
+ * Instantiates a {@link Managed.ManagedTransform} transform for the specified sink. The supported
Review Comment:
I think we wanted to introduced an outer "Managed" PTransform that will be returned here that may have Java-specific adapters etc. in the future.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIO.java:
##########
@@ -19,73 +19,116 @@
import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
import org.apache.beam.sdk.io.Read;
import org.apache.beam.sdk.transforms.PTransform;
import org.apache.beam.sdk.values.PBegin;
import org.apache.beam.sdk.values.PCollection;
import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
import org.apache.iceberg.Table;
import org.apache.iceberg.catalog.TableIdentifier;
import org.checkerframework.checker.nullness.qual.Nullable;
public class IcebergIO {
- public static WriteRows writeToDynamicDestinations(
- IcebergCatalogConfig catalog, DynamicDestinations dynamicDestinations) {
- return new WriteRows(catalog, dynamicDestinations);
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
Review Comment:
Is this interface change needed ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
Review Comment:
We discussed using camel case here for consistency (and to prevent a case-conversion) right ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
Review Comment:
This and proto conversion can be a separate PR.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
Review Comment:
we should avoid Java serialization of complex objects if possible. I've seen these easily running into serialization related errors when re-building across versions.
How about making `IcebergCatalogConfig` implement `Externalizable` and implementing `readExternal` and `writeExternal` that we can better reason about when it comes to serialization ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
Review Comment:
I was expecting that we could build the transform using the schema-aware transform Row constructor instead of using a builder here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571435142
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
After discussing offline, we decided it's best to not worry about serializing Java objects.
Instead of serializing at the IcebergIO level, we decided to serialize at the Iceberg SchemaTransform level (which doesn't include DynamicDestinations)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571618296
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
+ // The underlying schema-aware transform's identifier.
+ string underlying_transform_identifier = 1;
+ // The managed transform's configuration Schema.
+ Schema expansion_schema = 2;
+ // The configuration used to override the managed transform.
+ // Must be compatible with the expansion schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 3;
+ // The underlying transform's configuration, represented as a YAML string.
+ string yaml_config = 4;
Review Comment:
Both inline config (YAML string) and config URL are part of `expansion_payload`
I believe the rationale for having a separate `yaml_config` was to make it easier to display config values
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571754391
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
Review Comment:
I suppose I'm having trouble seeing the difference between SchemaTransforms and Schema-aware Transforms. (Also, the "stuff needed during expansion" *is* precisely this, the identifier and corresponding config.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571772809
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+// The information available in the payload can be used by runners to override the schema-aware transform.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to override the transform
+ Schema expansion_schema = 1;
+ // The configuration used to override this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a ManagedSchemaTransform.
+// The information available in the payload can be used by runners to override the ManagedSchemaTransform.
+message ManagedSchemaTransformPayload {
Review Comment:
I would like to keep `ManagedSchemaTransformPayload` separate since this has additional fields (and `underlying_transform_identifier` and `identifier` are different semantically).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574636070
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
Review Comment:
Adding it to the if-statement above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1575298682
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
The ExpansionService was updated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1573857432
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -92,6 +92,12 @@ private static Schema getDataFileAvroSchema(FileWriteResult fileWriteResult) {
@Override
public void encode(FileWriteResult value, OutputStream outStream)
throws CoderException, IOException {
+ // "version" of this coder.
+ // If breaking changes are introduced (e.g. from Beam, Iceberg, Avro, etc..),
+ // then update this version and create a fork in decode() below for the new decode logic.
+ // This helps keep the pipeline update-compatible
+ outStream.write(0);
Review Comment:
Let's make sure we have unit tests that exercise reading and writing from this coder including the version bit.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
Review Comment:
Let's also update `TransformTranslation` so that we add the SchemaTransform identifier as an annotation if the URN is `SCHEMA_TRANSFORM`. See below for an example (and you can get the identifier from the payload add the the annotation at the same location given below). This will allow us the identify schema-transforms that we want manage without parsing the payload.
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/core/src/main/java/org/apache/beam/sdk/util/construction/PTransformTranslation.java#L528
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIO.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.util.Preconditions.checkStateNotNull;
+
+import com.google.auto.value.AutoValue;
+import java.util.Arrays;
+import java.util.List;
+import org.apache.beam.sdk.io.Read;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.values.PBegin;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Predicates;
+import org.apache.iceberg.Table;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+public class IcebergIO {
+
+ public static WriteRows writeRows(IcebergCatalogConfig catalog) {
+ return new AutoValue_IcebergIO_WriteRows.Builder().setCatalogConfig(catalog).build();
Review Comment:
Do we still need the new builders here given that we are directly initializing schema-transforms in `TransformPayloadTranslator`s ?
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.model.pipeline.v1.ExternalTransforms.ExpansionMethods.Enum.SCHEMA_TRANSFORM;
+import static org.apache.beam.sdk.io.iceberg.IcebergReadSchemaTransformProvider.IcebergReadSchemaTransform;
+import static org.apache.beam.sdk.io.iceberg.IcebergWriteSchemaTransformProvider.IcebergWriteSchemaTransform;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import org.apache.beam.model.pipeline.v1.ExternalTransforms.SchemaTransformPayload;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.BeamUrns;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergSchemaTransformTranslation {
+ static class IcebergReadSchemaTransformTranslator
+ implements TransformPayloadTranslator<IcebergReadSchemaTransform> {
+ static final IcebergReadSchemaTransformProvider READ_PROVIDER =
+ new IcebergReadSchemaTransformProvider();
+ static final Schema READ_SCHEMA = READ_PROVIDER.configurationSchema();
+
+ @Override
+ public String getUrn() {
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
Review Comment:
Actually, we use the urn when registering `TransformProvider`s at the `ExpansionService`.
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/expansion-service/src/main/java/org/apache/beam/sdk/expansion/service/ExpansionService.java#L168
Hence' this will require us to do future modifications where we keep track of `TransformProvider`s using the schema-transform identifier. I think this is OK but wanted to call this out.
The other option will be go to back to using different URNs for different schema-aware transforms.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformTranslation.java:
##########
@@ -51,38 +53,47 @@ static class IcebergReadSchemaTransformTranslator
@Override
public String getUrn() {
- return READ_PROVIDER.identifier();
+ return BeamUrns.getUrn(SCHEMA_TRANSFORM);
}
@Override
public @Nullable FunctionSpec translate(
AppliedPTransform<?, ?, IcebergReadSchemaTransform> application, SdkComponents components)
throws IOException {
- SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Schema snakeCaseSchema = READ_SCHEMA.toSnakeCase();
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(snakeCaseSchema, true);
Row configRow = toConfigRow(application.getTransform());
ByteArrayOutputStream os = new ByteArrayOutputStream();
- RowCoder.of(READ_SCHEMA).encode(configRow, os);
+ RowCoder.of(snakeCaseSchema).encode(configRow, os);
return FunctionSpec.newBuilder()
.setUrn(getUrn())
.setPayload(
SchemaTransformPayload.newBuilder()
+ .setIdentifier(READ_PROVIDER.identifier())
.setConfigurationSchema(expansionSchema)
.setConfigurationRow(ByteString.copyFrom(os.toByteArray()))
- .setIdentifier(getUrn())
.build()
.toByteString())
.build();
}
@Override
public Row toConfigRow(IcebergReadSchemaTransform transform) {
- return transform.getConfigurationRow();
+ // Will retrieve a Row with snake_case naming convention.
+ // Transform expects camelCase convention, so convert back
+ // TODO(https://github.com/apache/beam/issues/31061): Remove conversion when
+ // TypedSchemaTransformProvider starts generating with snake_case convention
+ return transform.getConfigurationRow().toSnakeCase();
}
@Override
public IcebergReadSchemaTransform fromConfigRow(Row configRow, PipelineOptions options) {
Review Comment:
Sounds good. Thanks.
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergSchemaTransformCatalogConfig.java:
##########
@@ -0,0 +1,105 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.value.AutoValue;
+import java.util.Set;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.NoSuchSchemaException;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaRegistry;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.annotations.SchemaFieldDescription;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@DefaultSchema(AutoValueSchema.class)
+@AutoValue
+public abstract class IcebergSchemaTransformCatalogConfig {
+ public static Builder builder() {
+ return new AutoValue_IcebergSchemaTransformCatalogConfig.Builder();
+ }
+
+ public abstract String getCatalogName();
+
+ @SchemaFieldDescription("Valid types are: {hadoop, hive, rest}")
+ public abstract @Nullable String getCatalogType();
+
+ public abstract @Nullable String getCatalogImplementation();
+
+ public abstract @Nullable String getWarehouseLocation();
+
+ @AutoValue.Builder
+ public abstract static class Builder {
+
+ public abstract Builder setCatalogName(String catalogName);
+
+ public abstract Builder setCatalogType(String catalogType);
+
+ public abstract Builder setCatalogImplementation(String catalogImplementation);
+
+ public abstract Builder setWarehouseLocation(String warehouseLocation);
+
+ public abstract IcebergSchemaTransformCatalogConfig build();
+ }
+
+ public static final Schema SCHEMA;
+
+ static {
+ try {
+ SCHEMA =
+ SchemaRegistry.createDefault()
+ .getSchema(IcebergSchemaTransformCatalogConfig.class)
+ .sorted()
+ .toSnakeCase();
Review Comment:
Add a comment on why we use `toSnakeCase()` here and other schema-transform schemas we define in this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569460364
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
So:
Managed SchemaTransform: beam:transform:managed:v1
Iceberg Read Schematransform: beam:transform:iceberg_read:v1
Iceberg Write Schematransform: beam:transform:iceberg_write:v1
(for both URNs and schema-transform ids)
(document the reason for breaking the convention in the identifier() methods.)
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
So:
Managed SchemaTransform: `beam:transform:managed:v1`
Iceberg Read Schematransform: `beam:transform:iceberg_read:v1`
Iceberg Write Schematransform: `beam:transform:iceberg_write:v1`
(for both URNs and schema-transform ids)
(document the reason for breaking the convention in the identifier() methods.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569413654
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
Review Comment:
Should we change the name to `ManagedSchemaTransformPayload` ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569586395
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
`beam:transform:managed:v1` for ManagedSchemaTransform makes sense because it's a unique case.
For IO SchemaTransforms however, I'd think it makes more sense to stick to the "beam:schematransform:org.apache.beam:<io>:v1" convention. For example Iceberg read IO will have a urn like `beam:transform:iceberg_read:v1`, whereas its schematransform implementation will have the identifier `beam:schematransform:org.apache.beam:iceberg_read:v1`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569447460
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:transform:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:transform:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
+// This is a transform that is aware of its input and output PCollection schemas
+// and is configured using Beam Schema-compatible parameters.
+message SchemaAwareTransformPayload {
+ // The schema of the configuration row used to upgrade the transform
+ Schema expansion_schema = 1;
+ // The configuration used to build this transform.
+ // Must be compatible with the configuration schema, and decodable via beam:coder:row:v1.
+ bytes expansion_payload = 2;
+}
+
+// Payload for a Managed transform
Review Comment:
Yeah, that makes sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1569642022
##########
sdks/java/managed/src/main/java/org/apache/beam/sdk/managed/ManagedSchemaTransformProvider.java:
##########
@@ -51,14 +54,14 @@ public class ManagedSchemaTransformProvider
@Override
public String identifier() {
- return "beam:schematransform:org.apache.beam:managed:v1";
+ return "beam:transform:managed:v1";
Review Comment:
We agreed offline to keep the URNs the same for simplicity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Iceberg Write SchemaTransform [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1562688850
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/io/iceberg/IcebergWriteSchemaTransformProvider.java:
##########
@@ -0,0 +1,216 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.io.iceberg;
+
+import static org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Preconditions.checkArgument;
+
+import com.google.auto.service.AutoService;
+import com.google.auto.value.AutoValue;
+import java.util.*;
+import org.apache.beam.io.iceberg.IcebergWriteSchemaTransformProvider.Config;
+import org.apache.beam.sdk.schemas.AutoValueSchema;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.annotations.DefaultSchema;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransform;
+import org.apache.beam.sdk.schemas.transforms.SchemaTransformProvider;
+import org.apache.beam.sdk.schemas.transforms.TypedSchemaTransformProvider;
+import org.apache.beam.sdk.transforms.MapElements;
+import org.apache.beam.sdk.transforms.SimpleFunction;
+import org.apache.beam.sdk.util.Preconditions;
+import org.apache.beam.sdk.values.KV;
+import org.apache.beam.sdk.values.PCollection;
+import org.apache.beam.sdk.values.PCollectionRowTuple;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.annotations.VisibleForTesting;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.base.Strings;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.Sets;
+import org.apache.iceberg.CatalogUtil;
+import org.apache.iceberg.Snapshot;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@AutoService(SchemaTransformProvider.class)
+public class IcebergWriteSchemaTransformProvider extends TypedSchemaTransformProvider<Config> {
+
+ static final String INPUT_TAG = "input";
+ static final String OUTPUT_TAG = "output";
+
+ @Override
+ protected SchemaTransform from(Config configuration) {
+ configuration.validate();
+ return new IcebergWriteSchemaTransform(configuration);
+ }
+
+ @Override
+ public List<String> inputCollectionNames() {
+ return Collections.singletonList(INPUT_TAG);
+ }
+
+ @Override
+ public List<String> outputCollectionNames() {
+ return Collections.singletonList(OUTPUT_TAG);
+ }
+
+ @Override
+ public String identifier() {
+ return "beam:schematransform:org.apache.beam:iceberg_write:v1";
Review Comment:
I'm not sure what the question is here. SchemaTransform has the ability to query for its schema. Or is it that we want the schema to possibly evolve?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571769841
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
Review Comment:
Ok, I'm fine with using the`SchemaTransformPayload` message from 'external_transforms.proto' as the transform payload of SchemaTransforms and removing this message.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571733471
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
If there's complexity here that's all the more reason to not bake this into the implementation and tie our hands with update compatibility, right?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "robertwb (via GitHub)" <gi...@apache.org>.
robertwb commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571734103
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
That'll be dangerous. Let's see if we can remove this. (Fortunately we don't have any existing Iceberg to stay compatible with.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "chamikaramj (via GitHub)" <gi...@apache.org>.
chamikaramj commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571573242
##########
model/pipeline/src/main/proto/org/apache/beam/model/pipeline/v1/schema_aware_transforms.proto:
##########
@@ -0,0 +1,73 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+
+// Protocol Buffers describing available Schema-Aware transforms.
+// These are transforms that can be configured using Beam Schema-compatible parameters.
+// Runners can override these transforms with a native implementation.
+
+syntax = "proto3";
+
+package org.apache.beam.model.pipeline.v1;
+
+option go_package = "github.com/apache/beam/sdks/v2/go/pkg/beam/model/pipeline_v1;pipeline_v1";
+option java_package = "org.apache.beam.model.pipeline.v1";
+option java_outer_classname = "SchemaAwareTransforms";
+
+import "org/apache/beam/model/pipeline/v1/beam_runner_api.proto";
+import "org/apache/beam/model/pipeline/v1/schema.proto";
+
+
+message StandardSchemaAwareTransforms {
+ // Payload for all of these: SchemaAwareTransformPayload
+ enum IOs {
+ ICEBERG_READ = 0 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_read:v1"];
+ ICEBERG_WRITE = 1 [(beam_urn) = "beam:schematransform:org.apache.beam:iceberg_write:v1"];
+ }
+ enum Managed {
+ // Payload: ManagedSchemaTransformPayload
+ MANAGED = 0 [(beam_urn) = "beam:transform:managed:v1"];
+ }
+}
+
+
+// Payload for a Schema-aware PTransform.
Review Comment:
We decided to introduce a seperate proto for clarity exactly because some of the payloads look very similar but are functionality different.
* This file provides URNs and payloads for schema-aware transforms (ideally all this should be in beam_runner_api.proto but we couldn't due to the schema dependency).
* external_transforms.proto contains stuff needed during expansion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1574670744
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -92,6 +92,12 @@ private static Schema getDataFileAvroSchema(FileWriteResult fileWriteResult) {
@Override
public void encode(FileWriteResult value, OutputStream outStream)
throws CoderException, IOException {
+ // "version" of this coder.
+ // If breaking changes are introduced (e.g. from Beam, Iceberg, Avro, etc..),
+ // then update this version and create a fork in decode() below for the new decode logic.
+ // This helps keep the pipeline update-compatible
+ outStream.write(0);
Review Comment:
Tests already exist in FileWriteResultTest:
https://github.com/apache/beam/blob/93aa62c2900b210ee5e26849949154f19db51f4c/sdks/java/io/iceberg/src/test/java/org/apache/beam/io/iceberg/FileWriteResultTest.java#L137-L142
I'm adding another encoding-decoding test though to check the version number
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571131047
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergIOTranslation.java:
##########
@@ -0,0 +1,241 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.beam.sdk.io.iceberg;
+
+import static org.apache.beam.sdk.io.iceberg.IcebergIO.ReadRows;
+import static org.apache.beam.sdk.io.iceberg.IcebergIO.WriteRows;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.fromByteArray;
+import static org.apache.beam.sdk.util.construction.TransformUpgrader.toByteArray;
+
+import com.google.auto.service.AutoService;
+import java.io.ByteArrayOutputStream;
+import java.io.IOException;
+import java.io.InvalidClassException;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.stream.Collectors;
+import org.apache.beam.model.pipeline.v1.RunnerApi.FunctionSpec;
+import org.apache.beam.model.pipeline.v1.SchemaApi;
+import org.apache.beam.model.pipeline.v1.SchemaAwareTransforms.SchemaAwareTransformPayload;
+import org.apache.beam.sdk.coders.RowCoder;
+import org.apache.beam.sdk.options.PipelineOptions;
+import org.apache.beam.sdk.runners.AppliedPTransform;
+import org.apache.beam.sdk.schemas.Schema;
+import org.apache.beam.sdk.schemas.SchemaTranslation;
+import org.apache.beam.sdk.transforms.PTransform;
+import org.apache.beam.sdk.util.construction.PTransformTranslation.TransformPayloadTranslator;
+import org.apache.beam.sdk.util.construction.SdkComponents;
+import org.apache.beam.sdk.util.construction.TransformPayloadTranslatorRegistrar;
+import org.apache.beam.sdk.values.Row;
+import org.apache.beam.vendor.grpc.v1p60p1.com.google.protobuf.ByteString;
+import org.apache.beam.vendor.guava.v32_1_2_jre.com.google.common.collect.ImmutableMap;
+import org.apache.iceberg.catalog.TableIdentifier;
+import org.checkerframework.checker.nullness.qual.Nullable;
+
+@SuppressWarnings({"rawtypes", "nullness"})
+public class IcebergIOTranslation {
+ static class IcebergIOReadTranslator implements TransformPayloadTranslator<ReadRows> {
+
+ static final Schema READ_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .build();
+
+ public static final String ICEBERG_READ_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_read:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_READ_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, ReadRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(READ_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(READ_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(ReadRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+
+ return Row.withSchema(READ_SCHEMA).withFieldValues(fieldValues).build();
+ }
+
+ @Override
+ public ReadRows fromConfigRow(Row configRow, PipelineOptions options) {
+ try {
+ ReadRows.Builder builder = new AutoValue_IcebergIO_ReadRows.Builder();
+
+ byte[] catalogBytes = configRow.getBytes("catalog_config");
+ if (catalogBytes != null) {
+ builder = builder.setCatalogConfig((IcebergCatalogConfig) fromByteArray(catalogBytes));
+ }
+ String tableIdentifier = configRow.getString("table_identifier");
+ if (tableIdentifier != null) {
+ builder = builder.setTableIdentifier(TableIdentifier.parse(tableIdentifier));
+ }
+ return builder.build();
+ } catch (InvalidClassException e) {
+ throw new RuntimeException(e);
+ }
+ }
+ }
+
+ @AutoService(TransformPayloadTranslatorRegistrar.class)
+ public static class ReadRegistrar implements TransformPayloadTranslatorRegistrar {
+ @Override
+ @SuppressWarnings({
+ "rawtypes",
+ })
+ public Map<? extends Class<? extends PTransform>, ? extends TransformPayloadTranslator>
+ getTransformPayloadTranslators() {
+ return ImmutableMap.<Class<? extends PTransform>, TransformPayloadTranslator>builder()
+ .put(AutoValue_IcebergIO_ReadRows.class, new IcebergIOReadTranslator())
+ .build();
+ }
+ }
+
+ static class IcebergIOWriteTranslator implements TransformPayloadTranslator<WriteRows> {
+
+ static final Schema WRITE_SCHEMA =
+ Schema.builder()
+ .addByteArrayField("catalog_config")
+ .addNullableStringField("table_identifier")
+ .addNullableByteArrayField("dynamic_destinations")
+ .build();
+
+ public static final String ICEBERG_WRITE_TRANSFORM_URN =
+ "beam:transform:org.apache.beam:iceberg_write:v1";
+
+ @Override
+ public String getUrn() {
+ return ICEBERG_WRITE_TRANSFORM_URN;
+ }
+
+ @Override
+ public @Nullable FunctionSpec translate(
+ AppliedPTransform<?, ?, WriteRows> application, SdkComponents components)
+ throws IOException {
+ SchemaApi.Schema expansionSchema = SchemaTranslation.schemaToProto(WRITE_SCHEMA, true);
+ Row configRow = toConfigRow(application.getTransform());
+ ByteArrayOutputStream os = new ByteArrayOutputStream();
+ RowCoder.of(WRITE_SCHEMA).encode(configRow, os);
+
+ return FunctionSpec.newBuilder()
+ .setUrn(getUrn())
+ .setPayload(
+ SchemaAwareTransformPayload.newBuilder()
+ .setExpansionSchema(expansionSchema)
+ .setExpansionPayload(ByteString.copyFrom(os.toByteArray()))
+ .build()
+ .toByteString())
+ .build();
+ }
+
+ @Override
+ public Row toConfigRow(WriteRows transform) {
+
+ Map<String, Object> fieldValues = new HashMap<>();
+
+ if (transform.getCatalogConfig() != null) {
+ fieldValues.put("catalog_config", toByteArray(transform.getCatalogConfig()));
+ }
+ if (transform.getTableIdentifier() != null) {
+ TableIdentifier identifier = transform.getTableIdentifier();
+ List<String> identifierParts =
+ Arrays.stream(identifier.namespace().levels()).collect(Collectors.toList());
+ identifierParts.add(identifier.name());
+ fieldValues.put("table_identifier", String.join(".", identifierParts));
+ }
+ if (transform.getDynamicDestinations() != null) {
+ fieldValues.put("dynamic_destinations", toByteArray(transform.getDynamicDestinations()));
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571130805
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/DynamicDestinations.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
As discussed offline, will implement custom serialization for `OneTableDynamicDestinations` and make translation fail for other DynamicDestination classes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1571634913
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/FileWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Can't speak to this design decision, but it seems to be doing a lot of custom encoding with the help of some Apache Iceberg utils. I don't know if SchemaCoder would capture this complexity
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Managed Transform protos & translation; Iceberg SchemaTransforms & translation [beam]
Posted by "ahmedabu98 (via GitHub)" <gi...@apache.org>.
ahmedabu98 commented on code in PR #30910:
URL: https://github.com/apache/beam/pull/30910#discussion_r1572621753
##########
sdks/java/io/iceberg/src/main/java/org/apache/beam/sdk/io/iceberg/IcebergWriteResult.java:
##########
@@ -15,7 +15,7 @@
* See the License for the specific language governing permissions and
* limitations under the License.
*/
-package org.apache.beam.io.iceberg;
+package org.apache.beam.sdk.io.iceberg;
Review Comment:
Sounds good, I created a new AutoValue class (SnapshotInfo) to represent this data. Lmk if this is what you had in mind
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org