You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by GitBox <gi...@apache.org> on 2021/09/27 05:01:00 UTC

[GitHub] [flink] jherico opened a new pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

jherico opened a new pull request #17360:
URL: https://github.com/apache/flink/pull/17360


   Fix for https://issues.apache.org/jira/browse/FLINK-24379
   
   ## What is the purpose of the change
   
   Add missing components required to use the AWS Glue Schema registry in Table connectors.
   
   ## Brief change log
   
     - Added GluSchemaRegistryAvroFormatFactory class and reference to same in `flink-formats/flink-avro-glue-schema-registry/src/main/resources/META-INF/services/org.apache.flink.table.factories.Factory`
     - 
   
   ## Verifying this change
   
   This change added tests and can be verified as follows:
   
     - Added test class org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroDeserializationSchemaTest based off org.apache.flink.formats.avro.registry.confluent.RegistryAvroFormatFactoryTest
     - Verified that the test passed after some work correcting the options naming.
   
   ## Does this pull request potentially affect one of the following parts:
   
     - Dependencies (does it add or upgrade a dependency): yes
       - Adds internal dependency from flink-avro-glue-schema-registry to the table API jars
     - The public API, i.e., is any changed class annotated with `@Public(Evolving)`: no public API changed
     - The serializers: don't know
     - The runtime per-record code paths (performance sensitive): no
     - Anything that affects deployment or recovery: JobManager (and its components), Checkpointing, Kubernetes/Yarn, ZooKeeper: no
     - The S3 file system connector: no
   
   ## Documentation
   
     - Does this pull request introduce a new feature? No?  It's adding an existing feature that is present in all the other Avro formats to the AWS Glue Avro format
     - If yes, how is the feature documented? JavaDocs, tweaked from the similar docs in RegistryAvroFormatFactory
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r720131157



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro AWS Glue Schema Registry
+    maven: flink-avro-glue-schema-registry
+    category: format
+    sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/$version/flink-avro-glue-schema-registry-$version.jar

Review comment:
       This has recently changed, since 1.15 it will only support Scala 2.12. You can see the new artifact [here](https://repository.apache.org/content/repositories/snapshots/org/apache/flink/flink-avro-glue-schema-registry_2.12/1.15-SNAPSHOT/).




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688",
       "triggerID" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717342648



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       Same question as above

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       @hhkkxxx133 can you please double check these are the only required field?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroSchemaCoderProvider.java
##########
@@ -50,4 +51,29 @@ public GlueSchemaRegistryAvroSchemaCoderProvider(Map<String, Object> configs) {
     public GlueSchemaRegistryAvroSchemaCoder get() {
         return new GlueSchemaRegistryAvroSchemaCoder(transportName, configs);
     }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass()) {
+            return false;
+        }
+        GlueSchemaRegistryAvroSchemaCoderProvider that = (GlueSchemaRegistryAvroSchemaCoderProvider) o;
+        if (transportName == null)  {
+            if (that.transportName != null) {
+                return false;
+            }
+        } else if (!transportName.equals(that.transportName)) {
+            return false;
+        }
+        return configs.equals(that.configs);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(transportName, configs);
+    }
+

Review comment:
       Please confirm why you added these, and that it is acceptable per the coding guidelines:
   - https://flink.apache.org/contributing/code-style-and-quality-java.html#equals--hashcode

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       What does this mode refer to? Does this impact the type of connector? For instance, Kafka which supports upsert, and kinesis which does not (insert only)?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-929472860


   Responded to feedback, modified changelog mode to match existing code, added format document.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719126040



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       @dannycranmer I'm assuming AWS Glue could be used with Kinesis, what do you think? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "FAILURE",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 6c5edf94116d5e9cbea62d94719c32110e87fa0d Azure: [FAILURE](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717755865



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       Technically it [looks like](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/common/configs/GlueSchemaRegistryConfiguration.java#L40) the region is the only required parameter from the perspective of the underlying glue code.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716419484



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {

Review comment:
       fixed

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-932115740


   > Can you please add an e2e test for the Table API support?
   
   Bump: [Here](https://github.com/apache/flink/blob/master/flink-end-to-end-tests/flink-glue-schema-registry-avro-test/src/test/java/org/apache/flink/glue/schema/registry/test/GlueSchemaRegistryAvroKinesisITCase.java) is an example. Note that keys need to be provided via `IT_CASE_GLUE_SCHEMA_ACCESS_KEY` and `IT_CASE_GLUE_SCHEMA_SECRET_KEY`. The Flink CI provides keys to run the tests


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] hhkkxxx133 commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
hhkkxxx133 commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718011735



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro Schema Registry

Review comment:
       Can the name be distinguished from Confluent? For example Avro Glue Schema Registry?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));

Review comment:
       Duplicate?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       I agree. Looks good to me.

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format

Review comment:
       Should be updated as well to indicate Glue Schema Registry.

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro

Review comment:
       Title is incorrectly referring to "Confluent".

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [subject](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.subject`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.subject' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td><h5>format</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what format to use, here should be <code>'avro-glue'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.region</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what AWS region to use, such as <code>'us-east-1'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.registry.name</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The name (not the ARN) of the Glue schema registry in which to store the schemas.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.subject</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The subject name under which to store the schema in the registry.</td>

Review comment:
       Can this be named to `avro-glue.schema.name`?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/AvroGlueFormatOptions.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+@PublicEvolving
+public class AvroGlueFormatOptions {
+    public static final String PREFIX = "schema-registry.";
+
+    public static final ConfigOption<String> AWS_REGION = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.AWS_REGION)
+            .stringType().noDefaultValue().withDescription("AWS region");
+
+    public static final ConfigOption<String> REGISTRY_NAME = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.REGISTRY_NAME)
+            .stringType().noDefaultValue().withDescription("Registry name");
+
+    public static final ConfigOption<String> RECORD_TYPE = ConfigOptions
+            .key(PREFIX + AWSSchemaRegistryConstants.AVRO_RECORD_TYPE).stringType()
+            .defaultValue(AvroRecordType.GENERIC_RECORD.getName()).withDescription("Record type");
+
+    public static final ConfigOption<String> SCHEMA_REGISTRY_SUBJECT = ConfigOptions.key(PREFIX + "subject").stringType()

Review comment:
       Same as above: can this be named to `SCHEMA_NAME`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717739956



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       I'm actually a bit confused on this point.  The `Format` interface requires implementation of the `getChangelogMode` method, but I don't understand why this is tied to the format at all.  If I'm using a Kafka-Upsert connector then I should get upsert records... if I'm using a generic Kafka stream I should expect all kinds of records, both of which are totally orthogonal to the format used. 
   
   When I look at the other Format implementations they all seem to either be using the explicit form of `ChangelogMode.all()` or `ChangelogMode.insetOnly()` with no obvious (to me) pattern for why a given Format implementation gets one or the other.
   
   I've changed the code in these two instances to `ChangelogMode.all()` to match the subset of other formats that also does this, including the `DebeziumAvroFormatFactory`, but without a deeper understanding of how the `Format.getChangelogMode` is used, I can't reason more precisely about what this _should_ be. 
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "CANCELED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * d364ab24be51b23c260be42ddac97690384279ac Azure: [CANCELED](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520) 
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * d364ab24be51b23c260be42ddac97690384279ac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719846371



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro AWS Glue Schema Registry
+    maven: flink-avro-glue-schema-registry
+    category: format
+    sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/$version/flink-avro-glue-schema-registry-$version.jar

Review comment:
       The module is pure java and the only dependencies that it has that are scala-binary-versioned are `test` or `provided` scope.  The module is currently located [here](https://repo1.maven.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/) in the central repository, so I don't think this is the case.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70568032b21299c96c12505b8e7a413370318ae6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603) 
   * 72a98b6c81ca3972146961a485a37c48cbe2033e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718264199



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).

Review comment:
       > The Glue Schema Registry
   
   Please add "AWS" here:
   
   > The AWS Glue Schema Registry

##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro AWS Glue Schema Registry
+    maven: flink-avro-glue-schema-registry
+    category: format
+    sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-avro-glue-schema-registry/$version/flink-avro-glue-schema-registry-$version.jar

Review comment:
       I believe this needs to include the `$scala_version`, and if possible, only render for Scala 2.12

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       I had a pending comment to say the same. Yes I believe this format is connector agnostic, infact the e2e tests for datastream API use Kinesis

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       If we want to enforce a naming convention consistent with other Flink formats/connectors, we could transform them in the factory.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] Airblader commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718326427



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       The format can define a changelog, but it'll be up to the connector to defer to the format for providing the changelog mode. `SocketDynamicTableSource` in `flink-examples` would be an example where the connector just defers this to the format.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-929472860


   Responded to feedback, modified changelog mode to match existing code, added format document.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718067814



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro Schema Registry

Review comment:
       fixed

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro

Review comment:
       fixed

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718315208



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       I don't think this sentence should be in the docs?  

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       Why can't it be used together with Kinesis? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1e11514b73a035cad03c582c00e5a156ec7ab115 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594) 
   * 70568032b21299c96c12505b8e7a413370318ae6 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719847453



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).

Review comment:
       will fix.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718315208



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       I don't think this sentence should be in the docs?  

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       Why can't it be used together with Kinesis? 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70568032b21299c96c12505b8e7a413370318ae6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717739956



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       I'm actually a bit confused on this point.  The `Format` interface requires implementation of the `getChangelogMode` method, but I don't understand why this is tied to the format at all.  If I'm using a Kafka-Upsert connector then I should get upsert records... if I'm using a generic Kafka stream I should expect all kinds of records, both of which are totally orthogonal to the format used. 
   
   When I look at the other Format implementations they all seem to either be using the explicit form of `ChangelogMode.all()` or `ChangelogMode.insetOnly()` with no obvious (to me) pattern for why a given Format implementation gets one or the other.
   
   I've changed the code in these two instances to `ChangelogMode.all()` to match the subset of other formats that also does this, including the `DebeziumAvroFormatFactory`, but without a deeper understanding of how the `Format.getChangelogMode` is used, I can't reason more precisely about what this _should_ be. 
   

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       Technically it [looks like](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/common/configs/GlueSchemaRegistryConfiguration.java#L40) the region is the only required parameter from the perspective of the underlying glue code.  

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroSchemaCoderProvider.java
##########
@@ -50,4 +51,29 @@ public GlueSchemaRegistryAvroSchemaCoderProvider(Map<String, Object> configs) {
     public GlueSchemaRegistryAvroSchemaCoder get() {
         return new GlueSchemaRegistryAvroSchemaCoder(transportName, configs);
     }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass()) {
+            return false;
+        }
+        GlueSchemaRegistryAvroSchemaCoderProvider that = (GlueSchemaRegistryAvroSchemaCoderProvider) o;
+        if (transportName == null)  {
+            if (that.transportName != null) {
+                return false;
+            }
+        } else if (!transportName.equals(that.transportName)) {
+            return false;
+        }
+        return configs.equals(that.configs);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(transportName, configs);
+    }
+

Review comment:
       The test code relies on the existing implementations of [RegistryAvroSerializationSchema.equals](https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroSerializationSchema.java#L115) and [RegistryAvroDeserializationSchema.equals](https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L93), both of which then rely on their respective `schemaCoderProvider` having a valid `equals` implementation.  Although my only use case is testing, removing this function would break the equals contract for those containing classes.  
   
   Because the class is effectively immutable, I don't believe it contravenes the guidelines.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] Airblader commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716413338



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {

Review comment:
       Please mark this as `@Internal`.

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroOptions.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+public class GlueSchemaRegistryAvroOptions {

Review comment:
       Please follow the naming scheme here and name this `AvroGlueFormatOptions`. Also, this needs to be marked `@PublicEvolving`.

##########
File path: flink-formats/flink-avro-glue-schema-registry/pom.xml
##########
@@ -178,5 +192,33 @@ under the License.
 			<version>${glue.schema.registry.version}</version>
 		</dependency>
 
+		<!-- test dependencies -->
+		<dependency>
+			<groupId>org.apache.flink</groupId>
+			<artifactId>flink-table-api-java</artifactId>
+			<version>${project.version}</version>
+			<scope>test</scope>
+			<type>test-jar</type>
+		</dependency>
+		<dependency>
+			<groupId>org.apache.flink</groupId>
+			<artifactId>flink-table-runtime-blink_${scala.binary.version}</artifactId>

Review comment:
       This module doesn't exist since the legacy planner has been removed, and the Blink planner is now the only planner.
   
   ```suggestion
   			<artifactId>flink-table-runtime_${scala.binary.version}</artifactId>
   ```

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroOptions.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+public class GlueSchemaRegistryAvroOptions {
+    private GlueSchemaRegistryAvroOptions() {

Review comment:
       nit: please move the constructor to the end of the file (static constants come before the constructor).

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {

Review comment:
       There's a typo (glu → glue) in this class name.

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        var configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        var properties = new HashMap<String, Object>();

Review comment:
       Here, too, we cannot use `var`.

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        var configMap = buildConfigMap(formatOptions);

Review comment:
       We're on Java 8 and cannot use `var`. Please use `final Map<…>`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * d364ab24be51b23c260be42ddac97690384279ac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520) 
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-928976369






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1e11514b73a035cad03c582c00e5a156ec7ab115 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594) 
   * 70568032b21299c96c12505b8e7a413370318ae6 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718763920



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       reworded

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/AvroGlueFormatOptions.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+@PublicEvolving
+public class AvroGlueFormatOptions {
+    public static final String PREFIX = "schema-registry.";
+
+    public static final ConfigOption<String> AWS_REGION = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.AWS_REGION)
+            .stringType().noDefaultValue().withDescription("AWS region");
+
+    public static final ConfigOption<String> REGISTRY_NAME = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.REGISTRY_NAME)
+            .stringType().noDefaultValue().withDescription("Registry name");
+
+    public static final ConfigOption<String> RECORD_TYPE = ConfigOptions
+            .key(PREFIX + AWSSchemaRegistryConstants.AVRO_RECORD_TYPE).stringType()
+            .defaultValue(AvroRecordType.GENERIC_RECORD.getName()).withDescription("Record type");
+
+    public static final ConfigOption<String> SCHEMA_REGISTRY_SUBJECT = ConfigOptions.key(PREFIX + "subject").stringType()

Review comment:
       done

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [subject](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.subject`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.subject' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td><h5>format</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what format to use, here should be <code>'avro-glue'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.region</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what AWS region to use, such as <code>'us-east-1'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.registry.name</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The name (not the ARN) of the Glue schema registry in which to store the schemas.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.subject</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The subject name under which to store the schema in the registry.</td>

Review comment:
       done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718067814



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro Schema Registry

Review comment:
       fixed

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro

Review comment:
       fixed

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format

Review comment:
       fixed

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));

Review comment:
       fixed

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       reworded

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/AvroGlueFormatOptions.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+@PublicEvolving
+public class AvroGlueFormatOptions {
+    public static final String PREFIX = "schema-registry.";
+
+    public static final ConfigOption<String> AWS_REGION = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.AWS_REGION)
+            .stringType().noDefaultValue().withDescription("AWS region");
+
+    public static final ConfigOption<String> REGISTRY_NAME = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.REGISTRY_NAME)
+            .stringType().noDefaultValue().withDescription("Registry name");
+
+    public static final ConfigOption<String> RECORD_TYPE = ConfigOptions
+            .key(PREFIX + AWSSchemaRegistryConstants.AVRO_RECORD_TYPE).stringType()
+            .defaultValue(AvroRecordType.GENERIC_RECORD.getName()).withDescription("Record type");
+
+    public static final ConfigOption<String> SCHEMA_REGISTRY_SUBJECT = ConfigOptions.key(PREFIX + "subject").stringType()

Review comment:
       done

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [subject](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.subject`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.subject' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td><h5>format</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what format to use, here should be <code>'avro-glue'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.region</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what AWS region to use, such as <code>'us-east-1'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.registry.name</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The name (not the ARN) of the Glue schema registry in which to store the schemas.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.subject</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The subject name under which to store the schema in the registry.</td>

Review comment:
       done

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       This section is replicated from the closest analogue I could find, that of the avro-confluent.md.  Since the schema registry backend is mostly orthogonal to the connector type I assumed whatever restrictions applied to one applied to the other.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico removed a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico removed a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927595250


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719845596



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.schema-name' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).

Review comment:
       I'm doing another pass on the naming of the options and will have it in the next commit.  Should be more inline with other formats, and as you suggest I'm doing the translation inside the factory.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716419119



##########
File path: flink-formats/flink-avro-glue-schema-registry/pom.xml
##########
@@ -178,5 +192,33 @@ under the License.
 			<version>${glue.schema.registry.version}</version>
 		</dependency>
 
+		<!-- test dependencies -->
+		<dependency>
+			<groupId>org.apache.flink</groupId>
+			<artifactId>flink-table-api-java</artifactId>
+			<version>${project.version}</version>
+			<scope>test</scope>
+			<type>test-jar</type>
+		</dependency>
+		<dependency>
+			<groupId>org.apache.flink</groupId>
+			<artifactId>flink-table-runtime-blink_${scala.binary.version}</artifactId>

Review comment:
       didn't realize I'd been working off a fairly old checkout and based that on some outdated code from a sibling project.  Fixed.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico removed a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico removed a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927595250


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-929009820


   We're receiving a lot of community feedback that new features are not or not properly documented. It would be really good if we can get the documentation in with this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * d364ab24be51b23c260be42ddac97690384279ac Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520) 
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927595250


   @flinkbot run azure


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] Airblader commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
Airblader commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718326427



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       The format can define a changelog, but it'll be up to the connector to defer to the format for providing the changelog mode. `SocketDynamicTableSource` in `flink-examples` would be an example where the connector just defers this to the format.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718764921



##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: AWS Glue Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-glue.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# AWS Glue Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [value](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.schema-name`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).

Review comment:
       This section is replicated from the closest analogue I could find, that of the avro-confluent.md.  Since the schema registry backend is mostly orthogonal to the connector type I assumed whatever restrictions applied to one applied to the other.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 70568032b21299c96c12505b8e7a413370318ae6 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603) 
   * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-928976369


   Can you please add an e2e test for the Table API support?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718068149



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-929000972


   We will also need a documentation update similar to this:
   - https://nightlies.apache.org/flink/flink-docs-master/docs/connectors/table/formats/avro-confluent/
   
   If you do not have capacity to contribute this can you please raise a follow up Jira and link to this one


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r720134979



##########
File path: flink-formats/flink-avro-glue-schema-registry/pom.xml
##########
@@ -94,6 +94,19 @@ under the License.
 			<groupId>software.amazon.glue</groupId>
 			<artifactId>schema-registry-serde</artifactId>
 			<version>${glue.schema.registry.version}</version>
+			<exclusions>
+				<exclusion>
+					<groupId>com.kjetland</groupId>
+					<artifactId>mbknor-jackson-jsonschema_2.12</artifactId>
+				</exclusion>
+			</exclusions>
+		</dependency>
+
+		<!-- The above dependency hard-codes the Scala 2.12 binary version of this jar, so we replace it -->
+		<dependency>
+			<groupId>com.kjetland</groupId>
+			<artifactId>mbknor-jackson-jsonschema_${scala.binary.version}</artifactId>

Review comment:
       @jherico Does this change mean that the format will be compatible with Scala 2.11 also? Although 2.11 has been recently removed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523) 
   * 1e11514b73a035cad03c582c00e5a156ec7ab115 UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717764096



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroSchemaCoderProvider.java
##########
@@ -50,4 +51,29 @@ public GlueSchemaRegistryAvroSchemaCoderProvider(Map<String, Object> configs) {
     public GlueSchemaRegistryAvroSchemaCoder get() {
         return new GlueSchemaRegistryAvroSchemaCoder(transportName, configs);
     }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass()) {
+            return false;
+        }
+        GlueSchemaRegistryAvroSchemaCoderProvider that = (GlueSchemaRegistryAvroSchemaCoderProvider) o;
+        if (transportName == null)  {
+            if (that.transportName != null) {
+                return false;
+            }
+        } else if (!transportName.equals(that.transportName)) {
+            return false;
+        }
+        return configs.equals(that.configs);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(transportName, configs);
+    }
+

Review comment:
       The test code relies on the existing implementations of [RegistryAvroSerializationSchema.equals](https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroSerializationSchema.java#L115) and [RegistryAvroDeserializationSchema.equals](https://github.com/apache/flink/blob/master/flink-formats/flink-avro/src/main/java/org/apache/flink/formats/avro/RegistryAvroDeserializationSchema.java#L93), both of which then rely on their respective `schemaCoderProvider` having a valid `equals` implementation.  Although my only use case is testing, removing this function would break the equals contract for those containing classes.  
   
   Because the class is effectively immutable, I don't believe it contravenes the guidelines.  




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716423444



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        var configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        var properties = new HashMap<String, Object>();

Review comment:
       fixed

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroOptions.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+public class GlueSchemaRegistryAvroOptions {
+    private GlueSchemaRegistryAvroOptions() {

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 1e11514b73a035cad03c582c00e5a156ec7ab115 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     } ]
   }-->
   ## CI report:
   
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716424797



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroOptions.java
##########
@@ -0,0 +1,74 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+public class GlueSchemaRegistryAvroOptions {

Review comment:
       fixed




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r716423291



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GluSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.GlueSchemaRegistryAvroOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+public class GluSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        var configMap = buildConfigMap(formatOptions);

Review comment:
       fixed.  Will also remove `Set.of(...)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] dannycranmer commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
dannycranmer commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r717342648



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       Same question as above

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       @hhkkxxx133 can you please double check these are the only required field?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroSchemaCoderProvider.java
##########
@@ -50,4 +51,29 @@ public GlueSchemaRegistryAvroSchemaCoderProvider(Map<String, Object> configs) {
     public GlueSchemaRegistryAvroSchemaCoder get() {
         return new GlueSchemaRegistryAvroSchemaCoder(transportName, configs);
     }
+
+    @Override
+    public boolean equals(Object o) {
+        if (this == o) {
+            return true;
+        }
+        if (o == null || getClass() != o.getClass()) {
+            return false;
+        }
+        GlueSchemaRegistryAvroSchemaCoderProvider that = (GlueSchemaRegistryAvroSchemaCoderProvider) o;
+        if (transportName == null)  {
+            if (that.transportName != null) {
+                return false;
+            }
+        } else if (!transportName.equals(that.transportName)) {
+            return false;
+        }
+        return configs.equals(that.configs);
+    }
+
+    @Override
+    public int hashCode() {
+        return Objects.hash(transportName, configs);
+    }
+

Review comment:
       Please confirm why you added these, and that it is acceptable per the coding guidelines:
   - https://flink.apache.org/contributing/code-style-and-quality-java.html#equals--hashcode

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       What does this mode refer to? Does this impact the type of connector? For instance, Kafka which supports upsert, and kinesis which does not (insert only)?
   




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) 
   * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688",
       "triggerID" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 72a98b6c81ca3972146961a485a37c48cbe2033e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635) 
   * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] jherico commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
jherico commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r719847389



##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();

Review comment:
       So, I did a test of an actual flink job last night and discovered that unless this is set to `insertOnly` you can't create an upsert-kafka connector using it.  So I clearly don't understand what this changelog mode refers to, but I'm just going to cope.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "PENDING",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 9cac814b7a966b956bb3f8edc883b526e89c3cf9 Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523) 
   * 1e11514b73a035cad03c582c00e5a156ec7ab115 Azure: [PENDING](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "",
       "status" : "CANCELED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "UNKNOWN",
       "url" : "TBD",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   *  Unknown: [CANCELED](TBD) 
   * d364ab24be51b23c260be42ddac97690384279ac UNKNOWN
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run travis` re-run the last Travis build
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] MartijnVisser commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
MartijnVisser commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-929009820


   We're receiving a lot of community feedback that new features are not or not properly documented. It would be really good if we can get the documentation in with this PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot commented on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot commented on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927534884






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] hhkkxxx133 commented on a change in pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
hhkkxxx133 commented on a change in pull request #17360:
URL: https://github.com/apache/flink/pull/17360#discussion_r718011735



##########
File path: docs/data/sql_connectors.yml
##########
@@ -48,6 +48,12 @@ avro-confluent:
     category: format
     sql_url: https://repo.maven.apache.org/maven2/org/apache/flink/flink-sql-avro-confluent-registry/$version/flink-sql-avro-confluent-registry-$version.jar
 
+avro-glue:
+    name: Avro Schema Registry

Review comment:
       Can the name be distinguished from Confluent? For example Avro Glue Schema Registry?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.all();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));

Review comment:
       Duplicate?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/GlueSchemaRegistryAvroFormatFactory.java
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AUTO_REGISTRATION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.AWS_REGION;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_SIZE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.CACHE_TTL_MS;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPATIBILITY;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.COMPRESSION_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.ENDPOINT;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.RECORD_TYPE;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.REGISTRY_NAME;
+import static org.apache.flink.formats.avro.glue.schema.registry.AvroGlueFormatOptions.SCHEMA_REGISTRY_SUBJECT;
+
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+import java.util.Set;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+
+import org.apache.flink.annotation.Internal;
+import org.apache.flink.api.common.serialization.DeserializationSchema;
+import org.apache.flink.api.common.serialization.SerializationSchema;
+import org.apache.flink.api.common.typeinfo.TypeInformation;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ReadableConfig;
+import org.apache.flink.formats.avro.AvroRowDataDeserializationSchema;
+import org.apache.flink.formats.avro.AvroRowDataSerializationSchema;
+import org.apache.flink.formats.avro.AvroToRowDataConverters;
+import org.apache.flink.formats.avro.RowDataToAvroConverters;
+import org.apache.flink.formats.avro.typeutils.AvroSchemaConverter;
+import org.apache.flink.table.connector.ChangelogMode;
+import org.apache.flink.table.connector.format.DecodingFormat;
+import org.apache.flink.table.connector.format.EncodingFormat;
+import org.apache.flink.table.connector.sink.DynamicTableSink;
+import org.apache.flink.table.connector.source.DynamicTableSource;
+import org.apache.flink.table.data.RowData;
+import org.apache.flink.table.factories.DeserializationFormatFactory;
+import org.apache.flink.table.factories.DynamicTableFactory;
+import org.apache.flink.table.factories.FactoryUtil;
+import org.apache.flink.table.factories.SerializationFormatFactory;
+import org.apache.flink.table.types.DataType;
+import org.apache.flink.table.types.logical.RowType;
+
+/**
+ * Table format factory for providing configured instances of AWS Glue Schema
+ * Registry Avro to RowData {@link SerializationSchema} and
+ * {@link DeserializationSchema}.
+ */
+@Internal
+public class GlueSchemaRegistryAvroFormatFactory implements DeserializationFormatFactory, SerializationFormatFactory {
+    public static final String IDENTIFIER = "avro-glue";
+
+    @Override
+    public DecodingFormat<DeserializationSchema<RowData>> createDecodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+        final Map<String, Object> configMap = buildConfigMap(formatOptions);
+
+        return new DecodingFormat<DeserializationSchema<RowData>>() {
+            @Override
+            public DeserializationSchema<RowData> createRuntimeDecoder(DynamicTableSource.Context context,
+                    DataType producedDataType) {
+                final RowType rowType = (RowType) producedDataType.getLogicalType();
+                final TypeInformation<RowData> rowDataTypeInfo = context.createTypeInformation(producedDataType);
+                return new AvroRowDataDeserializationSchema(
+                        GlueSchemaRegistryAvroDeserializationSchema
+                                .forGeneric(AvroSchemaConverter.convertToSchema(rowType), configMap),
+                        AvroToRowDataConverters.createRowConverter(rowType), rowDataTypeInfo);
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public EncodingFormat<SerializationSchema<RowData>> createEncodingFormat(DynamicTableFactory.Context context,
+            ReadableConfig formatOptions) {
+        FactoryUtil.validateFactoryOptions(this, formatOptions);
+
+        return new EncodingFormat<SerializationSchema<RowData>>() {
+            @Override
+            public SerializationSchema<RowData> createRuntimeEncoder(DynamicTableSink.Context context,
+                    DataType consumedDataType) {
+                final RowType rowType = (RowType) consumedDataType.getLogicalType();
+                return new AvroRowDataSerializationSchema(rowType,
+                        GlueSchemaRegistryAvroSerializationSchema.forGeneric(
+                                AvroSchemaConverter.convertToSchema(rowType),
+                                formatOptions.get(SCHEMA_REGISTRY_SUBJECT), buildConfigMap(formatOptions)),
+                        RowDataToAvroConverters.createConverter(rowType));
+            }
+
+            @Override
+            public ChangelogMode getChangelogMode() {
+                return ChangelogMode.upsert();
+            }
+        };
+    }
+
+    @Override
+    public String factoryIdentifier() {
+        return IDENTIFIER;
+    }
+
+    private Map<String, Object> buildConfigMap(ReadableConfig formatOptions) {
+        final Map<String, Object> properties = new HashMap<String, Object>();
+        formatOptions.getOptional(AWS_REGION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_REGION, v));
+        formatOptions.getOptional(REGISTRY_NAME).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.REGISTRY_NAME, v));
+        formatOptions.getOptional(RECORD_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AVRO_RECORD_TYPE, v));
+        formatOptions.getOptional(COMPRESSION_TYPE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPRESSION_TYPE, v));
+        formatOptions.getOptional(ENDPOINT).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.AWS_ENDPOINT, v));
+        formatOptions.getOptional(COMPATIBILITY).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.COMPATIBILITY_SETTING, v));
+        formatOptions.getOptional(AUTO_REGISTRATION).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.SCHEMA_AUTO_REGISTRATION_SETTING, v));
+        formatOptions.getOptional(CACHE_SIZE).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_SIZE, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        formatOptions.getOptional(CACHE_TTL_MS).ifPresent(v -> properties.put(AWSSchemaRegistryConstants.CACHE_TIME_TO_LIVE_MILLIS, v));
+        return properties;
+    }
+
+    @Override
+    public Set<ConfigOption<?>> requiredOptions() {
+        Set<ConfigOption<?>> result = new HashSet<>();
+        result.add(REGISTRY_NAME);
+        result.add(AWS_REGION);
+        result.add(SCHEMA_REGISTRY_SUBJECT);
+        return result;
+    }

Review comment:
       I agree. Looks good to me.

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format

Review comment:
       Should be updated as well to indicate Glue Schema Registry.

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro

Review comment:
       Title is incorrectly referring to "Confluent".

##########
File path: docs/content/docs/connectors/table/formats/avro-glue.md
##########
@@ -0,0 +1,191 @@
+---
+title: Confluent Avro
+weight: 4
+type: docs
+aliases:
+  - /dev/table/connectors/formats/avro-confluent.html
+---
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements.  See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership.  The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License.  You may obtain a copy of the License at
+
+  http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied.  See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Confluent Avro Format
+
+{{< label "Format: Serialization Schema" >}}
+{{< label "Format: Deserialization Schema" >}}
+
+The Glue Schema Registry (``avro-glue``) format allows you to read records that were serialized by the ``com.amazonaws.services.schemaregistry.serializers.avro.AWSKafkaAvroSerializer`` and to write records that can in turn be read by the ``com.amazonaws.services.schemaregistry.deserializers.avro.AWSKafkaAvroDeserializer``.  These records have their schemas stored out-of-band in a configured registry provided by the AWS Glue Schema Registry [service](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas).
+
+When reading (deserializing) a record with this format the Avro writer schema is fetched from the configured AWS Glue Schema Registry based on the schema version id encoded in the record while the reader schema is inferred from table schema. 
+
+When writing (serializing) a record with this format the Avro schema is inferred from the table schema and used to retrieve a schema id to be encoded with the data. The lookup is performed against the configured AWS Glue Schema Registry under the [subject](https://docs.aws.amazon.com/glue/latest/dg/schema-registry.html#schema-registry-schemas) given in `avro-glue.subject`.
+
+The Avro Glue Schema Registry format can only be used in conjunction with the [Apache Kafka SQL connector]({{< ref "docs/connectors/table/kafka" >}}) or the [Upsert Kafka SQL Connector]({{< ref "docs/connectors/table/upsert-kafka" >}}).
+
+Dependencies
+------------
+
+{{< sql_download_table "avro-glue" >}}
+
+How to create tables with Avro-Glue format
+--------------
+
+Example of a table using raw UTF-8 string as Kafka key and Avro records registered in the Schema Registry as Kafka values:
+
+```sql
+CREATE TABLE user_created (
+
+  -- one column mapped to the Kafka raw UTF-8 key
+  the_kafka_key STRING,
+  
+  -- a few columns mapped to the Avro fields of the Kafka value
+  id STRING,
+  name STRING, 
+  email STRING
+
+) WITH (
+
+  'connector' = 'kafka',
+  'topic' = 'user_events_example1',
+  'properties.bootstrap.servers' = 'localhost:9092',
+
+  -- UTF-8 string as Kafka keys, using the 'the_kafka_key' table column
+  'key.format' = 'raw',
+  'key.fields' = 'the_kafka_key',
+
+  'value.format' = 'avro-glue',
+  'value.avro-glue.region' = 'us-east-1',
+  'value.avro-glue.registry.name' = 'my-schema-registry',
+  'value.avro-glue.subject' = 'my-schema-name',
+  'value.fields-include' = 'EXCEPT_KEY'
+)
+```
+
+Format Options
+----------------
+
+Yes, these options have inconsistent naming convnetions.  No, I can't fix it.  This is for consistentcy with the existing [AWS Glue client code](https://github.com/awslabs/aws-glue-schema-registry/blob/master/common/src/main/java/com/amazonaws/services/schemaregistry/utils/AWSSchemaRegistryConstants.java#L20).
+
+<table class="table table-bordered">
+    <thead>
+      <tr>
+        <th class="text-left" style="width: 25%">Option</th>
+        <th class="text-center" style="width: 8%">Required</th>
+        <th class="text-center" style="width: 7%">Default</th>
+        <th class="text-center" style="width: 10%">Type</th>
+        <th class="text-center" style="width: 50%">Description</th>
+      </tr>
+    </thead>
+    <tbody>
+        <tr>
+            <td><h5>format</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what format to use, here should be <code>'avro-glue'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.region</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>Specify what AWS region to use, such as <code>'us-east-1'</code>.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.registry.name</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The name (not the ARN) of the Glue schema registry in which to store the schemas.</td>
+        </tr>
+        <tr>
+            <td><h5>avro-glue.subject</h5></td>
+            <td>required</td>
+            <td style="word-wrap: break-word;">(none)</td>
+            <td>String</td>
+            <td>The subject name under which to store the schema in the registry.</td>

Review comment:
       Can this be named to `avro-glue.schema.name`?

##########
File path: flink-formats/flink-avro-glue-schema-registry/src/main/java/org/apache/flink/formats/avro/glue/schema/registry/AvroGlueFormatOptions.java
##########
@@ -0,0 +1,76 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements.  See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership.  The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License.  You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+ package org.apache.flink.formats.avro.glue.schema.registry;
+
+import java.time.Duration;
+
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants;
+import com.amazonaws.services.schemaregistry.utils.AWSSchemaRegistryConstants.COMPRESSION;
+import com.amazonaws.services.schemaregistry.utils.AvroRecordType;
+
+import org.apache.flink.annotation.PublicEvolving;
+import org.apache.flink.configuration.ConfigOption;
+import org.apache.flink.configuration.ConfigOptions;
+
+import software.amazon.awssdk.services.glue.model.Compatibility;
+
+@PublicEvolving
+public class AvroGlueFormatOptions {
+    public static final String PREFIX = "schema-registry.";
+
+    public static final ConfigOption<String> AWS_REGION = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.AWS_REGION)
+            .stringType().noDefaultValue().withDescription("AWS region");
+
+    public static final ConfigOption<String> REGISTRY_NAME = ConfigOptions.key(PREFIX + AWSSchemaRegistryConstants.REGISTRY_NAME)
+            .stringType().noDefaultValue().withDescription("Registry name");
+
+    public static final ConfigOption<String> RECORD_TYPE = ConfigOptions
+            .key(PREFIX + AWSSchemaRegistryConstants.AVRO_RECORD_TYPE).stringType()
+            .defaultValue(AvroRecordType.GENERIC_RECORD.getName()).withDescription("Record type");
+
+    public static final ConfigOption<String> SCHEMA_REGISTRY_SUBJECT = ConfigOptions.key(PREFIX + "subject").stringType()

Review comment:
       Same as above: can this be named to `SCHEMA_NAME`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [flink] flinkbot edited a comment on pull request #17360: [FLINK-24379][Formats] Add support for Glue schema registry in Table API

Posted by GitBox <gi...@apache.org>.
flinkbot edited a comment on pull request #17360:
URL: https://github.com/apache/flink/pull/17360#issuecomment-927544424


   <!--
   Meta data
   {
     "version" : 1,
     "metaDataEntries" : [ {
       "hash" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24518",
       "triggerID" : "6c5edf94116d5e9cbea62d94719c32110e87fa0d",
       "triggerType" : "PUSH"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "d364ab24be51b23c260be42ddac97690384279ac",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24520",
       "triggerID" : "d364ab24be51b23c260be42ddac97690384279ac",
       "triggerType" : "PUSH"
     }, {
       "hash" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24523",
       "triggerID" : "9cac814b7a966b956bb3f8edc883b526e89c3cf9",
       "triggerType" : "PUSH"
     }, {
       "hash" : "",
       "status" : "DELETED",
       "url" : "TBD",
       "triggerID" : "927595250",
       "triggerType" : "MANUAL"
     }, {
       "hash" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24594",
       "triggerID" : "1e11514b73a035cad03c582c00e5a156ec7ab115",
       "triggerType" : "PUSH"
     }, {
       "hash" : "70568032b21299c96c12505b8e7a413370318ae6",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24603",
       "triggerID" : "70568032b21299c96c12505b8e7a413370318ae6",
       "triggerType" : "PUSH"
     }, {
       "hash" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "status" : "DELETED",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24635",
       "triggerID" : "72a98b6c81ca3972146961a485a37c48cbe2033e",
       "triggerType" : "PUSH"
     }, {
       "hash" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "status" : "SUCCESS",
       "url" : "https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688",
       "triggerID" : "3bbe36d1af9b8830e4a8605dced3b1442c53d64e",
       "triggerType" : "PUSH"
     } ]
   }-->
   ## CI report:
   
   * 3bbe36d1af9b8830e4a8605dced3b1442c53d64e Azure: [SUCCESS](https://dev.azure.com/apache-flink/98463496-1af2-4620-8eab-a2ecc1a2e6fe/_build/results?buildId=24688) 
   
   <details>
   <summary>Bot commands</summary>
     The @flinkbot bot supports the following commands:
   
    - `@flinkbot run azure` re-run the last Azure build
   </details>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@flink.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org