[GitHub] [beam] apilloud commented on a change in pull request #14759: [WIP] Integration with Data Catalog physical schema API to enable Avro/Protobuf format in PubSub

[GitBox <gi...@apache.org>]

apilloud commented on a change in pull request #14759:
URL: https://github.com/apache/beam/pull/14759#discussion_r628420134



##########
File path: sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtils.java
##########
@@ -58,13 +60,66 @@
 
   /** Convert DataCatalog schema to Beam schema. */
   static Schema fromDataCatalog(com.google.cloud.datacatalog.v1beta1.Schema dcSchema) {
-    return fromColumnsList(dcSchema.getColumnsList());
+    if (dcSchema.hasPhysicalSchema()) {
+      return fromPhysicalSchema(dcSchema.getPhysicalSchema());
+    } else {
+      return fromColumnsList(dcSchema.getColumnsList());
+    }
   }
 
   private static Schema fromColumnsList(List<ColumnSchema> columnsMap) {
     return columnsMap.stream().map(SchemaUtils::toBeamField).collect(toSchema());
   }
 
+  private static Schema fromPhysicalSchema(PhysicalSchema physicalSchema) {
+    if (physicalSchema.hasAvro()) {
+      return fromAvroSchema(new Parser().parse(physicalSchema.getAvro().getText()));
+    } else if (physicalSchema.hasProtobuf()) {
+      return null; // TODO
+    } else {
+      throw new UnsupportedOperationException(
+          "Unsupported Data Catalog physical schema format. "
+              + "Currently we only support Avro and Protobuf.");
+    }
+  }
+
+  private static Schema fromAvroSchema(org.apache.avro.Schema avroSchema) {

Review comment:
       Why not use org.apache.beam.sdk.schemas.utils.AvroUtils.toBeamSchema()?
   https://github.com/apache/beam/blob/243128a8fc52798e1b58b0cf1a271d95ee7aa241/sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/utils/AvroUtils.java#L318




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org