[GitHub] [beam] apilloud commented on a diff in pull request #23620: Handle Avro schema generation for logical data types in BigQueryAvroUtils

[GitBox <gi...@apache.org>]

apilloud commented on code in PR #23620:
URL: https://github.com/apache/beam/pull/23620#discussion_r1017037762


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryAvroUtils.java:
##########
@@ -448,4 +449,32 @@ private static Field convertField(TableFieldSchema bigQueryField) {
         bigQueryField.getDescription(),
         (Object) null /* Cast to avoid deprecated JsonNode constructor. */);
   }
+
+  private static Schema handleAvroLogicalTypes(TableFieldSchema bigQueryField, Type avroType) {
+    String bqType = bigQueryField.getType();
+    switch (bqType) {
+      case "NUMERIC":
+        // Default value based on
+        // https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types
+        int precision = Optional.ofNullable(bigQueryField.getPrecision()).orElse(38L).intValue();
+        int scale = Optional.ofNullable(bigQueryField.getScale()).orElse(9L).intValue();
+        return LogicalTypes.decimal(precision, scale).addToSchema(Schema.create(Type.BYTES));
+      case "BIGNUMERIC":
+        // Default value based on
+        // https://cloud.google.com/bigquery/docs/reference/standard-sql/data-types#decimal_types
+        int precisionBigNumeric =
+            Optional.ofNullable(bigQueryField.getPrecision()).orElse(77L).intValue();

Review Comment:
   The documentation you linked says `76.76` and the API docs say `1 <= precision - scale <= 38 and 0 <= scale <= 38` which means the max value for precision is `76`. I'm not sure what will work in practice it seems like `76` would potentially fail to read from BigQuery and `77` will fail to write to BigQuery. Can you provide any more info on why 77 was chosen? Should there be tests for this edge case?
   https://cloud.google.com/java/docs/reference/google-cloud-bigquerystorage/2.8.2/com.google.cloud.bigquery.storage.v1.TableFieldSchema.Builder#com_google_cloud_bigquery_storage_v1_TableFieldSchema_Builder_getPrecision__



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org