[jira] [Created] (HUDI-6028) GCS incr source does not handle pubsub message properly

["Raymond Xu (Jira)" <ji...@apache.org>]

Raymond Xu created HUDI-6028:
--------------------------------

             Summary: GCS incr source does not handle pubsub message properly
                 Key: HUDI-6028
                 URL: https://issues.apache.org/jira/browse/HUDI-6028
             Project: Apache Hudi
          Issue Type: Bug
          Components: deltastreamer
            Reporter: Raymond Xu


Gcs event source uses schema converter from spark and won't handle field name with hyphen in nested column. a sample message


{code:java}
23/04/03 19:23:45 DEBUG GcsEventsSource: msg: {
  "kind": "storage#object",
  "id": "",
  "selfLink": "",
  "name": "",
  "bucket": "",
  "generation": "1680505551370137",
  "metageneration": "1",
  "contentType": "application/octet-stream",
  "timeCreated": "2023-04-03T07:05:51.373Z",
  "updated": "2023-04-03T07:05:51.373Z",
  "storageClass": "STANDARD",
  "timeStorageClassUpdated": "2023-04-03T07:05:51.373Z",
  "size": "6707",
  "md5Hash": "",
  "mediaLink": "",
  "metadata": {
    "goog-reserved-file-mtime": "1680503048"
  },
  "crc32c": "",
  "etag": ""
}
{code}

and it throws

{code}
Exception in thread "main" org.apache.avro.SchemaParseException: Illegal character in: goog-reserved-file-mtime
	at org.apache.avro.Schema.validateName(Schema.java:1571)
	at org.apache.avro.Schema.access$400(Schema.java:92)
	at org.apache.avro.Schema$Field.<init>(Schema.java:549)
	at org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2258)
	at org.apache.avro.SchemaBuilder$FieldBuilder.completeField(SchemaBuilder.java:2254)
	at org.apache.avro.SchemaBuilder$FieldBuilder.access$5100(SchemaBuilder.java:2150)
	at org.apache.avro.SchemaBuilder$GenericDefault.noDefault(SchemaBuilder.java:2557)
	at org.apache.hudi.org.apache.spark.sql.avro.SchemaConverters$.$anonfun$toAvroType$2(SchemaConverters.scala:205)
{code}


This is a problem with org.apache.spark.sql.avro.SchemaConverters#toAvroType



--
This message was sent by Atlassian Jira
(v8.20.10#820010)