[GitHub] [hudi] nsivabalan edited a comment on issue #2508: [SUPPORT] Error upserting bucketType UPDATE for partition

[GitBox <gi...@apache.org>]

nsivabalan edited a comment on issue #2508:
URL: https://github.com/apache/hudi/issues/2508#issuecomment-772655440


   I did ran some local tests w/ array column and things are fine. given below is the schema that I tested locally. Also, from the stack trace, don't think encoders are the issue. but I too don't have much idea as of now. need more time to triage.
   @n3nash / @bvaradar : Can you guys spot any issues based on information given buy author. 
   @rubenssote: I am sure you might have tested the schema and the stack trace also does not hint that. but anyways,  but wanted to confirm once. Did you ensure the schema is valid (and is backwards compatible ) ?  
   
   ```
   diagnostics: User class threw exception: java.lang.Exception: Error on Table: order, Error Message: org.apache.spark.SparkException: Job aborted due to stage failure: Task 7 in stage 28.0 failed 4 times, most recent failure: Lost task 7.3 in stage 28.0 (TID 530, ip-10-0-29-119.us-west-2.compute.internal, executor 5): org.apache.hudi.exception.HoodieUpsertException: Error upserting bucketType UPDATE for partition :7
   at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.handleUpsertPartition(BaseSparkCommitActionExecutor.java:279)
   at org.apache.hudi.table.action.commit.BaseSparkCommitActionExecutor.lambda$execute$ecf5068c$1(BaseSparkCommitActionExecutor.java:135)
   ```
   
   ```
   public static final StructType STRUCT_TYPE = new StructType(new StructField[] {
         new StructField(HoodieRecord.COMMIT_TIME_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.RECORD_KEY_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.PARTITION_PATH_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.FILENAME_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField("randomInt", DataTypes.IntegerType, false, Metadata.empty()),
         new StructField("randomLong", DataTypes.LongType, false, Metadata.empty()),
         new StructField("array_long", DataTypes.createArrayType(DataTypes.LongType), false, Metadata.empty())
     });public static final StructType STRUCT_TYPE = new StructType(new StructField[] {
         new StructField(HoodieRecord.COMMIT_TIME_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.COMMIT_SEQNO_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.RECORD_KEY_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.PARTITION_PATH_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField(HoodieRecord.FILENAME_METADATA_FIELD, DataTypes.StringType, false, Metadata.empty()),
         new StructField("randomInt", DataTypes.IntegerType, false, Metadata.empty()),
         new StructField("randomLong", DataTypes.LongType, false, Metadata.empty()),
         new StructField("array_long", DataTypes.createArrayType(DataTypes.LongType), false, Metadata.empty())
     });
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org