When append to Hudi, it wipe out all previous data and only add a few new data

["An, Hongguo (CORP)" <Ho...@ADP.com.INVALID>]

Hi:
I have code that write to hudi in append mode:

val hudiOptions = Map[String,String](
  HoodieWriteConfig.TABLE_NAME -> tableName,
  DataSourceWriteOptions.RECORDKEY_FIELD_OPT_KEY -> pk,
  DataSourceWriteOptions.PRECOMBINE_FIELD_OPT_KEY -> timestampKey,
  DataSourceWriteOptions.HIVE_SYNC_ENABLED_OPT_KEY -> (env != Options.LOCAL).toString,
  DataSourceWriteOptions.HIVE_TABLE_OPT_KEY -> tableName,
  DataSourceWriteOptions.HIVE_DATABASE_OPT_KEY -> db,
  DataSourceWriteOptions.HIVE_URL_OPT_KEY -> s"jdbc:hive2://${emsMaster(env)}:10000/;ssl=true"
)
df.write
  .format("org.apache.hudi")
  .options(hudiOptions)
.option(DataSourceWriteOptions.PARTITIONPATH_FIELD_OPT_KEY, partitionKey)
.option(DataSourceWriteOptions.HIVE_PARTITION_FIELDS_OPT_KEY, partitionKey)
.mode(SaveMode.Append)
.save(path)


The partition key is like 2021/05/03, the id and timestamps are unique.
The same code works for one table, but not for another

  1.  It only write a few records on the last day, all data in previous day got wipped out.
  2.  It has nested spark structure
  3.  it also failed to sync to hive (but no error reported). Eeven if I ran hive-sync tool, it will not work (no exception)

Is there anything else I need to take a look?

Thanks
Andrew


----------------------------------------------------------------------
This message and any attachments are intended only for the use of the addressee and may contain information that is privileged and confidential. If the reader of the message is not the intended recipient or an authorized representative of the intended recipient, you are hereby notified that any dissemination of this communication is strictly prohibited. If you have received this communication in error, notify the sender immediately by return email and delete the message and any attachments from your system.