[GitHub] [hudi] kazdy commented on pull request #8047: [HUDI-5272][SPARK] No PreCombineField mode - make COMBINE_BEFORE_UPSERT=false automatically

["kazdy (via GitHub)" <gi...@apache.org>]

kazdy commented on PR #8047:
URL: https://github.com/apache/hudi/pull/8047#issuecomment-1445377207

   Found out that I can't rely on HoodieWriteConfig as I need to check if user provided precombine filed. Rather should use HoodieTableConfig in HoodieSparkSqlWriter.
   
   I also found out that if user do not provide this config explicitly, it's not persisted in hoodie.properties but if "ts" field exists (which is optional but also default for precombine field) it uses it as if it was defined (still not persisted).
   A lot of tests run with assumption that "ts" is precombine field.
   
   A few usecases to consider:
   - no precombine field provided by the user + no "ts" (defaultValue) for precombine in the schema -> no precombine mode,
   - no precombine field provided by the user  + "ts" for precombine is available in the schema -> then what to do?, what if user does not want to use "ts" as precombine field?
   - precombine field provided by he user -> save to properties and use
   - precombine field not provided by the user and no "ts" in schema -> do not use "ts" to combine records
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org