[GitHub] [hudi] nsivabalan edited a comment on pull request #3820: [BUGFIX] Merge commit state from previous commit instead of current

[GitBox <gi...@apache.org>]

nsivabalan edited a comment on pull request #3820:
URL: https://github.com/apache/hudi/pull/3820#issuecomment-946024720


   thanks a lot for fixing this Dave. I would like to propose something here. 
   I am wondering why do we need to retrofit copying over delta streamer checkpoint into hoodie.write.meta.key.prefixes. to me, this new requirement is very simple and not really tied to `hoodie.write.meta.key.prefixes`. 
   
   Let me propose something and see how that looks like.
   Introduce a new config called `hoodie.copy.over.deltastreamer.checkpoints`. we can brainstorm on actual naming later. 
   When set to true,
   within TransactionUtils::overrideWithLatestCommitMetadata
   ```
      fetch value of "deltastreamer.checkpoint.key" from last committed transaction and copy to cur inflight commit extra metadata.
   ```
   This is very tight and not error prone. Users don't need to set two different config as below which is not very intuitive as to why they need to do this. 
   ```
   hoodie.write.meta.key.prefixes = 'deltastreamer.checkpoint.key'
   ```
   and optionally 
   ```
   deltastreamer.checkpoint.key =. ""
   ```
   
   All users have to do is, for all of their spark writers, they need to set `hoodie.copy.over.deltastreamer.checkpoints` to true. 
   
   welcome thoughts @n3nash @vinothchandar @davehagman 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org