[jira] [Created] (HUDI-2719) Add back support for copying over extra metadata from previous commit

["sivabalan narayanan (Jira)" <ji...@apache.org>]

sivabalan narayanan created HUDI-2719:
-----------------------------------------

             Summary: Add back support for copying over extra metadata from previous commit 
                 Key: HUDI-2719
                 URL: https://issues.apache.org/jira/browse/HUDI-2719
             Project: Apache Hudi
          Issue Type: Improvement
            Reporter: sivabalan narayanan


Add back support for copying over extra metadata from previous commit .

We had a config named 'hoodie.write.meta.key.prefixes'. when this config is added, hudi copies over any extra metadata from previous commit with matching key prefixes as per `hoodie.write.meta.key.prefixes`. for eg: if `hoodie.write.meta.key.prefixes` set to `abc` and if there are other config options as "abc.key1" and "abc.key2", these two will be copied over from previous commit metadata and get overridden with current writer's commit metadata. 

 

Our initial understanding is that this was added to support copying over delta streamer checkpoint for multi -writer scenairo. But since the config is not very straight forward to use, we simplified and introduced a new config as per [https://github.com/apache/hudi/pull/3820.] 

Added the old one "hoodie.write.meta.key.prefixes" as an alternative. Since multi-writer in itself is still evolving, we felt simplifying the config for deltastreamer checkpoint makes sense. 

But if someone out there was using the older config to copy some other extra metadata for debugging or some other purposes, we need to add the support back. 

 

 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)