[I] Add back support for copying over extra metadata from previous commit [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #14911:
URL: https://github.com/apache/hudi/issues/14911

   Add back support for copying over extra metadata from previous commit .
   
   We had a config named 'hoodie.write.meta.key.prefixes'. when this config is added, hudi copies over any extra metadata from previous commit with matching key prefixes as per `hoodie.write.meta.key.prefixes`. for eg: if `hoodie.write.meta.key.prefixes` set to `abc` and if there are other config options as "abc.key1" and "abc.key2", these two will be copied over from previous commit metadata and get overridden with current writer's commit metadata. 
   
    
   
   Our initial understanding is that this was added to support copying over delta streamer checkpoint for multi -writer scenairo. But since the config is not very straight forward to use, we simplified and introduced a new config as per [https://github.com/apache/hudi/pull/3820.] 
   
   Added the old one "hoodie.write.meta.key.prefixes" as an alternative. Since multi-writer in itself is still evolving, we felt simplifying the config for deltastreamer checkpoint makes sense. 
   
   But if someone out there was using the older config to copy some other extra metadata for debugging or some other purposes, we need to add the support back. 
   
    
   
    
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-2719
   - Type: Improvement


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add back support for copying over extra metadata from previous commit [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan closed issue #14911: Add back support for copying over extra metadata from previous commit 
URL: https://github.com/apache/hudi/issues/14911


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Add back support for copying over extra metadata from previous commit [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #14911:
URL: https://github.com/apache/hudi/issues/14911#issuecomment-4795281246

   This appears to be addressed on master by [PR #18421](https://github.com/apache/hudi/pull/18421) (commit `5066fcc4`, *"feat: Adding rolling extra metadata support"*, since `1.2.0`).
   
   A new config `hoodie.write.rolling.metadata.keys` (with companion `hoodie.write.rolling.metadata.timeline.lookback.commits`) was added in `HoodieWriteConfig` to auto-carry-forward configured extra metadata keys to every new commit and clean instant. Implementation lives in `BaseHoodieClient#mergeRollingMetadata` (called from `BaseHoodieWriteClient#preCommit`, `BaseHoodieTableServiceClient`, and `CleanActionExecutor`), with walkback up to the configured number of recent instants and tests in `TestRollingMetadata` / `TestRollingMetadataSpark`.
   
   A few caveats vs. the original ask:
   - The new config uses **exact metadata keys** (comma-separated) rather than the old **prefix matching** of `hoodie.write.meta.key.prefixes`. If prefix-style matching is still needed, a follow-up may be warranted.
   - The old key `hoodie.write.meta.key.prefixes` is **not** wired up as a `withAlternatives(...)` for the new config — it remains referenced only in javadoc on `SqlSource` / `SqlFileBasedSource`. That doc should probably be updated to point at the new config.
   
   If the exact-key behavior is sufficient, this issue can likely be closed. Otherwise, please confirm whether prefix-matching is still required.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org