You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 03:45:27 UTC

[I] Follow-up items for timeline-server-based marker files [hudi]

hudi-bot opened a new issue, #14832:
URL: https://github.com/apache/hudi/issues/14832

   Follow-up items after HUDI-1138:
    * Implement upgrade/downgrade, rollback regarding markers
    ** Rollback should be based on the markers in the file system, with marker type indicated by MARKERS.type, or assumed to be "direct", if no MARKERS.type
    ** New write follows the `hoodie.write.markers.type` config to create markers
    * Verify multiple concurrent writers with timeline-server-based markers
    * Let's ensure the parallelism passed for local engine context is reasonable
    * Fix `Task not serializable` due to new APIs in FSUtils for recursive, level by level listing (`java.io.NotSerializableException: org.apache.hudi.common.fs.FSUtils$$Lambda$4224/1845791682`)
    * Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
    * Optimize future timeouts due to I/O failure in the marker handler
    * Test new marker strategy for all engines (spark, flink and java) w/o any additional changes
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-2271
   - Type: Improvement


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow-up items for timeline-server-based marker files [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #14832: Follow-up items for timeline-server-based marker files
URL: https://github.com/apache/hudi/issues/14832


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Follow-up items for timeline-server-based marker files [hudi]

Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #14832:
URL: https://github.com/apache/hudi/issues/14832#issuecomment-4795268547

   This follow-up tracking ticket from RFC-15 / HUDI-1138 appears fully addressed in current master. The individual sub-items have been completed by subsequent PRs:
   
   - **Upgrade/downgrade & MARKERS.type rollback**: HUDI-2305 (`9056c68744a3` — Add MARKERS.type and fix marker-based rollback) and HUDI-1014 (`ff53e8f0b624` — upgrade/downgrade infra).
   - **Multi-writer with timeline-server-based markers**: HUDI-1575 (`c18d6153e105` — Early Conflict Detection).
   - **Parallelism for local engine context**: HUDI-9511 (`bb3f07965152` — Use local engine context for timeline server markers).
   - **`fs.exists` optimization in DirectMarkerFiles**: `DirectWriteMarkers.create()` now only calls `storage.exists()` when `checkIfExists` is set (see `DirectWriteMarkers.java:190`).
   - **Future timeouts on I/O failure in marker handler**: HUDI-8988 (`ffa4ca6f7130` — retry logic for HTTP calls including markers).
   - **Test new marker strategy for all engines**: #14202 (`53b33839fb5d` — TIMELINE_SERVER_BASED markers for Flink writer); Java/Spark already covered.
   - **Task not serializable in FSUtils recursive listing**: `FSUtils` recursive-listing helpers have been refactored; no current report of this NotSerializableException.
   
   Suggest closing this issue. If any specific sub-item still reproduces, please open a focused issue with a current repro on master.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org