You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "Ethan Guo (Jira)" <ji...@apache.org> on 2021/08/20 15:45:00 UTC
[jira] [Assigned] (HUDI-2271) Follow-up items for
timeline-server-based marker files
[ https://issues.apache.org/jira/browse/HUDI-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ethan Guo reassigned HUDI-2271:
-------------------------------
Assignee: Ethan Guo
> Follow-up items for timeline-server-based marker files
> ------------------------------------------------------
>
> Key: HUDI-2271
> URL: https://issues.apache.org/jira/browse/HUDI-2271
> Project: Apache Hudi
> Issue Type: Improvement
> Reporter: Ethan Guo
> Assignee: Ethan Guo
> Priority: Major
>
> Follow-up items after HUDI-1138:
> * Implement upgrade/downgrade, rollback regarding markers
> ** Rollback should be based on the markers in the file system, with marker type indicated by MARKERS.type, or assumed to be "direct", if no MARKERS.type
> ** New write follows the `hoodie.write.markers.type` config to create markers
> * Verify multiple concurrent writers with timeline-server-based markers
> * Let's ensure the parallelism passed for local engine context is reasonable
> * Fix `Task not serializable` due to new APIs in FSUtils for recursive, level by level listing (`java.io.NotSerializableException: org.apache.hudi.common.fs.FSUtils$$Lambda$4224/1845791682`)
> * Optimize fs.exists(dirPath) call in marker creation in DirectMarkerFiles
> * Optimize future timeouts due to I/O failure in the marker handler
> * Test new marker strategy for all engines (spark, flink and java) w/o any additional changes
--
This message was sent by Atlassian Jira
(v8.3.4#803005)