[jira] [Commented] (HUDI-309) General Redesign of Archived Timeline for efficient scan and management

["Balaji Varadarajan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HUDI-309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16971747#comment-16971747 ] 

Balaji Varadarajan commented on HUDI-309:
-----------------------------------------

[https://github.com/apache/incubator-hudi/blob/23b303e4b17c5f7b603900ee5b0d2e6718118014/hudi-client/src/main/java/org/apache/hudi/HoodieWriteClient.java#L860]
{code:java}
    if (!table.getActiveTimeline().getCleanerTimeline().empty()) {
        logger.info("Cleaning up older rollback meta files");
        // Cleanup of older cleaner meta files
        // TODO - make the commit archival generic and archive rollback metadata         
        FSUtils.deleteOlderRollbackMetaFiles(fs, table.getMetaClient().getMetaPath(),          
                    table.getActiveTimeline().getRollbackTimeline().getInstants());
    }
{code}
 

As part of PR-942, the above code is removed as it is handled elsewhere. Just noting that we need to ensure cleaner commits are also handled correctly for archiving

> General Redesign of Archived Timeline for efficient scan and management
> -----------------------------------------------------------------------
>
>                 Key: HUDI-309
>                 URL: https://issues.apache.org/jira/browse/HUDI-309
>             Project: Apache Hudi (incubating)
>          Issue Type: New Feature
>          Components: Common Core
>            Reporter: Balaji Varadarajan
>            Assignee: Vinoth Chandar
>            Priority: Major
>             Fix For: 0.5.1
>
>         Attachments: Archive TImeline Notes by Vinoth 1.jpg, Archived Timeline Notes by Vinoth 2.jpg
>
>
> As designed by Vinoth:
> Goals
>  # Archived Metadata should be scannable in the same way as data
>  # Provides more safety by always serving committed data independent of timeframe when the corresponding commit action was tried. Currently, we implicitly assume a data file to be valid if its commit time is older than the earliest time in the active timeline. While this works ok, any inherent bugs in rollback could inadvertently expose a possibly duplicate file when its commit timestamp becomes older than that of any commits in the timeline.
>  # We had to deal with lot of corner cases because of the way we treat a "commit" as special after it gets archived. Examples also include Savepoint handling logic by cleaner.
>  # Small Files : For Cloud stores, archiving simply moves fils from one directory to another causing the archive folder to grow. We need a way to efficiently compact these files and at the same time be friendly to scans
> Design:
>  The basic file-group abstraction for managing file versions for data files can be extended to managing archived commit metadata. The idea is to use an optimal format (like HFile) for storing compacted version of <commitTime, Metadata> pairs. Every archiving run will read <commitTime, Metadata> pairs from active timeline and append to indexable log files. We will run periodic minor compactions to merge multiple log files to a compacted HFile storing metadata for a time-range. It should be also noted that we will partition by the action types (commit/clean).  This design would allow for the archived timeline to be queryable for determining whether a timeline is valid or not.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)