You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2024/10/18 00:40:00 UTC

[jira] [Commented] (HUDI-8361) Revisit every fileSystemView.getLatestFileSlices in metadata writer/util and confirm if getLatestMergedFileSlices should be called

    [ https://issues.apache.org/jira/browse/HUDI-8361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17890764#comment-17890764 ] 

sivabalan narayanan commented on HUDI-8361:
-------------------------------------------

1. Functional and Sec index might need to be fixed. HoodieBackedTableMetadataWriter. 

 
{code:java}
private List<Pair<String, FileSlice>> getPartitionFileSlicePairs() throws IOException {
  try (HoodieMetadataFileSystemView fsView = getMetadataView()) {
    // Collect the list of latest file slices present in each partition
    List<String> partitions = metadata.getAllPartitionPaths();
    fsView.loadAllPartitions();
    List<Pair<String, FileSlice>> partitionFileSlicePairs = new ArrayList<>();
    partitions.forEach(partition -> fsView.getLatestFileSlices(partition).forEach(fs -> partitionFileSlicePairs.add(Pair.of(partition, fs))));
    return partitionFileSlicePairs;
  }
} {code}

fsView.getLatestFileSlices need to be replaced w/ fsView.getLatestMergedFileSlices

2. RLI initialization needs to be fixed 

initializeRecordIndexPartition 

3. HoodieBackedTableMetadataWriter.getFunctionalIndexUpdates -> 

HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight() 

 

 

 

> Revisit every fileSystemView.getLatestFileSlices in metadata writer/util and confirm if getLatestMergedFileSlices should be called
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: HUDI-8361
>                 URL: https://issues.apache.org/jira/browse/HUDI-8361
>             Project: Apache Hudi
>          Issue Type: Task
>            Reporter: Sagar Sumit
>            Assignee: sivabalan narayanan
>            Priority: Blocker
>             Fix For: 1.0.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)