You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "sivabalan narayanan (Jira)" <ji...@apache.org> on 2022/06/02 21:33:00 UTC

[jira] [Closed] (HUDI-4181) Async indexer fails for simple index creation

     [ https://issues.apache.org/jira/browse/HUDI-4181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

sivabalan narayanan closed HUDI-4181.
-------------------------------------
    Resolution: Duplicate

> Async indexer fails for simple index creation
> ---------------------------------------------
>
>                 Key: HUDI-4181
>                 URL: https://issues.apache.org/jira/browse/HUDI-4181
>             Project: Apache Hudi
>          Issue Type: Bug
>          Components: metadata
>            Reporter: sivabalan narayanan
>            Priority: Major
>
> I tried to ran async indexing to build out col stats and ran into issues. Had to make following fix to make progress. but this needs proper fix. 
>  
> Local fix that worked for me.
> {code:java}
> diff --git a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
> index f5a96fb676..1e67020810 100644
> --- a/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
> +++ b/hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java
> @@ -955,7 +955,6 @@ public abstract class HoodieBackedTableMetadataWriter implements HoodieTableMeta
>      HoodieTableFileSystemView fsView = HoodieTableMetadataUtil.getFileSystemView(metadataMetaClient);
>      for (Map.Entry<MetadataPartitionType, HoodieData<HoodieRecord>> entry : partitionRecordsMap.entrySet()) {
>        final String partitionName = entry.getKey().getPartitionPath();
> -      final int fileGroupCount = entry.getKey().getFileGroupCount();
>        HoodieData<HoodieRecord> records = entry.getValue();
>  
>        List<FileSlice> fileSlices =
> @@ -965,9 +964,10 @@ public abstract class HoodieBackedTableMetadataWriter implements HoodieTableMeta
>          // so if there are no committed file slices, look for inflight slices
>          fileSlices = HoodieTableMetadataUtil.getPartitionLatestFileSlicesIncludingInflight(metadataMetaClient, Option.ofNullable(fsView), partitionName);
>        }
> -      ValidationUtils.checkArgument(fileSlices.size() == fileGroupCount,
> +      final int fileGroupCount = fileSlices.size();
> +      /*ValidationUtils.checkArgument(fileSlices.size() == fileGroupCount,
>            String.format("Invalid number of file groups for partition:%s, found=%d, required=%d",
> -              partitionName, fileSlices.size(), fileGroupCount));
> +              partitionName, fileSlices.size(), fileGroupCount));*/
>  
>        List<FileSlice> finalFileSlices = fileSlices;
>        HoodieData<HoodieRecord> rddSinglePartitionRecords = records.map(r -> { {code}



--
This message was sent by Atlassian Jira
(v8.20.7#820007)