You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "HuangZhenQiu (via GitHub)" <gi...@apache.org> on 2026/07/06 21:49:45 UTC
[I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
HuangZhenQiu opened a new issue, #19207:
URL: https://github.com/apache/hudi/issues/19207
### Bug Description
**What happened:**
When we evaluate the Flink upsert internally, we find that the GlobalRecordIndexPartitioner get numFileGroupsForRecordIndexPartition = 8. It is different with the totally number of file groups in the record index partition.
**What you expected:**
getNumFileGroupsForRecordIndexPartition in GlobalRecordIndexPartitioner should get the correct totally number of file groups correctly, rather than use the getPartitionFileSlices that returns Get the latest file slices for a given partition the in org.apache.hudi.metadata.HoodieTableMetadataUtil.
**Steps to reproduce:**
1. Bootstrap a new table from a COW original table
2. Setup an Flink upsert job in MOR mode
3. Check the getNumFileGroupsForRecordIndexPartition in GlobalRecordIndexPartitioner.
### Environment
**Hudi version:**
**Query engine:** (Spark/Flink/Trino etc)
**Relevant configs:**
### Logs and Stack Trace
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4929983015
It is actually AbstractTableFileSystemView.buildFileGroups read the file groups and stored in the partitionToFileGroupsMap.
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L222
I think we need to set hoodie.table.multiple.base.file.formats.enable to true if record index in MET is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4931203616
Yes, you are right. The MDT table contains these configs:
hoodie.compaction.payload.class=org.apache.hudi.metadata.HoodieMetadataPayload
hoodie.table.initial.version=9
hoodie.table.keygenerator.type=HOODIE_TABLE_METADATA
hoodie.table.type=MERGE_ON_READ
hoodie.archivelog.folder=history
hoodie.timeline.layout.version=2
hoodie.timeline.history.path=history
hoodie.table.checksum=3970692307
hoodie.datasource.write.drop.partition.columns=false
hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000
hoodie.table.recordkey.fields=key
hoodie.timeline.path=timeline
hoodie.populate.meta.fields=false
hoodie.table.base.file.format=HFILE
hoodie.record.merge.mode=CUSTOM
hoodie.table.version=9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4929730829
IncrementalTimelineSyncFileSystemView.applyDeltaFileSlicesToPartitionView find the file groups and stored into storePartitionView.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu closed issue #19207: Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner
URL: https://github.com/apache/hudi/issues/19207
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4900190503
Agree. I think the major reason is this:
getLatestMergedFileSlicesBeforeOrOn can miss file groups that have been replaced or have no committed file slice at the queried instant.
@cshuo May I assign this issue to you? Honestly, I am not familiar with the file group calculation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4949979825
The call to getAllFilesInPartition is to build the file system view for partition
2026-07-11 23:01:35.358 [bucket_assigner (57/256)#1] INFO o.apache.hudi.common.table.view.AbstractTableFileSystemView - getLatestMergedFileSliceBeforeOrOnInternal partitionStr=record_index, maxInstantTime=20260711225001071, currentInstantTime=20260711225001071
2026-07-11 23:01:35.359 [bucket_assigner (57/256)#1] DEBUG o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (record_index)
2026-07-11 23:01:35.359 [bucket_assigner (57/256)#1] DEBUG o.apache.hudi.common.table.view.AbstractTableFileSystemView - Building file system view for partition (record_index)
2026-07-11 23:01:36.117 [bucket_assigner (57/256)#1] INFO org.apache.hudi.storage.HoodieStorage - Found 10712 files in path ***/.hoodie/metadata/record_index
2026-07-11 23:01:36.118 [bucket_assigner (57/256)#1] DEBUG o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in partition (record_index}) = 10712, Time taken =759
2026-07-11 23:01:36.118 [bucket_assigner (57/256)#1] DEBUG o.apache.hudi.common.table.view.AbstractTableFileSystemView - #files found in partition (record_index}) = 10712, Time taken =759
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4899810021
The file slice compaction is a file group scope operation and should not affect the num of file groups, is it because of the instant time filtering with "xxxBeforeOrOn"?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4901326257
@HuangZhenQiu yeah, i can take the issue. Could you share the detailed steps for “Bootstrap a new table from a COW original table”, along with the specific table configurations you used? That would make it easier for me to reproduce and investigate the problem.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4949260339
After investigation. the root cause is that the underneath file system is not able to return the right number of flles. The issue happens when the flink ingestion job write to 10k base file groups without quick log compaction for MDT. There are 100k+ log file created after several days. In this situation, the number of files returned by getAllFilesInPartition is not exactly the number in remote storage. It could be a bug in file system implementation rather than the issue in hudi.
tableMetadata.getAllFilesInPartition(partitionPath)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4929719058
The content of partitionToFileGroupsMap is only added by
@Override
protected void storePartitionView(String partitionPath, List<HoodieFileGroup> fileGroups) {
LOG.debug("Adding file-groups for partition :{}, #FileGroups={}", partitionPath, fileGroups.size());
List<HoodieFileGroup> newList = new ArrayList<>(fileGroups);
partitionToFileGroupsMap.put(partitionPath, newList);
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4931164430
@cshuo
https://github.com/apache/hudi/blob/master/hudi-common/src/main/java/org/apache/hudi/common/table/view/AbstractTableFileSystemView.java#L518
Does this mean only parquet files will be treated as base file?
This is the current table config
#Fri Jun 12 06:34:35 UTC 2026
hoodie.compaction.payload.class=com.uber.marmaray.uber.model.HooverAvroPayload
hoodie.table.initial.version=9
hoodie.table.keygenerator.type=SIMPLE
hoodie.table.type=MERGE_ON_READ
hoodie.table.metadata.partitions=files,record_index
hoodie.table.partition.fields=_hoodie_partition_path
hoodie.archivelog.folder=archived
hoodie.timeline.layout.version=2
hoodie.timeline.history.path=history
hoodie.table.checksum=889652317
hoodie.datasource.write.drop.partition.columns=false
hoodie.record.merge.strategy.id=00000000-0000-0000-0000-000000000000
hoodie.table.recordkey.fields=_row_key
hoodie.timeline.path=timeline
hoodie.table.ordering.fields=_row_key
hoodie.table.metadata.partitions.inflight=
hoodie.table.index.defs.path=.hoodie/.index_defs/index.json
hoodie.populate.meta.fields=true
hoodie.table.base.file.format=PARQUET
hoodie.record.merge.mode=CUSTOM
hoodie.table.version=9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4931069297
@HuangZhenQiu I'm wondering why `hoodie.table.multiple.base.file.formats.enable` is necessary, because the MDT uses HFile as its base-file format and does not contain multiple base-file formats. Log files are detected separately, so enabling this option should not change the discovered file-group count.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "danny0405 (via GitHub)" <gi...@apache.org>.
danny0405 commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4931197601
if multiple formats support is disabled and parquet is configured as the base file, then it is true, but for MDT, the base file should be just HFile format, no multiple format support as of now. Is the config you shared come from the MDT table.properties?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "cshuo (via GitHub)" <gi...@apache.org>.
cshuo commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4949666216
> After investigation. the root cause is that the underneath file system is not able to return the right number of flles. The issue happens when the flink ingestion job write to 10k base file groups without quick log compaction for MDT. There are 100k+ log file created after several days. In this situation, the number of files returned by getAllFilesInPartition is not exactly the number in remote storage. It could be a bug in file system implementation rather than the issue in hudi.
>
> tableMetadata.getAllFilesInPartition(partitionPath)
I think that's should not the root cause, because `getAllFilesInPartition` will not fetch files by file listing when MDT is enabled, but read the `FILES` partition of MDT to get all the files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-4929701181
Added more debug log. It shows this function returns 8.
/**
* Given a partition path, obtain all filegroups within that. All methods, that work at the partition level go through
* this.
*/
@Override
Stream<HoodieFileGroup> fetchAllStoredFileGroups(String partition) {
List<HoodieFileGroup> fileGroups = partitionToFileGroupsMap.get(partition);
if (fileGroups == null || fileGroups.isEmpty()) {
return Stream.empty();
}
return new ArrayList<>(partitionToFileGroupsMap.get(partition)).stream();
}
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Number File Group of Record Index Partition calculation is incorrect in GlobalRecordIndexPartitioner [hudi]
Posted by "HuangZhenQiu (via GitHub)" <gi...@apache.org>.
HuangZhenQiu commented on issue #19207:
URL: https://github.com/apache/hudi/issues/19207#issuecomment-5047226807
After internal debugging, we found it is not a bug in hudi.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org