You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "yihua (via GitHub)" <gi...@apache.org> on 2026/04/04 14:55:08 UTC
Re: [PR] feat(metadata): Optimize RLI initialization with size-weighted sampling for large tables [hudi]
yihua commented on code in PR #18354:
URL: https://github.com/apache/hudi/pull/18354#discussion_r3035668114
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -847,6 +887,195 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Estimates total record count using size-weighted sampling:
+ * 1. Sample a fraction of file slices and count their actual records (lightweight - no RLI record construction)
+ * 2. Calculate average records-per-byte ratio from samples
+ * 3. Use file sizes of ALL file slices to estimate total records
+ *
+ * If total file slices are below threshold, counts all file slices directly without sampling.
+ * Threshold: 50 for global RLI, 10 for partitioned RLI.
+ *
+ * @param partitionFileSliceList list of partition and file slice pairs
+ * @param samplingFraction fraction of file slices to sample
+ * @return estimated total record count
+ */
+ private long estimateRecordCountBySizeWeightedSampling(List<Pair<String, FileSlice>> partitionFileSliceList,
+ double samplingFraction) {
+ if (partitionFileSliceList.isEmpty()) {
+ return 0L;
+ }
+
+ int totalFileSlices = partitionFileSliceList.size();
+
+ // Determine threshold based on RLI type
+ // Global RLI: higher threshold (50) since it's one big partition
+ // Partitioned RLI: lower threshold (10) since we process per partition
+ int samplingThreshold = dataWriteConfig.isRecordLevelIndexEnabled() ? 10 : 50;
+
+ // For small number of file slices, skip sampling and count all
+ if (totalFileSlices <= samplingThreshold) {
+ LOG.info("Total file slices ({}) <= threshold ({}), counting all file slices without sampling",
+ totalFileSlices, samplingThreshold);
+ Pair<Long, Long> countAndSize = countRecordsFromFileSlices(partitionFileSliceList);
+ return countAndSize.getLeft();
+ }
+
+ int sampleSize = Math.max(1, (int) Math.ceil(totalFileSlices * samplingFraction));
+
+ LOG.info("Sampling {} out of {} file slices ({:.1f}%) to estimate record count",
Review Comment:
🤖 SLF4J uses `{}` for placeholders — `{:.1f}` is not valid and will be printed as literal text (the argument will be silently dropped). Same issue with `{:.6f}` a few lines below. Could you replace these with plain `{}` or use `String.format()` if you need decimal formatting?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -847,6 +887,195 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Estimates total record count using size-weighted sampling:
+ * 1. Sample a fraction of file slices and count their actual records (lightweight - no RLI record construction)
+ * 2. Calculate average records-per-byte ratio from samples
+ * 3. Use file sizes of ALL file slices to estimate total records
+ *
+ * If total file slices are below threshold, counts all file slices directly without sampling.
+ * Threshold: 50 for global RLI, 10 for partitioned RLI.
+ *
+ * @param partitionFileSliceList list of partition and file slice pairs
+ * @param samplingFraction fraction of file slices to sample
+ * @return estimated total record count
+ */
+ private long estimateRecordCountBySizeWeightedSampling(List<Pair<String, FileSlice>> partitionFileSliceList,
+ double samplingFraction) {
+ if (partitionFileSliceList.isEmpty()) {
+ return 0L;
+ }
+
+ int totalFileSlices = partitionFileSliceList.size();
+
+ // Determine threshold based on RLI type
+ // Global RLI: higher threshold (50) since it's one big partition
+ // Partitioned RLI: lower threshold (10) since we process per partition
+ int samplingThreshold = dataWriteConfig.isRecordLevelIndexEnabled() ? 10 : 50;
+
+ // For small number of file slices, skip sampling and count all
+ if (totalFileSlices <= samplingThreshold) {
+ LOG.info("Total file slices ({}) <= threshold ({}), counting all file slices without sampling",
+ totalFileSlices, samplingThreshold);
+ Pair<Long, Long> countAndSize = countRecordsFromFileSlices(partitionFileSliceList);
+ return countAndSize.getLeft();
+ }
+
+ int sampleSize = Math.max(1, (int) Math.ceil(totalFileSlices * samplingFraction));
+
+ LOG.info("Sampling {} out of {} file slices ({:.1f}%) to estimate record count",
+ sampleSize, totalFileSlices, samplingFraction * 100);
+
+ // Sample file slices uniformly
+ List<Pair<String, FileSlice>> sampledFileSlices = sampleFileSlicesUniformly(partitionFileSliceList, sampleSize);
+
+ // Count records from sampled file slices (lightweight - just counting, no RLI record construction)
+ // Returns: (recordCount, actualSampledBaseFileSize) - only counts file slices with base files
+ Pair<Long, Long> countAndSize = countRecordsFromFileSlices(sampledFileSlices);
+ long sampledRecordCount = countAndSize.getLeft();
+ long actualSampledSize = countAndSize.getRight();
+
+ LOG.info("Counted {} records from {} bytes of actual sampled base files", sampledRecordCount, actualSampledSize);
+
+ if (actualSampledSize == 0) {
+ LOG.warn("No base files found in sampled file slices, returning 0 as estimate");
+ return 0L;
+ }
+
+ double recordsPerByte = (double) sampledRecordCount / actualSampledSize;
+ LOG.info("Calculated records-per-byte ratio: {:.6f}", recordsPerByte);
+
+ // Estimate total records using ALL file slice base file sizes
+ long totalBaseFileSize = partitionFileSliceList.stream()
+ .mapToLong(p -> getFileSliceSize(p.getValue()))
+ .sum();
+
+ long estimatedTotal = (long) Math.ceil(totalBaseFileSize * recordsPerByte);
+ LOG.info("Estimated total record count: {} based on {} total base file bytes", estimatedTotal, totalBaseFileSize);
+
+ return estimatedTotal;
+ }
+
+ /**
+ * Counts records from file slices without constructing RLI records.
+ * This is a lightweight operation used only for estimation purposes.
+ * For MOR tables, only reads from base files (not log files) for faster estimation.
+ *
+ * @param partitionFileSlicePairs list of partition and file slice pairs
+ * @return Pair of (total record count, total base file size actually sampled)
+ */
+ private <T> Pair<Long, Long> countRecordsFromFileSlices(List<Pair<String, FileSlice>> partitionFileSlicePairs) {
+ if (partitionFileSlicePairs.isEmpty()) {
+ return Pair.of(0L, 0L);
+ }
+
+ // Filter to only file slices with base files
+ List<Pair<String, FileSlice>> fileSlicesWithBaseFiles = partitionFileSlicePairs.stream()
+ .filter(p -> p.getValue().getBaseFile().isPresent())
+ .collect(Collectors.toList());
+
+ if (fileSlicesWithBaseFiles.isEmpty()) {
+ LOG.warn("No file slices with base files found for record count estimation");
+ return Pair.of(0L, 0L);
+ }
+
+ // Calculate total base file size for the file slices we're actually counting
+ long totalBaseFileSize = fileSlicesWithBaseFiles.stream()
+ .mapToLong(p -> getFileSliceSize(p.getValue()))
+ .sum();
+
+ Option<String> instantTime = dataMetaClient.getActiveTimeline().getCommitsTimeline()
+ .filterCompletedInstants()
+ .lastInstant()
+ .map(HoodieInstant::requestedTime);
+ if (!instantTime.isPresent()) {
+ return Pair.of(0L, 0L);
+ }
+
+ final int parallelism = Math.min(fileSlicesWithBaseFiles.size(),
+ dataWriteConfig.getMetadataConfig().getRecordIndexMaxParallelism());
+ ReaderContextFactory<T> readerContextFactory = engineContext.getReaderContextFactory(dataMetaClient);
+
+ // Parallel count operation - just iterate through base file records without constructing RLI records
+ // For MOR tables, we skip log files to keep estimation fast
+ long totalCount = engineContext.parallelize(fileSlicesWithBaseFiles, parallelism)
+ .map(partitionAndFileSlice -> {
+ final FileSlice fileSlice = partitionAndFileSlice.getValue();
+ final HoodieBaseFile baseFile = fileSlice.getBaseFile().get();
+ HoodieReaderContext<T> readerContext = readerContextFactory.getContext();
+
+ try {
+ // Create a file slice with only the base file (no log files) for faster counting
+ FileSlice baseFileOnlySlice = new FileSlice(fileSlice.getPartitionPath(),
+ fileSlice.getBaseInstantTime(), fileSlice.getFileId());
+ baseFileOnlySlice.setBaseFile(baseFile);
+
+ HoodieFileGroupReader<T> fileGroupReader = HoodieFileGroupReader.<T>newBuilder()
+ .withReaderContext(readerContext)
+ .withHoodieTableMetaClient(dataMetaClient)
+ .withFileSlice(baseFileOnlySlice) // Base file only
+ .withLatestCommitTime(instantTime.get())
+ .withShouldUseRecordPosition(false)
+ .withProps(dataMetaClient.getTableConfig().getProps())
+ .build();
+ ClosableIterator closableIterator = fileGroupReader.getClosableKeyIterator();
+ // Just count records without materializing them
+ long count = 0;
+ while (closableIterator.hasNext()) {
+ closableIterator.next();
+ count++;
+ }
+ closableIterator.close();
+ fileGroupReader.close();
+ return count;
+ } catch (Exception e) {
+ LOG.warn("Failed to count records from file slice: " + fileSlice.getFileId(), e);
+ return 0L;
+ }
+ })
+ .sum();
+
Review Comment:
🤖 nit: `ClosableIterator` is used as a raw type here. Could you parameterize it (e.g., `ClosableIterator<?>`) to avoid unchecked warnings?
##########
hudi-client/hudi-client-common/src/main/java/org/apache/hudi/metadata/HoodieBackedTableMetadataWriter.java:
##########
@@ -847,6 +887,195 @@ private int estimateFileGroupCount(HoodieData<HoodieRecord> records) {
);
}
+ /**
+ * Estimates total record count using size-weighted sampling:
+ * 1. Sample a fraction of file slices and count their actual records (lightweight - no RLI record construction)
+ * 2. Calculate average records-per-byte ratio from samples
+ * 3. Use file sizes of ALL file slices to estimate total records
+ *
+ * If total file slices are below threshold, counts all file slices directly without sampling.
+ * Threshold: 50 for global RLI, 10 for partitioned RLI.
+ *
+ * @param partitionFileSliceList list of partition and file slice pairs
+ * @param samplingFraction fraction of file slices to sample
+ * @return estimated total record count
+ */
+ private long estimateRecordCountBySizeWeightedSampling(List<Pair<String, FileSlice>> partitionFileSliceList,
+ double samplingFraction) {
+ if (partitionFileSliceList.isEmpty()) {
+ return 0L;
+ }
+
+ int totalFileSlices = partitionFileSliceList.size();
+
+ // Determine threshold based on RLI type
+ // Global RLI: higher threshold (50) since it's one big partition
+ // Partitioned RLI: lower threshold (10) since we process per partition
+ int samplingThreshold = dataWriteConfig.isRecordLevelIndexEnabled() ? 10 : 50;
+
+ // For small number of file slices, skip sampling and count all
+ if (totalFileSlices <= samplingThreshold) {
+ LOG.info("Total file slices ({}) <= threshold ({}), counting all file slices without sampling",
+ totalFileSlices, samplingThreshold);
+ Pair<Long, Long> countAndSize = countRecordsFromFileSlices(partitionFileSliceList);
+ return countAndSize.getLeft();
+ }
+
+ int sampleSize = Math.max(1, (int) Math.ceil(totalFileSlices * samplingFraction));
+
+ LOG.info("Sampling {} out of {} file slices ({:.1f}%) to estimate record count",
+ sampleSize, totalFileSlices, samplingFraction * 100);
+
+ // Sample file slices uniformly
+ List<Pair<String, FileSlice>> sampledFileSlices = sampleFileSlicesUniformly(partitionFileSliceList, sampleSize);
+
+ // Count records from sampled file slices (lightweight - just counting, no RLI record construction)
+ // Returns: (recordCount, actualSampledBaseFileSize) - only counts file slices with base files
+ Pair<Long, Long> countAndSize = countRecordsFromFileSlices(sampledFileSlices);
+ long sampledRecordCount = countAndSize.getLeft();
+ long actualSampledSize = countAndSize.getRight();
+
+ LOG.info("Counted {} records from {} bytes of actual sampled base files", sampledRecordCount, actualSampledSize);
+
+ if (actualSampledSize == 0) {
+ LOG.warn("No base files found in sampled file slices, returning 0 as estimate");
+ return 0L;
+ }
+
+ double recordsPerByte = (double) sampledRecordCount / actualSampledSize;
+ LOG.info("Calculated records-per-byte ratio: {:.6f}", recordsPerByte);
Review Comment:
🤖 Same SLF4J issue here — `{:.6f}` won't be substituted. Should be `{}` or use `String.format("%.6f", recordsPerByte)`.
##########
hudi-common/src/main/java/org/apache/hudi/common/data/HoodieListData.java:
##########
@@ -238,6 +238,14 @@ public long count() {
return super.count();
}
+ @Override
+ public long sum() {
+ return asStream()
+ .map(obj -> (Long) obj)
+ .mapToLong(Long::longValue)
+ .sum();
+ }
Review Comment:
🤖 Agreed. The `collectAsList().stream().mapToLong(...).sum()` approach is much cleaner here — the result list has at most one Long per partition, so collecting is trivially cheap. It also avoids the empty-collection pitfall since `mapToLong().sum()` returns 0 for an empty stream, unlike Spark's `reduce()` which throws.
##########
hudi-spark-datasource/hudi-spark/src/test/java/org/apache/hudi/functional/TestHoodieBackedMetadata.java:
##########
@@ -4116,6 +4116,245 @@ private static void verifyMetadataColumnStatsRecords(HoodieStorage storage, List
}
}
+ /**
+ * Tests that RLI initialization uses sampling-based estimation when min != max file group count.
+ * Validates that file group count is properly estimated based on data size.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexSamplingBasedEstimation(HoodieTableType tableType) throws Exception {
+ init(tableType);
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+
+ // Configure RLI with different min (1) and max (10) to trigger sampling-based estimation
+ HoodieWriteConfig writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .withEnableRecordLevelIndex(true)
+ .withRecordIndexFileGroupCount(1, 10) // min != max triggers sampling
+ .build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder()
+ .withIndexType(HoodieIndex.IndexType.RECORD_INDEX)
+ .build())
+ .build();
+
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext, writeConfig)) {
+ // First commit
+ String firstCommit = metaClient.createNewInstantTime(false);
+ List<HoodieRecord> records = dataGen.generateInserts(firstCommit, 100);
+ WriteClientTestUtils.startCommitWithTime(client, firstCommit);
+ List<WriteStatus> writeStatuses = client.insert(jsc.parallelize(records, 1), firstCommit).collect();
+ assertNoWriteErrors(writeStatuses);
+ client.commit(firstCommit, jsc.parallelize(writeStatuses));
+
+ // Second commit - ensures RLI is initialized with sampling-based estimation
+ String secondCommit = metaClient.createNewInstantTime(false);
+ records = dataGen.generateInserts(secondCommit, 100);
+ WriteClientTestUtils.startCommitWithTime(client, secondCommit);
+ writeStatuses = client.insert(jsc.parallelize(records, 1), secondCommit).collect();
+ assertNoWriteErrors(writeStatuses);
+ client.commit(secondCommit, jsc.parallelize(writeStatuses));
+
+ // Verify RLI is initialized
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+ assertTrue(metaClient.getTableConfig().isMetadataPartitionAvailable(RECORD_INDEX),
+ "RLI should be initialized");
+
+ // Verify file group count is within bounds (1-10) and appropriate for data size
+ HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(getMetadataTableBasePath(basePath))
+ .setConf(storageConf.newInstance()).build();
+ int fileGroupCount = HoodieTableMetadataUtil.getPartitionLatestFileSlices(
+ mdtMetaClient, Option.empty(), RECORD_INDEX.getPartitionPath()).size();
+ assertTrue(fileGroupCount >= 1 && fileGroupCount <= 10,
+ "File group count should be within configured bounds: " + fileGroupCount);
+
+ // For small data (200 records), should use close to minimum
+ assertTrue(fileGroupCount <= 3,
+ "Small dataset should use small file group count: " + fileGroupCount);
+
+ // Verify RLI functionality - can look up records
+ validateMetadata(client);
+ }
+ }
+
+ /**
+ * Tests that RLI initialization uses user-configured value when min == max file group count.
+ * This bypasses sampling and uses the explicit value.
+ */
+ @ParameterizedTest
+ @EnumSource(HoodieTableType.class)
+ public void testRecordIndexWithFixedFileGroupCount(HoodieTableType tableType) throws Exception {
+ init(tableType);
+ HoodieSparkEngineContext engineContext = new HoodieSparkEngineContext(jsc);
+
+ // Configure RLI with same min and max (5) to bypass sampling
+ HoodieWriteConfig writeConfig = getWriteConfigBuilder(true, true, false)
+ .withMetadataConfig(HoodieMetadataConfig.newBuilder()
+ .enable(true)
+ .withEnableGlobalRecordLevelIndex(true)
+ .withRecordIndexFileGroupCount(2, 2) // min == max bypasses sampling
+ .build())
+ .withIndexConfig(HoodieIndexConfig.newBuilder()
+ .withIndexType(HoodieIndex.IndexType.RECORD_INDEX)
+ .build())
+ .build();
+
+ try (SparkRDDWriteClient client = new SparkRDDWriteClient(engineContext, writeConfig)) {
+ // First commit
+ String firstCommit = metaClient.createNewInstantTime(false);
+ List<HoodieRecord> records = dataGen.generateInserts(firstCommit, 100);
+ WriteClientTestUtils.startCommitWithTime(client, firstCommit);
+ List<WriteStatus> writeStatuses = client.insert(jsc.parallelize(records, 1), firstCommit).collect();
+ assertNoWriteErrors(writeStatuses);
+ client.commit(firstCommit, jsc.parallelize(writeStatuses));
+
+ // Second commit - triggers RLI initialization
+ String secondCommit = metaClient.createNewInstantTime(false);
+ records = dataGen.generateInserts(secondCommit, 100);
+ WriteClientTestUtils.startCommitWithTime(client, secondCommit);
+ writeStatuses = client.insert(jsc.parallelize(records, 1), secondCommit).collect();
+ assertNoWriteErrors(writeStatuses);
+ client.commit(secondCommit, jsc.parallelize(writeStatuses));
+
+ // Verify RLI is initialized with EXACTLY 2 file groups (user-specified value)
+ metaClient = HoodieTableMetaClient.reload(metaClient);
+ assertTrue(metaClient.getTableConfig().isMetadataPartitionAvailable(RECORD_INDEX),
+ "RLI should be initialized");
+
+ HoodieTableMetaClient mdtMetaClient = HoodieTableMetaClient.builder()
+ .setBasePath(getMetadataTableBasePath(basePath))
+ .setConf(storageConf.newInstance()).build();
+ int fileGroupCount = HoodieTableMetadataUtil.getPartitionLatestFileSlices(
+ mdtMetaClient, Option.empty(), RECORD_INDEX.getPartitionPath()).size();
Review Comment:
🤖 The assertion message says "exactly 5" but the config is `(2, 2)` and the expected value is `2`. Looks like a copy-paste oversight.
##########
hudi-client/hudi-spark-client/src/main/java/org/apache/hudi/data/HoodieJavaRDD.java:
##########
@@ -123,6 +123,13 @@ public long count() {
return rddData.count();
}
+ @Override
+ public long sum() {
+ return rddData
+ .map(obj -> (Long) obj)
+ .reduce(Long::sum);
+ }
Review Comment:
🤖 Good catch — Spark's `JavaRDD.reduce()` throws `UnsupportedOperationException` on an empty RDD. In the current call site (`countRecordsFromFileSlices`), the list is pre-filtered for non-empty base files, so it shouldn't hit this in practice. But since `sum()` is now a public interface method on `HoodieData`, it should handle the empty case gracefully rather than relying on callers to guard.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org