You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "nsivabalan (via GitHub)" <gi...@apache.org> on 2026/06/25 03:21:49 UTC

Re: [I] BaseTableMetadata::getRecordByKeys should accept HoodieData when a very large number of keys are to be looked up. [hudi]

nsivabalan commented on issue #16029:
URL: https://github.com/apache/hudi/issues/16029#issuecomment-4795520350

   This improvement appears to be implemented on current `master`.
   
   `HoodieTableMetadata` now provides `HoodieData`-based lookup APIs that don't require collecting keys to the driver:
   
   - `HoodiePairData<String, HoodieRecordGlobalLocation> readRecordIndexLocationsWithKeys(HoodieData<String> recordKeys)` — `HoodieTableMetadata.java:247`
   - `HoodieData<HoodieRecordGlobalLocation> readRecordIndexLocations(HoodieData<String> recordKeys)` — `HoodieTableMetadata.java:265`
   
   Implementations in `HoodieBackedTableMetadata` (`readRecordIndexLocationsWithKeys` at line 334, `readRecordIndexLocations` at line 428) wrap the keys as `HoodieData<RecordIndexRawKey>` and route through `lookupIndexRecords` / `repartitioningIfNeeded`, which performs file-group hashing and per-slice grouping internally — addressing pwason's concern that key-to-file-group hashing should not be exposed to callers. The old `SparkMetadataTableRecordIndex` that did its own hashing is no longer in the codebase.
   
   If this matches the intent of HUDI-6388, this issue can likely be closed. If there's a remaining caller still going through a `List<String>` path that you'd like simplified, please point to it and we can reopen with that scope.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org