[I] BaseTableMetadata::getRecordByKeys should accept HoodieData when a very large number of keys are to be looked up. [hudi]

["hudi-bot (via GitHub)" <gi...@apache.org>]

hudi-bot opened a new issue, #16029:
URL: https://github.com/apache/hudi/issues/16029

   This will help simplify SparkMetadataTableRecordIndex code which is hashing the keys into fileGroups itself.
   
    
   
    
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-6388
   - Type: Improvement
   
   
   ---
   
   
   ## Comments
   
   15/Jun/23 18:00;pwason;In theory, SparkMetadataTableRecordIndex can simply call getRecordsByKeys() without having to split the keys into fileGroups. But that gets limited due to driver memory limitations as getRecordsByKeys() requires a list of keys rather than a RDD. Also SparkMetadataTableRecordIndex should not be hashing the keys as that is internal implementation of RI in MDT.
   
   So the current implementation of SparkMetadataTableRecordIndex is actually a perf fix as getRecordsByKeys() accepts a list of keys rather than a RDD (HoodieData). For large upserts, we cannot collect all the keys from incoming records onto the driver to pass to SparkMetadataTableRecordIndex.;;;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BaseTableMetadata::getRecordByKeys should accept HoodieData when a very large number of keys are to be looked up. [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan closed issue #16029: BaseTableMetadata::getRecordByKeys should accept HoodieData when a very large number of keys are to be looked up.
URL: https://github.com/apache/hudi/issues/16029


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] BaseTableMetadata::getRecordByKeys should accept HoodieData when a very large number of keys are to be looked up. [hudi]

["nsivabalan (via GitHub)" <gi...@apache.org>]

nsivabalan commented on issue #16029:
URL: https://github.com/apache/hudi/issues/16029#issuecomment-4795520350

   This improvement appears to be implemented on current `master`.
   
   `HoodieTableMetadata` now provides `HoodieData`-based lookup APIs that don't require collecting keys to the driver:
   
   - `HoodiePairData<String, HoodieRecordGlobalLocation> readRecordIndexLocationsWithKeys(HoodieData<String> recordKeys)` — `HoodieTableMetadata.java:247`
   - `HoodieData<HoodieRecordGlobalLocation> readRecordIndexLocations(HoodieData<String> recordKeys)` — `HoodieTableMetadata.java:265`
   
   Implementations in `HoodieBackedTableMetadata` (`readRecordIndexLocationsWithKeys` at line 334, `readRecordIndexLocations` at line 428) wrap the keys as `HoodieData<RecordIndexRawKey>` and route through `lookupIndexRecords` / `repartitioningIfNeeded`, which performs file-group hashing and per-slice grouping internally — addressing pwason's concern that key-to-file-group hashing should not be exposed to callers. The old `SparkMetadataTableRecordIndex` that did its own hashing is no longer in the codebase.
   
   If this matches the intent of HUDI-6388, this issue can likely be closed. If there's a remaining caller still going through a `List<String>` path that you'd like simplified, please point to it and we can reopen with that scope.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org