You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 13:38:31 UTC

[I] Extend secondary index to work as non-global index [hudi]

hudi-bot opened a new issue, #17308:
URL: https://github.com/apache/hudi/issues/17308

   Currently, secondary index assumes that RLI is global and hence does not save patition info in its payload. To work with non-global RLI, we need to save partition in the value of the payload. And handle deletes accordingly. 
   
   ## JIRA info
   
   - Link: https://issues.apache.org/jira/browse/HUDI-8517
   - Type: Sub-task
   - Parent: https://issues.apache.org/jira/browse/HUDI-9101
   - Fix version(s):
     - 1.1.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Extend secondary index to work as non-global index [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #17308:
URL: https://github.com/apache/hudi/issues/17308#issuecomment-5365819865

   This issue was reviewed as part of the JIRA-migrated backlog triage (HUDI-8517).
   
   **Findings: confirmed, and the constraint is at the schema level - this is a payload format change, not a code tweak.**
   
   Your premise checks out exactly. `hudi-common/src/main/avro/HoodieMetadata.avsc` defines the secondary index value as:
   
   ```json
   {
       "name": "SecondaryIndexMetadata",
       "doc": "Metadata Index that contains information about secondary keys and the corresponding record keys in the dataset",
       "type": [
           "null",
           {
               "type": "record",
               "name": "HoodieSecondaryIndexInfo",
               "fields": [
                   {
                       "name": "isDeleted",
                       "type": "boolean",
                       "doc": "True if this entry has been deleted"
                   }
               ]
           }
       ],
       "default" : null
   }
   ```
   
   One field. There is nowhere to record which partition a record key lives in, which is exactly why the index only works against a global RLI.
   
   So the work is an MDT payload schema change with on-disk compatibility consequences: existing secondary-index partitions contain the one-field record, and readers must keep handling them. That probably means riding an index-version bump rather than evolving in place - and usefully, `SECONDARY_INDEX` already has version branching in `HoodieIndexVersion.getCurrentVersion`, returning `V2` at table version 9 and above and `V1` below. That is the natural seam for this.
   
   The delete handling the description mentions gets harder too: with partition in the value, a delete has to identify the right (secondary key, record key, partition) tuple rather than assuming global uniqueness.
   
   Keeping this open.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org