You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "YinChunGuang (via GitHub)" <gi...@apache.org> on 2025/09/12 08:00:17 UTC

[I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]

YinChunGuang opened a new issue, #13887:
URL: https://github.com/apache/hudi/issues/13887

   **Hudi record level  index dose not work when execute spark  sql**
   
   
   
   Hudi record level  index dose not work when execute spark  sql
   
   **To Reproduce**
   
   Steps to reproduce the behavior:
   
   1.  create table 
   create table if not exists hudix.hudihudi_table2(
     id string, 
     name string, 
     price string
   ) using hudi
   options (
     type = 'cow',
     primaryKey = 'id',
     'hoodie.index.type'='RECORD_INDEX',
     'hoodie.metadata.record.index.enable'='true'
   );
   
   
   2.  insert data 
   insert into hudix.hudihudi_table2 select  UUID() AS id, UUID() as id2, UUID() AS id3 from  range(0,10000007,1,200);
   
   3. execute query  
   
   3.1 SET hoodie.metadata.record.index.enable=true;
   3.2 select * from hudix.hudihudi_table2 where id='1';
   20250912145409224	20250912145409224_0_0	1		9922068f-73ab-4a2e-bc3f-dc241e4b0368-0_0-267-6144_20250912152837536.parquet	1	2	3
   20250912145553793	20250912145553793_0_2	1		9922068f-73ab-4a2e-bc3f-dc241e4b0368-0_0-267-6144_20250912152837536.parquet	1	2	3
   
   4. got spark dag scan all records as follows.
   
   >
   
   <img width="972" height="336" alt="Image" src="https://github.com/user-attachments/assets/b9d0e19c-ca77-4d31-bfab-bc0d099ccc02" />
   
   <img width="575" height="696" alt="Image" src="https://github.com/user-attachments/assets/08a1cdca-612b-496c-bc79-a942241aaeeb" />
   
   <img width="1784" height="709" alt="Image" src="https://github.com/user-attachments/assets/c7479136-4cfb-46b9-9101-b778ef451021" />
   
   
   
   **Expected behavior**
   
   Hudi Record Level Index dose not work with sparksql .  I expected that index filter work .
   
   **Environment Description**
   
   * Hudi version :  0.15
   
   * Spark version : 3.3.1
   
   * Hive version : 2.3.9
   
   * Hadoop version : 3.3.1
   
   * Storage (HDFS/S3/GCS..) : hdfs
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3299544954

   Hi Team,
   I will try to reproduce this after discussing with @rangareddy 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3322339330

   > [@YinChunGuang](https://github.com/YinChunGuang) Are you setting all of below - SET hoodie.metadata.enable=true; SET hoodie.enable.data.skipping=true; SET hoodie.metadata.record.index.enable=true;
   > 
   > Also, can you try to query using __hoodie_record_key once to check?
   
   after set the param as follows ,the job scan index work. Thank you very much! @ad1happy2go .
   SET hoodie.metadata.enable=true;
   SET hoodie.enable.data.skipping=true;
   SET hoodie.metadata.record.index.enable=true;


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3304008267

   @YinChunGuang 
   Are you setting all of below - 
   SET hoodie.metadata.enable=true;
   SET hoodie.enable.data.skipping=true;
   SET hoodie.metadata.record.index.enable=true;
   
   Also, can you try to query using __hoodie_record_key once to check?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3285829433

   Hi @YinChunGuang 
   
   We will try to reproduce this issue internally and will provide you with a status update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang closed issue #13887:  Hudi record level  index dose not work when execute spark  sql
URL: https://github.com/apache/hudi/issues/13887


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Hudi record level index dose not work when execute spark sql [hudi]

Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3313147450

   @ad1happy2go @rangareddy  I have tried to reproduce the above issue. the first screenshot is after I set the properties mentioned and query using _hoodie_record_key.
   The second screenshot is without any filters so no filter section there. BTW is there any way to check if record index key is used or not?
   
   
   
   <img width="1326" height="795" alt="Image" src="https://github.com/user-attachments/assets/4d1841d1-bf0e-45be-be61-f14714c685c3" />
   <img width="1416" height="642" alt="Image" src="https://github.com/user-attachments/assets/0e6a5ee5-e761-41af-947e-46dcd81d893e" />


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org