You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "YinChunGuang (via GitHub)" <gi...@apache.org> on 2025/09/12 08:00:17 UTC
[I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]
YinChunGuang opened a new issue, #13887:
URL: https://github.com/apache/hudi/issues/13887
**Hudi record level index dose not work when execute spark sql**
Hudi record level index dose not work when execute spark sql
**To Reproduce**
Steps to reproduce the behavior:
1. create table
create table if not exists hudix.hudihudi_table2(
id string,
name string,
price string
) using hudi
options (
type = 'cow',
primaryKey = 'id',
'hoodie.index.type'='RECORD_INDEX',
'hoodie.metadata.record.index.enable'='true'
);
2. insert data
insert into hudix.hudihudi_table2 select UUID() AS id, UUID() as id2, UUID() AS id3 from range(0,10000007,1,200);
3. execute query
3.1 SET hoodie.metadata.record.index.enable=true;
3.2 select * from hudix.hudihudi_table2 where id='1';
20250912145409224 20250912145409224_0_0 1 9922068f-73ab-4a2e-bc3f-dc241e4b0368-0_0-267-6144_20250912152837536.parquet 1 2 3
20250912145553793 20250912145553793_0_2 1 9922068f-73ab-4a2e-bc3f-dc241e4b0368-0_0-267-6144_20250912152837536.parquet 1 2 3
4. got spark dag scan all records as follows.
>
<img width="972" height="336" alt="Image" src="https://github.com/user-attachments/assets/b9d0e19c-ca77-4d31-bfab-bc0d099ccc02" />
<img width="575" height="696" alt="Image" src="https://github.com/user-attachments/assets/08a1cdca-612b-496c-bc79-a942241aaeeb" />
<img width="1784" height="709" alt="Image" src="https://github.com/user-attachments/assets/c7479136-4cfb-46b9-9101-b778ef451021" />
**Expected behavior**
Hudi Record Level Index dose not work with sparksql . I expected that index filter work .
**Environment Description**
* Hudi version : 0.15
* Spark version : 3.3.1
* Hive version : 2.3.9
* Hadoop version : 3.3.1
* Storage (HDFS/S3/GCS..) : hdfs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3299544954
Hi Team,
I will try to reproduce this after discussing with @rangareddy
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3322339330
> [@YinChunGuang](https://github.com/YinChunGuang) Are you setting all of below - SET hoodie.metadata.enable=true; SET hoodie.enable.data.skipping=true; SET hoodie.metadata.record.index.enable=true;
>
> Also, can you try to query using __hoodie_record_key once to check?
after set the param as follows ,the job scan index work. Thank you very much! @ad1happy2go .
SET hoodie.metadata.enable=true;
SET hoodie.enable.data.skipping=true;
SET hoodie.metadata.record.index.enable=true;
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3304008267
@YinChunGuang
Are you setting all of below -
SET hoodie.metadata.enable=true;
SET hoodie.enable.data.skipping=true;
SET hoodie.metadata.record.index.enable=true;
Also, can you try to query using __hoodie_record_key once to check?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "rangareddy (via GitHub)" <gi...@apache.org>.
rangareddy commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3285829433
Hi @YinChunGuang
We will try to reproduce this issue internally and will provide you with a status update.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "YinChunGuang (via GitHub)" <gi...@apache.org>.
YinChunGuang closed issue #13887: Hudi record level index dose not work when execute spark sql
URL: https://github.com/apache/hudi/issues/13887
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Hudi record level index dose not work when execute spark sql [hudi]
Posted by "vaibhavk1992 (via GitHub)" <gi...@apache.org>.
vaibhavk1992 commented on issue #13887:
URL: https://github.com/apache/hudi/issues/13887#issuecomment-3313147450
@ad1happy2go @rangareddy I have tried to reproduce the above issue. the first screenshot is after I set the properties mentioned and query using _hoodie_record_key.
The second screenshot is without any filters so no filter section there. BTW is there any way to check if record index key is used or not?
<img width="1326" height="795" alt="Image" src="https://github.com/user-attachments/assets/4d1841d1-bf0e-45be-be61-f14714c685c3" />
<img width="1416" height="642" alt="Image" src="https://github.com/user-attachments/assets/0e6a5ee5-e761-41af-947e-46dcd81d893e" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org