You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "RameshkumarChikoti123 (via GitHub)" <gi...@apache.org> on 2024/10/23 16:05:49 UTC
[I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
RameshkumarChikoti123 opened a new issue, #12152:
URL: https://github.com/apache/hudi/issues/12152
Added two record keys(customer_id,name) and configured Record index as below
`hudi_options = {
'hoodie.table.name': "hudi-table-with-rli-two-record-keys",
'hoodie.datasource.write.recordkey.field': "customer_id,name",
'hoodie.datasource.write.partitionpath.field': "state",
'hoodie.datasource.write.precombine.field': "created_at",
'hoodie.datasource.write.operation': "upsert", # Use upsert operation
'hoodie.index.type': "RECORD_INDEX",
'hoodie.metadata.enable': "true",
'hoodie.metadata.index.column.stats.enable': "true",
'hoodie.metadata.record.index.enable': "true"
}
df.write.format("hudi").options(**hudi_options).mode("append").save("s3a://bucket/var/proj/hudipoc-proj/hudi-table-with-rli-two-record-key/")`
**Reading record with composite keys**
` spark.read.format("hudi") \
.option("hoodie.enable.data.skipping", "true") \
.option("hoodie.metadata.enable", "true") \
.option("hoodie.metadata.record.index.enable", "true") \
.option("hoodie.metadata.index.column.stats.enable", "true") \
.load("s3a://bucket/var/proj/hudipoc-proj/hudi-table-with-rli-two-record-key/") \
.createOrReplaceTempView("hudi_snapshot1")
spark.sql("select * from hudi_snapshot1 where customer_id='04da8419-fb9e-47f1-a44f-3cf2199ad20a'and name='Customer_43680' ").show(truncate=False)`
**Observations**:
Spark is reading from all the partition as show in attached image
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "mzheng-plaid (via GitHub)" <gi...@apache.org>.
mzheng-plaid commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2447699322
Sorry, so is there currently any benefit to using RLI if you have a partitioned table? Or any composite record key?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go closed issue #12152: [SUPPORT]Spark reads from all partitions when using composite keys with a record index.
URL: https://github.com/apache/hudi/issues/12152
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2447005322
@mzheng-plaid Yes, this issue was always there with RLI as we didnt had this support.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2516377896
Fixed by https://github.com/apache/hudi/pull/12336
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2434609612
Thanks for raising this @RameshkumarChikoti123 . I checked with master code too and i confirmed the issue is still there.
Created tracking jira to fix - https://issues.apache.org/jira/browse/HUDI-8432
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "mzheng-plaid (via GitHub)" <gi...@apache.org>.
mzheng-plaid commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2436085304
@ad1happy2go I'm seeing the same behavior on Hudi 0.14.1 with a partitioned table with a single record key (`hoodie.datasource.write.recordkey.field`) - is that expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [SUPPORT]Spark reads from all partitions when using composite keys with a record index. [hudi]
Posted by "ad1happy2go (via GitHub)" <gi...@apache.org>.
ad1happy2go commented on issue #12152:
URL: https://github.com/apache/hudi/issues/12152#issuecomment-2462414515
@mzheng-plaid RLI is still very useful for fast upserts even in case of partitioned table and composite record key. We are working on adding support on query read side for composite key.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org