You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@hudi.apache.org by "hudi-bot (via GitHub)" <gi...@apache.org> on 2025/11/30 05:49:50 UTC
[I] Add read_optimize spark_session config to use in spark-sql [hudi]
hudi-bot opened a new issue, #15416:
URL: https://github.com/apache/hudi/issues/15416
When create a table not using hive catalog in spark, we can not easily do read_optimized query in sqark-sql(using global hudi config file is inconvenient),so I add the read_optimize spark_session config to use in spark-sql
## JIRA info
- Link: https://issues.apache.org/jira/browse/HUDI-4823
- Type: Improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add read_optimize spark_session config to use in spark-sql [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan closed issue #15416: Add read_optimize spark_session config to use in spark-sql
URL: https://github.com/apache/hudi/issues/15416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add read_optimize spark_session config to use in spark-sql [hudi]
Posted by "nsivabalan (via GitHub)" <gi...@apache.org>.
nsivabalan commented on issue #15416:
URL: https://github.com/apache/hudi/issues/15416#issuecomment-4795364846
This appears to be addressed in current master. Since #18205 (commit df5d7c8f), `DefaultSource.createRelation` picks up any SparkSQL session conf starting with `hoodie.` or `spark.hoodie.` and merges it into the read options, and `DataSourceOptionsHelper.parametersWithReadDefaults` normalizes `spark.hoodie.*` to `hoodie.*`. So in spark-sql you can now do:
```sql
SET spark.hoodie.datasource.query.type=read_optimized;
-- or
SET hoodie.datasource.query.type=read_optimized;
```
and subsequent reads against Hudi tables (including non-Hive-catalog tables) will run in read-optimized mode without needing a global `hudi-defaults.conf`. A companion fix #18297 extended the same handling to write paths. Could you re-check on current master and close this issue if it now meets your need?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Add read_optimize spark_session config to use in spark-sql [hudi]
Posted by "hudi-bot (via GitHub)" <gi...@apache.org>.
hudi-bot commented on issue #15416:
URL: https://github.com/apache/hudi/issues/15416#issuecomment-3630223806
### Linked PR(s)
- https://github.com/apache/hudi/pull/6643
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@hudi.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org