You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "yaooqinn (via GitHub)" <gi...@apache.org> on 2025/09/02 03:28:48 UTC
[PR] [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes [spark]
yaooqinn opened a new pull request, #52192:
URL: https://github.com/apache/spark/pull/52192
### What changes were proposed in this pull request?
Simplify interoperations between SQLConf and builtin Datasource Scan related classes (#51398 / #51398), following
[SPARK-52704](https://issues.apache.org/jira/browse/SPARK-52704)
[SPARK-53415](https://issues.apache.org/jira/browse/SPARK-53415)
### Why are the changes needed?
- Reduce code duplication
- Restore type annotation for IDE
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
existing tests
### Was this patch authored or co-authored using generative AI tooling?
no
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #52192:
URL: https://github.com/apache/spark/pull/52192#issuecomment-3244472331
Merged to master, thank you @peter-toth again
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes [spark]
Posted by "peter-toth (via GitHub)" <gi...@apache.org>.
peter-toth commented on code in PR #52192:
URL: https://github.com/apache/spark/pull/52192#discussion_r2315277269
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -327,7 +327,7 @@ trait FileSourceScanLike extends DataSourceScanExec {
relation.fileFormat.vectorTypes(
requiredSchema = requiredSchema,
partitionSchema = relation.partitionSchema,
- relation.sparkSession.sessionState.conf).map { vectorTypes =>
+ getSqlConf(relation.sparkSession)).map { vectorTypes =>
Review Comment:
I wonder if we could extract `getSqlConf(relation.sparkSession)` as you did in `DataSource`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53449][SQL] Simplify options for builtin Datasource Scan related classes [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on PR #52192:
URL: https://github.com/apache/spark/pull/52192#issuecomment-3247510762
Thank you very much, @dongjoon-hyun
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn closed pull request #52192: [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes
URL: https://github.com/apache/spark/pull/52192
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-53449][SQL] Simply options for builtin Datasource Scan related classes [spark]
Posted by "yaooqinn (via GitHub)" <gi...@apache.org>.
yaooqinn commented on code in PR #52192:
URL: https://github.com/apache/spark/pull/52192#discussion_r2315351136
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/DataSourceScanExec.scala:
##########
@@ -327,7 +327,7 @@ trait FileSourceScanLike extends DataSourceScanExec {
relation.fileFormat.vectorTypes(
requiredSchema = requiredSchema,
partitionSchema = relation.partitionSchema,
- relation.sparkSession.sessionState.conf).map { vectorTypes =>
+ getSqlConf(relation.sparkSession)).map { vectorTypes =>
Review Comment:
Thank you for the suggestion. @peter-toth
Because SparkPlan has also defined a field of SQLConf, and these `*FileSourceScan*` files visit conf very inconsistently, I decided to make in-place updates instead of extracting a common field to avoid potential regression.
I can verify this part in a separate PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org