You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/24 14:02:40 UTC
[GitHub] [spark] TonyDoen opened a new pull request #35963: [SPARK-38639] Support cleanedSourcePrint flag to print Codegen cleanedSource
TonyDoen opened a new pull request #35963:
URL: https://github.com/apache/spark/pull/35963
### What changes were proposed in this pull request?
adding a config: spark.sql.codegen.cleanedSourcePrint triggers to print codegen clean source
### Why are the changes needed?
When we use spark-sql, encountering problems in codegen source, we often have to change the log level to DEBUG, but there are too many logs in this mode (DEBUG) .
Then `spark.sql.codegen.cleanedSourcePrint` can ensure that just printing codegen source.
### Does this PR introduce _any_ user-facing change?
Yes, add new config: `spark.sql.codegen.cleanedSourcePrint`
### How was this patch tested?
Existed UT
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #35963: [SPARK-38639] Support cleanedSourcePrint flag to print Codegen cleanedSource
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #35963:
URL: https://github.com/apache/spark/pull/35963#discussion_r834860580
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala
##########
@@ -1472,6 +1472,14 @@ object SQLConf {
.booleanConf
.createWithDefault(true)
+ val WHOLESTAGE_CODEGEN_CLEAN_SOURCE_ENABLED = buildConf("spark.sql.codegen.cleanedSourcePrint")
+ .internal()
+ .doc("When true, and `spark.sql.codegen.wholeStage` is true, the whole stage " +
+ "(of multiple operators) will be compiled into single java method that will be printed")
+ .version("3.2.0")
Review comment:
Should be 3.4.0.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #35963: [SPARK-38639] Support cleanedSourcePrint flag to print Codegen cleanedSource
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #35963:
URL: https://github.com/apache/spark/pull/35963#discussion_r834860822
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
##########
@@ -709,7 +709,12 @@ case class WholeStageCodegenExec(child: SparkPlan)(val codegenStageId: Int)
val duration = System.nanoTime() - startTime
WholeStageCodegenExec.increaseCodeGenTime(duration)
- logDebug(s"\n${CodeFormatter.format(cleanedSource)}")
+ val formatCleanCode = CodeFormatter.format(cleanedSource)
+ if (conf.wholeStageCleanCodePrintEnabled) {
+ logInfo(s"\n$formatCleanCode")
Review comment:
I don't think it's worth introducing another PR. I believe you can already configure this via log4j configurations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] TonyDoen commented on a change in pull request #35963: [SPARK-38639] Support cleanedSourcePrint flag to print Codegen cleanedSource
Posted by GitBox <gi...@apache.org>.
TonyDoen commented on a change in pull request #35963:
URL: https://github.com/apache/spark/pull/35963#discussion_r834904383
##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/WholeStageCodegenExec.scala
##########
@@ -709,7 +709,12 @@ case class WholeStageCodegenExec(child: SparkPlan)(val codegenStageId: Int)
val duration = System.nanoTime() - startTime
WholeStageCodegenExec.increaseCodeGenTime(duration)
- logDebug(s"\n${CodeFormatter.format(cleanedSource)}")
+ val formatCleanCode = CodeFormatter.format(cleanedSource)
+ if (conf.wholeStageCleanCodePrintEnabled) {
+ logInfo(s"\n$formatCleanCode")
Review comment:
Thanks for reviewing. If I turn log level to DEBUG, there are logs more than I need. So I try to configure this on the other way
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org