You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/03/31 11:36:50 UTC
[GitHub] [spark] beliefer opened a new pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
beliefer opened a new pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081
### What changes were proposed in this pull request?
After my investigation, `SQLQueryTestSuite` spent a lot of time compiling the generated java code.
We should display the total compile time for generated java code.
This PR will add the following to `SQLQueryTestSuite`'s output.
```
=== Metrics of Whole Codegen ===
Total compile time: 80.564516529 seconds
```
Note: At first, I wanted to use `CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
### Why are the changes needed?
Display the total compile time for generated java code.
### Does this PR introduce any user-facing change?
'No'.
### How was this patch tested?
Jenkins test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402876561
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
OK. Thanks for your remind.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25361/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607711320
**[Test build #120717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120717/testReport)** for PR 28081 at commit [`eb27c3a`](https://github.com/apache/spark/commit/eb27c3acfcfe3b5e4ac86c6720bc02a287c4f584).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607843401
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120717/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607644468
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401313895
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
At first, I wanted to use` CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
Do you know the reason?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607842318
**[Test build #120717 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120717/testReport)** for PR 28081 at commit [`eb27c3a`](https://github.com/apache/spark/commit/eb27c3acfcfe3b5e4ac86c6720bc02a287c4f584).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607093188
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401305322
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
Probably, we need to reset this metric in `beforeAll`? btw , could we reuse `CodegenMetrics.METRIC_COMPILATION_TIME` for this purpose? Getting the same metric for the same purpose seems redundant.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606732588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120643/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607843390
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401424270
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
@maropu `Histogram` is a metric which calculates the distribution of a value.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607644472
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120704/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401313895
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
At first, I wanted to use` CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
I checked the type of ` CodegenMetrics.METRIC_COMPILATION_TIME` is `Histogram` that is it only preserves 1028 elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607843390
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401461266
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1466,10 +1473,12 @@ object CodeGenerator extends Logging {
val startTime = System.nanoTime()
val result = doCompile(code)
val endTime = System.nanoTime()
- def timeMs: Double = (endTime - startTime).toDouble / NANOS_PER_MILLIS
+ val duration = endTime - startTime
+ val timeMs: Double = duration.toDouble / NANOS_PER_MILLIS
CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
logInfo(s"Code generated in $timeMs ms")
+ _compileTime.add(duration)
Review comment:
`timeMs` lost a lot of precision. Because `SQLQueryTestSuite` contains 231 test cases contain a lot of SQL, tens of thousands of SQL are executed. The loss of accuracy will increase.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002334
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25361/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576724
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607843401
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120717/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576078
**[Test build #120643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120643/testReport)** for PR 28081 at commit [`41aa7e9`](https://github.com/apache/spark/commit/41aa7e90d742d0d2e58415ac7c926a96eccf20d4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606732571
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607712146
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25415/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401916067
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
Review comment:
Although this is designed for a test case, this is a general accumulator in the context of `CodeGenerator`. Could you revise into a more proper description?
cc @gatorsmile
Also, this doesn't `Display` anything here. The warning log happens in `afterAll` in `SQLQueryTestSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576736
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25344/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] rednaxelafx commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
rednaxelafx commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402874997
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
Hi @beliefer ,
I'm really sorry to post edit suggestions after the PR is merged. I noticed this PR just now...
If we have a chance to make follow-up changes, could you please update `java code` to `Java code`, and `whole codegen` to `whole-stage codegen`?
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607784269
@cloud-fan @maropu @dongjoon-hyun Thanks for all your help!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607576468
> Could we display this metric in `BenchmarkQueryTest`, too?
The run time of `BenchmarkQueryTest` is smaller. I'm not sure if I should consider it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574821
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002110
**[Test build #120662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120662/testReport)** for PR 28081 at commit [`3dd0adf`](https://github.com/apache/spark/commit/3dd0adf89e2ae736d0883b784bb077c40f7eb4e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607644468
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401432918
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1466,10 +1473,12 @@ object CodeGenerator extends Logging {
val startTime = System.nanoTime()
val result = doCompile(code)
val endTime = System.nanoTime()
- def timeMs: Double = (endTime - startTime).toDouble / NANOS_PER_MILLIS
+ val duration = endTime - startTime
+ val timeMs: Double = duration.toDouble / NANOS_PER_MILLIS
CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
Review comment:
`CodegenMetrics.METRIC_COMPILATION_TIME` is a `Histogram` is a metric which calculates the distribution of a value. We can't reuse it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576078
**[Test build #120643 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120643/testReport)** for PR 28081 at commit [`41aa7e9`](https://github.com/apache/spark/commit/41aa7e90d742d0d2e58415ac7c926a96eccf20d4).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607093195
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120662/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607712146
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25415/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574827
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25402/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r403846756
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
@beliefer can you send a followup PR to address it?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401450611
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1466,10 +1473,12 @@ object CodeGenerator extends Logging {
val startTime = System.nanoTime()
val result = doCompile(code)
val endTime = System.nanoTime()
- def timeMs: Double = (endTime - startTime).toDouble / NANOS_PER_MILLIS
+ val duration = endTime - startTime
+ val timeMs: Double = duration.toDouble / NANOS_PER_MILLIS
CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
logInfo(s"Code generated in $timeMs ms")
+ _compileTime.add(duration)
Review comment:
I think `timeMs` is good enough to be recorded.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574821
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607712130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401432918
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1466,10 +1473,12 @@ object CodeGenerator extends Logging {
val startTime = System.nanoTime()
val result = doCompile(code)
val endTime = System.nanoTime()
- def timeMs: Double = (endTime - startTime).toDouble / NANOS_PER_MILLIS
+ val duration = endTime - startTime
+ val timeMs: Double = duration.toDouble / NANOS_PER_MILLIS
CodegenMetrics.METRIC_SOURCE_CODE_SIZE.update(code.body.length)
CodegenMetrics.METRIC_COMPILATION_TIME.update(timeMs.toLong)
Review comment:
`CodegenMetrics.METRIC_COMPILATION_TIME` is a `Histogram` that is a metric which calculates the distribution of a value. We can't reuse it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402000199
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
+
+ def compileTime: Long = _compileTime.sum
Review comment:
OK.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401313895
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
At first, I wanted to use` CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
I checked the type of ` CodegenMetrics.METRIC_COMPILATION_TIME` is `Histogram` and the it only preserves 1028 elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401313895
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
At first, I wanted to use` CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
I checked the type of ` CodegenMetrics.METRIC_COMPILATION_TIME` is `Histogram` and the latter preserves 1028 elements.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402148398
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,21 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
+ // each Executor as the execution code to process data. This is very effective in processing
+ // large amounts of data in a distributed environment. However, in the test environment,
+ // because the amount of data is not large or not executed in parallel, the compilation time
+ // of these Java source code will become a major part of the entire test runtime. When
+ // running test cases, we summarize the total compilation time and output it to the execution
+ // log for easy analysis and view.
+ private val _compileTime = new LongAccumulator
+
+ // Visible for testing
+ def compileTime: Long = _compileTime.sum
Review comment:
OK. I will add it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607722912
The last commit just adds a line of comment. I'm merging it to master, thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002328
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r403877573
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
@cloud-fan @rednaxelafx I make a followup PR https://github.com/apache/spark/pull/28112.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574538
**[Test build #120704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120704/testReport)** for PR 28081 at commit [`1d7dc2f`](https://github.com/apache/spark/commit/1d7dc2fd340fe0fec6e8b435e3dca64fd1f19659).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401916067
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
Review comment:
Although this is designed for a test case, in this context, this is a general accumulator in `CodeGenerator`. Could you revise into a more proper description?
cc @gatorsmile
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606731021
**[Test build #120643 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120643/testReport)** for PR 28081 at commit [`41aa7e9`](https://github.com/apache/spark/commit/41aa7e90d742d0d2e58415ac7c926a96eccf20d4).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401452636
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
Review comment:
can we add a simple test to run queries with whole-stage-codegen on/off and see if this metrics increase? whole-stage-codegen compiles code at driver side.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606732588
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120643/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576724
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401916067
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
Review comment:
Although this is designed for a test case, in this context, this is a general accumulator in `CodeGenerator`. Could you revise into a more proper description?
cc @gatorsmile
Also, this doesn't `Display` anything here. The warning log happens in `afterAll` in `SQLQueryTestSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] gatorsmile commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
gatorsmile commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r410655567
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
+ // each Executor as the execution code to process data. This is very effective in processing
+ // large amounts of data in a distributed environment. However, in the test environment,
+ // because the amount of data is not large or not executed in parallel, the compilation time
+ // of these Java source code will become a major part of the entire test runtime. When
+ // running test cases, we summarize the total compilation time and output it to the execution
+ // log for easy analysis and view.
+ private val _compileTime = new LongAccumulator
+
+ // Returns the total compile time of Java source code in nanoseconds.
+ // Visible for testing
+ def compileTime: Long = _compileTime.sum
+
+ // Reset compile time.
+ // Visible for testing
+ def resetCompileTime: Unit = _compileTime.reset()
Review comment:
Methods should be declared with parentheses, unless they are accessors that have no side-effect. See: https://github.com/databricks/scala-style-guide#parentheses
Fixed in https://github.com/apache/spark/pull/28252
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on issue #28081: [SPARK-31315][SQL][TEST]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
maropu commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606972458
Could we display this metric in `BenchmarkQueryTest`, too?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607644472
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120704/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574827
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25402/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402000235
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
+
+ def compileTime: Long = _compileTime.sum
+
+ def resetCompileTime: Unit = _compileTime.reset()
Review comment:
OK
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607712130
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607093188
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607643869
**[Test build #120704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120704/testReport)** for PR 28081 at commit [`1d7dc2f`](https://github.com/apache/spark/commit/1d7dc2fd340fe0fec6e8b435e3dca64fd1f19659).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401914138
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
+
+ def compileTime: Long = _compileTime.sum
+
+ def resetCompileTime: Unit = _compileTime.reset()
Review comment:
ditto.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607092308
**[Test build #120662 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120662/testReport)** for PR 28081 at commit [`3dd0adf`](https://github.com/apache/spark/commit/3dd0adf89e2ae736d0883b784bb077c40f7eb4e6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607096899
cc @srowen @cloud-fan
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607093195
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/120662/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607574538
**[Test build #120704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120704/testReport)** for PR 28081 at commit [`1d7dc2f`](https://github.com/apache/spark/commit/1d7dc2fd340fe0fec6e8b435e3dca64fd1f19659).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607566810
> When a source code under `src/main` is changed, `[TESTS]` should not be used.
OK. Thank you. I get it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401998904
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
Review comment:
OK. I added the test data onto the description of this PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606576736
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/25344/
Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402006111
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
Review comment:
OK. I will supplement a more proper description.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002328
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan closed pull request #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan closed pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on issue #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-606732571
Merged build finished. Test PASSed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] cloud-fan commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r402084130
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,21 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
+ // each Executor as the execution code to process data. This is very effective in processing
+ // large amounts of data in a distributed environment. However, in the test environment,
+ // because the amount of data is not large or not executed in parallel, the compilation time
+ // of these Java source code will become a major part of the entire test runtime. When
+ // running test cases, we summarize the total compilation time and output it to the execution
+ // log for easy analysis and view.
+ private val _compileTime = new LongAccumulator
+
+ // Visible for testing
+ def compileTime: Long = _compileTime.sum
Review comment:
We should mention that it's time in nanoseconds
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401914068
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
+ private val _compileTime = new LongAccumulator
+
+ def compileTime: Long = _compileTime.sum
Review comment:
Shall we add `// Visible for testing`?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r404138071
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
Yes
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] beliefer commented on a change in pull request #28081:
[SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time
for generated java code
Posted by GitBox <gi...@apache.org>.
beliefer commented on a change in pull request #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401313895
##########
File path: sql/core/src/test/scala/org/apache/spark/sql/SQLQueryTestSuite.scala
##########
@@ -667,6 +669,13 @@ class SQLQueryTestSuite extends QueryTest with SharedSparkSession {
// For debugging dump some statistics about how much time was spent in various optimizer rules
logWarning(RuleExecutor.dumpTimeSpent())
+
+ val codegenInfo =
+ s"""
+ |=== Metrics of Whole Codegen ===
+ |Total compile time: ${CodeGenerator.compileTime.sum.toDouble / NANOS_PER_SECOND} seconds
Review comment:
At first, I wanted to use` CodegenMetrics.METRIC_COMPILATION_TIME` to do this. After many experiments, I found that `CodegenMetrics.METRIC_COMPILATION_TIME` is only effective for a single test case, and cannot play a role in the whole life cycle of `SQLQueryTestSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607711320
**[Test build #120717 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120717/testReport)** for PR 28081 at commit [`eb27c3a`](https://github.com/apache/spark/commit/eb27c3acfcfe3b5e4ac86c6720bc02a287c4f584).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST]
SQLQueryTestSuite: Display the total compile time for generated java code
Posted by GitBox <gi...@apache.org>.
SparkQA commented on issue #28081: [SPARK-31315][SQL][TEST] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#issuecomment-607002110
**[Test build #120662 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/120662/testReport)** for PR 28081 at commit [`3dd0adf`](https://github.com/apache/spark/commit/3dd0adf89e2ae736d0883b784bb077c40f7eb4e6).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r401916067
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,13 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // Display the total compile time for generating java code in the output of the test case.
Review comment:
Although this is designed for a test case, in this context, this is a general accumulator in `CodeGenerator`. Could you revise into a more proper description?
cc @gatorsmile
Also, this doesn't `Display` here. The warning log happens in `afterAll` in `SQLQueryTestSuite`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] maropu commented on a change in pull request #28081:
[SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for
generated java code
Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #28081: [SPARK-31315][SQL] SQLQueryTestSuite: Display the total compile time for generated java code
URL: https://github.com/apache/spark/pull/28081#discussion_r403877693
##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/codegen/CodeGenerator.scala
##########
@@ -1309,6 +1309,23 @@ object CodeGenerator extends Logging {
// bytecode instruction
final val MUTABLESTATEARRAY_SIZE_LIMIT = 32768
+ // The java source code generated by whole codegen is compiled on the Driver side and sent to
Review comment:
This one? https://github.com/apache/spark/pull/28112
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org