You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by maryannxue <gi...@git.apache.org> on 2018/09/12 21:35:40 UTC
[GitHub] spark pull request #22406: [SPARK-25415][SQL] Make plan change log in RuleEx...
GitHub user maryannxue opened a pull request:
https://github.com/apache/spark/pull/22406
[SPARK-25415][SQL] Make plan change log in RuleExecutor configurable by SQLConf
## What changes were proposed in this pull request?
In RuleExecutor, after applying a rule, if the plan has changed, the before and after plan will be logged using level "trace". At times, however, such information can be very helpful for debugging. Hence, making the log level configurable in SQLConf would allow users to turn on the plan change log independently and save the trouble of tweaking log4j settings. Meanwhile, filtering plan change log for specific rules can also be very useful.
So this PR adds two SQL configurations:
1. spark.sql.optimizer.planChangeLog.level - set a specific log level for logging plan changes after a rule is applied.
2. spark.sql.optimizer.planChangeLog.rules - enable plan change logging only for a set of specified rules, separated by commas.
## How was this patch tested?
Added UT.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/maryannxue/spark spark-25415
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/22406.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #22406
----
commit 3b98a0f6bbcf6bd56341276f1cdb20e32a743faf
Author: maryannxue <ma...@...>
Date: 2018-09-12T21:32:12Z
[SPARK-25415][SQL] Make plan change log in RuleExecutor configurable by SQLConf
----
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22406
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/testing-k8s-prb-make-spark-distribution-unified/3059/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22406
**[Test build #96005 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96005/testReport)** for PR 22406 at commit [`3b98a0f`](https://github.com/apache/spark/commit/3b98a0f6bbcf6bd56341276f1cdb20e32a743faf).
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22406
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22406
Merged build finished. Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22406: [SPARK-25415][SQL] Make plan change log in RuleEx...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/22406
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #22406: [SPARK-25415][SQL] Make plan change log in RuleEx...
Posted by gengliangwang <gi...@git.apache.org>.
Github user gengliangwang commented on a diff in the pull request:
https://github.com/apache/spark/pull/22406#discussion_r217249962
--- Diff: sql/catalyst/src/main/scala/org/apache/spark/sql/internal/SQLConf.scala ---
@@ -171,6 +171,26 @@ object SQLConf {
.intConf
.createWithDefault(10)
+ val OPTIMIZER_PLAN_CHANGE_LOG_LEVEL = buildConf("spark.sql.optimizer.planChangeLog.level")
--- End diff --
I think the configuration here is about all rules in analyzer/optimizer/.., not only optimizer.
We should revise the naming.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/22406
**[Test build #96005 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/96005/testReport)** for PR 22406 at commit [`3b98a0f`](https://github.com/apache/spark/commit/3b98a0f6bbcf6bd56341276f1cdb20e32a743faf).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #22406: [SPARK-25415][SQL] Make plan change log in RuleExecutor ...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/22406
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/96005/
Test PASSed.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org