You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/20 06:38:00 UTC
[GitHub] [spark] hddong opened a new pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
hddong opened a new pull request #32248:
URL: https://github.com/apache/spark/pull/32248
<!--
Thanks for sending a pull request! Here are some tips for you:
1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
4. Be sure to keep the PR description updated to reflect all changes.
5. Please write your PR title to summarize what this PR proposes.
6. If possible, provide a concise example to reproduce the issue for a faster review.
7. If you want to add a new configuration, please read the guideline first for naming configurations in
'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
-->
### What changes were proposed in this pull request?
Add default log config for spark-sql
### Why are the changes needed?
The default log level for spark-sql is `WARN`. How to change the log level is confusing, we need a default config.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Change config `log4j.logger.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver=INFO` in log4j.properties, then spark-sql's default log level changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823700901
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825415036
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823747962
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137704/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32248:
URL: https://github.com/apache/spark/pull/32248#discussion_r618943900
##########
File path: conf/log4j.properties.template
##########
@@ -22,10 +22,12 @@ log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
-# Set the default spark-shell log level to WARN. When running the spark-shell, the
-# log level for this class is used to overwrite the root logger's log level, so that
-# the user can have different defaults for the shell and regular Spark apps.
+# Set the default spark-shell/spark-sql log level to WARN. When running the
+# spark-shell/spark-sql, the log level for this class is used to overwrite
Review comment:
```suggestion
# spark-shell/spark-sql, the log level for these classes is used to overwrite
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #32248:
URL: https://github.com/apache/spark/pull/32248
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823027364
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823717673
cc @wangyum FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825338292
cc @sarutak
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823747344
**[Test build #137704 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137704/testReport)** for PR 32248 at commit [`cec327b`](https://github.com/apache/spark/commit/cec327b006a0b6c3ec68b66aeaf1552e52495851).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823721949
Yes. We already have `-S` mode.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823700901
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42232/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823698806
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823681401
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823681401
ok to test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825394381
**[Test build #137845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137845/testReport)** for PR 32248 at commit [`d95b28d`](https://github.com/apache/spark/commit/d95b28dd494a1d3e67ac80a9d1ac703c1bc560b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825468656
**[Test build #137845 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137845/testReport)** for PR 32248 at commit [`d95b28d`](https://github.com/apache/spark/commit/d95b28dd494a1d3e67ac80a9d1ac703c1bc560b6).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823027364
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823682380
**[Test build #137704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137704/testReport)** for PR 32248 at commit [`cec327b`](https://github.com/apache/spark/commit/cec327b006a0b6c3ec68b66aeaf1552e52495851).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #32248:
URL: https://github.com/apache/spark/pull/32248#discussion_r618936832
##########
File path: conf/log4j.properties.template
##########
@@ -22,10 +22,12 @@ log4j.appender.console.target=System.err
log4j.appender.console.layout=org.apache.log4j.PatternLayout
log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p %c{1}: %m%n
-# Set the default spark-shell log level to WARN. When running the spark-shell, the
-# log level for this class is used to overwrite the root logger's log level, so that
-# the user can have different defaults for the shell and regular Spark apps.
+# Set the default spark-shell/spark-sql log level to WARN. When running the
+# spark-shell/spark-sql, the log level for this class is used to overwrite
Review comment:
`this class` -> `these classes`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823681640
@hddong can you retrigger https://github.com/hddong/spark/actions/runs/766140297?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825394381
**[Test build #137845 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137845/testReport)** for PR 32248 at commit [`d95b28d`](https://github.com/apache/spark/commit/d95b28dd494a1d3e67ac80a9d1ac703c1bc560b6).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825415036
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42375/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825481856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137845/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823682380
**[Test build #137704 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/137704/testReport)** for PR 32248 at commit [`cec327b`](https://github.com/apache/spark/commit/cec327b006a0b6c3ec68b66aeaf1552e52495851).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823747962
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137704/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825410292
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825393624
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #32248: [SPARK-35143][SQL][SHELL] Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825481856
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/137845/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
sarutak commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825385066
The log level for `spark-shell` is determined [here](https://github.com/apache/spark/blob/4fcbf59079f591dc2214bd16aade340fd380f914/core/src/main/scala/org/apache/spark/internal/Logging.scala#L153).
Users can change the the log level by `log4j.logger.org.apache.spark.repl.Main` and `log4j.properties.template` contains an example.
On the other hand, the log level for `spark-sql` is determined at the same place as `spark-shell` and users can change the log level by `log4j.logger.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver`. But `log4j.properties.template` contains no example.
So, I think it's reasonable to have an example in `log4j.properties.template` like this PR suggests.
What do you think? @HyukjinKwon @wangyum
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hddong commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
hddong commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-825325249
@HyukjinKwon : Waht do you think about this PR and `-S` mode? If I have misunderstandings, please point it out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] hddong commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
Posted by GitBox <gi...@apache.org>.
hddong commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823742124
@HyukjinKwon : I think there some different with `-S`.
1. `-S` cannot change log to `INFO`(or other level)
2. with log4j we can change `spark-sql` init log level.
3. And they can worrk together.
With default:
```
> spark-sql
21/04/21 10:52:50 WARN Utils: Your hostname, hongddMac.local resolves to a loopback address: 127.0.0.1; using 172.20.30.177 instead (on interface en0)
21/04/21 10:52:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/04/21 10:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/04/21 10:52:54 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/04/21 10:52:54 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore hongdd@127.0.0.1
Spark master: local[*], Application Id: local-1618973572072
spark-sql> show databases;
default
Time taken: 1.719 seconds, Fetched 1 row(s)
```
And `-S` not out line `Time taken: 1.719 seconds, Fetched 1 row(s)`
With log4j changed `INFO`:
```
>bin/spark-sql
21/04/21 10:56:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/04/21 10:56:15 INFO HiveConf: Found configuration file null
21/04/21 10:56:15 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir.
21/04/21 10:56:15 INFO SharedState: Warehouse path is 'file:/user/hive/warehouse'.
21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd
21/04/21 10:56:15 INFO SessionState: Created local directory: /var/folders/xl/jv1jvw6s6hv42ht1ty6gnk100000gn/T/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd
21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd/_tmp_space.db
21/04/21 10:56:15 INFO SparkContext: Running Spark version 3.2.0-SNAPSHOT
21/04/21 10:56:15 INFO ResourceUtils: ==============================================================
21/04/21 10:56:15 INFO ResourceUtils: No custom resources configured for spark.driver.
21/04/21 10:56:15 INFO ResourceUtils: ==============================================================
21/04/21 10:56:15 INFO SparkContext: Submitted application: SparkSQL::172.20.30.177
....
spark-sql> show databases;
21/04/21 10:56:52 INFO HiveMetaStore: 0: get_databases: *
21/04/21 10:56:52 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_databases: *
21/04/21 10:56:53 INFO CodeGenerator: Code generated in 129.111772 ms
21/04/21 10:56:53 INFO CodeGenerator: Code generated in 5.292995 ms
default
Time taken: 1.735 seconds, Fetched 1 row(s)
21/04/21 10:56:53 INFO SparkSQLCLIDriver: Time taken: 1.735 seconds, Fetched 1 row(s)
```
And `-S` effect when you query but not init.
```
1/04/21 10:57:40 INFO HiveMetaStore: 0: get_all_functions
21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_all_functions
21/04/21 10:57:40 INFO HiveMetaStore: 0: get_database: default
21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_database: default
21/04/21 10:57:40 INFO SparkSQLCLIDriver: Spark master: local[*], Application Id: local-1618973857876
spark-sql> show databases;
default
21/04/21 10:57:46 INFO SparkSQLCLIDriver: Time taken: 1.698 seconds, Fetched 1 row(s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org