You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/09/03 01:09:27 UTC
[GitHub] [spark] wangyum opened a new pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
wangyum opened a new pull request #29631:
URL: https://github.com/apache/spark/pull/29631
### What changes were proposed in this pull request?
Remove legacy silent support mode for spark-sql CLI.
### Why are the changes needed?
https://github.com/apache/spark/pull/29619 add new silent mode. We can remove legacy silent support mode.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Manual test:
```
spark-sql> LM-SHC-16508156:spark yumwang$ bin/spark-sql -S
NOTE: SPARK_PREPEND_CLASSES is set, placing locally compiled Spark classes ahead of assembly.
20/09/03 09:06:12 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
20/09/03 09:06:16 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
20/09/03 09:06:16 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
20/09/03 09:06:19 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
20/09/03 09:06:19 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore yumwang@10.226.196.190
spark-sql> select * from test1;
1
spark-sql> select * from test1;
1
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum closed pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
wangyum closed pull request #29631:
URL: https://github.com/apache/spark/pull/29631
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
wangyum commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686834028
Merged to master.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #29631:
URL: https://github.com/apache/spark/pull/29631#discussion_r485344447
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
##########
@@ -320,9 +319,6 @@ private[hive] class SparkSQLCLIDriver extends CliDriver with Logging {
// because the Hive unit tests do not go through the main() code path.
if (!isRemoteMode) {
SparkSQLEnv.init()
- if (sessionState.getIsSilent) {
- SparkSQLEnv.sparkContext.setLogLevel(Level.WARN.toString)
Review comment:
Could you share an example?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686183129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29631:
URL: https://github.com/apache/spark/pull/29631#discussion_r485356997
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
##########
@@ -320,9 +319,6 @@ private[hive] class SparkSQLCLIDriver extends CliDriver with Logging {
// because the Hive unit tests do not go through the main() code path.
if (!isRemoteMode) {
SparkSQLEnv.init()
- if (sessionState.getIsSilent) {
- SparkSQLEnv.sparkContext.setLogLevel(Level.WARN.toString)
Review comment:
For example,
```bash
cd conf
echo "log4j.logger.org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver=INFO" >> log4j.properties.template
mv log4j.properties.template log4j.properties
cd ..
./bin/spark-sql -S
```
```
20/09/09 14:58:27 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
20/09/09 14:58:27 INFO HiveConf: Found configuration file null
20/09/09 14:58:28 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
20/09/09 14:58:28 INFO SharedState: Warehouse path is '/user/hive/warehouse'.
...
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686267963
cc @sarutak FYI
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-689990904
Let me just revert this @wangyum. I know https://github.com/apache/spark/pull/29631#discussion_r485356997 is a kind of unlikely scenario but I think the change is still valid.
Please feel free to bring it back if it's just something I misunderstood.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686172726
**[Test build #128218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128218/testReport)** for PR 29631 at commit [`6c48ebb`](https://github.com/apache/spark/commit/6c48ebb4f234dd414e678b19d84697c44fe8c5f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686182888
**[Test build #128218 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128218/testReport)** for PR 29631 at commit [`6c48ebb`](https://github.com/apache/spark/commit/6c48ebb4f234dd414e678b19d84697c44fe8c5f2).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686172726
**[Test build #128218 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/128218/testReport)** for PR 29631 at commit [`6c48ebb`](https://github.com/apache/spark/commit/6c48ebb4f234dd414e678b19d84697c44fe8c5f2).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686173050
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] sarutak commented on a change in pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
sarutak commented on a change in pull request #29631:
URL: https://github.com/apache/spark/pull/29631#discussion_r485955588
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
##########
@@ -320,9 +319,6 @@ private[hive] class SparkSQLCLIDriver extends CliDriver with Logging {
// because the Hive unit tests do not go through the main() code path.
if (!isRemoteMode) {
SparkSQLEnv.init()
- if (sessionState.getIsSilent) {
- SparkSQLEnv.sparkContext.setLogLevel(Level.WARN.toString)
Review comment:
Hmm, silent mode should be given preference over the static configuration.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686173050
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #29631:
URL: https://github.com/apache/spark/pull/29631#issuecomment-686183129
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on a change in pull request #29631: [SPARK-32772][SQL][FOLLOWUP] Remove legacy silent support mode for spark-sql CLI
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #29631:
URL: https://github.com/apache/spark/pull/29631#discussion_r485320783
##########
File path: sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala
##########
@@ -320,9 +319,6 @@ private[hive] class SparkSQLCLIDriver extends CliDriver with Logging {
// because the Hive unit tests do not go through the main() code path.
if (!isRemoteMode) {
SparkSQLEnv.init()
- if (sessionState.getIsSilent) {
- SparkSQLEnv.sparkContext.setLogLevel(Level.WARN.toString)
Review comment:
@wangyum, I was just double checking. Shouldn't we still respect this option when the log level of this class is explicitly set? It's a bit unlikely but I just wanted to check for doubly sure.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org