You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/04/21 03:01:48 UTC
[GitHub] [spark] hddong commented on pull request #32248: [SPARK-35143][SQL][SHELL]Add default log level config for spark-sql
hddong commented on pull request #32248:
URL: https://github.com/apache/spark/pull/32248#issuecomment-823742124
@HyukjinKwon : I think there some different with `-S`.
1. `-S` cannot change log to `INFO`(or other level)
2. with log4j we can change `spark-sql` init log level.
3. And they can worrk together.
With default:
```
> spark-sql
21/04/21 10:52:50 WARN Utils: Your hostname, hongddMac.local resolves to a loopback address: 127.0.0.1; using 172.20.30.177 instead (on interface en0)
21/04/21 10:52:50 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
Setting default log level to "WARN".
To adjust logging level use sc.setLogLevel(newLevel). For SparkR, use setLogLevel(newLevel).
21/04/21 10:52:51 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
21/04/21 10:52:53 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
21/04/21 10:52:54 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
21/04/21 10:52:54 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore hongdd@127.0.0.1
Spark master: local[*], Application Id: local-1618973572072
spark-sql> show databases;
default
Time taken: 1.719 seconds, Fetched 1 row(s)
```
And `-S` not out line `Time taken: 1.719 seconds, Fetched 1 row(s)`
With log4j changed `INFO`:
```
>bin/spark-sql
21/04/21 10:56:14 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
21/04/21 10:56:15 INFO HiveConf: Found configuration file null
21/04/21 10:56:15 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir.
21/04/21 10:56:15 INFO SharedState: Warehouse path is 'file:/user/hive/warehouse'.
21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd
21/04/21 10:56:15 INFO SessionState: Created local directory: /var/folders/xl/jv1jvw6s6hv42ht1ty6gnk100000gn/T/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd
21/04/21 10:56:15 INFO SessionState: Created HDFS directory: /tmp/hive/hongdd/91f89606-aa0c-4514-96a7-1ff3bbff46bd/_tmp_space.db
21/04/21 10:56:15 INFO SparkContext: Running Spark version 3.2.0-SNAPSHOT
21/04/21 10:56:15 INFO ResourceUtils: ==============================================================
21/04/21 10:56:15 INFO ResourceUtils: No custom resources configured for spark.driver.
21/04/21 10:56:15 INFO ResourceUtils: ==============================================================
21/04/21 10:56:15 INFO SparkContext: Submitted application: SparkSQL::172.20.30.177
....
spark-sql> show databases;
21/04/21 10:56:52 INFO HiveMetaStore: 0: get_databases: *
21/04/21 10:56:52 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_databases: *
21/04/21 10:56:53 INFO CodeGenerator: Code generated in 129.111772 ms
21/04/21 10:56:53 INFO CodeGenerator: Code generated in 5.292995 ms
default
Time taken: 1.735 seconds, Fetched 1 row(s)
21/04/21 10:56:53 INFO SparkSQLCLIDriver: Time taken: 1.735 seconds, Fetched 1 row(s)
```
And `-S` effect when you query but not init.
```
1/04/21 10:57:40 INFO HiveMetaStore: 0: get_all_functions
21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_all_functions
21/04/21 10:57:40 INFO HiveMetaStore: 0: get_database: default
21/04/21 10:57:40 INFO audit: ugi=hongdd ip=unknown-ip-addr cmd=get_database: default
21/04/21 10:57:40 INFO SparkSQLCLIDriver: Spark master: local[*], Application Id: local-1618973857876
spark-sql> show databases;
default
21/04/21 10:57:46 INFO SparkSQLCLIDriver: Time taken: 1.698 seconds, Fetched 1 row(s)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org