You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Maxim Gekk (Jira)" <ji...@apache.org> on 2020/09/01 11:59:00 UTC
[jira] [Resolved] (SPARK-32108) Silent mode of spark-sql is broken
[ https://issues.apache.org/jira/browse/SPARK-32108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Maxim Gekk resolved SPARK-32108.
--------------------------------
Resolution: Not A Problem
> Silent mode of spark-sql is broken
> ----------------------------------
>
> Key: SPARK-32108
> URL: https://issues.apache.org/jira/browse/SPARK-32108
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Maxim Gekk
> Priority: Major
>
> 1. I download the recent release Spark 3.0 from http://spark.apache.org/downloads.html
> 2. Run bin/spark-sql -S, it prints a lot of INFO
> {code}
> ➜ ~ ./spark-3.0/bin/spark-sql -S
> 20/06/26 20:43:38 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
> log4j:WARN No appenders could be found for logger (org.apache.hadoop.hive.conf.HiveConf).
> log4j:WARN Please initialize the log4j system properly.
> log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
> Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
> 20/06/26 20:43:39 INFO SharedState: spark.sql.warehouse.dir is not set, but hive.metastore.warehouse.dir is set. Setting spark.sql.warehouse.dir to the value of hive.metastore.warehouse.dir ('/user/hive/warehouse').
> 20/06/26 20:43:39 INFO SharedState: Warehouse path is '/user/hive/warehouse'.
> 20/06/26 20:43:39 INFO SessionState: Created HDFS directory: /tmp/hive/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a
> 20/06/26 20:43:39 INFO SessionState: Created local directory: /var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a
> 20/06/26 20:43:39 INFO SessionState: Created HDFS directory: /tmp/hive/maximgekk/a47e882c-86a3-42b9-b43f-9dab0dd8492a/_tmp_space.db
> 20/06/26 20:43:39 INFO SparkContext: Running Spark version 3.0.0
> 20/06/26 20:43:39 INFO ResourceUtils: ==============================================================
> 20/06/26 20:43:39 INFO ResourceUtils: Resources for spark.driver:
> 20/06/26 20:43:39 INFO ResourceUtils: ==============================================================
> 20/06/26 20:43:39 INFO SparkContext: Submitted application: SparkSQL::192.168.1.78
> 20/06/26 20:43:39 INFO SecurityManager: Changing view acls to: maximgekk
> 20/06/26 20:43:39 INFO SecurityManager: Changing modify acls to: maximgekk
> 20/06/26 20:43:39 INFO SecurityManager: Changing view acls groups to:
> 20/06/26 20:43:39 INFO SecurityManager: Changing modify acls groups to:
> 20/06/26 20:43:39 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(maximgekk); groups with view permissions: Set(); users with modify permissions: Set(maximgekk); groups with modify permissions: Set()
> 20/06/26 20:43:39 INFO Utils: Successfully started service 'sparkDriver' on port 59414.
> 20/06/26 20:43:39 INFO SparkEnv: Registering MapOutputTracker
> 20/06/26 20:43:39 INFO SparkEnv: Registering BlockManagerMaster
> 20/06/26 20:43:39 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
> 20/06/26 20:43:39 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
> 20/06/26 20:43:39 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
> 20/06/26 20:43:39 INFO DiskBlockManager: Created local directory at /private/var/folders/p3/dfs6mf655d7fnjrsjvldh0tc0000gn/T/blockmgr-c1d041ad-dd46-4d11-bbd0-e8ba27d3bf69
> 20/06/26 20:43:39 INFO MemoryStore: MemoryStore started with capacity 408.9 MiB
> 20/06/26 20:43:39 INFO SparkEnv: Registering OutputCommitCoordinator
> 20/06/26 20:43:40 INFO Utils: Successfully started service 'SparkUI' on port 4040.
> 20/06/26 20:43:40 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://192.168.1.78:4040
> 20/06/26 20:43:40 INFO Executor: Starting executor ID driver on host 192.168.1.78
> 20/06/26 20:43:40 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 59415.
> 20/06/26 20:43:40 INFO NettyBlockTransferService: Server created on 192.168.1.78:59415
> 20/06/26 20:43:40 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
> 20/06/26 20:43:40 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO BlockManagerMasterEndpoint: Registering block manager 192.168.1.78:59415 with 408.9 MiB RAM, BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, 192.168.1.78, 59415, None)
> 20/06/26 20:43:40 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('file:/Users/maximgekk/spark-warehouse/').
> 20/06/26 20:43:40 INFO SharedState: Warehouse path is 'file:/Users/maximgekk/spark-warehouse/'.
> 20/06/26 20:43:40 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using Spark classes.
> 20/06/26 20:43:40 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is file:/Users/maximgekk/spark-warehouse/
> 20/06/26 20:43:41 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
> 20/06/26 20:43:41 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
> 20/06/26 20:43:41 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
> 20/06/26 20:43:41 INFO ObjectStore: ObjectStore, initialize called
> 20/06/26 20:43:41 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
> 20/06/26 20:43:41 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
> 20/06/26 20:43:42 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
> 20/06/26 20:43:42 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
> 20/06/26 20:43:42 INFO ObjectStore: Initialized ObjectStore
> 20/06/26 20:43:42 ERROR ObjectStore: Version information found in metastore differs 1.2.0 from expected schema version 2.3.0. Schema verififcation is disabled hive.metastore.schema.verification
> 20/06/26 20:43:42 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore maximgekk@192.168.1.78
> 20/06/26 20:43:42 INFO HiveMetaStore: Added admin role in metastore
> 20/06/26 20:43:43 INFO HiveMetaStore: Added public role in metastore
> 20/06/26 20:43:43 INFO HiveMetaStore: No user is added in admin role, since config is empty
> 20/06/26 20:43:43 INFO HiveMetaStore: 0: get_all_functions
> 20/06/26 20:43:43 INFO audit: ugi=maximgekk ip=unknown-ip-addr cmd=get_all_functions
> 20/06/26 20:43:43 INFO HiveMetaStore: 0: get_database: default
> 20/06/26 20:43:43 INFO audit: ugi=maximgekk ip=unknown-ip-addr cmd=get_database: default
> {code}
> but it should output logs up to WARN:
> https://github.com/apache/spark/blob/178ca961fe2b27a821b55d2231532804f1d8b68f/sql/hive-thriftserver/src/main/scala/org/apache/spark/sql/hive/thriftserver/SparkSQLCLIDriver.scala#L323
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org