You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/18 12:25:28 UTC
[GitHub] [spark] juliuszsompolski commented on pull request #28544: [SPARK-31387][SQL][test-maven] Handle unknown operation/session ID in HiveThriftServer2Listener
juliuszsompolski commented on pull request #28544:
URL: https://github.com/apache/spark/pull/28544#issuecomment-630147656
@dongjoon-hyun this PR passes "SPARK-29604 external listeners should be initialized with Spark classloader" in https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122763/testReport that was failing before in https://github.com/apache/spark/pull/28155. I have no idea how this change could have been related to that failure...
```
20/05/17 20:19:47 INFO SparkSQLEnvSuite:
===== TEST OUTPUT FOR o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' =====
SparkSQLEnvSuite:
20/05/17 20:19:47 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT
20/05/17 20:19:47 INFO ResourceUtils: ==============================================================
20/05/17 20:19:47 INFO ResourceUtils: No custom resources configured for spark.driver.
20/05/17 20:19:47 INFO ResourceUtils: ==============================================================
20/05/17 20:19:47 INFO SparkContext: Submitted application: testApp
20/05/17 20:19:47 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
20/05/17 20:19:47 INFO ResourceProfile: Limiting resource is cpu
20/05/17 20:19:47 INFO ResourceProfileManager: Added ResourceProfile id: 0
20/05/17 20:19:47 INFO SecurityManager: Changing view acls to: jenkins
20/05/17 20:19:47 INFO SecurityManager: Changing modify acls to: jenkins
20/05/17 20:19:47 INFO SecurityManager: Changing view acls groups to:
20/05/17 20:19:47 INFO SecurityManager: Changing modify acls groups to:
20/05/17 20:19:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(jenkins); groups with view permissions: Set(); users with modify permissions: Set(jenkins); groups with modify permissions: Set()
20/05/17 20:19:47 INFO Utils: Successfully started service 'sparkDriver' on port 37165.
20/05/17 20:19:47 INFO SparkEnv: Registering MapOutputTracker
20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMaster
20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
20/05/17 20:19:47 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/blockmgr-1d9a2431-0019-41bb-9fbe-c8ca6dd6bc1f
20/05/17 20:19:47 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB
20/05/17 20:19:47 INFO SparkEnv: Registering OutputCommitCoordinator
20/05/17 20:19:47 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-03.amp
20/05/17 20:19:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40300.
20/05/17 20:19:47 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-03.amp:40300
20/05/17 20:19:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
20/05/17 20:19:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-03.amp:40300 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
20/05/17 20:19:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
20/05/17 20:19:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
20/05/17 20:19:47 INFO SharedState: loading hive config file: file:/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive/target/scala-2.12/test-classes/hive-site.xml
20/05/17 20:19:47 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd').
20/05/17 20:19:47 INFO SharedState: Warehouse path is '/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd'.
20/05/17 20:19:47 WARN SharedState: Not allowing to set spark.sql.warehouse.dir or hive.metastore.warehouse.dir in SparkSession's options, it should be set statically for cross-session usages
20/05/17 20:19:47 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using maven.
20/05/17 20:19:53 INFO IsolatedClientLoader: Downloaded metastore jars to /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/hive-v2_3-a7e1202d-698a-456d-8609-2859a43dfba9
20/05/17 20:19:54 INFO HiveConf: Found configuration file null
20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0
20/05/17 20:19:54 INFO SessionState: Created local directory: /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0
20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0/_tmp_space.db
20/05/17 20:19:54 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd
20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
20/05/17 20:19:55 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
20/05/17 20:19:55 INFO ObjectStore: ObjectStore, initialize called
20/05/17 20:19:55 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
20/05/17 20:19:55 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
20/05/17 20:19:57 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
20/05/17 20:19:59 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
20/05/17 20:20:00 INFO ObjectStore: Initialized ObjectStore
20/05/17 20:20:00 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
20/05/17 20:20:00 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore jenkins@192.168.10.23
20/05/17 20:20:00 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
20/05/17 20:20:00 INFO HiveMetaStore: Added admin role in metastore
20/05/17 20:20:00 INFO HiveMetaStore: Added public role in metastore
20/05/17 20:20:00 INFO HiveMetaStore: No user is added in admin role, since config is empty
20/05/17 20:20:00 INFO HiveMetaStore: 0: get_all_functions
20/05/17 20:20:00 INFO audit: ugi=jenkins ip=unknown-ip-addr cmd=get_all_functions
20/05/17 20:20:00 INFO HiveMetaStore: 0: get_database: default
20/05/17 20:20:00 INFO audit: ugi=jenkins ip=unknown-ip-addr cmd=get_database: default
20/05/17 20:20:00 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
20/05/17 20:20:00 INFO StreamingQueryManager: Registered listener test.custom.listener.DummyStreamingQueryListener
20/05/17 20:20:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
20/05/17 20:20:00 INFO MemoryStore: MemoryStore cleared
20/05/17 20:20:00 INFO BlockManager: BlockManager stopped
20/05/17 20:20:00 INFO BlockManagerMaster: BlockManagerMaster stopped
20/05/17 20:20:00 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
20/05/17 20:20:00 INFO SparkContext: Successfully stopped SparkContext
20/05/17 20:20:00 INFO SparkSQLEnvSuite:
===== FINISHED o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' =====
- SPARK-29604 external listeners should be initialized with Spark classloader
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org