You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2020/05/18 12:25:28 UTC

[GitHub] [spark] juliuszsompolski commented on pull request #28544: [SPARK-31387][SQL][test-maven] Handle unknown operation/session ID in HiveThriftServer2Listener

juliuszsompolski commented on pull request #28544:
URL: https://github.com/apache/spark/pull/28544#issuecomment-630147656


   @dongjoon-hyun this PR passes "SPARK-29604 external listeners should be initialized with Spark classloader" in https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/122763/testReport that was failing before in https://github.com/apache/spark/pull/28155. I have no idea how this change could have been related to that failure...
   
   ```
   20/05/17 20:19:47 INFO SparkSQLEnvSuite: 
   
   ===== TEST OUTPUT FOR o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' =====
   
   SparkSQLEnvSuite:
   20/05/17 20:19:47 INFO SparkContext: Running Spark version 3.1.0-SNAPSHOT
   20/05/17 20:19:47 INFO ResourceUtils: ==============================================================
   20/05/17 20:19:47 INFO ResourceUtils: No custom resources configured for spark.driver.
   20/05/17 20:19:47 INFO ResourceUtils: ==============================================================
   20/05/17 20:19:47 INFO SparkContext: Submitted application: testApp
   20/05/17 20:19:47 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
   20/05/17 20:19:47 INFO ResourceProfile: Limiting resource is cpu
   20/05/17 20:19:47 INFO ResourceProfileManager: Added ResourceProfile id: 0
   20/05/17 20:19:47 INFO SecurityManager: Changing view acls to: jenkins
   20/05/17 20:19:47 INFO SecurityManager: Changing modify acls to: jenkins
   20/05/17 20:19:47 INFO SecurityManager: Changing view acls groups to: 
   20/05/17 20:19:47 INFO SecurityManager: Changing modify acls groups to: 
   20/05/17 20:19:47 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(jenkins); groups with view permissions: Set(); users  with modify permissions: Set(jenkins); groups with modify permissions: Set()
   20/05/17 20:19:47 INFO Utils: Successfully started service 'sparkDriver' on port 37165.
   20/05/17 20:19:47 INFO SparkEnv: Registering MapOutputTracker
   20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMaster
   20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
   20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
   20/05/17 20:19:47 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
   20/05/17 20:19:47 INFO DiskBlockManager: Created local directory at /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/blockmgr-1d9a2431-0019-41bb-9fbe-c8ca6dd6bc1f
   20/05/17 20:19:47 INFO MemoryStore: MemoryStore started with capacity 2.1 GiB
   20/05/17 20:19:47 INFO SparkEnv: Registering OutputCommitCoordinator
   20/05/17 20:19:47 INFO Executor: Starting executor ID driver on host amp-jenkins-worker-03.amp
   20/05/17 20:19:47 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 40300.
   20/05/17 20:19:47 INFO NettyBlockTransferService: Server created on amp-jenkins-worker-03.amp:40300
   20/05/17 20:19:47 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
   20/05/17 20:19:47 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
   20/05/17 20:19:47 INFO BlockManagerMasterEndpoint: Registering block manager amp-jenkins-worker-03.amp:40300 with 2.1 GiB RAM, BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
   20/05/17 20:19:47 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
   20/05/17 20:19:47 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, amp-jenkins-worker-03.amp, 40300, None)
   20/05/17 20:19:47 INFO SharedState: loading hive config file: file:/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive/target/scala-2.12/test-classes/hive-site.xml
   20/05/17 20:19:47 INFO SharedState: Setting hive.metastore.warehouse.dir ('null') to the value of spark.sql.warehouse.dir ('/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd').
   20/05/17 20:19:47 INFO SharedState: Warehouse path is '/home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd'.
   20/05/17 20:19:47 WARN SharedState: Not allowing to set spark.sql.warehouse.dir or hive.metastore.warehouse.dir in SparkSession's options, it should be set statically for cross-session usages
   20/05/17 20:19:47 INFO HiveUtils: Initializing HiveMetastoreConnection version 2.3.7 using maven.
   20/05/17 20:19:53 INFO IsolatedClientLoader: Downloaded metastore jars to /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/hive-v2_3-a7e1202d-698a-456d-8609-2859a43dfba9
   20/05/17 20:19:54 INFO HiveConf: Found configuration file null
   20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0
   20/05/17 20:19:54 INFO SessionState: Created local directory: /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0
   20/05/17 20:19:54 INFO SessionState: Created HDFS directory: /tmp/hive/jenkins/eb4b5224-dc13-48e3-8b2b-e1fa57f042e0/_tmp_space.db
   20/05/17 20:19:54 INFO HiveClientImpl: Warehouse location for Hive client (version 2.3.7) is /home/jenkins/workspace/SparkPullRequestBuilder/sql/hive-thriftserver/target/tmp/warehouse-49d39f38-dc7f-4a52-93cd-782c0d1231cd
   20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.jdbc.timeout does not exist
   20/05/17 20:19:55 WARN HiveConf: HiveConf of name hive.stats.retries.wait does not exist
   20/05/17 20:19:55 INFO HiveMetaStore: 0: Opening raw store with implementation class:org.apache.hadoop.hive.metastore.ObjectStore
   20/05/17 20:19:55 INFO ObjectStore: ObjectStore, initialize called
   20/05/17 20:19:55 INFO Persistence: Property hive.metastore.integral.jdo.pushdown unknown - will be ignored
   20/05/17 20:19:55 INFO Persistence: Property datanucleus.cache.level2 unknown - will be ignored
   20/05/17 20:19:57 INFO ObjectStore: Setting MetaStore object pin classes with hive.metastore.cache.pinobjtypes="Table,StorageDescriptor,SerDeInfo,Partition,Database,Type,FieldSchema,Order"
   20/05/17 20:19:59 INFO MetaStoreDirectSql: Using direct SQL, underlying DB is DERBY
   20/05/17 20:20:00 INFO ObjectStore: Initialized ObjectStore
   20/05/17 20:20:00 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 2.3.0
   20/05/17 20:20:00 WARN ObjectStore: setMetaStoreSchemaVersion called but recording version is disabled: version = 2.3.0, comment = Set by MetaStore jenkins@192.168.10.23
   20/05/17 20:20:00 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
   20/05/17 20:20:00 INFO HiveMetaStore: Added admin role in metastore
   20/05/17 20:20:00 INFO HiveMetaStore: Added public role in metastore
   20/05/17 20:20:00 INFO HiveMetaStore: No user is added in admin role, since config is empty
   20/05/17 20:20:00 INFO HiveMetaStore: 0: get_all_functions
   20/05/17 20:20:00 INFO audit: ugi=jenkins       ip=unknown-ip-addr      cmd=get_all_functions   
   20/05/17 20:20:00 INFO HiveMetaStore: 0: get_database: default
   20/05/17 20:20:00 INFO audit: ugi=jenkins       ip=unknown-ip-addr      cmd=get_database: default       
   20/05/17 20:20:00 INFO StateStoreCoordinatorRef: Registered StateStoreCoordinator endpoint
   20/05/17 20:20:00 INFO StreamingQueryManager: Registered listener test.custom.listener.DummyStreamingQueryListener
   20/05/17 20:20:00 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
   20/05/17 20:20:00 INFO MemoryStore: MemoryStore cleared
   20/05/17 20:20:00 INFO BlockManager: BlockManager stopped
   20/05/17 20:20:00 INFO BlockManagerMaster: BlockManagerMaster stopped
   20/05/17 20:20:00 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
   20/05/17 20:20:00 INFO SparkContext: Successfully stopped SparkContext
   20/05/17 20:20:00 INFO SparkSQLEnvSuite: 
   
   ===== FINISHED o.a.s.sql.hive.thriftserver.SparkSQLEnvSuite: 'SPARK-29604 external listeners should be initialized with Spark classloader' =====
   
   - SPARK-29604 external listeners should be initialized with Spark classloader
   ```


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org