You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2016/10/10 06:11:20 UTC
[jira] [Resolved] (SPARK-11936) Logs are flooded with HiveMetaStore
warnings in 2.0.0-mr1-cdh4.1.2 Jenkins builds
[ https://issues.apache.org/jira/browse/SPARK-11936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Xiao Li resolved SPARK-11936.
-----------------------------
Resolution: Not A Problem
The problem sounds gone. Please reopen it if you still hit it. Thanks!
> Logs are flooded with HiveMetaStore warnings in 2.0.0-mr1-cdh4.1.2 Jenkins builds
> ---------------------------------------------------------------------------------
>
> Key: SPARK-11936
> URL: https://issues.apache.org/jira/browse/SPARK-11936
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.0
> Reporter: Josh Rosen
>
> In https://amplab.cs.berkeley.edu/jenkins/job/Spark-Master-Maven-pre-YARN/HADOOP_VERSION=2.0.0-mr1-cdh4.1.2,label=spark-test/4876/consoleFull and other Hadoop 2.0.0 builds, many of the test logs are flooded with warning messages from Hadoop. For instance, from HiveCompatibilitySuite:
> {code}
> - sort_merge_join_desc_5
> 17:04:25.130 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/src specified for non-external table:src
> 17:04:25.395 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcpart specified for non-external table:srcpart
> 17:04:26.365 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcbucket_mapjoin_part_1 specified for non-external table:srcbucket_mapjoin_part_1
> 17:04:26.511 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
> 17:04:26.691 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcbucket_mapjoin_part_2 specified for non-external table:srcbucket_mapjoin_part_2
> - sort_merge_join_desc_6
> 17:04:27.597 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/src specified for non-external table:src
> 17:04:27.773 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcpart specified for non-external table:srcpart
> 17:04:28.804 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcbucket_mapjoin_part_1 specified for non-external table:srcbucket_mapjoin_part_1
> 17:04:28.963 WARN org.apache.hadoop.conf.Configuration: fs.default.name is deprecated. Instead, use fs.defaultFS
> 17:04:29.374 WARN org.apache.hadoop.hive.metastore.HiveMetaStore: Location: file:/home/jenkins/workspace/Spark-Master-Maven-pre-YARN/HADOOP_VERSION/2.0.0-mr1-cdh4.1.2/label/spark-test/sql/hive/target/tmp/warehouse--5147e4ab-8c0f-4952-b63c-44228a6c9879/srcbucket_mapjoin_part_2 specified for non-external table:srcbucket_mapjoin_part_2
> {code}
> My hunch is that this is somehow related to SPARK-9596, a change where we now share the Hadoop classes between the isolated metastore classloader and Spark.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org