[jira] [Assigned] (SPARK-22463) Missing hadoop/hive/hbase/etc configuration files in SPARK_CONF_DIR to distributed archive

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-22463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-22463:
------------------------------------

    Assignee:     (was: Apache Spark)

> Missing hadoop/hive/hbase/etc configuration files in SPARK_CONF_DIR to distributed archive
> ------------------------------------------------------------------------------------------
>
>                 Key: SPARK-22463
>                 URL: https://issues.apache.org/jira/browse/SPARK-22463
>             Project: Spark
>          Issue Type: Bug
>          Components: YARN
>    Affects Versions: 2.1.2, 2.2.0
>            Reporter: Kent Yao
>
> When I ran self contained sql apps, such as
> {code:java}
> import org.apache.spark.sql.SparkSession
> object ShowHiveTables {
>   def main(args: Array[String]): Unit = {
>     val spark = SparkSession
>       .builder()
>       .appName("Show Hive Tables")
>       .enableHiveSupport()
>       .getOrCreate()
>     spark.sql("show tables").show()
>     spark.stop()
>   }
> }
> {code}
> with **yarn cluster** mode and `hive-site.xml` correctly within `$SPARK_HOME/conf`,they failed to connect the right hive metestore for not seeing hive-site.xml in AM/Driver's classpath.
> Although submitting them with `--files/--jars local/path/to/hive-site.xml` or puting it to `$HADOOP_CONF_DIR/YARN_CONF_DIR` can make these apps works well in cluster mode as client mode, according to the official doc, see @ http://spark.apache.org/docs/latest/sql-programming-guide.html#hive-tables
> > Configuration of Hive is done by placing your hive-site.xml, core-site.xml (for security configuration), and hdfs-site.xml (for HDFS configuration) file in conf/.
> We may respect these configuration files too or modify the doc for hive-tables in cluster mode.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org