[jira] [Commented] (HIVE-14369) submit a task with hive on spark to other yarn cluster failed

["Rui Li (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-14369?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15398610#comment-15398610 ] 

Rui Li commented on HIVE-14369:
-------------------------------

[~Huang Xiaomeng] discussed this with me offline and we think the possible reason is sparkcluster cannot resolve the logical URI of the hive table.
I'm not sure if this is specific to HoS. Xiaomeng, how did you submit the spark job to sparkcluster? Can you try the same thing with HoMR?

> submit a task with hive on spark to other yarn cluster failed
> -------------------------------------------------------------
>
>                 Key: HIVE-14369
>                 URL: https://issues.apache.org/jira/browse/HIVE-14369
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Xiaomeng Huang
>
> In our environment, we have two hadoop clusters with HA,  named hivecluster and sparkcluster.
> hivecluster is a HA hadoop cluster for hive, which has large hard disk.
> sparkcluster is a HA hadoop cluster for spark, which has large memory.
> e.g. below is a hdfs-site.xml of hivecluster:
> {code}
> <property>
>         <name>dfs.ha.namenodes.hivecluster</name>
>         <value>nn1,nn2</value>
> </property>
> <property>
>         <name>dfs.namenode.rpc-address.hivecluster.nn1</name>
>         <value>10.17.21.32:9000</value>
> </property>
> <property>
>         <name>dfs.namenode.rpc-address.hivecluster.nn2</name>
>         <value>10.17.21.77:9000</value>
> </property>
> <property>
>         <name>dfs.namenode.http-address.hivecluster.nn1</name>
>         <value>10.17.21.32:50070</value>
> </property>
> <property>
>         <name>dfs.namenode.http-address.hivecluster.nn2</name>
>         <value>10.17.21.77:50070</value>
> </property>
> {code}
> Firstly, I created a hive table located as hdfs://hivecluster/hive/warehouse/xxx
> If I use hive on mr, it will run successfully.
> But if I use hive on spark to submit a task to yarn cluster of sparkcluster, it says:
> {code}
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exec.spark.SparkTask
> yarn日志显示：
> Diagnostics: java.lang.IllegalArgumentException: java.net.UnknownHostException: hivecluster
> Failing this attempt. Failing the application.
> {code}
> I didn't set host of hivecluster into hdfs-site.xml of sparkcluster



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)