You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alexander Pivovarov (JIRA)" <ji...@apache.org> on 2016/02/27 22:31:18 UTC
[jira] [Updated] (SPARK-13532) Spark yarn executor container fails
if yarn.nodemanager.local-dirs starts with file://
[ https://issues.apache.org/jira/browse/SPARK-13532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Pivovarov updated SPARK-13532:
----------------------------------------
Description:
Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with file://
{code}
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
</property>
{code}
other application, e.g. Hadoop MR and Hive work normally
Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
e.g.
{code}
<value>/data01/yarn/nm,/data02/yarn/nm</value>
{code}
to reproduce the issue
open spark-shell
run
{code}
$ spark-shell
> sc.parallelize(1 to 10).count
{code}
was:
Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with file://
{code}
<property>
<name>yarn.nodemanager.local-dirs</name>
<value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
</property>
{code}
other application, e.g. Hadoop MR and Hive work normally
Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
e.g.
{code}
<value>/data01/yarn/nm,/data02/yarn/nm</value>
{code}
> Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with file://
> --------------------------------------------------------------------------------------
>
> Key: SPARK-13532
> URL: https://issues.apache.org/jira/browse/SPARK-13532
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.6.0
> Reporter: Alexander Pivovarov
>
> Spark yarn executor container fails if yarn.nodemanager.local-dirs starts with file://
> {code}
> <property>
> <name>yarn.nodemanager.local-dirs</name>
> <value>file:///data01/yarn/nm,file:///data02/yarn/nm</value>
> </property>
> {code}
> other application, e.g. Hadoop MR and Hive work normally
> Spark works only if yarn.nodemanager.local-dirs does not have file:// prefix
> e.g.
> {code}
> <value>/data01/yarn/nm,/data02/yarn/nm</value>
> {code}
> to reproduce the issue
> open spark-shell
> run
> {code}
> $ spark-shell
> > sc.parallelize(1 to 10).count
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org