You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Thomas Graves (JIRA)" <ji...@apache.org> on 2015/09/29 17:53:04 UTC
[jira] [Closed] (SPARK-6508) error handling issue running python in
yarn cluster mode
[ https://issues.apache.org/jira/browse/SPARK-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves closed SPARK-6508.
--------------------------------
Resolution: Duplicate
> error handling issue running python in yarn cluster mode
> ---------------------------------------------------------
>
> Key: SPARK-6508
> URL: https://issues.apache.org/jira/browse/SPARK-6508
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.3.0
> Reporter: Thomas Graves
>
> I was running python in yarn cluster mode and didn't have the SPARK_HOME envirnoment variables set. The client reported a failure of:
> ava.io.FileNotFoundException: File does not exist: hdfs://axonitered-nn1.red.ygrid.yahoo.com:8020/user/tgraves/.sparkStaging/application_1425530846697_59578/pi.py
> at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1201)
> at org.apache.hadoop.hdfs.DistributedFileSystem$19.doCall(DistributedFileSystem.java:1193)
> at org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
>
> But when you look at the application master log:
> Log Contents:
> Traceback (most recent call last):
> File "pi.py", line 29, in <module>
> sc = SparkContext(appName="PythonPi")
> File "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py", line 108, in __init__
> File "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/context.py", line 222, in _ensure_initialized
> File "/grid/11/tmp/yarn-local/usercache/tgraves/filecache/37/spark-assembly-1.3.0.0-hadoop2.6.0.6.1502061521.jar/pyspark/java_gateway.py", line 32, in launch_gateway
> File "/usr/lib64/python2.6/UserDict.py", line 22, in __getitem__
> raise KeyError(key)
> KeyError: 'SPARK_HOME'
> But the application master thought it succeeded and removed the staging directory when it shouldn't have:
> 15/03/24 14:50:07 INFO yarn.ApplicationMaster: Waiting for spark context initialization ...
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Final app status: SUCCEEDED, exitCode: 0, (reason: Shutdown hook called before final status was reported.)
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Unregistering ApplicationMaster with SUCCEEDED (diag message: Shutdown hook called before final status was reported.)
> 15/03/24 14:50:08 INFO yarn.ApplicationMaster: Deleting staging directory .sparkStaging/application_1425530846697_59578
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org