[jira] [Resolved] (SPARK-2325) Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly

["Josh Rosen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-2325?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Josh Rosen resolved SPARK-2325.
-------------------------------

    Resolution: Duplicate

Resolving as duplicate of SPARK-2974, which I'm working on fixing.

> Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly
> -----------------------------------------------------------------------------------------------------------------
>
>                 Key: SPARK-2325
>                 URL: https://issues.apache.org/jira/browse/SPARK-2325
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>            Reporter: YanTang Zhai
>
> If the first directory of spark.local.dir is bad, application will exit with the exception:
> Exception in thread "main" java.io.IOException: Failed to create a temp directory (under /data1/sparkenv/local) after 10 attempts!
>         at org.apache.spark.util.Utils$.createTempDir(Utils.scala:258)
>         at org.apache.spark.broadcast.HttpBroadcast$.createServer(HttpBroadcast.scala:154)
>         at org.apache.spark.broadcast.HttpBroadcast$.initialize(HttpBroadcast.scala:127)
>         at org.apache.spark.broadcast.HttpBroadcastFactory.initialize(HttpBroadcastFactory.scala:31)
>         at org.apache.spark.broadcast.BroadcastManager.initialize(BroadcastManager.scala:48)
>         at org.apache.spark.broadcast.BroadcastManager.<init>(BroadcastManager.scala:35)
>         at org.apache.spark.SparkEnv$.create(SparkEnv.scala:218)
>         at org.apache.spark.SparkContext.<init>(SparkContext.scala:202)
>         at JobTaskJoin$.main(JobTaskJoin.scala:9)
>         at JobTaskJoin.main(JobTaskJoin.scala)
>         at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>         at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:601)
>         at org.apache.spark.deploy.SparkSubmit$.launch(SparkSubmit.scala:292)
>         at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:55)
>         at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
> Utils.getLocalDir had better check the directory and choose a good one instead of choosing the first one directly. For example, spark.local.dir is /data1/sparkenv/local,/data2/sparkenv/local. The disk data1 is bad while the disk data2 is good, we could choose the data2 not data1.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org