You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Andrew Or (JIRA)" <ji...@apache.org> on 2014/12/05 06:59:12 UTC
[jira] [Updated] (SPARK-4757) Yarn-client failed to start due to
Wrong FS error in distCacheMgr.addResource
[ https://issues.apache.org/jira/browse/SPARK-4757?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Andrew Or updated SPARK-4757:
-----------------------------
Affects Version/s: 1.3.0
> Yarn-client failed to start due to Wrong FS error in distCacheMgr.addResource
> -----------------------------------------------------------------------------
>
> Key: SPARK-4757
> URL: https://issues.apache.org/jira/browse/SPARK-4757
> Project: Spark
> Issue Type: Bug
> Components: YARN
> Affects Versions: 1.2.0, 1.3.0
> Reporter: Jianshi Huang
> Fix For: 1.2.0, 1.3.0
>
>
> I got the following error during Spark startup (Yarn-client mode):
> 14/12/04 19:33:58 INFO Client: Uploading resource file:/x/home/jianshuang/spark/spark-latest/lib/datanucleus-api-jdo-3.2.6.jar -> hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar
> java.lang.IllegalArgumentException: Wrong FS: hdfs://stampy/user/jianshuang/.sparkStaging/application_1404410683830_531767/datanucleus-api-jdo-3.2.6.jar, expected: file:///
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:643)
> at org.apache.hadoop.fs.RawLocalFileSystem.pathToFile(RawLocalFileSystem.java:79)
> at org.apache.hadoop.fs.RawLocalFileSystem.deprecatedGetFileStatus(RawLocalFileSystem.java:506)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:724)
> at org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:501)
> at org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:397)
> at org.apache.spark.deploy.yarn.ClientDistributedCacheManager.addResource(ClientDistributedCacheManager.scala:67)
> at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:257)
> at org.apache.spark.deploy.yarn.ClientBase$$anonfun$prepareLocalResources$5.apply(ClientBase.scala:242)
> at scala.Option.foreach(Option.scala:236)
> at org.apache.spark.deploy.yarn.ClientBase$class.prepareLocalResources(ClientBase.scala:242)
> at org.apache.spark.deploy.yarn.Client.prepareLocalResources(Client.scala:35)
> at org.apache.spark.deploy.yarn.ClientBase$class.createContainerLaunchContext(ClientBase.scala:350)
> at org.apache.spark.deploy.yarn.Client.createContainerLaunchContext(Client.scala:35)
> at org.apache.spark.deploy.yarn.Client.submitApplication(Client.scala:80)
> at org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:57)
> at org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:140)
> at org.apache.spark.SparkContext.<init>(SparkContext.scala:335)
> at org.apache.spark.repl.SparkILoop.createSparkContext(SparkILoop.scala:986)
> at $iwC$$iwC.<init>(<console>:9)
> at $iwC.<init>(<console>:18)
> at <init>(<console>:20)
> at .<init>(<console>:24)
> According to Liancheng and Andrew, this hotfix might be the root cause:
> https://github.com/apache/spark/commit/38cb2c3a36a5c9ead4494cbc3dde008c2f0698ce
> Jianshi
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org