You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Alex Nederlof (JIRA)" <ji...@apache.org> on 2016/01/14 14:42:39 UTC
[jira] [Created] (SPARK-12825) Spark-submit Jar URL loading fails
on redirect
Alex Nederlof created SPARK-12825:
-------------------------------------
Summary: Spark-submit Jar URL loading fails on redirect
Key: SPARK-12825
URL: https://issues.apache.org/jira/browse/SPARK-12825
Project: Spark
Issue Type: Bug
Affects Versions: 1.6.0
Reporter: Alex Nederlof
Priority: Minor
When you use spark-submit, and pass the jar as a URL, it fails when the URL redirects.
The log prints:
{code}
16/01/14 14:26:43 INFO Utils: Fetching http://myUrl/my.jar to /opt/spark/spark-1.6.0-bin-hadoop2.6/work/driver-20160114142642-0010/fetchFileTemp8495494631100918254.tmp
{code}
However, that file doesn't exist, but a file called "redirect" is created, with the appropriate content.
After that, the driver fails with
{code}
16/01/14 14:26:43 WARN Worker: Driver driver-20160114142642-0010 failed with unrecoverable exception: java.lang.Exception: Did not see expected jar my.jar in /opt/spark/spark-1.6.0-bin-hadoop2.6/work/driver-20160114142642-0010
{code}
Here's the related code:
https://github.com/apache/spark/blob/56cdbd654d54bf07a063a03a5c34c4165818eeb2/core/src/main/scala/org/apache/spark/util/Utils.scala#L583-L603
My Scala chops aren't up to this challenge, otherwise I would have made a patch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org