[jira] [Resolved] (LIVY-429) Livy skips url fragment identifier in spark.yarn.dist.archives setting

["Saisai Shao (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/LIVY-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Saisai Shao resolved LIVY-429.
------------------------------
       Resolution: Fixed
    Fix Version/s: 0.5.0

Issue resolved by pull request 71
[https://github.com/apache/incubator-livy/pull/71]

> Livy skips url fragment identifier in spark.yarn.dist.archives setting
> ----------------------------------------------------------------------
>
>                 Key: LIVY-429
>                 URL: https://issues.apache.org/jira/browse/LIVY-429
>             Project: Livy
>          Issue Type: Bug
>          Components: Server
>    Affects Versions: 0.3
>            Reporter: Artem Plotnikov
>             Fix For: 0.5.0
>
>
> When submitting Spark session via Livy API with the following parameters:
> {code}
> {
> 	"kind": "pyspark",
> 	"name": "my-app"
> 	"conf": {
> 		"spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
> 	},
> 	"archives": ["/path/to/custom-python.zip#ENVS"]
> }
> {code}
> Spark session fails with:
> {code}
> java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory
> {code}
> Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)