[jira] [Created] (LIVY-429) Livy skips url fragment identifier in spark.yarn.dist.archives setting

["Artem Plotnikov (JIRA)" <ji...@apache.org>]

Artem Plotnikov created LIVY-429:
------------------------------------

             Summary: Livy skips url fragment identifier in spark.yarn.dist.archives setting
                 Key: LIVY-429
                 URL: https://issues.apache.org/jira/browse/LIVY-429
             Project: Livy
          Issue Type: Bug
          Components: API
    Affects Versions: 0.3
            Reporter: Artem Plotnikov


When submitting Spark session via Livy API with the following parameters:
```
{
	"kind": "pyspark",
	"name": "my-app"
	"conf": {
		"spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
	},
	"archives": ["/path/to/custom-python.zip#ENVS"]
}
```
Spark session fails with:
```
java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory
```
Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)