You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@livy.apache.org by "Artem Plotnikov (JIRA)" <ji...@apache.org> on 2017/12/26 10:02:00 UTC
[jira] [Updated] (LIVY-429) Livy skips url fragment identifier in
spark.yarn.dist.archives setting
[ https://issues.apache.org/jira/browse/LIVY-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Artem Plotnikov updated LIVY-429:
---------------------------------
Description:
When submitting Spark session via Livy API with the following parameters:
{code}
{
"kind": "pyspark",
"name": "my-app"
"conf": {
"spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
},
"archives": ["/path/to/custom-python.zip#ENVS"]
}
{code}
Spark session fails with:
{code}
java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory
{code}
Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.
was:
When submitting Spark session via Livy API with the following parameters:
```
{
"kind": "pyspark",
"name": "my-app"
"conf": {
"spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
},
"archives": ["/path/to/custom-python.zip#ENVS"]
}
```
Spark session fails with:
```
java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory
```
Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.
> Livy skips url fragment identifier in spark.yarn.dist.archives setting
> ----------------------------------------------------------------------
>
> Key: LIVY-429
> URL: https://issues.apache.org/jira/browse/LIVY-429
> Project: Livy
> Issue Type: Bug
> Components: API
> Affects Versions: 0.3
> Reporter: Artem Plotnikov
>
> When submitting Spark session via Livy API with the following parameters:
> {code}
> {
> "kind": "pyspark",
> "name": "my-app"
> "conf": {
> "spark.yarn.appMasterEnv.PYSPARK_PYTHON": "./ENVS/custom-python/bin/python"
> },
> "archives": ["/path/to/custom-python.zip#ENVS"]
> }
> {code}
> Spark session fails with:
> {code}
> java.io.IOException: Cannot run program "./ENVS/custom-python/bin/python": error=2, No such file or directory
> {code}
> Because Livy uses java.net.URI#getPath method under the hood, which just skips URI fragment identifier part and Spark is being submitted with spark.yarn.dist.archives=/path/to/custom-python.zip and does not even extracts the archive in this case.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)