[jira] [Commented] (AIRFLOW-3647) Contributed SparkSubmitOperator doesn't honor --archives configuration

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-3647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16784932#comment-16784932 ] 

ASF subversion and git services commented on AIRFLOW-3647:
----------------------------------------------------------

Commit 9c11ac92fdda0e8ba623cb0be543ff5058cd9ce2 in airflow's branch refs/heads/v1-10-stable from Penumbra69
[ https://gitbox.apache.org/repos/asf?p=airflow.git;h=9c11ac9 ]

[AIRFLOW-3647] Add archives config option to SparkSubmitOperator (#4467)

To enable to spark behavior of transporting and extracting an archive
on job launch,  making the _contents_ of the archive available to the
driver as well as the workers (not just the jar or archive as a zip
file) - this configuration attribute is necessary.

This is required if you have no ability to modify the Python env on
the worker / driver nodes, but you wish to use versions, modules, or
features not installed.

We transport a full Python 3.5 environment to our CDH cluster using
this option and the alias "#PYTHON" paired an additional configuration
to spark to use it:

    --archives "hdfs:///user/myuser/my_python_env.zip#PYTHON"
    --conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PYTHON/python35/bin/python3"

> Contributed SparkSubmitOperator doesn't honor --archives configuration
> ----------------------------------------------------------------------
>
>                 Key: AIRFLOW-3647
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-3647
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>    Affects Versions: 1.10.1
>         Environment: Linux (RHEL 7)
> Python 3.5 (using a virtual environment)
> spark-2.1.3-bin-hadoop26
> Airflow 1.10.1
> CDH 5.14 Hadoop [Yarn] cluster (no end user / dev modifications allowed)
>            Reporter: Ken Melms
>            Priority: Minor
>              Labels: easyfix, newbie
>             Fix For: 1.10.3
>
>   Original Estimate: 1h
>  Remaining Estimate: 1h
>
> The contributed SparkSubmitOperator has no ability to honor the spark-submit configuration field "--archives" which is treated subtly different than "files" or "-py-files" in that it will unzip the archive into the application's working directory, and can optionally add an alias to the unzipped folder so that you can refer to it elsewhere in your submission.
> EG:
> spark-submit  --archives=hdfs:////user/someone/python35_venv.zip#PYTHON --conf "spark.yarn.appMasterEnv.PYSPARK_PYTHON=./PYTHON/python35/bin/python3" run_me.py  
> In our case - this behavior allows for multiple python virtual environments to be sourced from HDFS without incurring the penalty of pushing the whole python virtual env to the cluster each submission.  This solves (for us) using python-based spark jobs on a cluster that the end user has no ability to define the python modules in use.
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)