[jira] [Assigned] (AIRFLOW-1331) Contrib.SparkSubmitOperator should allow --packages parameter

["chie hayashida (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-1331?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

chie hayashida reassigned AIRFLOW-1331:
---------------------------------------

    Assignee: chie hayashida

> Contrib.SparkSubmitOperator should allow --packages parameter
> -------------------------------------------------------------
>
>                 Key: AIRFLOW-1331
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1331
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: contrib
>            Reporter: manuel garrido
>            Assignee: chie hayashida
>             Fix For: 1.9.0
>
>
> Right now SparkSubmitOperator (and its related hook SparkSubmitHook) does not allow for the parameter packages, an option very useful to pull packages from the spark-packages repository.
> I am not an expert by no means , but given how SparkSubmitHook builds the command to submit a spark job this could be as easy as adding 
> {code:python}
>         if self._jars:
>             connection_cmd += ["--jars", self._jars]
> {code}
> Right under [this line](https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L167), as well as adding the *packages* parameter (defaulting to None) both in the SparkSubmitHook and SparkSubmitOperator init methods (basically, anywhere where the jars parameter is called).
> To be honest I would not mind doing a pull request to fix this, however I am not knowledgeable enough both about Airflow and how the Contribution guidelines are setup. I the community thinks this could be an easy fix that a newbie like me can do (i do believe this) then please let me know and I will do my best.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)