[jira] [Commented] (AIRFLOW-1319) Fix misleading SparkSubmitOperator and SparkSubmitHook docstring

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-1319?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16359589#comment-16359589 ] 

ASF subversion and git services commented on AIRFLOW-1319:
----------------------------------------------------------

Commit fe73f22153118601a22300b21c15975632321338 in incubator-airflow's branch refs/heads/master from chrissng
[ https://git-wip-us.apache.org/repos/asf?p=incubator-airflow.git;h=fe73f22 ]

[AIRFLOW-1319] Fix misleading SparkSubmitOperator and SparkSubmitHook docstring

- Reword docstring to reduce ambigiuity in the
usage of `--files` option.
- See JIRA issue
https://issues.apache.org/jira/browse/AIRFLOW-1319
for more info.

Closes #2377 from chrissng/airflow-1319


> Fix misleading SparkSubmitOperator and SparkSubmitHook docstring
> ----------------------------------------------------------------
>
>                 Key: AIRFLOW-1319
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1319
>             Project: Apache Airflow
>          Issue Type: Improvement
>          Components: contrib
>            Reporter: Chris Sng
>            Assignee: Chris Sng
>            Priority: Trivial
>              Labels: documentation
>             Fix For: 2.0.0
>
>   Original Estimate: 5m
>  Remaining Estimate: 5m
>
> In the community-contributed Spark submit hook and operator, it support {{spark-submit}}'s {{\--files}} command line option. The {{\--files}} option is used to submit file to each executor to be used. A good example of such files are serialized objects.
> However, in both docstrings (https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/operators/spark_submit_operator.py#L37 and https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L36), it provided {{hive-site.xml}} as an example. This may mislead less-informed developers into assuming that Hive configuration files can be submitted to the cluster in this manner. According to Apache Hive's documentation, hive configuration files are located in the directory located in the {{HIVE_CONF_DIR}} environment variable.
> I propose excluding this example from the docstrings.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)