[jira] [Updated] (AIRFLOW-6384) Custom Spark Livy Operator

["Lightning Analytics (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/AIRFLOW-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Lightning Analytics updated AIRFLOW-6384:
-----------------------------------------
    Summary: Custom Spark Livy Operator  (was: Cusomt Spark Livy Operator)

> Custom Spark Livy Operator
> --------------------------
>
>                 Key: AIRFLOW-6384
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6384
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: operators
>    Affects Versions: 1.10.5, 1.10.6
>            Reporter: Lightning Analytics
>            Priority: Minor
>
> Currently, Livy Rest API to submit a job to spark cluster is not available.
> We have developed a Spark Livy Operator and thoroughly tested by submitting spark jobs to the cluster using the Livy Rest API. Livy provides two sessions namely, Batch and Interactive. The custom Spark Livy Operator developed by us provides Interactive session submission and additionally sends heart beats to check the status of the batch. Extensive testing has been performed using Celery executor, and all available hooks and libraries in Airflow have been integrated.
> We would like to publish the operator to the Apache open source community for enhancement, use and distribution. Please inform about the steps for creating a Git pull request for Airflow.
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)