[jira] [Commented] (AIRFLOW-4188) Customized LivySparkOperator

["Angad Singh (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-4188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16947239#comment-16947239 ] 

Angad Singh commented on AIRFLOW-4188:
--------------------------------------

hey [~debdhar] can you share a link to the LivySparkOperator you've written, perhaps on your company/personal github?

> Customized LivySparkOperator
> ----------------------------
>
>                 Key: AIRFLOW-4188
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-4188
>             Project: Apache Airflow
>          Issue Type: New Feature
>          Components: DagRun
>            Reporter: Debashish Dhar
>            Priority: Minor
>              Labels: features
>
> Airflow provides operators for various APIs, however there is no operator available for Livy Rest API. Here at Deloitte we have developed and tested an operator which can connect to Livy and submit jobs on EMR. Livy provides two sessions, Batch and Interactive, the custom Livy spark operator provides Interactive session submission as well as sends heart beats to check the batch status. The operator has been successfully implemented at one of our current client project.
>  We would like to publish our code to Airflow GitHub to enable further use, research, change and distribution. The code is a standalone operator developed using hooks and libraries present in the Airflow implementation. Please let us know the next steps to create a Pull Request for the code  and submit to the open source community.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)