You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Lightning Analytics (Jira)" <ji...@apache.org> on 2019/12/28 19:51:00 UTC
[jira] [Updated] (AIRFLOW-6384) Cusomt Spark Livy Operator
[ https://issues.apache.org/jira/browse/AIRFLOW-6384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Lightning Analytics updated AIRFLOW-6384:
-----------------------------------------
Description:
Currently, Livy Rest API to submit a job to spark cluster is not available.
We have developed a Spark Livy Operator and thoroughly tested by submitting spark jobs to the cluster using the Livy Rest API. Livy provides two sessions namely, Batch and Interactive. The custom Spark Livy Operator developed by us provides Interactive session submission and additionally sends heart beats to check the status of the batch. Extensive testing has been performed using Celery executor, and all available hooks and libraries in Airflow have been integrated.
We would like to publish the operator to the Apache open source community for enhancement, use and distribution. Please inform about the steps for creating a Git pull request for Airflow.
was:
Here at lightning analytics, we always thrive for innovation. Keeping in mind the challenges posed by open source technologies in terms of integration, the aim is to build custom solutions to bridge the gap, thereby enabling ease of deployment. The orchestration tool, Airflow, presents certain challenges for integration with components in the cloud environment. Airflow has in-built support for operators which is an integral component for integration. It was observed that the operator for Livy Rest API is not available.
An operator, Spark Livy Operator, has been developed and tested that submits spark jobs to the cluster using the Livy Rest API. Livy provides two sessions namely, Batch and Interactive. The custom Spark Livy Operator provides Interactive session submission and additionally sends heart beats to check the status of the batch. Extensive testing has been performed using Celery executor, and all available hooks and libraries in Airflow has been integrated.
We would like to publish the same to the Apache open source community for enhancement, use and distribution. Please inform about the steps for creating a Git pull request for Airflow.
> Cusomt Spark Livy Operator
> --------------------------
>
> Key: AIRFLOW-6384
> URL: https://issues.apache.org/jira/browse/AIRFLOW-6384
> Project: Apache Airflow
> Issue Type: New Feature
> Components: operators
> Affects Versions: 1.10.5, 1.10.6
> Reporter: Lightning Analytics
> Priority: Minor
>
> Currently, Livy Rest API to submit a job to spark cluster is not available.
> We have developed a Spark Livy Operator and thoroughly tested by submitting spark jobs to the cluster using the Livy Rest API. Livy provides two sessions namely, Batch and Interactive. The custom Spark Livy Operator developed by us provides Interactive session submission and additionally sends heart beats to check the status of the batch. Extensive testing has been performed using Celery executor, and all available hooks and libraries in Airflow have been integrated.
> We would like to publish the operator to the Apache open source community for enhancement, use and distribution. Please inform about the steps for creating a Git pull request for Airflow.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)