You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Zav Shotan (Jira)" <ji...@apache.org> on 2020/02/14 21:08:00 UTC

[jira] [Created] (AIRFLOW-6812) KubernetesJobOperator for airflow

Zav Shotan created AIRFLOW-6812:
-----------------------------------

             Summary: KubernetesJobOperator for airflow
                 Key: AIRFLOW-6812
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-6812
             Project: Apache Airflow
          Issue Type: Improvement
          Components: DAG, operators
    Affects Versions: 1.10.9, 1.10.3, 2.0.0
            Reporter: Zav Shotan


Dear reader,

I have created an Airflow Kubernetes Job operator that allows running a Job in kubernetes. I follows similar to the KubernetesPodOperator.

What are the steps, and what would be required to add this to airflow?

*Improves*: Adds an operator for kubernetes Jobs.

*Repo*: [https://github.com/LamaAni/KubernetesJobOperator]

*Why/when (to) use this over the KubernetesPodOperator:*
The kubernetes Job allows for more execution options such as retries/timeouts/deadlines/replicas/etc.. which cannot be defined directly on a pod.

Also, the connection between the kubernetes pod and the airflow worker can be lost, due to communication issues,
pod deletions or just pod scheduling problems in the cluster. The Kubernetes Job is a "definition" like resource, and therefore would lose its execution state only if deliberately deleted. A job will also recover automatically from pod manual deletions and pod scheduling errors.

You can find a description of the kubernetes Job resource [here](https://kubernetes.io/docs/concepts/workloads/controllers/jobs-run-to-completion/)



--
This message was sent by Atlassian Jira
(v8.3.4#803005)