You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/19 20:17:01 UTC
[GitHub] [airflow] dimberman commented on a change in pull request #13405: KubernetesPodOperator Guide
dimberman commented on a change in pull request #13405:
URL: https://github.com/apache/airflow/pull/13405#discussion_r560459900
##########
File path: docs/apache-airflow-providers-cncf-kubernetes/operators.rst
##########
@@ -15,114 +15,1015 @@
specific language governing permissions and limitations
under the License.
+###########################
+KubernetesPodOperator Guide
+###########################
+
+.. contents:: :local:
.. _howto/operator:KubernetesPodOperator:
-KubernetesPodOperator
-=====================
+Kubernetes Pod Operator "Hello World!"
+======================================
+
+- Creating Task using KubernetesPodOperator which prints "Hello World [HOSTNAME]". Add ``in_cluster=False`` to
Review comment:
Hi Vivek, could you put examples of using pod_template file and pod_template_spec on top? Both of those could make this operator a lot easier to use.
##########
File path: docs/apache-airflow-providers-cncf-kubernetes/operators.rst
##########
@@ -15,114 +15,1015 @@
specific language governing permissions and limitations
under the License.
+###########################
+KubernetesPodOperator Guide
+###########################
+
+.. contents:: :local:
.. _howto/operator:KubernetesPodOperator:
-KubernetesPodOperator
-=====================
+Kubernetes Pod Operator "Hello World!"
+======================================
+
+- Creating Task using KubernetesPodOperator which prints "Hello World [HOSTNAME]". Add ``in_cluster=False`` to
+ KubernetesPodOperator constructor when running examples with minikube.
+
+
+.. code-block:: python
+
+ with DAG(dag_id="example_k8s_operator", start_date=days_ago(1),
+ schedule_interval='@once', tags=["example"]) as dag:
+ task1 = KubernetesPodOperator(task_id='k8s_pod_operator_task',
+ name='airflow_pod_operator',
+ namespace='default',
+ image='alpine',
+ cmds=["sh", "-c",
+ 'echo "Hello World from pod [$HOSTNAME]"'],
+ startup_timeout_seconds=60,
+ )
+
+- Executing / Debugging example and checking the logs
Review comment:
Is there something you were going to write here? I just see logs...
##########
File path: docs/apache-airflow-providers-cncf-kubernetes/operators.rst
##########
@@ -15,114 +15,1015 @@
specific language governing permissions and limitations
under the License.
+###########################
+KubernetesPodOperator Guide
+###########################
+
+.. contents:: :local:
.. _howto/operator:KubernetesPodOperator:
-KubernetesPodOperator
-=====================
+Kubernetes Pod Operator "Hello World!"
+======================================
+
+- Creating Task using KubernetesPodOperator which prints "Hello World [HOSTNAME]". Add ``in_cluster=False`` to
+ KubernetesPodOperator constructor when running examples with minikube.
+
+
+.. code-block:: python
+
+ with DAG(dag_id="example_k8s_operator", start_date=days_ago(1),
+ schedule_interval='@once', tags=["example"]) as dag:
+ task1 = KubernetesPodOperator(task_id='k8s_pod_operator_task',
+ name='airflow_pod_operator',
+ namespace='default',
+ image='alpine',
+ cmds=["sh", "-c",
+ 'echo "Hello World from pod [$HOSTNAME]"'],
+ startup_timeout_seconds=60,
+ )
+
+- Executing / Debugging example and checking the logs
+
+.. code-block:: bash
+
+ {taskinstance.py:1230} INFO - Exporting the following env vars:
+ AIRFLOW_CTX_DAG_OWNER=airflow
+ AIRFLOW_CTX_DAG_ID=example_k8s_operator
+ AIRFLOW_CTX_TASK_ID=k8s_pod_operator_task
+ AIRFLOW_CTX_EXECUTION_DATE=2020-12-03T00:00:00+00:00
+ AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-03T00:00:00+00:00
+
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event
+ of type Pending
+ {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of type
+ Succeeded
+ {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded
+
+ {pod_launcher.py:136} INFO - Hello World from pod [airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c]
+
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of
+ type Succeeded
+ {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded
+ {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_operator, task_id=k8s_pod_operator_task,
+ execution_date=20201203T000000, start_date=20201204T140331, end_date=20201204T140345
+ ................................................................................................................
+
+- Getting kubernetes pods using labels ``dag_id`` and ``task_id`` automatically assigned by Airflow and Describing it.
+
+.. code-block:: bash
+
+ $ kubectl get pods -l dag_id=example_k8s_operator,task_id=k8s_pod_operator_task
Review comment:
good catch on this one!
##########
File path: docs/apache-airflow-providers-cncf-kubernetes/operators.rst
##########
@@ -15,114 +15,1015 @@
specific language governing permissions and limitations
under the License.
+###########################
+KubernetesPodOperator Guide
+###########################
+
+.. contents:: :local:
.. _howto/operator:KubernetesPodOperator:
-KubernetesPodOperator
-=====================
+Kubernetes Pod Operator "Hello World!"
+======================================
+
+- Creating Task using KubernetesPodOperator which prints "Hello World [HOSTNAME]". Add ``in_cluster=False`` to
+ KubernetesPodOperator constructor when running examples with minikube.
Review comment:
I'd recommend we push people towards KinD instead of minikube.
##########
File path: docs/apache-airflow-providers-cncf-kubernetes/operators.rst
##########
@@ -15,114 +15,1015 @@
specific language governing permissions and limitations
under the License.
+###########################
+KubernetesPodOperator Guide
+###########################
+
+.. contents:: :local:
.. _howto/operator:KubernetesPodOperator:
-KubernetesPodOperator
-=====================
+Kubernetes Pod Operator "Hello World!"
+======================================
+
+- Creating Task using KubernetesPodOperator which prints "Hello World [HOSTNAME]". Add ``in_cluster=False`` to
+ KubernetesPodOperator constructor when running examples with minikube.
+
+
+.. code-block:: python
+
+ with DAG(dag_id="example_k8s_operator", start_date=days_ago(1),
+ schedule_interval='@once', tags=["example"]) as dag:
+ task1 = KubernetesPodOperator(task_id='k8s_pod_operator_task',
+ name='airflow_pod_operator',
+ namespace='default',
+ image='alpine',
+ cmds=["sh", "-c",
+ 'echo "Hello World from pod [$HOSTNAME]"'],
+ startup_timeout_seconds=60,
+ )
+
+- Executing / Debugging example and checking the logs
+
+.. code-block:: bash
+
+ {taskinstance.py:1230} INFO - Exporting the following env vars:
+ AIRFLOW_CTX_DAG_OWNER=airflow
+ AIRFLOW_CTX_DAG_ID=example_k8s_operator
+ AIRFLOW_CTX_TASK_ID=k8s_pod_operator_task
+ AIRFLOW_CTX_EXECUTION_DATE=2020-12-03T00:00:00+00:00
+ AIRFLOW_CTX_DAG_RUN_ID=backfill__2020-12-03T00:00:00+00:00
+
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event
+ of type Pending
+ {pod_launcher.py:113} WARNING - Pod not yet started: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of type
+ Succeeded
+ {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded
+
+ {pod_launcher.py:136} INFO - Hello World from pod [airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c]
+
+ {pod_launcher.py:176} INFO - Event: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c had an event of
+ type Succeeded
+ {pod_launcher.py:289} INFO - Event with job id airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c Succeeded
+ {taskinstance.py:1136} INFO - Marking task as SUCCESS. dag_id=example_k8s_operator, task_id=k8s_pod_operator_task,
+ execution_date=20201203T000000, start_date=20201204T140331, end_date=20201204T140345
+ ................................................................................................................
+
+- Getting kubernetes pods using labels ``dag_id`` and ``task_id`` automatically assigned by Airflow and Describing it.
+
+.. code-block:: bash
+
+ $ kubectl get pods -l dag_id=example_k8s_operator,task_id=k8s_pod_operator_task
-The :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator` allows
-you to create and run Pods on a Kubernetes cluster.
+ NAME READY STATUS RESTARTS AGE
+ airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c 0/1 Completed 0 14m
-.. contents::
- :depth: 1
- :local:
+ $ kubectl describe pod airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c
-.. note::
- If you use `Google Kubernetes Engine <https://cloud.google.com/kubernetes-engine/>`__, consider
- using the
- :ref:`GKEStartPodOperator <howto/operator:GKEStartPodOperator>` operator as it
- simplifies the Kubernetes authorization process.
+ Name: airflow-pod-operator-aed97ecd64854367ad7d0ff39f37859c
+ Namespace: default
+ Priority: 0
+ Node: minikube/192.168.49.2
+ Start Time: Fri, 04 Dec 2020 19:33:36 +0530
+ Labels: airflow_version=2.0.0b2
+ dag_id=example_k8s_operator
+ execution_date=2020-12-03T0000000000-767fcb862
+ kubernetes_pod_operator=True
+ task_id=k8s_pod_operator_task
+ try_number=1
+ Annotations: <none>
+ Status: Succeeded
+ IP: 172.17.0.7
+ IPs:
+ IP: 172.17.0.7
+ Containers:
+ base:
+ Container ID: docker://56c91324dc925b0bad0d60474e35d8c7eb7fad7d8410ca123b657f1416207504
+ Image: alpine
+ Image ID: docker-pullable://alpine@sha256:c0e9560cda118f9ec63ddefb4a173a2b2a0347082d7dff7dc14272e7841a5b5a
+ Port: <none>
+ Host Port: <none>
+ Command:
+ sh
+ -c
+ echo "Hello World from pod [$HOSTNAME]"
+ State: Terminated
+ Reason: Completed
+ Exit Code: 0
+ Started: Fri, 04 Dec 2020 19:33:43 +0530
+ Finished: Fri, 04 Dec 2020 19:33:43 +0530
+ Ready: False
+ Restart Count: 0
+ Environment: <none>
+ Mounts:
+ /var/run/secrets/kubernetes.io/serviceaccount from default-token-ltgdm (ro)
+ Conditions:
+ Type Status
+ Initialized True
+ Ready False
+ ContainersReady False
+ PodScheduled True
+ Volumes:
+ default-token-ltgdm:
+ Type: Secret (a volume populated by a Secret)
+ SecretName: default-token-ltgdm
+ Optional: false
+ QoS Class: BestEffort
+ Node-Selectors: <none>
+ Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
+ node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
+ Events:
+ Type Reason Age From Message
+ ---- ------ ---- ---- -------
+ Normal Scheduled 15m default-scheduler Successfully assigned default/airflow-pod-operator
+ -aed97ecd64854367ad7d0ff39f37859c to minikube
+ Normal Pulling 15m kubelet Pulling image "alpine"
+ Normal Pulled 15m kubelet Successfully pulled image "alpine" in 4.214686688s
+ Normal Created 15m kubelet Created container base
+ Normal Started 15m kubelet Started container base
-.. note::
- The :doc:`Kubernetes executor <apache-airflow:executor/kubernetes>` is **not** required to use this operator.
-How does this operator work?
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-The :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator` uses the
-Kubernetes API to launch a pod in a Kubernetes cluster. By supplying an
-image URL and a command with optional arguments, the operator uses the Kube Python Client to generate a Kubernetes API
-request that dynamically launches those individual pods.
-Users can specify a kubeconfig file using the ``config_file`` parameter, otherwise the operator will default
-to ``~/.kube/config``.
-The :class:`~airflow.providers.cncf.kubernetes.operators.kubernetes_pod.KubernetesPodOperator` enables task-level
-resource configuration and is optimal for custom Python
-dependencies that are not available through the public PyPI repository. It also allows users to supply a template
-YAML file using the ``pod_template_file`` parameter.
-Ultimately, it allows Airflow to act a job orchestrator - no matter the language those jobs are written in.
+Defining Environment Variables for Pod
+======================================
-How to use cluster ConfigMaps, Secrets, and Volumes with Pod?
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-To add ConfigMaps, Volumes, and other Kubernetes native objects, we recommend that you import the Kubernetes model API
-like this:
+- Creating Task using KubernetesPodOperator with given environment variables.
.. code-block:: python
- from kubernetes.client import models as k8s
-
-With this API object, you can have access to all Kubernetes API objects in the form of python classes.
-Using this method will ensure correctness
-and type safety. While we have removed almost all Kubernetes convenience classes, we have kept the
-:class:`~airflow.kubernetes.secret.Secret` class to simplify the process of generating secret volumes/env variables.
-
-.. exampleinclude:: /../../airflow/providers/cncf/kubernetes/example_dags/example_kubernetes.py
- :language: python
- :start-after: [START howto_operator_k8s_cluster_resources]
- :end-before: [END howto_operator_k8s_cluster_resources]
-
-Difference between ``KubernetesPodOperator`` and Kubernetes object spec
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
Review comment:
Why are you getting rid of this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org