You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "mayorblock (via GitHub)" <gi...@apache.org> on 2024/08/23 21:43:13 UTC

[I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

mayorblock opened a new issue, #41706:
URL: https://github.com/apache/airflow/issues/41706

   ### Apache Airflow version
   
   2.10.0
   
   ### If "Other Airflow 2 version" selected, which one?
   
   _No response_
   
   ### What happened?
   
   I try to run a Kubernetes Pod Operator (v8.4.0) with `deferrable=True` (and `do_xcom_push=True` which was also an issue in [earlier versions](https://github.com/apache/airflow/issues/32458) but not now). I define my kubernetes cluster connection in the code using `airflow.models.connection.Connection` and supply it via an [environment variable](https://airflow.apache.org/docs/apache-airflow/stable/howto/connection.html#storing-connections-in-environment-variables).
   
   After all containers (init-container, sidecar and main container) have completed succesfully I get
   ```python
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 767, in _execute_task
       result = _execute_callable(context=context, **execute_callable_kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/taskinstance.py", line 733, in _execute_callable
       return ExecutionCallableRunner(
              ^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/utils/operator_helpers.py", line 252, in run
       return self.func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/models/baseoperator.py", line 1792, in resume_execution
       return execute_callable(context)
              ^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/airflow/.local/lib/python3.12/site-packages/airflow/providers/cncf/kubernetes/operators/pod.py", line 773, in trigger_reentry
       raise AirflowException(message)
   airflow.exceptions.AirflowException: Traceback (most recent call last):
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/triggers/pod.py", line 162, in run
       state = await self._wait_for_pod_start()
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/triggers/pod.py", line 223, in _wait_for_pod_start
       pod = await self.hook.get_pod(self.pod_name, self.pod_namespace)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 747, in get_pod
       async with self.get_conn() as connection:
     File "/usr/local/lib/python3.12/contextlib.py", line 210, in __aenter__
       return await anext(self.gen)
              ^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 734, in get_conn
       kube_client = await self._load_config() or async_client.ApiClient()
                     ^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 664, in _load_config
       in_cluster = self._coalesce_param(self.in_cluster, await self._get_field("in_cluster"))
                                                          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 724, in _get_field
       extras = await self.get_conn_extras()
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 712, in get_conn_extras
       connection = await sync_to_async(self.get_connection)(self.conn_id)
                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/asgiref/sync.py", line 468, in __call__
       ret = await asyncio.shield(exec_coro)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/usr/local/lib/python3.12/concurrent/futures/thread.py", line 58, in run
       result = self.fn(*self.args, **self.kwargs)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/asgiref/sync.py", line 522, in thread_handler
       return func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/providers/cncf/kubernetes/hooks/kubernetes.py", line 169, in get_connection
       return super().get_connection(conn_id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/hooks/base.py", line 83, in get_connection
       conn = Connection.get_connection_from_secrets(conn_id)
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "/home/***/.local/lib/python3.12/site-packages/***/models/connection.py", line 537, in get_connection_from_secrets
       raise AirflowNotFoundException(f"The conn_id `{conn_id}` isn't defined")
   ***.exceptions.AirflowNotFoundException: The conn_id `k8s_conn_id` isn't defined
   ```
   
   Now if I:
   - set `deferrable=False` it works with environment variable connection 
   - make an explicit k8s cluster connection in the UI, then `deferrable=True` works
   <img width="703" alt="image" src="https://github.com/user-attachments/assets/7cb40a9f-d3c0-4371-9672-e92a8a060c9b">
   
   
   ### What you think should happen instead?
   
   supplying connection via environment variable as described in docs should work exactly like connections via UI
   
   ### How to reproduce
   
   make a kubernetes cluster connection 
   ```python
   # create connection
   import os
   from airflow import DAG
   from airflow.operators.bash import BashOperator
   from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import (
       KubernetesPodOperator,
   )
   
   extra = {
           "kube_config": <kube_config to your cluser>,
           "namespace": <your namespace>,  
           "in_cluster": False,
       }
   conn = Connection(conn_id="k8s_conn_id", conn_type="kubernetes", description="k8s connection", extra=extra)
   os.environ[f"AIRFLOW_CONN_{conn.conn_id.upper()}"] = conn.get_uri()
   
   # define minimal DAG with  deferrable=True and do_xcom_push=True
   with DAG( dag_id="k8s-example") as dag:
       k = KubernetesPodOperator(
           kubernetes_conn_id="k8s_conn_id",
           name="k8s-pod",
           cmds=["bash", "-cx"],
           arguments=["mkdir -p /airflow/xcom/;cat /tmp/mnt/hello.txt > /airflow/xcom/return.json"],
           task_id="k8s-pod",
           startup_timeout_seconds=1000,
           do_xcom_push=True,
           pod_template_file="k8s_tempalte.yaml"),
           on_finish_action="keep_pod", 
           deferrable=True
       )
   
       b = BashOperator(
           bash_command="echo \"{{ task_instance.xcom_pull('k8s-pod')[0] }}\"",
           task_id="pod_task_xcom_result",
       )
   
       k >> b
   ```
   and the manifest
   ```yaml
   apiVersion: v1
   kind: Pod
   spec:
     initContainers:
       - name: init-container
         image: "ubuntu:latest"
         command: ["bash", "-cx"]
         args: ["echo '[1,2,3,4]' > /tmp/mnt/hello.txt"]
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 100m
             memory: 1Gi
         volumeMounts:
           - name: shared-volume
             mountPath: "/tmp/mnt"
     containers:
       - name: base
         image: "ubuntu:latest"
         imagePullPolicy: IfNotPresent
         ports: []
         resources:
           limits:
             cpu: 500m
             memory: 1Gi
           requests:
             cpu: 100m
             memory: 1Gi
         volumeMounts:
           - name: shared-volume
             mountPath: "/tmp/mnt"
     restartPolicy: Never
     volumes:
     - name: shared-volume
       emptyDir: {}
   ```
   
   
   
   ### Operating System
   
   docker
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-cncf-kubernetes==8.4.0
   apache-airflow-providers-microsoft-azure==10.3.0
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   I am running Docker locally with the official [docker-compose](https://airflow.apache.org/docs/apache-airflow/2.10.0/docker-compose.yaml) and deploying pods on Azure Kubernetes Services
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2307848038

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2381033294

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "mayorblock (via GitHub)" <gi...@apache.org>.
mayorblock commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2309385064

   @tirkarthi good catch :) but unfortunately just a typo from during transfer from inhouse code to to this example. 
   
   My issue still stands. With kubernetes cluster connection served via environment variable the KPO with `deferrable=True` fails even when the pod runs successfully. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "mayorblock (via GitHub)" <gi...@apache.org>.
mayorblock commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2351107508

   Hi @RNHTTR, the bot message seems to imply that I should somehow respond. I am unsure what I am supposed to do? You added a pending-response tag; what does it imply?
   
   thank you in advance


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "mayorblock (via GitHub)" <gi...@apache.org>.
mayorblock commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2317730031

   Hi @RNHTTR, politely nudging to hear if there is anything I can do to advance the issue? I am very willing to contribute but will need some guidance :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2347443442

   This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "romsharon98 (via GitHub)" <gi...@apache.org>.
romsharon98 closed issue #41706: kubernetes connection defined via env variable does not work with deferrable=True
URL: https://github.com/apache/airflow/issues/41706


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2307878364

   Connection name uses hypen k8s-conn-id. Should it be k8s_conn_id ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] kubernetes connection defined via env variable does not work with deferrable=True [airflow]

Posted by "mayorblock (via GitHub)" <gi...@apache.org>.
mayorblock commented on issue #41706:
URL: https://github.com/apache/airflow/issues/41706#issuecomment-2424702731

   Hi @RNHTTR , no problem :)
   
   - yes, it does work with `deferrable=False` (or with connection from airflow DB)
   - yes, the issue occur for the latest airflow and kubernetes provider versions
   
   I will step through and try to see where the chain falls off. I'll be back (sometime)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org