[jira] [Created] (AIRFLOW-5809) Kubernetes executor never returns when run with 'airflow run ...`

["Francesco Macagno (Jira)" <ji...@apache.org>]

Francesco Macagno created AIRFLOW-5809:
------------------------------------------

             Summary: Kubernetes executor never returns when run with 'airflow run ...`
                 Key: AIRFLOW-5809
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5809
             Project: Apache Airflow
          Issue Type: Bug
          Components: executor-kubernetes
    Affects Versions: 1.10.3
            Reporter: Francesco Macagno
            Assignee: Daniel Imberman


When running a task using `airflow run dag_id task_id date` from a minikube instance, the command never returns, even after the child pod has completed and been deleted.

 
{code:java}
airflow@airflow-85464bb66-c4j9m:~$ airflow run k8s-test python 20191010 -int
[2019-10-26 13:31:32,258] {settings.py:182} INFO - settings.configure_orm(): Using pool settings. pool_size=5, pool_recycle=1800, pid=38
/usr/local/lib/python3.6/site-packages/psycopg2/__init__.py:144: UserWarning: The psycopg2 wheel package will be renamed from release 2.8; in order to keep installing from binary please use "pip install psycopg2-binary" instead. For details see: <http://initd.org/psycopg/docs/install.html#binary-install-from-pypi>.
  """)
[2019-10-26 13:31:33,745] {__init__.py:51} INFO - Using executor KubernetesExecutor
[2019-10-26 13:31:34,019] {__init__.py:305} INFO - Filling up the DagBag from /usr/local/airflow/dags
[2019-10-26 13:31:34,235] {cli.py:517} INFO - Running <TaskInstance: k8s-test.python 2019-10-10T00:00:00+00:00 [success]> on host airflow-85464bb66-c4j9m
[2019-10-26 13:31:34,235] {kubernetes_executor.py:690} INFO - Start Kubernetes executor
[2019-10-26 13:31:34,338] {kubernetes_executor.py:304} INFO - Event: and now my watch begins starting at resource_version: 0
[2019-10-26 13:31:34,371] {kubernetes_executor.py:627} INFO - When executor started up, found 0 queued task instances
Sending to executor.
[2019-10-26 13:31:34,378] {base_executor.py:59} INFO - Adding to queue: ['airflow', 'run', 'k8s-test', 'python', '2019-10-10T00:00:00+00:00', '--local', '-sd', 'DAGS_FOLDER/hdfs_example.py']
[2019-10-26 13:31:34,381] {kubernetes_executor.py:711} INFO - Add task ('k8s-test', 'python', <Pendulum [2019-10-10T00:00:00+00:00]>, 2) with command ['airflow', 'run', 'k8s-test', 'python', '2019-10-10T00:00:00+00:00', '--local', '-sd', 'DAGS_FOLDER/hdfs_example.py'] with executor_config {'KubernetesExecutor': {'image': 'registry.prod.factual.com/airflow-cdh5:1', 'image_pull_policy': 'Always', 'annotations': {'injector.factual.com/request': 'airflow-kinit'}}}
[2019-10-26 13:31:34,383] {kubernetes_executor.py:405} INFO - Kubernetes job is (('k8s-test', 'python', <Pendulum [2019-10-10T00:00:00+00:00]>, 2), ['airflow', 'run', 'k8s-test', 'python', '2019-10-10T00:00:00+00:00', '--local', '-sd', 'DAGS_FOLDER/hdfs_example.py'], KubernetesExecutorConfig(image=registry.prod.factual.com/airflow-cdh5:1, image_pull_policy=Always, request_memory=None, request_cpu=None, limit_memory=None, limit_cpu=None, gcp_service_account_key=None, node_selectors=None, affinity=None, annotations={'injector.factual.com/request': 'airflow-kinit'}, volumes=[], volume_mounts=[], tolerations=None))
/usr/local/lib/python3.6/site-packages/airflow/contrib/kubernetes/kubernetes_request_factory/pod_request_factory.py:45: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
  req = yaml.load(self._yaml)
[2019-10-26 13:31:34,408] {kubernetes_executor.py:318} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e had an event of type ADDED
[2019-10-26 13:31:34,409] {kubernetes_executor.py:350} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e Pending
[2019-10-26 13:31:34,422] {kubernetes_executor.py:770} INFO - Shutting down Kubernetes executor
[2019-10-26 13:31:34,427] {kubernetes_executor.py:318} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e had an event of type MODIFIED
[2019-10-26 13:31:34,427] {kubernetes_executor.py:350} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e Pending
[2019-10-26 13:31:34,438] {kubernetes_executor.py:318} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e had an event of type MODIFIED
[2019-10-26 13:31:34,439] {kubernetes_executor.py:350} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e Pending
[2019-10-26 13:31:37,402] {kubernetes_executor.py:318} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e had an event of type MODIFIED
[2019-10-26 13:31:37,403] {kubernetes_executor.py:358} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e is Running
[2019-10-26 13:31:40,443] {kubernetes_executor.py:318} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e had an event of type MODIFIED
[2019-10-26 13:31:40,444] {kubernetes_executor.py:355} INFO - Event: k8stestpython-b8743ba7b8904b8482e364dccfdd879e Succeeded
{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)