You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/05/16 18:17:02 UTC

[GitHub] [airflow] fengyu05 opened a new issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

fengyu05 opened a new issue #15883:
URL: https://github.com/apache/airflow/issues/15883


   
   
   **Apache Airflow version**:
   
   ```
   root@airflow-865c58dc6-cl754:~/airflow# airflow version
   2.0.2
   ```
   
   **Kubernetes version (if you are using kubernetes)** (use `kubectl version`):
   
   ```
   kubectl version
   Client Version: version.Info{Major:"1", Minor:"19", GitVersion:"v1.19.3", GitCommit:"1e11e4a2108024935ecfcb2912226cedeafd99df", GitTreeState:"clean", BuildDate:"2020-10-14T12:50:19Z", GoVersion:"go1.15.2", Compiler:"gc", Platform:"darwin/amd64"}
   Server Version: version.Info{Major:"1", Minor:"18+", GitVersion:"v1.18.9-eks-d1db3c", GitCommit:"d1db3c46e55f95d6a7d3e5578689371318f95ff9", GitTreeState:"clean", BuildDate:"2020-10-20T22:18:07Z", GoVersion:"go1.13.15", Compiler:"gc", Platform:"linux/amd64"}
   ```
   
   **Environment**:
   
   
   
   - **Cloud provider or hardware configuration**:
   - **OS** (e.g. from /etc/os-release):
   - **Kernel** (e.g. `uname -a`):
   - **Install tools**:
   - **Others**:
   
   **What happened**:
   After upgrade to airflow 2.0.2 kubernetes_executor always report `airflow.exceptions.AirflowException: Could not get scheduler_job_id`
   
   ```
   root@airflow-865c58dc6-cl754:~/airflow# airflow tasks run demo_bash_op bash_op 2021-05-16
   /usr/local/lib/python3.8/site-packages/airflow/configuration.py:332 DeprecationWarning: The logging_level option in [core] has been moved to the logging_level option in [logging] - the old setting has been used, but please update your config.
   /usr/local/lib/python3.8/site-packages/airflow/configuration.py:332 DeprecationWarning: The base_log_folder option in [core] has been moved to the base_log_folder option in [logging] - the old setting has been used, but please update your config.
   /usr/local/lib/python3.8/site-packages/airflow/configuration.py:332 DeprecationWarning: The statsd_on option in [scheduler] has been moved to the statsd_on option in [metrics] - the old setting has been used, but please update your config.
   [2021-05-16 18:15:53,229] {dagbag.py:451} INFO - Filling up the DagBag from /root/airflow/dags
   Running <TaskInstance: demo_bash_op.bash_op 2021-05-16T00:00:00+00:00 [None]> on host airflow-865c58dc6-cl754
   Traceback (most recent call last):
     File "/usr/local/bin/airflow", line 8, in <module>
       sys.exit(main())
     File "/usr/local/lib/python3.8/site-packages/airflow/__main__.py", line 40, in main
       args.func(args)
     File "/usr/local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
       return func(*args, **kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/utils/cli.py", line 89, in wrapper
       return f(*args, **kwargs)
     File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 235, in task_run
       _run_task_by_selected_method(args, dag, ti)
     File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 68, in _run_task_by_selected_method
       _run_task_by_executor(args, dag, ti)
     File "/usr/local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 91, in _run_task_by_executor
       executor.start()
     File "/usr/local/lib/python3.8/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
       raise AirflowException("Could not get scheduler_job_id")
   airflow.exceptions.AirflowException: Could not get scheduler_job_id
   ```
   
   **What you expected to happen**:
   
   <!-- What do you think went wrong? -->
   
   **How to reproduce it**:
   <!---
   
   As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
   
   If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
   
   ## Install minikube/kind
   
   - Minikube https://minikube.sigs.k8s.io/docs/start/
   - Kind https://kind.sigs.k8s.io/docs/user/quick-start/
   
   If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
   
   You can include images using the .md style of
   ![alt text](http://url/to/img.png)
   
   To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
   
   --->
   
   
   **Anything else we need to know**:
   
   <!--
   
   How often does this problem occur? Once? Every time etc?
   
   Any relevant logs to include? Put them here in side a detail tag:
   <details><summary>x.log</summary> lots of stuff </details>
   
   -->
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-841855234


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] fengyu05 commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

fengyu05 commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-841855995


   seems to be relative to https://github.com/apache/airflow/issues/13805 which suppose to be fixed in the 2.0.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-909748859


   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

boring-cyborg[bot] commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-841855234


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-905055696


   This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] fengyu05 commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

fengyu05 commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-841855995


   seems to be relative to https://github.com/apache/airflow/issues/13805 which suppose to be fixed in the 2.0.2


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

potiuk commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-877622489


   I believe there were a few fixes to CeleryKubernetesExecutor including some final fixes in 2.1.2 (which will be released soon). Could you plase double-check @Reddy1990 and @fengyu05 if the issue will be fixed there after we release the Release Candidate in the next few days?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

github-actions[bot] closed issue #15883:
URL: https://github.com/apache/airflow/issues/15883


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] closed issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

github-actions[bot] closed issue #15883:
URL: https://github.com/apache/airflow/issues/15883


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] github-actions[bot] commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

github-actions[bot] commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-909748859


   This issue has been closed because it has not received response from the issue author.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Reddy1990 commented on issue #15883: airflow(2.0.2): kubernetes_executor task never recived a scheduler_job_id

Posted by GitBox <gi...@apache.org>.

Reddy1990 commented on issue #15883:
URL: https://github.com/apache/airflow/issues/15883#issuecomment-877100623


   HI All, I'm still getting the same error. While using CeleryKubernetesExecutor
   
   ____ |( )_______ / /________ __
   ____ /| |_ /__ / / __ / __ _ | /| / /
   ___ ___ | / _ / _ / _ / / // / |/ |/ /
   // |// // // // _/____/|__/
   [2021-07-09 10:31:52,972] {scheduler_job.py:1251} INFO - Starting the scheduler
   [2021-07-09 10:31:52,972] {scheduler_job.py:1256} INFO - Processing each file at most -1 times
   [2021-07-09 10:31:52,978] {kubernetes_executor.py:471} INFO - Start Kubernetes executor
   [2021-07-09 10:31:52,978] {scheduler_job.py:1302} ERROR - Exception when executing SchedulerJob._run_scheduler_loop
   Traceback (most recent call last):
   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/jobs/scheduler_job.py", line 1276, in _execute
   self.executor.start()
   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/celery_kubernetes_executor.py", line 60, in start
   self.kubernetes_executor.start()
   File "/home/airflow/.local/lib/python3.6/site-packages/airflow/executors/kubernetes_executor.py", line 473, in start
   raise AirflowException("Could not get scheduler_job_id")
   airflow.exceptions.AirflowException: Could not get scheduler_job_id
   [2021-07-09 10:31:52,979] {dag_processing.py:453} WARNING - Ending without manager process.
   [2021-07-09 10:31:52,979] {scheduler_job.py:1305} INFO - Exited execute loop
   
   Environment:
   
   Using Airflow 2.0.2 Version
   Kubernetes v1.20.7
   Cloud provider or hardware configuration: AWS Linux
   
   I was using KubernetesExecutor (all was working fine , faced no issues and able to run DAG's/Jobs) and now experimenting with CeleryKubernetesExecutor the issue started showing up and I check if any job/Dag/task is causing the issue where the Scheduler looks for but I tried after deleting DAG's and placing any empty dir but the issue is still there.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org