You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/03 20:14:44 UTC
[GitHub] [airflow] bparhy opened a new issue #14590: Tasks getting Queued but not running
bparhy opened a new issue #14590:
URL: https://github.com/apache/airflow/issues/14590
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
<!--
IMPORTANT!!!
PLEASE CHECK "SIMILAR TO X EXISTING ISSUES" OPTION IF VISIBLE
NEXT TO "SUBMIT NEW ISSUE" BUTTON!!!
PLEASE CHECK IF THIS ISSUE HAS BEEN REPORTED PREVIOUSLY USING SEARCH!!!
Please complete the next sections or the issue will be closed.
These questions are the first thing we need to know to understand the context.
-->
**Apache Airflow version**: 1.10.14
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`):v1.20.2
**Environment**: airflow on k8s
- **Cloud provider or hardware configuration**: aws
- **OS** (e.g. from /etc/os-release):
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
We are currently running on 1.10.10 and everything works fine. I am working on upgrading it to 2.0.1. I am following this documentation from airflow(https://airflow.apache.org/docs/apache-airflow/stable/upgrading-to-2.html) and starting with the bridge version 1.10.14. Everything up(scheduler,WS and metadata DB) but when I am activating and triggering any jobs they are giving the below error.
ERROR - Executor reports task instance <TaskInstance: X> [queued]> finished (failed) although the task says its queued. Was the task killed externally?
No tasks is running. I am seeing a bunch of warning related to Airflow 2.0.
I am trying to see the corresponding pod but its immediately erroring out.
Please let me know if this is related to any airflow.cfg I have.
<!-- (please include exact error messages if you can) -->
**What you expected to happen**:
<!-- What do you think went wrong? -->
**How to reproduce it**:
I just get the image apache-airflow==1.10.14 and create the docker image and trying to run it.
<!---
As minimally and precisely as possible. Keep in mind we do not have access to your cluster or dags.
If you are using kubernetes, please attempt to recreate the issue using minikube or kind.
## Install minikube/kind
- Minikube https://minikube.sigs.k8s.io/docs/start/
- Kind https://kind.sigs.k8s.io/docs/user/quick-start/
If this is a UI bug, please provide a screenshot of the bug or a link to a youtube video of the bug in action
You can include images using the .md style of
To record a screencast, mac users can use QuickTime and then create an unlisted youtube video with the resulting .mov file.
--->
**Anything else we need to know**:
<!--
How often does this problem occur? Once? Every time etc?
Any relevant logs to include? Put them here in side a detail tag:
<details><summary>x.log</summary> lots of stuff </details>
-->
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-852591345
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-790025052
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-814185607
Can you try setting [delete_worker_pods](https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#delete-worker-pods) to false temporarily so you can inspect the failed pod?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] bparhy commented on issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
bparhy commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-799516277
I just see in my scheduler logs. Please let me know if this helps. The task just immediately fails and there is no logs. The pod just goes to ERROR state.
[2021-03-15 15:24:05,573] {kubernetes_executor.py:853} INFO - Deleted pod: ('test_dmart_bi_case_dly', 'proc_start', datetime.datetime(2021, 3, 3, 19, 47, 58, 415513, tzinfo=tzlocal()), 1) in namespace XXX
[2021-03-15 15:24:23,584] {scheduler_job.py:946} INFO - 1 tasks up for execution:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00 [scheduled]>
[2021-03-15 15:24:23,591] {scheduler_job.py:980} INFO - Figuring out tasks to run in Pool(name=default_pool) with 128 open slots and 1 task instances ready to be queued
[2021-03-15 15:24:23,592] {scheduler_job.py:1008} INFO - DAG hello_world has 0/16 running and queued tasks
[2021-03-15 15:24:23,598] {scheduler_job.py:1068} INFO - Setting the following tasks to queued state:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00 [scheduled]>
[2021-03-15 15:24:23,607] {scheduler_job.py:1142} INFO - Setting the following 1 tasks to queued state:
<TaskInstance: hello_world.hello_task 2021-03-15 15:21:40.190964+00:00 [queued]>
[2021-03-15 15:24:23,608] {scheduler_job.py:1178} INFO - Sending ('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40, 190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) to executor with priority 1 and queue default
[2021-03-15 15:24:23,608] {base_executor.py:58} INFO - Adding to queue: ['airflow', 'run', 'hello_world', 'hello_task', '2021-03-15T15:21:40.190964+00:00', '--local', '--pool', 'default_pool', '-sd', '/usr/local/airflow/dags/network/hello_world.py']
[2021-03-15 15:24:23,609] {kubernetes_executor.py:793} INFO - Add task ('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40, 190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1) with command ['airflow', 'run', 'hello_world', 'hello_task', '2021-03-15T15:21:40.190964+00:00', '--local', '--pool', 'default_pool', '-sd', '/usr/local/airflow/dags/network/hello_world.py'] with executor_config {'KubernetesExecutor': {'request_cpu': '200m', 'limit_cpu': '400m', 'request_memory': '558Mi', 'limit_memory': '600Mi', 'image_pull_policy': 'IfNotPresent'}}
[2021-03-15 15:24:23,613] {kubernetes_executor.py:429} INFO - Kubernetes job is (('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40, 190964, tzinfo=<TimezoneInfo [UTC, GMT, +00:00:00, STD]>), 1), ['airflow', 'run', 'hello_world', 'hello_task', '2021-03-15T15:21:40.190964+00:00', '--local', '--pool', 'default_pool', '-sd', '/usr/local/airflow/dags/network/hello_world.py'], {'api_version': 'v1',
'kind': 'Pod',
'metadata': {'annotations': None,
'cluster_name': None,
'creation_timestamp': None,
'deletion_grace_period_seconds': None,
'deletion_timestamp': None,
'finalizers': None,
'generate_name': None,
'generation': None,
'initializers': None,
'labels': None,
'managed_fields': None,
'name': None,
'namespace': None,
'owner_references': None,
'resource_version': None,
'self_link': None,
'uid': None},
'spec': {'active_deadline_seconds': None,
'affinity': None,
'automount_service_account_token': None,
'containers': [{'args': [],
'command': [],
'env': [],
'env_from': [],
'image': None,
'image_pull_policy': 'IfNotPresent',
'lifecycle': None,
'liveness_probe': None,
'name': 'base',
'ports': [],
'readiness_probe': None,
'resources': {'limits': {'cpu': '400m',
'memory': '600Mi'},
'requests': {'cpu': '200m',
'memory': '558Mi'}},
'security_context': None,
'stdin': None,
'stdin_once': None,
'termination_message_path': None,
'termination_message_policy': None,
'tty': None,
'volume_devices': None,
'volume_mounts': [],
'working_dir': None}],
'dns_config': None,
'dns_policy': None,
'enable_service_links': None,
'host_aliases': None,
'host_ipc': None,
'host_network': False,
'host_pid': None,
'hostname': None,
'image_pull_secrets': [],
'init_containers': None,
'node_name': None,
'node_selector': None,
'preemption_policy': None,
'priority': None,
'priority_class_name': None,
'readiness_gates': None,
'restart_policy': None,
'runtime_class_name': None,
'scheduler_name': None,
'security_context': None,
'service_account': None,
'service_account_name': None,
'share_process_namespace': None,
'subdomain': None,
'termination_grace_period_seconds': None,
'tolerations': None,
'volumes': []},
'status': None})
/usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:331: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod`.
security_context=_extract_security_context(pod.spec.security_context)
/usr/local/lib/python3.6/site-packages/airflow/kubernetes/pod_launcher.py:77: DeprecationWarning: Using `airflow.contrib.kubernetes.pod.Pod` is deprecated. Please use `k8s.V1Pod` instead.
pod = self._mutate_pod_backcompat(pod)
[2021-03-15 15:24:23,659] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type ADDED
[2021-03-15 15:24:23,659] {kubernetes_executor.py:369} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:23,674] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type MODIFIED
[2021-03-15 15:24:23,674] {kubernetes_executor.py:369} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:23,696] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type MODIFIED
[2021-03-15 15:24:23,697] {kubernetes_executor.py:369} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Pending
[2021-03-15 15:24:25,226] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type MODIFIED
[2021-03-15 15:24:25,226] {kubernetes_executor.py:377} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 is Running
[2021-03-15 15:24:27,353] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type MODIFIED
[2021-03-15 15:24:27,353] {kubernetes_executor.py:371} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
[2021-03-15 15:24:27,566] {kubernetes_executor.py:500} INFO - Attempting to finish pod; pod_id: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5; state: failed; labels: {'airflow-worker': '38cee1a9-9e0a-406a-a3d9-d7132b1769ff', 'airflow_version': '1.10.14', 'dag_id': 'hello_world', 'execution_date': '2021-03-15T15_21_40.190964_plus_00_00', 'kubernetes_executor': 'True', 'task_id': 'hello_task', 'try_number': '1'}
[2021-03-15 15:24:27,572] {kubernetes_executor.py:600} INFO - Found matching task hello_world-hello_task (2021-03-15 15:21:40.190964+00:00) with current state of queued
[2021-03-15 15:24:27,574] {kubernetes_executor.py:814} INFO - Changing state of (('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40, 190964, tzinfo=tzlocal()), 1), 'failed', 'helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5', 'xxx', '634337857') to failed
[2021-03-15 15:24:27,589] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type MODIFIED
[2021-03-15 15:24:27,589] {kubernetes_executor.py:371} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
[2021-03-15 15:24:27,592] {kubernetes_executor.py:853} INFO - Deleted pod: ('hello_world', 'hello_task', datetime.datetime(2021, 3, 15, 15, 21, 40, 190964, tzinfo=tzlocal()), 1) in namespace xxx
[2021-03-15 15:24:27,596] {kubernetes_executor.py:327} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 had an event of type DELETED
[2021-03-15 15:24:27,597] {kubernetes_executor.py:371} INFO - Event: helloworldhellotask-6c4037cdf28a4f3b992f6a0f87b7a6a5 Failed
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #14590:
URL: https://github.com/apache/airflow/issues/14590
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #14590: Tasks getting Queued but not running
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #14590:
URL: https://github.com/apache/airflow/issues/14590#issuecomment-848361487
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org