[GitHub] [airflow] karthik-raparthi commented on issue #10790: Copy of [AIRFLOW-5071] JIRA: Thousands of Executor reports task instance X finished (success) although the task says its queued. Was the task killed externally?

[GitBox <gi...@apache.org>]

karthik-raparthi commented on issue #10790:
URL: https://github.com/apache/airflow/issues/10790#issuecomment-1254506118

   Hello,
   
   We are facing a similar issue but it looks likes for me a combination of EFS(Throughput-Provisioned(25MiB/s) + Worker not sending back an exception to Scheduler that unable to read the dag file and **task struck in queue state forever.**
   
   **Scheduler Log**
   `[^[[34m2022-09-16 22:38:31,848^[[0m] {^[[34mscheduler_job.py:^[[0m572} ERROR^[[0m - Executor reports task instance <TaskInstance: dagName.TaskName scheduled__2022-09-16T21:30:00+00:00 [queued]> finished (failed) although the task says its queued. (Info: None) Was the task killed externally?^[[0m`
   
   **Worker Log**
   
   `File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/slack_sdk/__init__.py", line 44, in <module>\n    from .web import WebClient  # noqa\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/slack_sdk/web/__init__.py", line 3, in <module>\n    from .client import WebClient  # noqa\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/slack_sdk/web/client.py", line 8, in <module>\n    from slack_sdk.models.views import View\n  File "<frozen importlib._bootstrap>", line 991, in _find_and_load\n  File "<frozen importlib._bootstrap>", line 971, in _find_and_load_unlocked\n  File "<frozen importlib._bootstrap>", line 914, in _find_spec\n  File "<frozen importlib._bootstrap_external>", line 1342, in find_spec\n  File "<frozen importlib._bootstrap_external>", line 1314, in _get_spec\n  File "<frozen importlib._bootstrap_external>", line 1458, in find_spec\n  File "<frozen importlib._bootstrap_external>", line 101, in _path_isfile\n  File "<frozen importlib._bootstrap
 _external>", line 93, in _path_is_mode_type\n  File "<frozen importlib._bootstrap_external>", line 87, in _path_stat\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/utils/timeout.py", line 37, in handle_timeout\n    raise AirflowTaskTimeout(self.error_message)\nairflow.exceptions.AirflowTaskTimeout: DagBag import timeout for /mnt/airflow/config2/dags/DagName.py after 30.0s.\nPlease take a look at these docs to improve your DAG import time:\n* https://airflow.apache.org/docs/apache-airflow/2.2.2/best-practices.html#top-level-python-code\n* https://airflow.apache.org/docs/apache-airflow/2.2.2/best-practices.html#reducing-dag-complexity, PID: 13809\nTraceback (most recent call last):\n  File "/mnt/airflow/airflow2/bin/airflow", line 8, in <module>\n    sys.exit(main())\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/__main__.py", line 48, in main\n    args.func(args)\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/cli/cli_p
 arser.py", line 48, in command\n    return func(*args, **kwargs)\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/utils/cli.py", line 92, in wrapper\n    return f(*args, **kwargs)\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/cli/commands/task_command.py", line 276, in task_run\n    dag = get_dag(args.subdir, args.dag_id)\n  File "/mnt/airflow/airflow2/lib64/python3.8/site-packages/airflow/utils/cli.py", line 192, in get_dag\n    raise AirflowException(\nairflow.exceptions.AirflowException: Dag \'DagName\' could not be found; either it does not exist or it failed to parse.\n'
   [2022-09-16 22:38:12,536: ERROR/ForkPoolWorker-16] execute_command encountered a CalledProcessError`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org