You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/10/27 20:20:15 UTC
[GitHub] [airflow] gurjit-sandhu opened a new issue, #27332: airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found
gurjit-sandhu opened a new issue, #27332:
URL: https://github.com/apache/airflow/issues/27332
### Official Helm Chart version
1.6.0
### Apache Airflow version
airflow 2
### Kubernetes Version
1.22
### Helm Chart configuration
i have set PYTHONPATH as below in extraENV and verified after login to worker pod
extraEnv: |
- name: PYTHONPATH
value: "/opt/airflow/dags:/opt/airflow"
also have setup PYTHONPATH in docker image
# Setting python path for importing dag modules
ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"
### Docker Image customisations
also have setup PYTHONPATH in docker image
# Setting python path for importing dag modules
ENV PYTHONPATH="/opt/airflow/dags:/opt/airflow"
### What happened
dags are showing up on UI however after executing it get terminated and below are logs from worker showing its not able to find python modules -- if we re-run same tasks it succeeds however randomly it fails with below error module not found
_execute_in_fork(command_to_exec, celery_task_id)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 108, in _execute_in_fork
raise AirflowException(msg)
airflow.exceptions.AirflowException: Celery command failed on host: airflow-worker-1.airflow-worker.airflow.svc.cluster.local with celery_task_id f672c24c-d54c-4401-a1c1-8892d54f90f4
[2022-10-27 18:48:32,001: INFO/MainProcess] Task airflow.executors.celery_executor.execute_command[1fdb829f-755a-4ac0-98c5-362e6b6c8a44] received
[2022-10-27 18:48:32,011: INFO/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Executing command in Celery: ['airflow', 'tasks', 'run', 'yipit_fsv_data_export_dag', 'verify_all_exports_were_successful', 'manual__2022-10-27T13:42:55-05:00', '--local', '--subdir', 'DAGS_FOLDER/partner_exports/data_export_dag_builder.py']
[2022-10-27 18:48:32,087: INFO/ForkPoolWorker-15] Filling up the DagBag from /opt/airflow/dags/partner_exports/data_export_dag_builder.py
[2022-10-27 18:48:32,144: ERROR/ForkPoolWorker-15] Failed to import: /opt/airflow/dags/partner_exports/data_export_dag_builder.py
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/dagbag.py", line 317, in parse
loader.exec_module(new_module)
File "<frozen importlib._bootstrap_external>", line 843, in exec_module
File "<frozen importlib._bootstrap>", line 219, in _call_with_frames_removed
File "/opt/airflow/dags/partner_exports/data_export_dag_builder.py", line 20, in <module>
from utils.generic_callbacks import on_failure_callback, on_success_callback
ModuleNotFoundError: No module named 'utils'
[2022-10-27 18:48:32,145: ERROR/ForkPoolWorker-15] [1fdb829f-755a-4ac0-98c5-362e6b6c8a44] Failed to execute task Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse..
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 128, in _execute_in_fork
args.func(args)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 51, in command
return func(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 99, in wrapper
return f(*args, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 360, in task_run
dag = get_dag(args.subdir, args.dag_id)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 203, in get_dag
raise AirflowException(
airflow.exceptions.AirflowException: Dag 'yipit_fsv_data_export_dag' could not be found; either it does not exist or it failed to parse.
### What you think should happen instead
since PYTHONPATH is set in both docker and helm chart environment variables - dag should be able to find modules
### How to reproduce
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #27332: airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #27332:
URL: https://github.com/apache/airflow/issues/27332#issuecomment-1294020775
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #27332: airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #27332: airflow tasks are getting randomly terminated with no errors in UI and logs on worker shows module not found
URL: https://github.com/apache/airflow/issues/27332
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org