You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/02/09 12:36:49 UTC
[GitHub] [airflow] zorzigio opened a new issue #21459: Tasks failing often with no logs
zorzigio opened a new issue #21459:
URL: https://github.com/apache/airflow/issues/21459
### Apache Airflow version
2.2.3 (latest released)
### What happened
I often have tasks failing in Airflow and no logs are produced.
If I clear the tasks, it will then run successfully.
Some tasks are stuck in queue state and even if cleared, will get stuck again in queue state.
Happy to provide more details if needed.
### What you expected to happen
Tasks running successfully
### How to reproduce
_No response_
### Operating System
Debian GNU/Linux 10 (buster)
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==2.4.0
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==2.2.0
apache-airflow-providers-docker==2.3.0
apache-airflow-providers-elasticsearch==2.1.0
apache-airflow-providers-ftp==2.0.1
apache-airflow-providers-google==6.2.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http==2.0.1
apache-airflow-providers-imap==2.0.1
apache-airflow-providers-microsoft-azure==3.4.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.4.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.3.0
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-sqlite==2.0.1
apache-airflow-providers-ssh==2.3.0
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
I am using the Celery Executor with KEDA enables on Kubernetes. The nodepool is set on autoscaling.
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] zorzigio commented on issue #21459: Tasks failing often with no logs
Posted by GitBox <gi...@apache.org>.
zorzigio commented on issue #21459:
URL: https://github.com/apache/airflow/issues/21459#issuecomment-1034163857
Not sure if this is related, but I am also seeing an "Import error" every now and then, apparently appearing out of thin air
```bash
Broken DAG: [/opt/airflow/dags/repo/dags/documents/DEV/file_pipeline.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 574, in serialize_operator
serialize_op['params'] = cls._serialize_params_dict(op.params)
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 447, in _serialize_params_dict
if f'{v.__module__}.{v.__class__.__name__}' == 'airflow.models.param.Param':
AttributeError: 'str' object has no attribute '__module__'
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 935, in to_dict
json_dict = {"__version": cls.SERIALIZER_VERSION, "dag": cls.serialize_dag(var)}
File "/home/airflow/.local/lib/python3.9/site-packages/airflow/serialization/serialized_objects.py", line 847, in serialize_dag
raise SerializationError(f'Failed to serialize DAG {dag.dag_id!r}: {e}')
airflow.exceptions.SerializationError: Failed to serialize DAG 'DEV_file_pipeline': 'str' object has no attribute '__module__'
```
This disappears when I refresh the page
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chenglongyan commented on issue #21459: Tasks failing often with no logs
Posted by GitBox <gi...@apache.org>.
chenglongyan commented on issue #21459:
URL: https://github.com/apache/airflow/issues/21459#issuecomment-1039940714
related: #20648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #21459: Tasks failing often with no logs
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #21459:
URL: https://github.com/apache/airflow/issues/21459#issuecomment-1042334632
Some logs should be present - can you also check if you have enough resources / kubernetes logs ? It is lilkely your tasks are killed due to lack of resources (memory most likely).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] chenglongyan commented on issue #21459: Tasks failing often with no logs
Posted by GitBox <gi...@apache.org>.
chenglongyan commented on issue #21459:
URL: https://github.com/apache/airflow/issues/21459#issuecomment-1039940714
related: #20648
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org