You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/08 14:39:58 UTC
[GitHub] [airflow] theagilehacker opened a new issue #18089: KeyError when ECS failed to start image
theagilehacker opened a new issue #18089:
URL: https://github.com/apache/airflow/issues/18089
### Apache Airflow version
2.1.3 (latest released)
### Operating System
Linux
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon==1.4.0
apache-airflow-providers-docker==1.2.0
apache-airflow-providers-ftp==1.1.0
apache-airflow-providers-http==1.1.1
apache-airflow-providers-imap==1.0.1
apache-airflow-providers-postgres==2.0.0
apache-airflow-providers-sqlite==1.0.2
### Deployment
Other
### Deployment details
_No response_
### What happened
[2021-09-08 00:30:00,035] {taskinstance.py:1462} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1164, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1282, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1312, in _execute_task
result = task_copy.execute(context=context)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/session.py", line 70, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 230, in execute
self._check_success_task()
File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 380, in _check_success_task
if container.get('lastStatus') == 'STOPPED' and container['exitCode'] != 0:
KeyError: 'exitCode'
### What you expected to happen
I Expect to see the error reason reported by the ECS operator not the KeyError when the code tries to access the exit code from the container.
Maybe something like:
` if container.get('lastStatus') == 'STOPPED' and container.get('exitCode', 1) != 0:`
to avoid the exception and some error handling that would make is easier to find the reason the container failed.
currently we have to search through the big huge info log with the Failed INFO.
### How to reproduce
Run ECS Task with bad creds to pull docker image from gitlab repo
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] theagilehacker commented on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
theagilehacker commented on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-944992737
Oh nice, I guess that makes sense if you kill it before it is about to load the container should cause the same issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-915298373
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-916758390
@theagilehacker feel free to submit PR with a fix
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-944887234
@theagilehacker
I was able to reproduce same error:
1. start task from airflow
2. manually stop ECS task from ECS console before the task reached `RUNNING` state.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal closed issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
eladkal closed issue #18089:
URL: https://github.com/apache/airflow/issues/18089
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] theagilehacker commented on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
theagilehacker commented on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-933556241
I have a fix, (super simple) but I need to re-produce it manually to make sure that I do the right thing here.
For example the cloudwall logs will not be there as there is no container, and hence no possible logs. So I am going to have to make a separate case and run a manual run at work (the setup that did cause it is there) to see what the results contain, or not so I can make sure it will not cause another issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif edited a comment on issue #18089: KeyError when ECS failed to start image
Posted by GitBox <gi...@apache.org>.
codenamestif edited a comment on issue #18089:
URL: https://github.com/apache/airflow/issues/18089#issuecomment-944887234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org