You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/07 19:18:03 UTC
[GitHub] [airflow] TamGB opened a new issue, #24904: AWS ECSOperator -
TamGB opened a new issue, #24904:
URL: https://github.com/apache/airflow/issues/24904
### Apache Airflow Provider(s)
amazon
### Versions of Apache Airflow Providers
4.00
### Apache Airflow version
2.3.2 (latest released)
### Operating System
Ubuntu
### Deployment
Docker-Compose
### Deployment details
_No response_
### What happened
Using the `EcsOperator` fails with misconfiguration errors even though the AWS connection is setup as such:
```
[2022-07-07, 08:00:02 UTC] {ecs.py:294} INFO - Running ECS Task - Task definition: XXXX - on cluster YYYY
[2022-07-07, 08:00:02 UTC] {ecs.py:296} INFO - EcsOperator overrides: {'containerOverrides': [{'name': 'XXXX', 'command': ['mycommands'], 'environment': [{'AWS_ACCESS_KEY_ID': 'XXXX', 'AWS_SECRET_ACCESS_KEY': 'ZZZZ', 'AWS_REGION': 'us-west-2'}]}]}
[2022-07-07, 08:00:02 UTC] {base.py:68} INFO - Using connection ID 'aws_ecs_conn' for task execution.
[2022-07-07, 08:00:02 UTC] {base_aws.py:206} INFO - Credentials retrieved from login
[2022-07-07, 08:00:02 UTC] {base_aws.py:100} INFO - Retrieving region_name from Connection.extra_config['region_name']
[2022-07-07, 08:00:02 UTC] {taskinstance.py:1889} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/utils/session.py", line 71, in wrapper
return func(*args, session=session, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 303, in execute
self._start_wait_check_task(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/hooks/base_aws.py", line 582, in decorator_f
return fun(self, *args, **kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 321, in _start_wait_check_task
self._start_task(context)
File "/home/airflow/.local/lib/python3.7/site-packages/airflow/providers/amazon/aws/operators/ecs.py", line 369, in _start_task
response = self.client.run_task(**run_opts)
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line 508, in _api_call
return self._make_api_call(operation_name, kwargs)
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line 875, in _make_api_call
api_params, operation_model, context=request_context
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/client.py", line 936, in _convert_to_request_dict
api_params, operation_model
File "/home/airflow/.local/lib/python3.7/site-packages/botocore/validate.py", line 381, in serialize_to_request
raise ParamValidationError(report=report.generate_report())
botocore.exceptions.ParamValidationError: Parameter validation failed:
Unknown parameter in overrides.containerOverrides[0].environment[0]: "AWS_ACCESS_KEY_ID", must be one of: name, value
Unknown parameter in overrides.containerOverrides[0].environment[0]: "AWS_SECRET_ACCESS_KEY", must be one of: name, value
Unknown parameter in overrides.containerOverrides[0].environment[0]: "AWS_REGION", must be one of: name, value
[2022-07-07, 08:00:02 UTC] {taskinstance.py:1400} INFO - Marking task as UP_FOR_RETRY. dag_id=ecs_dag, task_id=a_task, execution_date=20220706T080000, start_date=20220707T080001, end_date=20220707T080002
```
### What you think should happen instead
The task should not fail.
### How to reproduce
Setup an `Amazon Web Service` connection and setup an ECSOperator Task:
```
ope = EcsOperator(
task_id=f'docker_{test}',
cluster=cluster_name,
task_definition=task_definition_name,
launch_type="EXTERNAL|FARGATE",
aws_conn_id="aws_ecs",
overrides={
"containerOverrides": [
{
"name": "XXXX,
"command": ["mycommand"],
"environment": [
{
'AWS_ACCESS_KEY_ID': *****,
'AWS_SECRET_ACCESS_KEY': ****,
'AWS_REGION': *****,
},
],
},
],
},
tags={
"Project": "Test",
},
awslogs_group="/ecs/hello-world",
awslogs_region=s3_logs_region,
awslogs_stream_prefix="ecs/hello-world-container"
)
```
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TamGB closed issue #24904: AWS ECSOperator -
Posted by GitBox <gi...@apache.org>.
TamGB closed issue #24904: AWS ECSOperator -
URL: https://github.com/apache/airflow/issues/24904
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] TamGB commented on issue #24904: AWS ECSOperator -
Posted by GitBox <gi...@apache.org>.
TamGB commented on issue #24904:
URL: https://github.com/apache/airflow/issues/24904#issuecomment-1178145953
Issue fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org