You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/09/27 15:09:49 UTC

[GitHub] [airflow] Echronix opened a new issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Echronix opened a new issue #18549:
URL: https://github.com/apache/airflow/issues/18549


   ### Apache Airflow version
   
   2.1.2
   
   ### Operating System
   
   Ubuntu Docker
   
   ### Versions of Apache Airflow Providers
   
   apache-airflow-providers-amazon==2.0.0
   apache-airflow-providers-celery==2.0.0
   apache-airflow-providers-cncf-kubernetes==2.0.0
   apache-airflow-providers-docker==2.0.0
   apache-airflow-providers-elasticsearch==2.0.2
   apache-airflow-providers-ftp==2.0.0
   apache-airflow-providers-google==4.0.0
   apache-airflow-providers-grpc==2.0.0
   apache-airflow-providers-hashicorp==2.0.0
   apache-airflow-providers-http==2.0.0
   apache-airflow-providers-imap==2.0.0
   apache-airflow-providers-microsoft-azure==3.0.0
   apache-airflow-providers-mysql==2.0.0
   apache-airflow-providers-neo4j==2.0.0
   apache-airflow-providers-postgres==2.0.0
   apache-airflow-providers-redis==2.0.0
   apache-airflow-providers-sendgrid==2.0.0
   apache-airflow-providers-sftp==2.0.0
   apache-airflow-providers-slack==4.0.0
   apache-airflow-providers-sqlite==2.0.0
   apache-airflow-providers-ssh==2.0.0
   
   ### Deployment
   
   Official Apache Airflow Helm Chart
   
   ### Deployment details
   
   ```
       config:
         core:
           executor: 'CeleryExecutor'
         logging:
           remote_logging: 'True'
           remote_base_log_folder: 'cloudwatch://arn:aws:logs:eu-central-1:<redacted>:log-group:airflow'
           remote_log_conn_id: 'cloudwatch'
   ```
   
   The log group is already created in Terraform.
   
   ### What happened
   
   ```
   
   [2021-09-27 15:03:58,541: ERROR/ForkPoolWorker-15] Failed to execute task An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts::<redacted>:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:.
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/executors/celery_executor.py", line 117, in _execute_in_fork
       args.func(args)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/cli_parser.py", line 48, in command
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/cli.py", line 91, in wrapper
       return f(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/cli/commands/task_command.py", line 228, in task_run
       ti.init_run_context(raw=args.raw)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 2031, in init_run_context
       self._set_context(self)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 54, in _set_context
       set_context(self.log, context)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/utils/log/logging_mixin.py", line 173, in set_context
       handler.set_context(value)
     File "/home/airflow/.local/lib/python3.8/site-packages/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py", line 81, in set_context
       self.handler = watchtower.CloudWatchLogHandler(
     File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 148, in __init__
       _idempotent_create(self.cwl_client, "create_log_group", logGroupName=self.log_group)
     File "/home/airflow/.local/lib/python3.8/site-packages/watchtower/__init__.py", line 15, in _idempotent_create
       method_callable(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 386, in _api_call
       return self._make_api_call(operation_name, kwargs)
     File "/home/airflow/.local/lib/python3.8/site-packages/botocore/client.py", line 705, in _make_api_call
       raise error_class(parsed_response, operation_name)
   botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:<redacted>}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:
   
   ```
   
   ### What you expected to happen
   
   I would expect Airflow to create a stream and write the logs, not to create a log-group inside a log-group.
   
   ### How to reproduce
   
   Run any dag with the chart config.
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Echronix commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
Echronix commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959905825


   Hey folks,
   
   sorry that I didn't respond for that long. 
   
   I also figured out that this looks like an authentication problem but even with all privileges given to airflow the error still occurred. I was wondering if airflow maybe tries to create the log group inside the log stream. The error `arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:` looks like the correct path, but I don't see a reason why AWS would require the sub-resouce `:log-stream:` when calling to create a log-group.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
anaynayak commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-937851583


   @potiuk  I think I can try this one. 
   
   My current understanding of the code + changes required:
   1. `CloudwatchTaskHandler` delegates to `watchtower.CloudWatchLogHandler` which provides the option to create a log group if it doesn't exist (true by default) https://github.com/kislyuk/watchtower/blob/develop/watchtower/__init__.py#L84
   2. Do we intend to continue with the existing behavior but allow users to override based on config values defined ? e.g.  `conf.get('logging', 'CLOUDWATCH_CREATE_LOG_GROUP', fallback=True)` similar to https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py#L209 ?
   
   Let me know if thats the approach. Can do the changes accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] o-nikolas commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959506843






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Echronix commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
Echronix commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959905825


   Hey folks,
   
   sorry that I didn't respond for that long. 
   
   I also figured out that this looks like an authentication problem but even with all privileges given to airflow the error still occurred. I was wondering if airflow maybe tries to create the log group inside the log stream. The error `arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:` looks like the correct path, but I don't see a reason why AWS would require the sub-resouce `:log-stream:` when calling to create a log-group.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] potiuk commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-939556165


   Hard to say. I am not expert on cloud-watch - but maybe others can help :)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] o-nikolas commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959506843


   Given the discussion in [19022](https://github.com/apache/airflow/pull/19022), I think this issue can be resolved/closed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] o-nikolas commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-960021734


   What I'm quite sure of is that the issue is constrained to your configuration. Otherwise we'd have this exception for all users with logging to cloudwatch.
   
   I don't think the boto3 api would return an authentication issue if the log group name was formatted incorrectly. The airflow-service-account-role you're using likely is still missing the correct policies.
   As for the strangely formatted loggroup name, check the config you're using to specify it. You can even paste that here to get a second pair of eyes on it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] o-nikolas commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-954102099


   Hey folks,
   
   watchtower does an idempotent create for the log group as you can see [here.](https://github.com/kislyuk/watchtower/blob/6ee1e54017f024dc2607571afe06c1d44b23da72/watchtower/__init__.py#L12-L17). If the log group already exists it will catch that error and ignore it silently.
   
   The issue you actually hit is an authentication issue, as seen in your exception message:
   
   > botocore.exceptions.ClientError: An error occurred (AccessDeniedException) when calling the CreateLogGroup operation: User: arn:aws:sts:<redacted>}:assumed-role/airflow-service-account-role/botocore-session-1632755037 is not authorized to perform: logs:CreateLogGroup on resource: arn:aws:logs:eu-central-1:<redacted>:log-group:airflow:log-stream:
   
   The role you're using doesn't have permission for the log group creation. It's worth double checking that role has all the required permissions for cloudwatch logging (creating groups, streams and uploading records).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] Echronix commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
Echronix commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959905825






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
anaynayak commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-940677505


   Was curious more from a config standpoint. I believe the way to specify/override the default config is from airflow_local_settings.py. Will create a PR. Will make it easier to iterate on it. 
   
   If anyone else gets to reviewing the approach, please let me know :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] boring-cyborg[bot] commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-927968278


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] anaynayak commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
anaynayak commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-937851583


   @potiuk  I think I can try this one. 
   
   My current understanding of the code + changes required:
   1. `CloudwatchTaskHandler` delegates to `watchtower.CloudWatchLogHandler` which provides the option to create a log group if it doesn't exist (true by default) https://github.com/kislyuk/watchtower/blob/develop/watchtower/__init__.py#L84
   2. Do we intend to continue with the existing behavior but allow users to override based on config values defined ? e.g.  `conf.get('logging', 'CLOUDWATCH_CREATE_LOG_GROUP', fallback=True)` similar to https://github.com/apache/airflow/blob/main/airflow/config_templates/airflow_local_settings.py#L209 ?
   
   Let me know if thats the approach. Can do the changes accordingly.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



[GitHub] [airflow] o-nikolas commented on issue #18549: CloudWatch CreateLogGroup error on already existing LogGroup

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #18549:
URL: https://github.com/apache/airflow/issues/18549#issuecomment-959506843






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org