You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "nwsparks (via GitHub)" <gi...@apache.org> on 2023/10/24 18:35:54 UTC
[I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
nwsparks opened a new issue, #35164:
URL: https://github.com/apache/airflow/issues/35164
### Description
Please add an option to disable attempting to create the cloudwatch log group when remote logging is enabled or just disable the behavior and require the the group already exists.
### Use case/motivation
We have enabled remote logging to CloudWatch. After doing so, we see that some of our dags with a lot of parallelization are hitting this error and then never executing:
`stderr F botocore.exceptions.ClientError: An error occurred (ThrottlingException) when calling the CreateLogGroup operation (reached max retries: 4): Rate exceeded`
I am guessing that on the launch of each task, it is trying to create the log group defined here https://airflow.apache.org/docs/apache-airflow/stable/configurations-ref.html#remote-base-log-folder
This is unnecessary if the log group was already created.
### Related issues
_No response_
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas closed issue #35164: logging/remote_logging - option to disable creation of cloudwatch log groups
URL: https://github.com/apache/airflow/issues/35164
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1788043297
I don't think the best course of action is to disable this feature. For the vast majority of the time this feature works great and sets up the log groups seamlessly for users. It would be yet another config that we'd have to test and maintain.
If anything I wonder if this is really a feature request/bug report for Watchtower, which should maybe log the Exception message they receive so that the user has better visibility into why the log group is being setup so frequently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1777810733
Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1852604590
Closing this one for now, since we lost engagement a bit. If we see this again we could think about adding more logging, but so far we just have one data point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1778018702
> This is unnecessary if the log group was already created.
Yeah, I think better to create configuration option in AWS Provider which could control this behaviour. [`watchtower.CloudWatchLogHandler`](https://kislyuk.github.io/watchtower/#watchtower.CloudWatchLogHandler) has the option `create_log_group` which by default set to True.
As current workaround I would recommend increase retry timeout: https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/connections/aws.html#avoid-throttling-exceptions
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "nwsparks (via GitHub)" <gi...@apache.org>.
nwsparks commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1779293956
good suggestion, thanks @Taragolis
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1788121752
I think we're trying to resolve a non-existing problem here. The only reason throttling happened in this case was because of the issues with the Role. We do not generally see reports of hitting this throttling naturally very often (and we have some VERY big users with lots of tasks).
I think the best course of action is to simply increase your quota if your task throughput is so massive that you do legitimately hit that throttling error. And in this specific case, I think some better logging of the errors with the describe request could help users like @nwsparks root cause their issue earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1788106289
My 50 cents. It is a classical Chicken And Egg problem.
For write into the CloudWatch Logs you need to have Log Groups, so you need to check is this log group exists or not and create it but both requests to limited by 5 TPS https://docs.aws.amazon.com/AmazonCloudWatch/latest/logs/cloudwatch_limits_cwl.html as result with default retries strategy it is quite easy to have a Throttling error if spawn multiple task and since mapping task quite popular it become easier to got this error.
The suggestion here, and just give optional opportunity do not check if is cloudwatch group exist or not, which is already supported by watchtower. No need any other stuff because if we would try to manually check is this cloudwatch group exists or not then we would spawn same error. Just because there is no better way to check it.
Another option is configure retries, side effect of this it might take significant amount of time due to exponential backoff just for logger initialization
Yet another option would be request AWS Support to increase quotes for this API Call, according to documentation it could be done because it is not hard quota.
And final solution it is switch from cloudwatch remote logger to s3 remote logger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] logging/remote_logging - option to disable creation of cloudwatch log groups [airflow]
Posted by "nwsparks (via GitHub)" <gi...@apache.org>.
nwsparks commented on issue #35164:
URL: https://github.com/apache/airflow/issues/35164#issuecomment-1779320595
so it does seem watchtower was updated to not do this every time, however the code for it is wrapped in a try/catch that first does describe log groups, then swallows any errors and the catch block just tries to create the log group...which is not ideal.
we found via manual testing that the problem is we don't have permissions on the IAM role airflow is using to list all log groups, it just has `*` on the group we created....so per the above behavior it was calling createloggroup for every task.
that said, I still think that the option to just disable creation of the group should be considered.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org