[GitHub] [airflow] o-nikolas commented on a change in pull request #19022: CloudwatchTaskHandler params to create log group/stream

[GitBox <gi...@apache.org>]

o-nikolas commented on a change in pull request #19022:
URL: https://github.com/apache/airflow/pull/19022#discussion_r784233810



##########
File path: airflow/config_templates/airflow_local_settings.py
##########
@@ -194,13 +193,17 @@
 
         DEFAULT_LOGGING_CONFIG['handlers'].update(S3_REMOTE_HANDLERS)
     elif REMOTE_BASE_LOG_FOLDER.startswith('cloudwatch://'):
+        create_log_group = conf.getboolean('logging', 'CREATE_CLOUDWATCH_LOG_GROUP', fallback=True)
+        create_log_stream = conf.getboolean('logging', 'CREATE_CLOUDWATCH_LOG_STREAM', fallback=True)

Review comment:
       @nik-davis 
   Hey Nik, This is a good callout and it's something we ran into while developing MWAA (which leverages Cloudawatch for all log types and thus creates a **LOT** of log groups and log stream requests). The correct solution here is to actually not call create log group until you're sure it doesn't exist. Which involves first calling describe_log_groups (which has a much higher quota) and if the log group in question doesn't exist, then create it. This is what we do in MWAA and this is what the Watchtower package should be doing as well. And in fact, it is what they [do here](https://github.com/kislyuk/watchtower/blob/a2a723787c3a0c84a1c708fd0b850d08d2a18c9d/watchtower/__init__.py#L263) since v2.0.0. And I recently upgraded Airflow to [use v2.0.1 here](https://github.com/apache/airflow/pull/19907). So this problem should already be solved! 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org