You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/08 15:10:41 UTC
[GitHub] [airflow] codenamestif opened a new issue #15279: Error on logging empty line to Cloudwatch
codenamestif opened a new issue #15279:
URL: https://github.com/apache/airflow/issues/15279
**Apache Airflow version**: 2.0.1
**Environment**:
- **Cloud provider or hardware configuration**: AWS
**What happened**:
I have Airflow with Cloudwatch-based remote logging running. I also have `BashOperator` that does, for example, `rsync` with invalid parameters, for example `rsync -av test test`. The output of the `rsync` error is formatted and contains empty line. Once that empty line is logged to the Cloudwatch, i receive an error:
```
2021-04-06 19:29:22,318] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:154 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[5].message, value: 0, valid range: 1-inf
[2021-04-06 19:29:22,320] /home/airflow/.local/lib/python3.6/site-packages/watchtower/__init__.py:158 WatchtowerWarning: Failed to deliver logs: None
```
So basically empty lines can't be submitted to the Cloudwatch and as result the whole output of the process doesn't appear in logs.
**What you expected to happen**:
I expect to have an output of the bash command in logs. Empty lines can be skipped or replaced with something.
**How to reproduce it**:
For example: run `BashOperator` with `rsync` command that fails on Airflow with Cloudwatch-based remote logging. It could be any other command that produces empty line in the output.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-909509739
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-816507842
@jedcunningham no, it wasn't a running container. I updated my image and started a new application. I also see in build logs:
```
#14 6.959 Installing collected packages: watchtower
#14 6.959 Attempting uninstall: watchtower
#14 6.961 Found existing installation: watchtower 0.7.3
#14 6.965 Uninstalling watchtower-0.7.3:
#14 6.974 Successfully uninstalled watchtower-0.7.3
#14 7.095 ERROR: apache-airflow-providers-amazon 1.1.0 requires watchtower~=0.7.3, but you'll have watchtower 1.0.6 which is incompatible.
#14 7.096 Successfully installed watchtower-1.0.6
```
So i assume i updated the lib properly and still see the error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelkhan3 commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
michaelkhan3 commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-937418131
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm edited a comment on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm edited a comment on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-909509739
i am also seeing this error even with apparently the latest versions:
```
apache-airflow==2.1.3
apache-airflow-providers-amazon==2.1.0
...
watchtower==1.0.6
```
```
[2021-08-31 18:33:50,305] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:201 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[19].message, value: 0, valid min length: 1
Invalid length for parameter logEvents[21].message, value: 0, valid min length: 1
[2021-08-31 18:33:50,308] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:205 WatchtowerWarning: Failed to deliver logs: None
```
it is difficult to debug as the log gets truncated. not sure where the blanks are coming from. the underlying `BashOperator` `TaskInstance` should not be generating any blank log lines.
is it easy to setup airflow to log to two locations simultaneously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michieldhadamus commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
michieldhadamus commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-924883452
We are also seeing this issue, airflow version 2.1.4, using the kubernetesPodOperator. Is there a workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] afischer-opentext-com commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
afischer-opentext-com commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-867666277
Another approach is to wrap the LogHandler into something like this:
```(python)
import watchtower
class NoEmptyMessageHandler(watchtower.CloudWatchLogHandler):
def emit(self, message):
if isinstance(message.msg, str) and not message.msg:
pass
else:
super().emit(message)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelkhan3 commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
michaelkhan3 commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-937418131
> i've gotten around it by creating a `log_config.py` with the following:
>
> ```
> from copy import deepcopy
> import logging
>
> from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
>
> LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
> class EmptyLogFilter(logging.Filter):
> def filter(self, record: logging.LogRecord) -> bool:
> if not record.getMessage():
> return False
> else:
> return True
>
>
> LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}
>
> CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
>
> TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
> if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
> # This thing blows up on empty log messages so apply filter
> filters = TASK_HANDLER.get('filters', [])
> filters.append('remove_blanks')
> TASK_HANDLER['filters'] = filters
> ```
>
> don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`
Where did you add the log_config.py file? In the airflow docker image or with the dags?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BMalaichik edited a comment on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
BMalaichik edited a comment on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-992342616
> > i've gotten around it by creating a `log_config.py` with the following:
> > ```
> > from copy import deepcopy
> > import logging
> >
> > from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
> >
> > LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
> > class EmptyLogFilter(logging.Filter):
> > def filter(self, record: logging.LogRecord) -> bool:
> > if not record.getMessage():
> > return False
> > else:
> > return True
> >
> >
> > LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}
> >
> > CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
> >
> > TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
> > if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
> > # This thing blows up on empty log messages so apply filter
> > filters = TASK_HANDLER.get('filters', [])
> > filters.append('remove_blanks')
> > TASK_HANDLER['filters'] = filters
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`
>
> Where did you add the log_config.py file? In the airflow docker image or with the dags?
@michaelkhan3
Setup reference from official docs - https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
This file must be on `PYTHONPATH` env variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sicarul commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
sicarul commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-874727033
I had a lot of jobs freezing with cloudwatch, migrated to using an EFS filesystem to keep logs myself (using ECS Fargate to host Airflow)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sicarul commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
sicarul commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-867730058
I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i'll test it later :smile:
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-909509739
i am also seeing this error even with apparently the latest versions:
```
apache-airflow==2.1.3
apache-airflow-providers-amazon==2.1.0
...
watchtower==1.0.6
...
```
[2021-08-31 18:33:50,305] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:201 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[19].message, value: 0, valid min length: 1
Invalid length for parameter logEvents[21].message, value: 0, valid min length: 1
[2021-08-31 18:33:50,308] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:205 WatchtowerWarning: Failed to deliver logs: None
```
it is difficult to debug as the log gets truncated. not sure where the blanks are coming from. the underlying `BashOperator` `TaskInstance` should not be generating any blank log lines.
is it easy to setup airflow to log to two locations simultaneously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-816097846
@jedcunningham I run airflow from the docker image based on `apache/airflow:2.0.1`. I installed `watchtower==1.0.6` under `airflow` user and `pip freeze` confirms it. However I still see the same error. Dunno exactly if this is a correct way to update the lib for the docker image.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-973159337
I cut [an issue](https://github.com/kislyuk/watchtower/issues/154) to the watchtower repo (since the other was closed) with some possible fixes I had tested. But it seems like there is already a fix that *just* got released into Watchtower 2.0.
I have yet to do some testing with the new release, but once I get around to (and assuming it works as expected) it should be an easy fix to bump the watchtower version in Airflow to 2.0.
I'm happy to be assigned this issue if no one else is working on it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-815969747
Airflow currently pins watchtower to `~=0.7.3`, so the fix (only available in 1.x, I believe) cannot be used in this context.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
potiuk closed issue #15279:
URL: https://github.com/apache/airflow/issues/15279
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-909528992
> I had a lot of jobs freezing with cloudwatch, migrated to using an EFS filesystem to keep logs myself (using ECS Fargate to host Airflow)
I have noticed some jobs randomly freezing with nothing in the logs. how were you able to pinpoint the issue to cloudwatch logging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-816383423
To clarify, you did it in your image custom, not in an already running container, right? If so, then yeah that is the right way to test it. Bummer. Might be worth an issue over in the watchtower repo.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-874524193
> I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i'll test it later 😄
I just tested it, the error is still there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-981950448
Update: My testing revealed another edge case where empty messages were failing watchtower/cloudwatch, I've submitted a PR to the watchtower repo to fix it: https://github.com/kislyuk/watchtower/pull/162
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] BMalaichik commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
BMalaichik commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-992342616
> > i've gotten around it by creating a `log_config.py` with the following:
> > ```
> > from copy import deepcopy
> > import logging
> >
> > from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
> >
> > LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
> > class EmptyLogFilter(logging.Filter):
> > def filter(self, record: logging.LogRecord) -> bool:
> > if not record.getMessage():
> > return False
> > else:
> > return True
> >
> >
> > LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}
> >
> > CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
> >
> > TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
> > if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
> > # This thing blows up on empty log messages so apply filter
> > filters = TASK_HANDLER.get('filters', [])
> > filters.append('remove_blanks')
> > TASK_HANDLER['filters'] = filters
> > ```
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`
>
> Where did you add the log_config.py file? In the airflow docker image or with the dags?
@michaelkhan3
Setup reference from official docs - https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/logging-tasks.html#advanced-configuration
This file must be on `PYTHON_PATH` env variable.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-815972387
Ah, good catch. @codenamestif, would it be possible for you to test 1.x in your environment? I suspect there wasn't an issue, just no one has tried it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] codenamestif commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
codenamestif commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-874524193
> I just saw that a new version of apache-airflow-providers-amazon was released (2.0.0) with an updated watchtower, i'll test it later 😄
I just tested it, the error is still there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] sicarul commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
sicarul commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-867631634
Hey y'all, i'm having the same issue now, but i found a workaround, you can use this awk command to delete the newlines in the Bash command:
`my_cool_command | awk NF`
In case you have the same problem as i did (In my case it happens with running dbt, which puts empty newlines for styling between it's messages)
Have a great day!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm edited a comment on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm edited a comment on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-909509739
i am also seeing this error even with apparently the latest versions:
```
apache-airflow==2.1.3
apache-airflow-providers-amazon==2.1.0
...
watchtower==1.0.6
```
```
[2021-08-31 18:33:50,305] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:201 WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[19].message, value: 0, valid min length: 1
Invalid length for parameter logEvents[21].message, value: 0, valid min length: 1
[2021-08-31 18:33:50,308] /<home dir>/lib/python3.9/site-packages/watchtower/__init__.py:205 WatchtowerWarning: Failed to deliver logs: None
```
it is difficult to debug as the log gets truncated. not sure where the blanks are coming from. the underlying `BashOperator` `TaskInstance` should not be generating any blank log lines.
is it easy to setup airflow to log to two locations simultaneously?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-815968161
Are you on the latest version of watchtower? Sounds like they should be ignored now: kislyuk/watchtower#43
Either way, I'm not sure Airflow should be responsible for skipping/replacing empty lines.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] kevlubkcm commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
kevlubkcm commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-924912967
i've gotten around it by creating a `log_config.py` with the following:
```
from copy import deepcopy
import logging
from airflow.config_templates.airflow_local_settings import DEFAULT_LOGGING_CONFIG
LOGGING_CONFIG = deepcopy(DEFAULT_LOGGING_CONFIG)
class EmptyLogFilter(logging.Filter):
def filter(self, record: logging.LogRecord) -> bool:
if not record.getMessage():
return False
else:
return True
LOGGING_CONFIG['filters']['remove_blanks'] = {'()': 'log_config.EmptyLogFilter'}
CLOUDWATCH_HANDLER = 'airflow.providers.amazon.aws.log.cloudwatch_task_handler.CloudwatchTaskHandler'
TASK_HANDLER = LOGGING_CONFIG['handlers']['task']
if TASK_HANDLER['class'] == CLOUDWATCH_HANDLER:
# This thing blows up on empty log messages so apply filter
filters = TASK_HANDLER.get('filters', [])
filters.append('remove_blanks')
TASK_HANDLER['filters'] = filters
```
don't forget to enable this logger in `airflow.cfg` with `logging_config_class = log_config.LOGGING_CONFIG`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] michaelkhan3 commented on issue #15279: Error on logging empty line to Cloudwatch
Posted by GitBox <gi...@apache.org>.
michaelkhan3 commented on issue #15279:
URL: https://github.com/apache/airflow/issues/15279#issuecomment-937477547
@kevlubkcm ☝️
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org