You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/03/08 18:36:12 UTC
[GitHub] [airflow] maxcountryman opened a new issue #14667: Cloudwatch parsing fails for log groups containing slashes
maxcountryman opened a new issue #14667:
URL: https://github.com/apache/airflow/issues/14667
**Apache Airflow version**: 2.0.1
**Environment**:
- **Cloud provider or hardware configuration**: AWS + ECS
- **OS** (e.g. from /etc/os-release): Ubuntu
- **Kernel** (e.g. `uname -a`):
- **Install tools**:
- **Others**:
**What happened**:
When enabling remote logging via Cloudwatch we have observed that you cannot use log groups with slashes in their name, e.g. "/foo/bar/baz" or a more practical example like "/ecs/your-app". This is in our experience a common convention in AWS and doesn't seem to be a documented limitation so we wonder if this is a bug.
**What you expected to happen**:
We believe that unless this is explicitly designed in this manner this is likely not behaving as desired.
<!-- What do you think went wrong? -->
**How to reproduce it**:
Simply create a log group with forward slashes and you should observe the same effect.
It's even easier to observe this in the REPL, for instance [here is the culprit](https://github.com/apache/airflow/blob/97b5e4cd6c001ec1a1597606f4e9f1c0fbea20d2/airflow/config_templates/airflow_local_settings.py#L192) in our code and in the REPL we can reproduce like so:
```python
>>> urlparse('cloudwatch://arn:aws:logs:us-west-2:123:log-group:/ecs/my-app:*').netloc
'arn:aws:logs:us-west-2:123:log-group:'
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-1060046439
> I am also having this problem on Apache Airflow 2.1.4 and Amazon Linux 2 as the OS. Thank you maxcountryman for outlining the problem however a fix would be appreciated.
Indeed. anyone providing fix would be great.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mini1989cloud commented on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
mini1989cloud commented on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-804709424
When I try to enable remote logging in airflow, it fails with following message
Could not read remote logs from Cloudwatch log_group: <group name> log_stream: xxxx/xxx/<timestamp>/1.log.
I have created a connection with "connection type" AWS, and its credentials in "Extra" field (Created a user for this).
In airflow.cfg file, my entries are :-
remote_logging = True
remote_log_conn_id = <Connection name from airflow UI>
remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:airflow-task-log
Am I missing something with this configuration?
I have created log-group before running the DAG, it is showing same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] HybridNeos commented on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
HybridNeos commented on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-965231923
I am also having this problem on Apache Airflow 2.1.4 and Amazon Linux 2 as the OS. Thank you maxcountryman for outlining the problem however a fix would be appreciated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] maxcountryman commented on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
maxcountryman commented on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-804866243
@mini1989cloud I had to create a set of AWS credentials which are injected as the standard boto3 environment variables in order for Cloudwatch to work. I don't think it knows about or will use an Airflow connection.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mini1989cloud commented on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
mini1989cloud commented on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-805757189
Luckily logs are getting pushed to cloudwatch. Anyone faced below issue ?
WatchtowerWarning: Failed to deliver logs: Parameter validation failed:
Invalid length for parameter logEvents[4].message, value: 0, valid range: 1-inf
Invalid length for parameter logEvents[6].message, value: 0, valid range: 1-inf
etc?
Im getting this exception for one of my dags.
Any reason why I am getting this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mini1989cloud edited a comment on issue #14667: Cloudwatch parsing fails for log groups containing slashes
Posted by GitBox <gi...@apache.org>.
mini1989cloud edited a comment on issue #14667:
URL: https://github.com/apache/airflow/issues/14667#issuecomment-804709424
When I try to enable remote logging in airflow, it fails with following message
Could not read remote logs from Cloudwatch log_group: <group name> log_stream: xxxx/xxx/<timestamp>/1.log.
I have created a connection with "connection type" AWS, and its credentials in "Extra" field (Created a user for this).
In airflow.cfg file, my entries are :-
remote_logging = True
remote_log_conn_id = Connection name from airflow UI
remote_base_log_folder = cloudwatch://arn:aws:logs:<region name>:<account id>:log-group:airflow-task-log
Am I missing something with this configuration?
I have created log-group before running the DAG, it is showing same error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org