You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/11/23 16:48:51 UTC

[GitHub] [airflow] joumenharzli opened a new issue #19783: Bad parsing for Cloudwatch Group ARN for logging

joumenharzli opened a new issue #19783:
URL: https://github.com/apache/airflow/issues/19783


   ### Apache Airflow version
   
   2.1.3
   
   ### Operating System
   
   All
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Other
   
   ### Deployment details
   
   _No response_
   
   ### What happened
   
   I am deploying Airflow in AWS ECS,
   I am trying to send tasks logs to cloudwatch.
   Usually log groups in AWS have this format /aws/name_of_service/name_of_component
   
   I configured my env variables as follow
   ```
       {
         name  = "AIRFLOW__LOGGING__REMOTE_LOGGING",
         value = "true"
       },
       {
         name = "AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER",
         value = "cloudwatch://arn:aws:logs:aaaa:bbbbb:log-group:/aws/ecs/ccccc"
       },
       {
         name  = "AIRFLOW__LOGGING__REMOTE_LOG_CONN_ID",
         value = "aws_default"
       },
   ```
   I am getting this error in my tasks logs
   ```
   *** Reading remote log from Cloudwatch log_group:  log_stream: hello_airflow2/hello_task/2021-11-23T16_29_21.191922+00_00/1.log.
   Could not read remote logs from log_group:  log_stream: hello_airflow2/hello_task/2021-11-23T16_29_21.191922+00_00/1.log.
   ```
   It's throwed because the log group is empty
   The reason behind this error is this line
   https://github.com/apache/airflow/blob/c4fd84accd143977cba57e4daf6daef2af2ff457/airflow/config_templates/airflow_local_settings.py#L202
   
   the result of netloc is "arn:aws:logs:aaaa:bbbbb:log-group:"
   ```
   urlparse("cloudwatch://arn:aws:logs:aaaa:bbbbb:log-group:/aws/ecs/ccccc")
   ParseResult(scheme='cloudwatch', netloc='arn:aws:logs:aaaa:bbbbb:log-group:', path='/aws/ecs/ccccc', params='', query='', fragment='')
   ```
   and this line
   https://github.com/apache/airflow/blob/86a2a19ad2bdc87a9ad14bb7fde9313b2d7489bb/airflow/providers/amazon/aws/log/cloudwatch_task_handler.py#L53
   which will result an empty log group
   
   
   
   
   ### What you expected to happen
   
   _No response_
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas commented on issue #19783: Bad parsing for Cloudwatch Group ARN for logging

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #19783:
URL: https://github.com/apache/airflow/issues/19783#issuecomment-984219690


   This is being addressed in #19700


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #19783: Bad parsing for Cloudwatch Group ARN for logging

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #19783:
URL: https://github.com/apache/airflow/issues/19783#issuecomment-976839633


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk closed issue #19783: Bad parsing for Cloudwatch Group ARN for logging

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #19783:
URL: https://github.com/apache/airflow/issues/19783


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org