You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/09/20 02:11:59 UTC

[GitHub] [airflow] ChirangaL opened a new issue, #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

ChirangaL opened a new issue, #26511:
URL: https://github.com/apache/airflow/issues/26511

   ### Apache Airflow version
   
   2.4.0
   
   ### What happened
   
   We are using cross account AWS connection to invoke a Glue job using GlueOperator. When the verbose flag is set we expected the cloudwatch logs to be retrieved from the AWS account where glue job was executed and displayed in Airflow logs.
   We constantly receive below error on Airflow logs rather than getting cloudwatch logs
   `Polling for AWS Glue Job <Job name> current run state with status RUNNING
   No new Glue driver logs found. This might be because there are no new logs, or might be an error.
   If the error persists, check the CloudWatch dashboard at: https://ap-southeast-1.console.aws.amazon.com/cloudwatch/home`
   
   ### What you think should happen instead
   
   When AWS GlueOperator is called with a aws_conn_id parameter and verbose flag we expect
   
   1. Glue job to be executed in the aws account as per aws_conn_id
   2. Cloudwatch logs to be relieved from aws account as per aws_conn_id and displayed in Airflow logs
   
   ### How to reproduce
   
   1. submit a glue job in a different AWS account using GlueOperator as below
    `with DAG(
       dag_id="sample_dag",
       description="Sample DAG testing",
       schedule_interval=None,
       start_date=datetime(2022, 9, 1),
       catchup=False,
   ) as dag:
       submit_glue_job = SeekGlueJobOperator(
           aws_conn_id="different_aws_account",
           task_id="submit_glue_job",
           job_name=job_name,
           wait_for_completion=True,
           retry_limit=1,
           script_location=None,
           iam_role_name="iam-role-for-glue-job-invocation",
           script_args={},
           run_job_kwargs={"NumberOfWorkers": 2, 'WorkerType': "Standard"},
           verbose=True,
           region_name="ap-southeast-1"
       )
   `
   2. Check Airflow logs to see if cloudwatch logs are retrieved and displayed
   
   ### Operating System
   
   Mac 12.6
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Docker-Compose
   
   ### Deployment details
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] ChirangaL commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
ChirangaL commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1252078824

   Hi @Taragolis You are correct. The updates in mentioned PR will resolve our issue. But given the below proposed update is performed.
   https://github.com/apache/airflow/pull/26269/files/3e4e6db6a2e199483022f27dc4681dfae293b2ed#r969486988


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1330029164

   > @o-nikolas - given there is no response for more than a month, can we close this issue for now? 
   > @ChirangaL - feel free to open it again if the solution doesn't work for your use case.
   
   Agreed, I will close the issue


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] Taragolis commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1252054378

   I've made some suggestion to current existed PR: https://github.com/apache/airflow/pull/26269#discussion_r969486988
   If I get you correct it should help in your case. It also possible to make this changes with individual PR (quite simple)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] shubham22 commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
shubham22 commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1330024053

   @o-nikolas - given there is no response for more than a month, can we close this issue for now? 
   @ChirangaL - feel free to open it again if the solution doesn't work for your use case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1251751569

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] potiuk commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1254240610

   I linked the issue to the PR.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas commented on issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #26511:
URL: https://github.com/apache/airflow/issues/26511#issuecomment-1281557704

   > Hi @Taragolis You are correct. The updates in mentioned PR will resolve our issue. But given the below proposed update is performed. https://github.com/apache/airflow/pull/26269/files/3e4e6db6a2e199483022f27dc4681dfae293b2ed#r969486988
   
   The PR has been updated, the access key and secret are now fetched from the hook credentials. @ChirangaL please have a look at the current state of the PR to see if it should suffice for your usecase?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] o-nikolas closed issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator

Posted by GitBox <gi...@apache.org>.
o-nikolas closed issue #26511: Retrieve AWS Cloudwatch logs using aws_conn_id connection for GlueOperator
URL: https://github.com/apache/airflow/issues/26511


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org