[GitHub] [airflow] jmp-aws commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook

[GitBox <gi...@apache.org>]

jmp-aws commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-820434122


   Hello @eladkal thanks for the fast reply! I think I might not understand this operator and hook completely as I thought that this get_job call should also be passed the region passed in(at least that's what the developer was expecting)
   
   https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/glue.py#L166
   
   However, one of my colleagues pointed out that it is in the connection passed in as a parameter the region also needs to be set. To elaborate more, the operator can take both aws_conn_id and region_name as arguments. aws_conn_id is set to aws_default by default, so will just use the region of env as it doesn't specify region. Thus, default region overrides region_name passed with operator.
   So, if we modify aws_default or create new AWS connection with Extra as {"region_name": "<glue-job-region>"} , operator will create Glue client with correct region and will work.
   
   please feel free to close if you don't see any issue and this is the expected behavior


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org