You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/04/14 17:24:36 UTC
[GitHub] [airflow] jmp-aws opened a new issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
jmp-aws opened a new issue #15372:
URL: https://github.com/apache/airflow/issues/15372
**Description**
As a developer I want to be able to run AWS Glue jobs in different regions from a single Dag with different tasks.
**Use case / motivation**
The developer needs a workflow where the single dag invokes an AWS Glue job in every region.
**Are you willing to submit a PR?** Maybe if I have time
**Related Issues**
NA
**Related code**
Currently in this method
https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/hooks/glue.html#AwsGlueJobHook
```
def get_or_create_glue_job(self) -> str:
"""
Creates(or just returns) and returns the Job name
:return:Name of the Job
"""
glue_client = self.get_conn()
try:
get_job_response = glue_client.get_job(JobName=self.job_name)
self.log.info("Job Already exist. Returning Name of the job")
return get_job_response['Job']['Name']
except glue_client.exceptions.EntityNotFoundException:
self.log.info("Job doesnt exist. Now creating and running AWS Glue Job")
if self.s3_bucket is None:
raise AirflowException('Could not initialize glue job, error: Specify Parameter `s3_bucket`')
s3_log_path = f's3://{self.s3_bucket}/{self.s3_glue_logs}{self.job_name}'
execution_role = self.get_iam_execution_role()
try:
create_job_response = glue_client.create_job(
Name=self.job_name,
Description=self.desc,
LogUri=s3_log_path,
Role=execution_role['Role']['RoleName'],
ExecutionProperty={"MaxConcurrentRuns": self.concurrent_run_limit},
Command={"Name": "glueetl", "ScriptLocation": self.script_location},
MaxRetries=self.retry_limit,
AllocatedCapacity=self.num_of_dpus,
**self.create_job_kwargs,
)
return create_job_response['Name']
except Exception as general_error:
self.log.error("Failed to create aws glue job, error: %s", general_error)
raise
```
The get_job method does not accept a region flag here so the region flag passed into the [AwsGlueJobOperator](https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_modules/airflow/providers/amazon/aws/operators/glue.html#AwsGlueJobOperator.execute) doesn't apply. This results in the method failing if we're expecting the job to be in a different region other than the default one in the credential chain.
**Work around**
we can set the AWS_DEFAULT_REGION environment variable in python using this line before the task runs. However this might have issues if there are concurrent dags running on the same machine and race conditions might happen.
`os.environ["AWS_DEFAULT_REGION"] = "us-east-1"`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jmp-aws commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
jmp-aws commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-820434122
Hello @eladkal thanks for the fast reply! I think I might not understand this operator and hook completely as I thought that this get_job call should also be passed the region passed in(at least that's what the developer was expecting)
https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/glue.py#L166
However, one of my colleagues pointed out that it is in the connection passed in as a parameter the region also needs to be set. To elaborate more, the operator can take both aws_conn_id and region_name as arguments. aws_conn_id is set to aws_default by default, so will just use the region of env as it doesn't specify region. Thus, default region overrides region_name passed with operator.
So, if we modify aws_default or create new AWS connection with Extra as {"region_name": "<glue-job-region>"} , operator will create Glue client with correct region and will work.
please feel free to close if you don't see any issue and this is the expected behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jmp-aws edited a comment on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
jmp-aws edited a comment on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-820434122
Hello @eladkal thanks for the fast reply! I think I might not have understood this operator and hook completely. I thought that this get_job call should also be passed the region provided in the region_name parameter(at least that's what the developer was expecting)
https://github.com/apache/airflow/blob/master/airflow/providers/amazon/aws/hooks/glue.py#L166
However, one of my colleagues pointed out that it is in the connection passed in as a parameter the region also needs to be set. To elaborate more, the operator can take both aws_conn_id and region_name as arguments. aws_conn_id is set to aws_default by default, so will just use the region of env as it doesn't specify region. Thus, default region overrides region_name passed with operator.
So, if we modify aws_default or create new AWS connection with Extra as {"region_name": "<glue-job-region>"} , operator will create Glue client with correct region and will work.
please feel free to close if you don't see any issue and this is the expected behavior
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ashb commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
ashb commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-820462016
Looking at the code, the `region_name` from the AwsGlueOperator is handled in the constructor -- none of the _methods_ in the Aws hooks should take a region_name parameter.
And the operator take a region_name https://airflow.apache.org/docs/apache-airflow-providers-amazon/stable/_api/airflow/providers/amazon/aws/operators/glue/index.html?highlight=glue#airflow.providers.amazon.aws.operators.glue.AwsGlueJobOperator and passes it on, so from looking at the code, it should work like this:
- If an explicit region_name is passed via the operator, that is used
- If not, the default from whatever connection is used is taken.
- All access via the AwsGlueHook should already respect the `self.region_name` property of the hook.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-841742443
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-846479229
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-819688172
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-841742443
This issue has been automatically marked as stale because it has been open for 30 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #15372:
URL: https://github.com/apache/airflow/issues/15372#issuecomment-820362046
Wasn't this fixed in https://github.com/apache/airflow/pull/14251 ?
can you please verify ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] closed issue #15372: Set region flag before making getJob API call in AwsGlueJobHook
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed issue #15372:
URL: https://github.com/apache/airflow/issues/15372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org