[GitHub] [airflow] Ritika-Singhal commented on issue #19207: AWSGlueJobOperator using 'AllocatedCapacity' variable even after specifying the "WorkerType" and "NumberOfWorkers" causing issues with the AWS Glue version 2.0 and 3.0

[GitBox <gi...@apache.org>]

Ritika-Singhal commented on issue #19207:
URL: https://github.com/apache/airflow/issues/19207#issuecomment-951566516


   Thanks for acknowledging this issue. The solution which I have in mind to correct this is:
   
   In the AWSGlueJobHook script where `glue_client.create_job` is being called, we need to insert an if else block which checks whether the `create_job_kwargs` dict has the `WorkerType`  and `NumberOfWorkers` parameters. If it has these 2 parameters, then in the `glue_client.create_job` function call, we need to omit the `AllocatedCapacity` parameter. However, if these 2 parameters are not there win the `create_job_kwargs` dict , then we should have the function call assigning the `num_of_dpus` to either `AllocatedCapacity` or `MaxCapacity` parameter. So, my plan includes modifying function call based on the if else condition.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org