[GitHub] [airflow] Ritika-Singhal commented on pull request #19787: Bug fix in AWS glue operator when specifying the WorkerType & NumberOfWorkers

[GitBox <gi...@apache.org>]

Ritika-Singhal commented on pull request #19787:
URL: https://github.com/apache/airflow/pull/19787#issuecomment-977161677


   > Instead of sliently dropping `num_of_dpus`, I feel we should explicitly fail with `ValueError` in `__init__`.
   
   In the `AwsGlueJobHook` `__init__`, the `num_of_dpus` are being initialized by default to 10 if the user did not specify this parameter. So, to throw a `ValueError`, we would need to remove the default initialization of `num_of_dpus` (otherwise it will always throw the error because `num_of_dpus` will always have a valid value). It will also need to check if the user has specified one of the following arguments or not i.e. either (`num_of_dpus`) or (`WorkerType` and `NumberOfWorkers`) but not both. The default initialization of the `num_of_dpus` is keeping the job from failing if the user did not specify either of the parameters.
   
   Currently, the `num_of_dpus` parameter works as a default option in the `else` block if the user did not specify the `WorkerType` and `NumberOfWorkers`. So, it's not completed dropped.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org