You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "0x26res (via GitHub)" <gi...@apache.org> on 2023/05/09 10:15:29 UTC

[GitHub] [airflow] 0x26res opened a new issue, #31148: Pass custom schedulingPriorityOverride in aws BatchOperator

0x26res opened a new issue, #31148:
URL: https://github.com/apache/airflow/issues/31148

   ### Description
   
   The aws [BatchOperator](https://github.com/apache/airflow/blob/521dae534dd0b906e4dd9a7446c6bec3f9022ac3/airflow/providers/amazon/aws/operators/batch.py#L47) uses boto3 batch [submit_job](https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch/client/submit_job.html).
   
   At the moment the constructor of BatchOperator allows you to customise some aspect of the batch job (eg: job_definition, job_queue etc). But this doesn't cover all the options that can be passed to boto3 batch submit_job. 
   
   In particular I'd like to be able to pass the parameter `schedulingPriorityOverride`
   
   ### Use case/motivation
   
   I submit a lot of array jobs to aws batch using airflow. As there can only be a limited number of job running at the same time, aws schedule jobs on a mix of FIFO logic and using the `schedulingPriorityOverride`. 
   
   As I have jobs with higher priority than other, I'd like to be able to customise that argument for my high priority jobs. So they don't get blocked until the other lower priority jobs are done.
   
   ### Related issues
   
   _No response_
   
   ### Are you willing to submit a PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] 0x26res commented on issue #31148: Pass custom schedulingPriorityOverride in aws BatchOperator

Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res commented on issue #31148:
URL: https://github.com/apache/airflow/issues/31148#issuecomment-1540053736

   Oh my bad, I must have been looking at the wrong version. This is from 2 weeks ago, I guess it must be available in 8.0.0. Thanks I'll close this now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] 0x26res closed issue #31148: Pass custom schedulingPriorityOverride in aws BatchOperator

Posted by "0x26res (via GitHub)" <gi...@apache.org>.
0x26res closed issue #31148: Pass custom schedulingPriorityOverride in aws BatchOperator
URL: https://github.com/apache/airflow/issues/31148


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] pankajastro commented on issue #31148: Pass custom schedulingPriorityOverride in aws BatchOperator

Posted by "pankajastro (via GitHub)" <gi...@apache.org>.
pankajastro commented on issue #31148:
URL: https://github.com/apache/airflow/issues/31148#issuecomment-1540027166

   I can see `scheduling_priority_override` did you try to use it? https://github.com/apache/airflow/blob/521dae534dd0b906e4dd9a7446c6bec3f9022ac3/airflow/providers/amazon/aws/operators/batch.py#L135


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #31148: Pass custom schedulingPriorityOverride in aws BatchOperator

Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #31148:
URL: https://github.com/apache/airflow/issues/31148#issuecomment-1539865194

   Thanks for opening your first issue here! Be sure to follow the issue template! If you are willing to raise PR to address this issue please do so, no need to wait for approval.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org