You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/03 23:14:13 UTC
[GitHub] [airflow] camilleanne opened a new issue, #25522: Support AWS Batch multinode job types
camilleanne opened a new issue, #25522:
URL: https://github.com/apache/airflow/issues/25522
### Description
Support [multinode job types](https://docs.aws.amazon.com/batch/latest/userguide/multi-node-parallel-jobs.html) in the [AWS Batch Operator](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/operators/batch.py).
The [boto3 `submit_job` method](https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/batch.html#Batch.Client.submit_job) supports container, multinode, and array batch jobs with the mutually exclusive `nodeOverrides` and `containerOverrides` (+ `arrayProperties`) parameters. But currently the AWS Batch Operator only supports submission of container jobs and array jobs by hardcoding the boto3 `submit_job` parameter `containerOverrides`: https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/operators/batch.py#L200 & https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/hooks/batch_client.py#L99
The [`get_job_awslogs_info`](https://github.com/apache/airflow/blob/main/airflow/providers/amazon/aws/hooks/batch_client.py#L419) method in the batch client hook is also hardcoded for the container type job: https://github.com/apache/airflow/blob/3c08cefdfd2e2636a714bb835902f0cb34225563/airflow/providers/amazon/aws/hooks/batch_client.py#L425
To support multinode jobs the `get_job_awslogs_info` method would need to access `nodeProperties` from the [`describe_jobs`](https://boto3.amazonaws.com/v1/documentation/api/1.9.88/reference/services/batch.html#Batch.Client.describe_jobs) response.
### Use case/motivation
Multinode jobs are a supported job type of AWS Batch, are supported by the underlying boto3 library, and should be also be available to be managed by Airflow. I've extended the AWS Batch Operator for our own use cases, but would prefer to not maintain a separate operator.
### Related issues
_No response_
### Are you willing to submit a PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25522: Support AWS Batch multinode job types
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25522:
URL: https://github.com/apache/airflow/issues/25522#issuecomment-1204572695
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #25522: Support AWS Batch multinode job types
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25522:
URL: https://github.com/apache/airflow/issues/25522#issuecomment-1204861378
Feel free. I assigned you. You might also sync this up with the work done by @ferruzzi @o-nikolas and @vincbeck from the AWS team -
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #25522: Support AWS Batch multinode job types
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #25522:
URL: https://github.com/apache/airflow/issues/25522#issuecomment-1263988152
@camilleanne are you working on this issue?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] dimberman closed issue #25522: Support AWS Batch multinode job types
Posted by "dimberman (via GitHub)" <gi...@apache.org>.
dimberman closed issue #25522: Support AWS Batch multinode job types
URL: https://github.com/apache/airflow/issues/25522
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on issue #25522: Support AWS Batch multinode job types
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #25522:
URL: https://github.com/apache/airflow/issues/25522#issuecomment-1347521669
@camilleanne has put out a PR (thanks Camille!) reassigning
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on issue #25522: Support AWS Batch multinode job types
Posted by GitBox <gi...@apache.org>.
o-nikolas commented on issue #25522:
URL: https://github.com/apache/airflow/issues/25522#issuecomment-1281588649
Unassigning Camille, so that others are free to take this task, since we have not heard back.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org