You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Jacopo Sabbatini (JIRA)" <ji...@apache.org> on 2018/10/09 08:24:00 UTC
[jira] [Comment Edited] (AIRFLOW-3156) AWS Batch executor
[ https://issues.apache.org/jira/browse/AIRFLOW-3156?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16642949#comment-16642949 ]
Jacopo Sabbatini edited comment on AIRFLOW-3156 at 10/9/18 8:24 AM:
--------------------------------------------------------------------
Well in a sense it's like the use case for the Mesos and Kubernetes executor.
Correct me if wrong but when using the AWS Batch operator all the task does is continuously polling for the task status until done. This seems a very inefficient use of a worker. Furthermore if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the polling API requests being throttled by AWS.
Having a Batch executor it would mean that the polling of the job status can be batched using the DescribeJobs Batch API this would make having many tasks more efficient. Using Batch as executor would also simplify deployments since it remove the needs for workers that are now managed by AWS Batch.
was (Author: dedalusj):
Well in a sense it's like the use case for the Mesos and Kubernetes executor.
Correct me if wrong but when using the AWS Batch operator all the task does is continuously polling for the task status until done. This seems a very inefficient use of a worker. Furthermore if I wanted to submit many tasks using the Batch operator I suspect I would start seeing the polling API requests being throttled by AWS.
Having a Batch executor it would mean that the polling of the job status can be batched using the DescribeJobs Batch API this would make having many tasks more efficient.
> AWS Batch executor
> ------------------
>
> Key: AIRFLOW-3156
> URL: https://issues.apache.org/jira/browse/AIRFLOW-3156
> Project: Apache Airflow
> Issue Type: New Feature
> Components: executor
> Reporter: Jacopo Sabbatini
> Priority: Major
> Labels: aws
>
> Add support for AWS Batch as executor.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)