You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Tim Mottershead (JIRA)" <ji...@apache.org> on 2019/08/12 20:50:00 UTC
[jira] [Updated] (AIRFLOW-5178) AWS Batch Operator improvement to
support batch job parameters
[ https://issues.apache.org/jira/browse/AIRFLOW-5178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Mottershead updated AIRFLOW-5178:
-------------------------------------
Description:
AWSBatchOperator does not currently support AWS Batch Job parameters.
When creating an AWS Batch Job Definition and when submitting a job to AWS Batch, it's possible to define and supply job parameters. Most of our AWS Batch jobs take parameters but we are not able to pass them using the AWSBatchOperator.
In order to support batch job parameters, a new argument to __init__(self) could be added called *job_parameters*, saved to an instance variable and supplied to self.client.submit_job() in the execute() method:
{code}
self.client.submit_job(
jobName=self.job_name,
jobQueue=self.job_queue,
jobDefinition=self.job_definition,
containerOverrides=self.overrides,
parameters=self.job_parameters)
{code}
See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job
was:
AWSBatchOperator does not currently support AWS Batch Job parameters.
When creating an AWS Batch Job Definition and when submitting a job to AWS Batch, it's possible to define and supply job parameters. Most of our AWS Batch jobs take parameters but we are not able to pass them using the AWSBatchOperator.
In order to support batch job parameters, a new argument to __init__(self) could be added called *_job_parameters*_, saved to an instance variable and supplied to self.client.submit_job() in the execute() method:
{code}
self.client.submit_job(
jobName=self.job_name,
jobQueue=self.job_queue,
jobDefinition=self.job_definition,
containerOverrides=self.overrides,
parameters=self.job_parameters)
{code}
See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job
> AWS Batch Operator improvement to support batch job parameters
> --------------------------------------------------------------
>
> Key: AIRFLOW-5178
> URL: https://issues.apache.org/jira/browse/AIRFLOW-5178
> Project: Apache Airflow
> Issue Type: Improvement
> Components: aws, operators
> Affects Versions: 1.10.4
> Reporter: Tim Mottershead
> Priority: Major
> Labels: newbie
>
> AWSBatchOperator does not currently support AWS Batch Job parameters.
> When creating an AWS Batch Job Definition and when submitting a job to AWS Batch, it's possible to define and supply job parameters. Most of our AWS Batch jobs take parameters but we are not able to pass them using the AWSBatchOperator.
> In order to support batch job parameters, a new argument to __init__(self) could be added called *job_parameters*, saved to an instance variable and supplied to self.client.submit_job() in the execute() method:
> {code}
> self.client.submit_job(
> jobName=self.job_name,
> jobQueue=self.job_queue,
> jobDefinition=self.job_definition,
> containerOverrides=self.overrides,
> parameters=self.job_parameters)
> {code}
> See https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/batch.html#Batch.Client.submit_job
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)