[jira] [Comment Edited] (AIRFLOW-5819) AWSBatchOperator has invalid default value for array_properties

["Dom (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/AIRFLOW-5819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16963217#comment-16963217 ] 

Dom edited comment on AIRFLOW-5819 at 10/30/19 4:55 PM:
--------------------------------------------------------

I am not entirely sure why this issue is not capture by the existing test case (which does have None in the arguments):
 [https://github.com/apache/airflow/blob/73bf718358c01cc41e5f23b914a8824a2665a28c/tests/contrib/operators/test_awsbatch_operator.py#L39]


was (Author: domantasjurkus):
I am not entirely sure why this issue is not capture by the existing test case:
https://github.com/apache/airflow/blob/73bf718358c01cc41e5f23b914a8824a2665a28c/tests/contrib/operators/test_awsbatch_operator.py#L39

> AWSBatchOperator has invalid default value for array_properties
> ---------------------------------------------------------------
>
>                 Key: AIRFLOW-5819
>                 URL: https://issues.apache.org/jira/browse/AIRFLOW-5819
>             Project: Apache Airflow
>          Issue Type: Bug
>          Components: operators
>    Affects Versions: 1.10.6
>            Reporter: Dom
>            Priority: Major
>
> We upgraded our Airflow version from 1.10.3 to 1.10.6 and saw our AWSBatchOperator throwing the following error (easily fixed by changing the default param from None to {}):
> {code:java}
>  [2019-10-30 13:10:15.392] INFO:airflow.task.operators.execute:93 Running AWS Batch Job - Job definition: batch-jobdef-1 - on queue AwsBatch-batch-queue
> [2019-10-30 13:10:15.393] INFO:airflow.task.operators.execute:95 AWSBatchOperator overrides: {'command': ['--start_datetime', '2019-10-29']}
> [2019-10-30 13:10:15.433] INFO:airflow.task.operators.execute:121 AWS Batch Job has failed executed
> [2019-10-30 13:10:15.445] ERROR:airflow.task.handle_failure:1058 Parameter validation failed:
> Invalid type for parameter arrayProperties, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>
> Traceback (most recent call last):
>   File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py", line 108, in execute
>     containerOverrides=self.overrides)
>   File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 357, in _api_call
>     return self._make_api_call(operation_name, kwargs)
>   File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 634, in _make_api_call
>     api_params, operation_model, context=request_context)
>   File "/usr/local/lib/python3.7/site-packages/botocore/client.py", line 682, in _convert_to_request_dict
>     api_params, operation_model)
>   File "/usr/local/lib/python3.7/site-packages/botocore/validate.py", line 297, in serialize_to_request
>     raise ParamValidationError(report=report.generate_report())
> botocore.exceptions.ParamValidationError: Parameter validation failed:
> Invalid type for parameter arrayProperties, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>During handling of the above exception, another exception occurred:Traceback (most recent call last):
>   File "/usr/local/lib/python3.7/site-packages/airflow/models/taskinstance.py", line 930, in _run_raw_task
>     result = task_copy.execute(context=context)
>   File "/usr/local/lib/python3.7/site-packages/airflow/contrib/operators/awsbatch_operator.py", line 122, in execute
>     raise AirflowException(e)
> airflow.exceptions.AirflowException: Parameter validation failed:
> Invalid type for parameter arrayProperties, value: None, type: <class 'NoneType'>, valid types: <class 'dict'>
> [2019-10-30 13:10:15.447] INFO:airflow.task.handle_failure:1087 All retries failed; marking task as FAILED{code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)