You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/22 09:34:32 UTC
[GitHub] [airflow] Taragolis opened a new pull request, #25228: Fix BatchOperator links on wait_for_completion = True
Taragolis opened a new pull request, #25228:
URL: https://github.com/apache/airflow/pull/25228
Fix incidentally wrong usage of `list.extend` method in `operator_extra_links` property of _BatchOperator_
```
Broken DAG: [/files/dags/example_batch.py] Traceback (most recent call last):
File "/opt/airflow/airflow/serialization/serialized_objects.py", line 658, in _serialize_node
if op.operator_extra_links:
File "/opt/airflow/airflow/providers/amazon/aws/operators/batch.py", line 118, in operator_extra_links
op_extra_links.extend(BatchJobDefinitionLink(), BatchJobQueueLink())
TypeError: extend() takes exactly one argument (2 given)
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/opt/airflow/airflow/serialization/serialized_objects.py", line 1125, in to_dict
json_dict = {"__version": cls.SERIALIZER_VERSION, "dag": cls.serialize_dag(var)}
File "/opt/airflow/airflow/serialization/serialized_objects.py", line 1033, in serialize_dag
raise SerializationError(f'Failed to serialize DAG {dag.dag_id!r}: {e}')
airflow.exceptions.SerializationError: Failed to serialize DAG 'example_batch_submit_job_2': extend() takes exactly one argument (2 given)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on pull request #25228: Fix BatchOperator links on wait_for_completion = True
Posted by GitBox <gi...@apache.org>.
josh-fell commented on PR #25228:
URL: https://github.com/apache/airflow/pull/25228#issuecomment-1192909115
@Taragolis Have you happened to use the `BatchOperator` with Dynamic Task Mapping? The reason I ask is there was a [previously reported issue](https://github.com/apache/airflow/issues/24653) with operator links declared this way and it breaking dynamically mapped tasks. There are some fixes out there as well: #24676 and #25215.
Going to block this in the meantime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell merged pull request #25228: Fix BatchOperator links on wait_for_completion = True
Posted by GitBox <gi...@apache.org>.
josh-fell merged PR #25228:
URL: https://github.com/apache/airflow/pull/25228
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on pull request #25228: Fix BatchOperator links on wait_for_completion = True
Posted by GitBox <gi...@apache.org>.
josh-fell commented on PR #25228:
URL: https://github.com/apache/airflow/pull/25228#issuecomment-1192941483
> I thought it required changes in DAG serialisation, if attribute is property it is required get actual value of property rather than object and only affect serialisation error for Dynamic Task Mapping since Airflow 2.3.
>
>
>
> I could also look on issue related to DAG serialisation (separate PR), however this PR fix general Operator issue.
No worries! Just curious if you had tried it. Good points. The issue can be tackled later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on pull request #25228: Fix BatchOperator links on wait_for_completion = True
Posted by GitBox <gi...@apache.org>.
Taragolis commented on PR #25228:
URL: https://github.com/apache/airflow/pull/25228#issuecomment-1192915347
I thought it required changes in DAG serialisation, if attribute is property it is required get actual value of property rather than object and only affect serialisation error for Dynamic Task Mapping since Airflow 2.3.
I could also look on issue related to DAG serialisation (separate PR), however this PR fix general Operator issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on pull request #25228: Fix BatchOperator links on wait_for_completion = True
Posted by GitBox <gi...@apache.org>.
josh-fell commented on PR #25228:
URL: https://github.com/apache/airflow/pull/25228#issuecomment-1192933878
I did a quick test. Definitely the same serialization issue. See #25243.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org