You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "agomez-etsy (via GitHub)" <gi...@apache.org> on 2023/08/21 21:53:33 UTC
[GitHub] [airflow] agomez-etsy opened a new issue, #33600: Dynamic tasks fail when default_task_args contains custom fields
agomez-etsy opened a new issue, #33600:
URL: https://github.com/apache/airflow/issues/33600
### Apache Airflow version
Other Airflow 2 version (please specify below)
### What happened
Airflow version: **2.6.3**
Related to: https://github.com/apache/airflow/issues/29366
**DAG**
```python3
DEFAULT_TASK_ARGS = {
"owner": "myteam",
"start_date": "2023-08-18",
"project_id": "myproject",
"custom_arg": "foo",
}
@task
def foo() -> list[str]:
return ["a", "b", "c"]
@task
def bar(val):
logging.info(val)
with DAG(
...
default_args=DEFAULT_TASK_ARGS,
) as dag:
dynamic_tasks = bar.expand(val=foo())
# Some more tasks that use `project_id` and `custom_arg`
...
```
If a DAG specifies **default_task_args** and the result of one of its tasks (**foo**) is used to expand another task (**bar**), expanded tasks fail at runtime with the following error:
```
[2023-08-21, 21:07:16 UTC] {taskinstance.py:1824} ERROR - Task failed with exception
Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1407, in _run_raw_task
self._execute_task_with_callbacks(context, test_mode)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 1531, in _execute_task_with_callbacks
task_orig = self.render_templates(context=context)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/taskinstance.py", line 2179, in render_templates
original_task.render_template_fields(context)
...
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 429, in apply_defaults
result = func(self, **kwargs, default_args=default_args)
File "/home/airflow/.local/lib/python3.10/site-packages/airflow/models/baseoperator.py", line 788, in __init__
raise AirflowException(
airflow.exceptions.AirflowException: Invalid arguments were passed to _PythonDecoratedOperator (task_id: bar__1). Invalid arguments were:
**kwargs: {'project_id': "myproject", 'custom_arg': "foo"}
```
Tried to add **kwargs** to the list of arguments, but got the same result
```python3
@task
def bar(val, **kwargs):
logging.info(val)
```
Finally, tried calling **partial** before expanding to test if [this](https://github.com/apache/airflow/pull/29913) could fix it
```python3
dynamic_tasks = bar.partial().expand(val=foo())
```
### What you think should happen instead
Mapped tasks ignore non-generic default task arguments and run tasks accordingly.
### How to reproduce
Create a DAG similar to the one in the `What happened` section and run it.
### Operating System
Debian GNU/Linux 11 (bullseye)
### Versions of Apache Airflow Providers
apache-airflow-providers-airbyte==3.3.1
apache-airflow-providers-alibaba==2.4.1
apache-airflow-providers-amazon==8.2.0
apache-airflow-providers-apache-beam==5.1.1
apache-airflow-providers-apache-cassandra==3.2.1
apache-airflow-providers-apache-drill==2.4.1
apache-airflow-providers-apache-druid==3.4.1
apache-airflow-providers-apache-flink==1.1.1
apache-airflow-providers-apache-hdfs==4.1.0
apache-airflow-providers-apache-hive==6.1.1
apache-airflow-providers-apache-impala==1.1.1
apache-airflow-providers-apache-kylin==3.2.1
apache-airflow-providers-apache-livy==3.5.1
apache-airflow-providers-apache-pig==4.1.1
apache-airflow-providers-apache-pinot==4.1.1
apache-airflow-providers-apache-spark==4.1.1
apache-airflow-providers-apache-sqoop==3.2.1
apache-airflow-providers-arangodb==2.2.1
apache-airflow-providers-asana==2.2.1
apache-airflow-providers-atlassian-jira==2.1.1
apache-airflow-providers-celery==3.2.1
apache-airflow-providers-cloudant==3.2.1
apache-airflow-providers-cncf-kubernetes==7.4.1
apache-airflow-providers-common-sql==1.5.2
apache-airflow-providers-databricks==4.3.0
apache-airflow-providers-datadog==3.3.1
apache-airflow-providers-dbt-cloud==3.2.1
apache-airflow-providers-dingding==3.2.1
apache-airflow-providers-discord==3.3.0
apache-airflow-providers-docker==3.7.1
apache-airflow-providers-elasticsearch==4.5.1
apache-airflow-providers-exasol==4.2.1
apache-airflow-providers-facebook==3.2.1
apache-airflow-providers-ftp==3.4.2
apache-airflow-providers-github==2.3.1
apache-airflow-providers-google==10.5.0
apache-airflow-providers-grpc==3.2.1
apache-airflow-providers-hashicorp==3.4.1
apache-airflow-providers-http==4.4.2
apache-airflow-providers-imap==3.2.2
apache-airflow-providers-influxdb==2.2.1
apache-airflow-providers-jdbc==4.0.0
apache-airflow-providers-jenkins==3.3.1
apache-airflow-providers-microsoft-azure==6.1.2
apache-airflow-providers-microsoft-mssql==3.4.1
apache-airflow-providers-microsoft-psrp==2.3.1
apache-airflow-providers-microsoft-winrm==3.2.1
apache-airflow-providers-mongo==3.2.1
apache-airflow-providers-mysql==5.1.1
apache-airflow-providers-neo4j==3.3.1
apache-airflow-providers-odbc==4.0.0
apache-airflow-providers-openfaas==3.2.1
apache-airflow-providers-opsgenie==5.1.1
apache-airflow-providers-oracle==3.7.1
apache-airflow-providers-pagerduty==3.3.0
apache-airflow-providers-papermill==3.2.1
apache-airflow-providers-plexus==3.2.1
apache-airflow-providers-postgres==5.5.1
apache-airflow-providers-presto==5.1.1
apache-airflow-providers-qubole==3.4.1
apache-airflow-providers-redis==3.2.1
apache-airflow-providers-salesforce==5.4.1
apache-airflow-providers-samba==4.2.1
apache-airflow-providers-segment==3.2.1
apache-airflow-providers-sendgrid==3.2.1
apache-airflow-providers-sftp==4.3.1
apache-airflow-providers-singularity==3.2.1
apache-airflow-providers-slack==7.3.1
apache-airflow-providers-smtp==1.2.0
apache-airflow-providers-snowflake==4.2.0
apache-airflow-providers-sqlite==3.4.2
apache-airflow-providers-ssh==3.7.1
apache-airflow-providers-tableau==4.2.1
apache-airflow-providers-tabular==1.2.1
apache-airflow-providers-telegram==4.1.1
apache-airflow-providers-trino==5.1.1
apache-airflow-providers-vertica==3.4.1
apache-airflow-providers-zendesk==4.3.1
### Deployment
Other 3rd-party Helm chart
### Deployment details
Chart based on the official helm chart. Airflow running on Google Kubernetes Engine (GKE) using KubernetesExecutor.
### Anything else
_No response_
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "hkc-8010 (via GitHub)" <gi...@apache.org>.
hkc-8010 commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-1971838010
Hey @agomez-etsy ,
How about defining custom_args as dag_level params in default_args and then use it in creating Dynamic tasks?
Below is an example. I tested this on my local and its working.
```
from airflow.decorators import dag, task
from datetime import datetime
from airflow.operators.python import get_current_context
import logging
DEFAULT_TASK_ARGS = {
"owner": "myteam",
"params": {
"project_id": "myproject",
"custom_arg": "foo",
}
}
@task
def foo() -> list[str]:
return ["a", "b", "c"]
@task
def bar(val):
context = get_current_context()
project_id = context['params']['project_id']
custom_arg = context['params']['custom_arg']
logging.info(val)
logging.info(project_id)
logging.info(custom_arg)
@dag(default_args=DEFAULT_TASK_ARGS, schedule_interval=None, start_date=datetime(2021, 1, 1), catchup=False, dag_id='params_example')
def params_example():
dynamic_tasks = bar.expand(val=foo())
dynamic_tasks
params_example_dag = params_example()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "kaxil (via GitHub)" <gi...@apache.org>.
kaxil commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-2296517138
@uranusjr / @Lee-W -- you guys might know
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "agomez-etsy (via GitHub)" <gi...@apache.org>.
agomez-etsy commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-1969976367
hey @dadadima94 I just tried it on 2.7.3 and I still see the same error. I was working on this in my free time, but I've been kinda busy lately, so feel free to pick it up!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "dadadima94 (via GitHub)" <gi...@apache.org>.
dadadima94 commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-1900215365
Is there any known workaround to this issue?
I see that #29913 was merged to address this issue, am I wrong?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-2296313081
This fixes it, but I don't know why.
```properties
AIRFLOW__OPERATORS__ALLOW_ILLEGAL_ARGUMENTS=true
```
I think that if this is the cause, the error should occur in all `BaseOperator`s, not just `MappedOperator`.
Of course, it's possible that this doesn't fix the cause of the problem, just the symptom.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "agomez-etsy (via GitHub)" <gi...@apache.org>.
agomez-etsy commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-1969467213
hey @dadadima94. I don't think that change addressed the bug because it was merged on `2.6.0` and I tested it on `2.6.3`. We're now using `2.7.3`, so I can give it a try once again and will let you know how it goes
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "kyleburke-meq (via GitHub)" <gi...@apache.org>.
kyleburke-meq commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-2245871154
Are there any updates here? this appears to still be an issue on 2.9
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-2296515249
Using this property `_hook_apply_defaults` seems to use the wrong `default_args`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dynamic tasks fail when default_args contains custom fields [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-2296621844
Most operators are passed only the parameters specified when defining the class using `BaseOperatorMeta._apply_defaults()`.
However, `_TaskDecorator` bypasses that here and passes the entire `default_args`.
(At this point, `kwargs` already has the entire `default_args`.)
https://github.com/apache/airflow/blob/8def71479a8a4e943f8c39f07793c9fb2cf1fcc6/airflow/models/mappedoperator.py#L807-L818
This is where the entire `default_args` is passed.
https://github.com/apache/airflow/blob/8def71479a8a4e943f8c39f07793c9fb2cf1fcc6/airflow/decorators/base.py#L434-L439
https://github.com/apache/airflow/blob/8def71479a8a4e943f8c39f07793c9fb2cf1fcc6/airflow/decorators/base.py#L484-L488
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #33600: Dynamic tasks fail when default_args contains custom fields
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on issue #33600:
URL: https://github.com/apache/airflow/issues/33600#issuecomment-1691879533
@agomez-etsy assigned to you
waiting for your PR to address the issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org