You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "ldacey (via GitHub)" <gi...@apache.org> on 2023/09/08 23:56:08 UTC
[GitHub] [airflow] ldacey opened a new issue, #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
ldacey opened a new issue, #34227:
URL: https://github.com/apache/airflow/issues/34227
### Apache Airflow version
2.7.1
### What happened
120 DAGs broke when I updated to Airflow 2.7.1. I had to revert to 2.7.0 again, so the tests below are from a local deployment.
The error message is the same for all DAGs:
```python
Broken DAG: [/opt/airflow/dags/dag-name.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dag.py", line 640, in __init__
self.validate_schedule_and_params()
File "/home/airflow/.local/lib/python3.11/site-packages/airflow/models/dag.py", line 3289, in validate_schedule_and_params
raise AirflowException(
airflow.exceptions.AirflowException: DAG Schedule must be None, if there are any required params without default values
```
After testing, I found that if I change the blob_prefix param to "" instead of None, the DAG imports successfully. I have various other params which are None that I will probably need to assign default values for (in reality, the function which Airflow runs handles cases if the value is None, but I guess I need to add a default in the DAG). All 120 DAGs with errors had a field like this in the "params" dictionary I pass to Airflow params.
```python
params = {
"exclude_pattern": "*errors*",
"blob_prefix": None,
```
### What you think should happen instead
These are legacy DAGs which have run for a long time without changes (old Airflow instance), so I did not expect an upgrade to break them.
### How to reproduce
I believe the root cause is passing a param with the value None to the DAG params:
```python
params = {
subfolder=None,
conn_id="gcs_conn_id",
}
@dag(
dag_id="dag-example",
start_date=datetime(2023, 2, 2, tz="America/New_York"),
schedule="41 2 * * *",
params=params,
)
```
### Operating System
Ubuntu 22.04
### Versions of Apache Airflow Providers
Providers info
apache-airflow-providers-celery | 3.3.3
apache-airflow-providers-common-sql | 1.7.1
apache-airflow-providers-docker | 3.7.4
apache-airflow-providers-ftp | 3.5.1
apache-airflow-providers-google | 10.7.0
apache-airflow-providers-http | 4.5.1
apache-airflow-providers-imap | 3.3.1
apache-airflow-providers-microsoft-azure | 6.3.0
apache-airflow-providers-mysql | 5.3.0
apache-airflow-providers-odbc | 4.0.0
apache-airflow-providers-openlineage | 1.0.2
apache-airflow-providers-postgres | 5.6.0
apache-airflow-providers-redis | 3.3.1
apache-airflow-providers-sftp | 4.6.0
apache-airflow-providers-sqlite | 3.4.3
apache-airflow-providers-ssh | 3.7.2
### Deployment
Other Docker-based deployment
### Deployment details
Docker Swarm using docker stack deploy
### Anything else
DAGs fail to import when deployed on 2.7.1
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
URL: https://github.com/apache/airflow/issues/34227
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on issue #34227:
URL: https://github.com/apache/airflow/issues/34227#issuecomment-1712544057
> Yes. I think the behaviour when param value was set to None has not been defined. And to be honest i am not sure we should interpret it as 'No default".. The None being default value seems to be pretty standard.
IMHO we should consider the params defined without a default value as **required** params, and the fail/exception should be raised when trying to create a DagRun without providing these required params.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] yermalov-here commented on issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "yermalov-here (via GitHub)" <gi...@apache.org>.
yermalov-here commented on issue #34227:
URL: https://github.com/apache/airflow/issues/34227#issuecomment-1712520975
looks like it was explicitly changed here #33141
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jens-scheffler-bosch commented on issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34227:
URL: https://github.com/apache/airflow/issues/34227#issuecomment-1712562403
I assume currently it is caused by an inconsistency. Because the check assumes to have a `Params` object but this is not enforced. In the trigger UI there is some wrapping being made to auto-detect typed making a `Param` object out of standard python data. But not when parsing a DAG.
`None` as default value is in my eyes totally right and as the bug is reported this is a valid use case. Also relates to my (still pending revision and open) PR #31301 .
I am currently looking though the history of the code, understanding when the problem has been introduced. I volunteer for a fix but want to understand the history first :-D
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #34227:
URL: https://github.com/apache/airflow/issues/34227#issuecomment-1712542074
Yes. I think the behaviour when param value was set to `None` has not been defined. And to be honest i am not sure we should interpret it as 'No default".. The None being default value seems to be pretty standard.
@jens-scheffler-bosch @hussein-awala @uranusjr -> WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jens-scheffler-bosch commented on issue #34227: 100+ DAGs fail to import after update to 2.7.1 due to "params without default values"
Posted by "jens-scheffler-bosch (via GitHub)" <gi...@apache.org>.
jens-scheffler-bosch commented on issue #34227:
URL: https://github.com/apache/airflow/issues/34227#issuecomment-1712562780
> looks like it was explicitly changed here #33141
Not related to this, as in the reported case there is no `Param` object used.
In #33141 the fix was for `param = { "key": Param(None) }` which was badly serialized, but here it is `param = { "key": None }`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org