You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "phi-friday (via GitHub)" <gi...@apache.org> on 2024/07/25 07:34:38 UTC
[I] `airflow.Param` fails to get the default value of `array`. [airflow]
phi-friday opened a new issue, #41019:
URL: https://github.com/apache/airflow/issues/41019
### Apache Airflow version
2.9.3
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
When I try to run `DAG`, it fails to get the default values, so I can't run `DAG`.
### What you think should happen instead?
The `has_default_list` should default to an empty array.
### How to reproduce
```python
from __future__ import annotations
from airflow.decorators import dag
from airflow.models.param import Param
from airflow.operators.empty import EmptyOperator
from pendulum import datetime
@dag(
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
params={
"has_default_list": Param(
default=[], items={"type": "string"}, title="Has default list", type="array"
)
},
)
def render_test() -> None:
EmptyOperator(task_id="empty_task")
render_test()
```
### Operating System
PRETTY_NAME="Ubuntu 22.04.3 LTS" NAME="Ubuntu" VERSION_ID="22.04" VERSION="22.04.3 LTS (Jammy Jellyfish)" VERSION_CODENAME=jammy ID=ubuntu ID_LIKE=debian HOME_URL="https://www.ubuntu.com/" SUPPORT_URL="https://help.ubuntu.com/" BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/" PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy" UBUNTU_CODENAME=jammy
### Versions of Apache Airflow Providers
```shell
➜ dags pip freeze | grep apache-airflow-providers
apache-airflow-providers-celery==3.7.2
apache-airflow-providers-common-io==1.3.2
apache-airflow-providers-common-sql==1.14.2
apache-airflow-providers-docker==3.12.2
apache-airflow-providers-fab==1.1.1
apache-airflow-providers-ftp==3.10.0
apache-airflow-providers-http==4.12.0
apache-airflow-providers-imap==3.6.1
apache-airflow-providers-jdbc==4.3.1
apache-airflow-providers-odbc==4.6.2
apache-airflow-providers-postgres==5.11.2
apache-airflow-providers-redis==3.7.1
apache-airflow-providers-smtp==1.7.1
apache-airflow-providers-sqlite==3.8.1
```
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2295037909
This issue has been automatically marked as stale because it has been open for 14 days with no response from the author. It will be closed in next 7 days if no further activity occurs from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2265979007
Okay, then this is a similar issue like we also had discussed in #33923 about how to handle "empty lists" and "empty strings" which are treated as `None` value when re-parsing the form.
I agree this is rather a "structural" problem. No easy fix. But we are currently planning to re-vamp the UI for an upcoming Airflow 3. If you want to contribute a fix or idea... helping hands are always welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2251786048
When the default value of `has_default_list` was `“[]”`, there was no problem
because it was unscheduled and the parameter validation procedure was ignored.
When a schedule is specified, the validation process is started and an error is thrown.
For `array`, it seems that if the value is empty, it is automatically rendered as `null`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2249662622
I've noticed that
if I pass a serialized value as a `json` string instead of a `python` object,
I can get the default value.
Does the default value for `airflow.Param` only accept `json` strings?
```python
➜ dags cat render_test.py
from __future__ import annotations
from airflow.decorators import dag
from airflow.models.param import Param
from airflow.operators.empty import EmptyOperator
from pendulum import datetime
@dag(
start_date=datetime(2024, 1, 1),
schedule=None,
catchup=False,
params={
"has_default_list": Param(
default="[]", items={"type": "string"}, title="Has default list", type="array"
)
},
)
def render_test() -> None:
EmptyOperator(task_id="empty_task")
render_test()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "techolga (via GitHub)" <gi...@apache.org>.
techolga commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2265947375
Hi,
I introduced the change of the UI element in `2.9.3`. with 60243e77ddf1ecea2aad524cebbd174fb03d40f5. The JSON input element was previously shown on simple string arrays because of a bug in the template. I was not aware of the side effect it has on the handling of an empty array as default, but I think now it conforms [AIP-50](https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-50+Trigger+DAG+UI+Extension+with+Flexible+User+Form+Concept).
> - Type of the field derived from the JSON schema type and formatting clause
> * Type boolean can render a checkbox or on/off control
> * Elements of JSON format date and date-time can render a date picker
> * Type of arrays can be made as lists of text string and object types as a JSON sub-element entry
> - If JSON schema allows to have types of null can control if a field is optional or required.
As far as I understand the current mechanism of the trigger page it looks like this:
1. the default value is inserted into UI input element, if the element is not a sequence nothing is inserted (what else should we do with an empty list as default value?)
2. the content of the input element is extracted and displayed in the `Generated Configuration JSON` element (how should we now distinguish between an empty array and null?)
At the moment I do not see a clever way to handle this edge case with the current structure...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2309064063
This issue has been closed because it has not received response from the issue author.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2256834438
Can you please describe a bit more in detail what "So I can't run DAG" means? Does it not start or do you get a specific error message?
What I can tell is if the array field is empty, an empty array is treated as "None"/NULL Value and if you define the field as being array I assume it should complain that the (required, see a red start sign at the label) is missing a value. DAG should reject to be running as long as you don't enter at least one entry.
Otherwise if you have a default value of `"[]"` I would expect that the string `[]` is used as default which is an array of one element with 1 string as default - which is more than NULL and represents one value, so is valid.
Don't know if I mis-understand or if this is a race / corner case you hit. Either it is a mis-understanding or really a bug. I don't get it from the error description.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2266357262
If we don't allow empty strings and empty lists as defaults, I think users will add handling for `null` when defining `DAG`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2257221667
> Can you please describe a bit more in detail what "So I can't run DAG" means? Does it not start or do you get a specific error message?
>
> What I can tell is if the array field is empty, an empty array is treated as "None"/NULL Value and if you define the field as being array I assume it should complain that the (required, see a red start sign at the label) is missing a value. DAG should reject to be running as long as you don't enter at least one entry.
>
> Otherwise if you have a default value of `"[]"` I would expect that the string `[]` is used as default which is an array of one element with 1 string as default - which is more than NULL and represents one value, so is valid.
>
> Don't know if I mis-understand or if this is a race / corner case you hit. Either it is a mis-understanding or really a bug. I don't get it from the error description.
If treating an empty array as a `null` value is an intentional implementation, then this is not a bug.
I passed an empty array as the default, but I couldn't run `DAG` because it didn't accept an empty array and required at least one element.
As you said, I get a message that a value is missing.
I passed in the string `“[]”`, just in case.
If `schedule` is not specified, the default value of the parameter is not validated.
So I was mistaken. What is rendered as `[“[”, “]”]` is not what I expected.
If this is what was intended, I don't see a problem.
However, it seems a bit odd that in a parameter that has a default value and that I defined to not allow `null` values, an empty array is treated as `null` without allowing it.
Currently, I've redefined the parameter to allow `null` values, and added handling for `null` values in some of the `Task`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "phi-friday (via GitHub)" <gi...@apache.org>.
phi-friday commented on issue #41019:
URL: https://github.com/apache/airflow/issues/41019#issuecomment-2257239855
Also, in `2.8.4`, it works like this
In `2.9.3`, it looks like this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `airflow.Param` fails to get the default value of `array`. [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #41019: `airflow.Param` fails to get the default value of `array`.
URL: https://github.com/apache/airflow/issues/41019
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org