You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "raphaelauv (via GitHub)" <gi...@apache.org> on 2024/10/28 09:27:23 UTC
[I] param object do not accept array [airflow]
raphaelauv opened a new issue, #43433:
URL: https://github.com/apache/airflow/issues/43433
### Apache Airflow version
2.10.2
### If "Other Airflow 2 version" selected, which one?
_No response_
### What happened?
I can't trigger a dag with `[ ]` or `[{"toto": 5}]` ( that are valid JSON )
```python
with DAG(
dag_id="a",
params={
"x": Param(None, type="object"),
})
```
it log
```log
Invalid input for param batch: [{'toto': 5}] is not of type 'object' Failed validating 'type' in schema: {'type': 'object'} On instance: [{'toto': 5}]
```
### What you think should happen instead?
_No response_
### How to reproduce
-----
### Operating System
----
### Versions of Apache Airflow Providers
_No response_
### Deployment
Official Apache Airflow Helm Chart
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "bbovenzi (via GitHub)" <gi...@apache.org>.
bbovenzi closed issue #43433: param object do not accept array
URL: https://github.com/apache/airflow/issues/43433
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2454217893
thanks for your help , but I don't want to trigger the dag with an array of string or integer, but an array of dict/map
`[{"toto": 5,"tata":"hello"}]`
I tried
```python
"x": Param(None, type="array",items={"type": "object"}),
```
but the front fail with
```
An invalid form control with name='element_x' is not focusable.
<textarea class="form-control" name="element_x" id="element_x" valuetype="advancedarray" rows="6" required="" style="display: none;"></textarea>
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2453092870
I assume this is correct.
The term "object" in the JSON schema validation we run refers to "dict" object types. See: https://json-schema.org/understanding-json-schema/reference/object.
So in your case you need to define it as array. Note that empty arrays in the UI form are always triggered with `none` value... there was also a ticket open for this to discuss how the form shall handle empty fields...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2457950263
Yeah - a small hint/clarificatio nin the docs would be good - there might be others running into the same pitfall.
Also if the default is None as you initially had... might be a small bug. For DAGs that are not scheduled "non valid defaults" are possible for all other fields. So is a small bug in the form generation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2455643752
Okay, @raphaelauv maybe you were hitting a side effect... I did a check on my side adding to airflow/example_dags/example_params_ui_tutorial.py with:
```
# An array of numbers
"array_of_dicts": Param(
[{}, {"key": "value"}, {"one": "two"}],
"Only dicts are accepted in this array",
type="array",
title="Array of dicts",
items={"type": "object"},
),
```
..alongside to other parameters.
This renders correctly on my side with:
Can you check adding a "valid" default value? Because you define `None`per default - which is not valid according to specs. So maybe this special case with a invalid/empty default was not tested. If you want to permit `None` as valid value then you need to set `type=["none", "array"]`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2453609553
Yea... and no. Please check the details in https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html#use-params-to-provide-a-trigger-ui-form -might not be obvious but exactly for this case a feature was added:
> If you add the attribute items with a
dictionary that contains a field type
with a value other than “string”, a JSON entry
field will be generated for more array types and
additional type validation as described in
[JSON Schema Array Items](https://json-schema.org/understanding-json-schema/reference/array.html#items).
...means in case you want to have non-string items be added define an item type and then a JSON style input box will be added just not validating to an object.
One example is contained in the example DAGs: https://github.com/apache/airflow/blob/main/airflow/example_dags/example_params_ui_tutorial.py#L136 - just using numbers and not dict's.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "jscheffl (via GitHub)" <gi...@apache.org>.
jscheffl commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2455742806
> so with `type=["none", "array"]` dag parsing is failing
aaah, sorry RTFM.... https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/params.html --> `type=["null", "array"]` is the right typing in JSON schema...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2455707046
so with `type=["none", "array"]` dag parsing is failing
```log
Broken DAG: [/opt/airflow/dags/dags/aaaa.py] Traceback (most recent call last):
File "/home/airflow/.local/lib/python3.12/site-packages/jsonschema/validators.py", line 1328, in validate
cls.check_schema(schema)
File "/home/airflow/.local/lib/python3.12/site-packages/jsonschema/validators.py", line 317, in check_schema
raise exceptions.SchemaError.create_from(error)
jsonschema.exceptions.SchemaError: ['none', 'array'] is not valid under any of the given schemas
```
----
but using a not none default value is working
```python
params={
"x": Param([{}], type="array", items={"type": "object"}),
}
```
it let me trigger the dag with the argument `[{"toto": 5,"tata":"hello"}]`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "JuliaBars (via GitHub)" <gi...@apache.org>.
JuliaBars commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2750986236
Dear all, i have almost the same issue i am passing [{"toto": 5}] from UI and DAG fails when does data = json.loads() for params
Ive checked that it convert [{"toto": 5}] to "[{"toto": 5}]" and later cant parse it from {"my_param": 1, "my_list_param": "[{"toto": 5}]"} exactly because this double quotes which airflow added by itself
If you will locally try json.loads('{"my_param": 1, "my_list_param": "[{"toto": 5}]"}') it will fails
and once you remove double quotes around list it will work json.loads('{"my_param": 1, "my_list_param": [{"toto": 5}]}')
So dont know how to prevent this behaviur or to fix on my side, using exactly what @raphaelauv "x": Param("", type=["array"], items={"type": "object"}),
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2453556417
hey , using "array" change my value(array) to a string in another array
`['[{"toto": 5}]']`
```python
from airflow.models import Param
from airflow.operators.python import PythonOperator
from pendulum import today
from airflow import DAG
with DAG(
dag_id="a",
schedule_interval=None,
start_date=today("UTC").add(days=-1),
params={
"x": Param([], type="array"),
}
):
def a(toto):
print(toto)
print(type(toto))
PythonOperator(task_id="toto", python_callable=a, op_kwargs={"toto": "{{params.x}}"})
```
so I must use `string` until a better support
```python
with DAG(
dag_id="a",
schedule_interval=None,
start_date=today("UTC").add(days=-1),
params={
"x": Param(None, type="string"),
}
):
def a(toto):
toto = json.loads(toto)
print(toto)
PythonOperator(task_id="toto", python_callable=a, op_kwargs={"toto": "{{params.x}}"})
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] param object do not accept array [airflow]
Posted by "raphaelauv (via GitHub)" <gi...@apache.org>.
raphaelauv commented on issue #43433:
URL: https://github.com/apache/airflow/issues/43433#issuecomment-2455814004
yeah it work
```python
params={
"x": Param(None, type=["null", "array"], items={"type": "object"}),
}
```
so conclusion it's not obvious, if we decide to not let accept array in the type `object`
than I will add a little example in the doc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org