You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/01/30 19:04:26 UTC
[GitHub] [airflow] YangMuye opened a new issue #13988: List and Dict template fields are rendered as JSON.
YangMuye opened a new issue #13988:
URL: https://github.com/apache/airflow/issues/13988
OracleOperator
**Apache Airflow version**: 2.0.0
**Kubernetes version (if you are using kubernetes)** (use `kubectl version`): n/a
**Environment**: Linux
- **Cloud provider or hardware configuration**: amd64
- **OS** (e.g. from /etc/os-release): Centos 7
- **Kernel** (e.g. `uname -a`):
- **Install tools**: pip
- **Others**:
**What happened**:
The field `sql` is rendered as a serialized json `["select 1 from dual", "select 2 from dual"]` instead of a list of syntax-highlighted SQL statements.
**What you expected to happen**:
`lists` and `dicts` should be rendered as lists and dicts rather than serialized json unless the `template_field_renderer` is `json`
**How to reproduce it**:
```
from airflow import DAG
from airflow.providers.oracle.operators.oracle import OracleOperator
with DAG("demo", default_args={owner='airflow'}, start_date= pendulum.yesterday(), schedule_interval='@daily',) as dag:
OracleOperator(task_id='single', sql='select 1 from dual')
OracleOperator(task_id='list', sql=['select 1 from dual', 'select 2 from dual'])
```
**Anything else we need to know**:
Introduced by #11061, .
A quick and dirty work-around:
Edit file [airflow/www/views.py](https://github.com/PolideaInternal/airflow/blob/13ba1ec5494848d4a54b3291bd8db5841bfad72e/airflow/www/views.py#L673)
```
if renderer in renderers:
- if isinstance(content, (dict, list)):
+ if isinstance(content, (dict, list)) and renderer is renderers['json']:
content = json.dumps(content, sort_keys=True, indent=4)
html_dict[template_field] = renderers[renderer](content)
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770265005
Thanks for opening your first issue here! Be sure to follow the issue template!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
VBhojawala edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770793786
Hi @mik-laj
I have already developed solution for this. If you @anushkrishnav have not started work on this can i create a PR for the same?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
turbaszek closed issue #13988:
URL: https://github.com/apache/airflow/issues/13988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the renderer, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----
EDIT:
After checking the latest source code, I was wrong. The behavior applies to all renderers.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771629797
> The breaking to individual item was the original behavior before #11061 ? I don't remember this
I don't remember it too, but it may have been only issue of `sql`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] anushkrishnav commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
anushkrishnav commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770618359
Awesome thanks !! @mik-laj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the renderer, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----
EDIT:
After checking the latest source code, I was wrong. The behavior applies to all renders.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] anushkrishnav commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
anushkrishnav commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770605160
I would like to work on the issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the render, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771598844
Hi @VBhojawala, why don't you just create a PR for it first?
If we do not need a sophisticated solution, it is mostly just about a one-line change as mentioned in my original post. If anyone has
a better solution, we can merge it later. I hope the patch is available in 2.0.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the renderer, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
turbaszek commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771629797
> The breaking to individual item was the original behavior before #11061 ? I don't remember this
I don't remember it too, but it may have been only issue of `sql`.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek closed issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
turbaszek closed issue #13988:
URL: https://github.com/apache/airflow/issues/13988
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770614698
@anushkrishnav Assigned!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771598844
Hi @VBhojawala, why don't you just create a PR for it first?
If we do not need a sophisticated solution, it is mostly just about a one-line change as mentioned in my original post. If anyone has better solution, we can merge them later. I hope the patch is available in 2.0.1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771598844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771598844
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the render, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771614496
The breaking to individual item was the original behavior before https://github.com/apache/airflow/pull/11061 ? I don't remember this
In the Json example https://github.com/apache/airflow/pull/11061 it doesn't break each key to it's own item @turbaszek
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] anushkrishnav commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
anushkrishnav commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770798787
I have already finished 50% of the solution
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the renderer, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----
After checking the latest source code, I was wrong. The behavior applies to all renders.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] YangMuye edited a comment on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
YangMuye edited a comment on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771628888
@RosterIn it is the behavior of some renderers (e.g. the SQL renderer), but not the others (such as the JSON renderer).
In the past, the field was directly passed to the renderer, so a SQL renderer would render a SQL field as a list as illustrated in my example, and a JSON renderer would render a JSON field as JSON as shown in his example. After the patch, anything that is a list is serialized into a string so the SQL render does not have a chance to get a list at all.
----
After checking the latest source code, I was wrong. The behavior applies to all renders.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] RosterIn commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
RosterIn commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-771614496
The breaking to individual item was the original behavior before https://github.com/apache/airflow/pull/11061 ? I don't remember this
In the Json example https://github.com/apache/airflow/pull/11061 it doesn't break each key to it's own item @turbaszek
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VBhojawala commented on issue #13988: List and Dict template fields are rendered as JSON.
Posted by GitBox <gi...@apache.org>.
VBhojawala commented on issue #13988:
URL: https://github.com/apache/airflow/issues/13988#issuecomment-770793786
Hi @anushkrishnav
I have already developed solution for this. If you have not started work on this can i create a PR for the same?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org