You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/08 13:50:31 UTC
[GitHub] [airflow] sd274 opened a new issue, #24325: `render_template_as_native_obj` breaks posting data with `SimpleHttpOperator`
sd274 opened a new issue, #24325:
URL: https://github.com/apache/airflow/issues/24325
### Apache Airflow version
2.3.0
### What happened
I'm trying to create a dag that uses the `SimpleHttpOperator` to post data to various endpoints and pass the responses to other endpoints. As I am passing the responses to other endpoints it is convenient to have `render_template_as_native_obj=True`. However, when I have this set to True it seems to cause issues with the data that is posted. Example below (works when `render_template_as_native_obj=False`) the http_conn_id is set up with `https://jsonplaceholder.typicode.com/`):
```
from datetime import datetime
import yaml
from airflow import DAG
from airflow.operators.bash import BashOperator
from airflow.providers.http.operators.http import SimpleHttpOperator
from airflow.decorators import dag, task
dag_id = f'http_test'
@dag(dag_id=dag_id, start_date=datetime(2022, 6,7), render_template_as_native_obj=True)
def http_test():
first_task = BashOperator(
task_id='first_task',
bash_command='echo helllo',
)
get_some_data_task = SimpleHttpOperator(
task_id='get_some_data',
method='POST',
endpoint='/posts',
http_conn_id='jsonplaceholder',
headers={"Content-Type": "application/json"},
data=json.dumps({
'title': 'foo',
'body': 'bar',
'userId': 1,
}),
)
first_task >> get_some_data_task
dag = http_test()
```
### What you think should happen instead
Having `render_template_as_native_obj` should only effect jinja templates, not other parts of the code.
### How to reproduce
Dag defined above works with `render_template_as_native_obj` as False, but fails with it as True.
### Operating System
mac-os
### Versions of Apache Airflow Providers
apache-airflow-providers-amazon @ file:///pip-packages/apache_airflow_providers_amazon-3.4.0-py3-none-any.whl
apache-airflow-providers-celery==2.1.0
apache-airflow-providers-cncf-kubernetes==2.1.0
apache-airflow-providers-docker==2.3.0
apache-airflow-providers-elasticsearch==2.1.0
apache-airflow-providers-ftp @ file:///pip-packages/apache_airflow_providers_ftp-2.1.2-py3-none-any.whl
apache-airflow-providers-google==6.1.0
apache-airflow-providers-grpc==2.0.1
apache-airflow-providers-hashicorp==2.1.1
apache-airflow-providers-http @ file:///pip-packages/apache_airflow_providers_http-2.1.2-py3-none-any.whl
apache-airflow-providers-imap @ file:///pip-packages/apache_airflow_providers_imap-2.2.3-py3-none-any.whl
apache-airflow-providers-microsoft-azure==3.3.0
apache-airflow-providers-mysql==2.1.1
apache-airflow-providers-odbc==2.0.1
apache-airflow-providers-postgres==2.3.0
apache-airflow-providers-redis==2.0.1
apache-airflow-providers-sendgrid==2.0.1
apache-airflow-providers-sftp==2.2.0
apache-airflow-providers-slack==4.1.0
apache-airflow-providers-snowflake @ file:///pip-packages/apache_airflow_providers_snowflake-2.7.0-py3-none-any.whl
apache-airflow-providers-sqlite @ file:///pip-packages/apache_airflow_providers_sqlite-2.1.3-py3-none-any.whl
apache-airflow-providers-ssh==2.3.0
### Deployment
Docker-Compose
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24325: `render_template_as_native_obj` breaks posting data with `SimpleHttpOperator`
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24325:
URL: https://github.com/apache/airflow/issues/24325#issuecomment-1149965252
What errors do you see? you have not written that. Please clarify. I expect this is a problem with some of the templated fields of the opearator that get a different value than string - but it's hard to say without you showing the error. My guess is that you are passing some data that looks like a Dict to a templated field and it gets rendered as a dict..
There are various ways you can deal with it - for eample you can extend SimpleHttpOperator and remove the field that causes the problem from templated_fields, pass your data in a different format, or disable retndering as native_obj,
By definition, render_as_nativeobj will render all templated_fields as native objects and the fact that it might not work in all cases is the whole reason why it is disabled by default - then you can decide in your DAG how to deal with it on your own after you enable it. Moving it to discussions awaiting more comments, and clairfications, but it seems like ti's working as expected for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24325: `render_template_as_native_obj` breaks posting data with `SimpleHttpOperator`
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24325:
URL: https://github.com/apache/airflow/issues/24325#issuecomment-1149943484
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org