You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/07/29 18:20:16 UTC
[GitHub] [airflow] charlesdong1991 opened a new issue, #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
charlesdong1991 opened a new issue, #25409:
URL: https://github.com/apache/airflow/issues/25409
### Apache Airflow version
2.2.2
### What happened
Use a custom operator inherited from BaseOperator, and try to assign a list of sql files to an attribute that is not defined in `template_fields`, but it still gets rendered, unless the value assigned to the attribute is a string.
Below is the minimum reproducible example and its output, except with a query folder next to dag file, and in this example,
- I expect all attributes starting with suffix of original shouldn’t be rendered
- In fact, only `self.original_inputs_str` doesn’t get rendered (a string is assigned to it), but the other two, which are assigned a list, somehow both get rendered.
Not sure if I am misunderstanding the templated_fields, or this might be a bug.
Version: 2.2.2
Python: 3.7
### What you think should happen instead
Might be a bug? Or I misunderstood how to use it.
### How to reproduce
```
from typing import Sequence
from airflow import DAG
from airflow.models import BaseOperator
from pendulum import datetime
class TestOperator(BaseOperator):
template_fields: Sequence[str] = ("inputs", "inputs_one", "inputs_two")
template_ext: Sequence[str] = (".sql",)
def __init__(self, inputs, inputs_one, inputs_two, **base_operator_kwargs):
super().__init__(**base_operator_kwargs)
# should render
self.inputs = inputs
self.inputs_one = inputs_one
self.inputs_two = inputs_two
# shouldn't render
self.original_inputs_str = inputs
self.original_inputs_one = inputs_one
self.original_inputs_two = inputs_two
def execute(self, context):
# render correctly
print(f"input: {self.inputs}")
print(f"input 1: {self.inputs_one}")
print(f"input 2: {self.inputs_two}")
# expected to be NOT rendered and doesn't get rendered
print(f"original input str: {self.original_inputs_str}")
# both expected to be NOT rendered but get rendered
print(f"original input 1: {self.original_inputs_one}")
print(f"original input 2: {self.original_inputs_two}")
with DAG(
dag_id="test",
description="testing template fields and ext",
start_date=datetime(2022, 7, 28),
catchup=False,
default_args=dict(owner="test"),
tags=["test"],
) as dag:
inputs_str = "query/hahaha.sql"
inputs_one = ["query/hahaha.sql"]
inputs_two = ["query/haha.sql", "query/hahaha.sql"]
TestOperator(
dag=dag,
task_id="test",
inputs=inputs_str,
inputs_one=inputs_one,
inputs_two=inputs_two,
)
```
### Operating System
Amazon Linux AMI
### Versions of Apache Airflow Providers
_No response_
### Deployment
MWAA
### Deployment details
_No response_
### Anything else
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1200063773
Yes, your `original_inputs_one` and `inputs_one` (same as `two`) ultimately point to the same value, and it’s just how Python works. Airflow can’t blindly do `copy.copy` either since while it solves the problem for built-in types, you’d still hit the same issue if you pass in custom class objects. So I’m inclined to close this as won’t fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1200123611
@charlesdong1991 Just a silly question, why you need keep original values of rendered attributes?
Due to my experience last time when someone want to keep both value did it only for logging purpose, however original values before rendering shown in UI: `Task Instances -> Task Instance Detail -> Task Attributes`
Might be it also your case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] charlesdong1991 commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
charlesdong1991 commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1200139086
Hey, @Taragolis it isn't silly at all, probably I am doing something very silly, haha 😅
So the use case at my company is, when we get the parsed sql query from for instance the file `haha.sql`, we also want to get this filename used for other logics in our codebase. However, when setting the `template_fields` and `template_ext`, we cannot get that original filename anymore, and then we take a work it around to try to assign to a different attribute.
So, if I understand correctly, for this case, we could use `ti.xcom_pull` for instance, to also let us get the original name for the attribute in `template_fields`?
Thanks a lot for your help!! 🙇 And happy weekend!! 🤗
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1199826806
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr closed issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
uranusjr closed issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
URL: https://github.com/apache/airflow/issues/25409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] Taragolis commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
Taragolis commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1199925757
I think the root of the problem is that Python strings is immutable, but list is mutable.
```python
from copy import copy
class TestOperator(BaseOperator):
template_fields: Sequence[str] = ("inputs", "inputs_one", "inputs_two")
template_ext: Sequence[str] = (".sql",)
def __init__(self, inputs, inputs_one, inputs_two, **base_operator_kwargs):
super().__init__(**base_operator_kwargs)
def __init__(self, inputs, inputs_one, inputs_two, **base_operator_kwargs):
super().__init__(**base_operator_kwargs)
# should render, create shallow copy of original objects
# might required deepcopy, if nested object also mutable.
# from copy import deepcopy
self.inputs = copy(inputs)
self.inputs_one = copy(inputs_one)
self.inputs_two = copy(inputs_two)
# shouldn't render
self.original_inputs_str = inputs
self.original_inputs_one = inputs_one
self.original_inputs_two = inputs_two
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] charlesdong1991 commented on issue #25409: [Templating]: attributes not defined in `templated_fields` also get value rendered
Posted by GitBox <gi...@apache.org>.
charlesdong1991 commented on issue #25409:
URL: https://github.com/apache/airflow/issues/25409#issuecomment-1200110218
ah, I see where it went wrong!!
Great thanks for the answer!!! Appreciate it a lot!! @Taragolis @uranusjr 🙇
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org