You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/03 13:02:10 UTC
[GitHub] [airflow] shane-lab opened a new issue, #24164: AbstractOperator should have broader support Jinja templates
shane-lab opened a new issue, #24164:
URL: https://github.com/apache/airflow/issues/24164
### Description
At the moment the Jinja templates are resolved at the `AbstractOperator` class during [render_template](3632dc0ede695827d1ebdbd091401e94d/airflow/models/abstractoperator.py#L357). It is best practice to use the provided Kubernetes client package for defining custom resources when using the derived `KubernetesPodOperator`. This gives issues when passing through class instances from the k8s client package because the assertions are currently only present for native types and instances of `DagParam` or `XComArg`.
### Use case/motivation
To give a better example: say I would like to invoke an init container with templated values for my KubernetesPodOperator task using the corresponding classes from the provided k8s client package. However, the current assertions will ignore the class instances and forces me to convert the objects over to dictionaries..
```py
from airflow.providers.cncf.kubernetes.operators.kubernetes_pod import KubernetesPodOperator
from kubernetes.client import models as k8s_models
....
init_container = k8s_models.V1Container(
name="init",
image="alpine:latest",
args=["echo", "{{ ti.xcom_pull(key=\"my_xcom_value\") }}"]
)
task = KubernetesPodOperator(
init_containers=[init_container],
...
)
task.template_fields = task.template_fields + ("init_containers",)
```
Fortunately, the KubernetesPodOperator is able to map from dict to the corresponding classes of the kubernetes client package already, which solves my "issue", but maybe a two-way conversion could be a solution to the mentioned inconvenience above.
### Related issues
It would also be great if it was possible to overload/extend the `template_fields` through the constructor. At the moment one has to extend this object after creating an instance of the operator (or create a custom derived class).
e.g.
```py
class MyKubernetesPodOperator(KubernetesPodOperator):
def __init__(self, template_fields=(), **kwargs):
super().__init__(**kwargs)
self.template_fields = self.template_fields + template_fields
```
### Are you willing to submit a PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24164: AbstractOperator should have broader support Jinja templates
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24164:
URL: https://github.com/apache/airflow/issues/24164#issuecomment-1146226315
Draft PR would be great. You can even start with draft PR for the templated_fields and see if you can make it works first.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on issue #24164: AbstractOperator should have broader support Jinja templates
Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #24164:
URL: https://github.com/apache/airflow/issues/24164#issuecomment-1145941033
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #24164: AbstractOperator should have broader support Jinja templates
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #24164:
URL: https://github.com/apache/airflow/issues/24164#issuecomment-1146225616
Why don't you make a POC for that. I don't quite get what you really want to do, but you seem to be well versed in Python and discussion about actual code is much better than clarifying the issue in discussion.
That could also give you a chance to understanda how it works and see if what you propose is doable at all @shane-lab
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] uranusjr commented on issue #24164: AbstractOperator should have broader support Jinja templates
Posted by GitBox <gi...@apache.org>.
uranusjr commented on issue #24164:
URL: https://github.com/apache/airflow/issues/24164#issuecomment-1146477265
I beleive some parts of the scheduler and task runner code uses `template_fields` as a _class variable_, so run-time per-instance `template_fields` additions will break certain features.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org