You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/10 00:49:34 UTC
[GitHub] [airflow] iantbutler01 opened a new issue #10270: KubernetesPodOperator annotations kwarg should support templating.
iantbutler01 opened a new issue #10270:
URL: https://github.com/apache/airflow/issues/10270
<!--
Welcome to Apache Airflow! For a smooth issue process, try to answer the following questions.
Don't worry if they're not all applicable; just try to include what you can :-)
If you need to include code snippets or logs, please put them in fenced code
blocks. If they're super-long, please use the details tag like
<details><summary>super-long log</summary> lots of stuff </details>
Please delete these comment blocks before submitting the issue.
-->
**Description**
I would like the annotations kwarg in the KubernetesPodOperator to support templating.
**Use case / motivation**
With how the KubernetesExecutor runs tasks right now I can't generate an id in my dag file and use it for each task in the dag because it will be regenerated for each subsequent task. When the dag files are copied and ran on the individual Kubernetes pod the generation step is rerun.
For example if I set a variable x = str(uuid.uuid4()) at the beginning of my dag file and then pass x as an arg to 3 tasks each task will have a different value for that parameter on the KubernetesExecutor. To get around that we are using an external metadata DB and xcoms.
We would love to be able to template xcom data into the pod annotations.
Our specific use case is we're doing versioned web crawls using Airflow as our job scheduler for crawling and loading data in spark and we use annotations on the pod and xcoms to know which pod to communicate with over HTTP from different tasks for various monitoring and management tasks.
<!-- What do you want to happen?
Rather than telling us how you might implement this solution, try to take a
step back and describe what you are trying to achieve.
-->
**Related Issues**
<!-- Is there currently another issue associated with this? -->
I have not seen one.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-671126064
Great! I assigned you to this ticket. 😺
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iantbutler01 commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
iantbutler01 commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-830991921
I am not, my work has carried me far away from Airflow and I do not have the time these days @eladkal
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] iantbutler01 commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
iantbutler01 commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-671125684
Yes I am happy to work on this!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-671124794
@iantbutler01 It makes sense. Wouldn't you like to prepare a PR? I am happy to help with the review and merging the change.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] eladkal commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-830791256
@iantbutler01 are you still working on it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #10270: KubernetesPodOperator annotations kwarg should support templating.
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10270:
URL: https://github.com/apache/airflow/issues/10270#issuecomment-671126064
Great! I assigned you to this ticket.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org