[GitHub] [airflow] hterik commented on issue #19135: PythonKubernetesOperator and kubernetes taskflow decorator

[GitBox <gi...@apache.org>]

hterik commented on issue #19135:
URL: https://github.com/apache/airflow/issues/19135#issuecomment-1029776459


   Did you see the [CeleryKubernetes executor](https://airflow.apache.org/docs/apache-airflow/stable/executor/celery_kubernetes.html) ? That allows mixing K8 with celery 
   
   Can also recommend `["executor_config"]["pod_override"]` on taskflow api if you have need to specify exact dependencies per task. We have custom wrappers on top of this so you just have to write:
   ```
   @node_type=HighPerformance
   @task
   def foo():
      ....
   ```
   Actually it is a lot more granular than this but you get the idea.
   
   For small lightweight sensors there is possibility to batch them into https://airflow.apache.org/docs/apache-airflow/stable/concepts/smart-sensors.html or https://airflow.apache.org/docs/apache-airflow/stable/concepts/deferring.html
   
   Not against your idea of a custom operator, just handing out tips :) 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org