[I] Using `start_from_trigger` with `context` and/or `template_fields` [airflow]

["karenbraganz (via GitHub)" <gi...@apache.org>]

karenbraganz opened a new issue, #51245:
URL: https://github.com/apache/airflow/issues/51245

   
   ### Discussed in https://github.com/apache/airflow/discussions/41944
   
   <div type='discussions-op-text'>
   
   <sup>Originally posted by **gil-tober** September  2, 2024</sup>
   Hey all,
   
   We are running Airflow on AWS EKS running `CeleryKubernetesExecutor` and have ~100K daily tasks and ~1500 daily dag_runs.
   
   In Airflow 2.10.0 the `start_from_trigger` feature was introduced for deferrable tasks and it's a feature that could greatly improve our performance and scaling of the cluster.
   
   We have a lot of tasks that are deferrable (mainly Redshift and BigQuery queries).
   Looking and the docs for this feature, trying to implement it and looking at the feature code, it seems that
   1.  `template_fields` are not being rendered before they are passed to the trigger
   2. There is no way to pass `context` to the trigger in order to render the fields that we need to render.
   
   Is there any way to achieve this? is there any future plans to implement fields rendering before the trigger is called?</div>


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Using `start_from_trigger` with `context` and/or `template_fields` [airflow]

["jroachgolf84 (via GitHub)" <gi...@apache.org>]

jroachgolf84 commented on issue #51245:
URL: https://github.com/apache/airflow/issues/51245#issuecomment-2922737346

   This is something that I'm seeing with another user, I'm going to see if I can get some eyes on it. Before that, is this the expected behavior?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] Using `start_from_trigger` with `context` and/or `template_fields` [airflow]

["raphaelauv (via GitHub)" <gi...@apache.org>]

raphaelauv commented on issue #51245:
URL: https://github.com/apache/airflow/issues/51245#issuecomment-5016692351

   it should works in airflow 3.3 @karenbraganz 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org