[GitHub] [airflow] ecodina commented on issue #28647: Intermitent log on deferrable operator

[GitBox <gi...@apache.org>]

ecodina commented on issue #28647:
URL: https://github.com/apache/airflow/issues/28647#issuecomment-1368017243

   Hi @pankajastro 
   
   We use SLURM to distribute tasks in different nodes of our cluster. The Airflow operator we use submits a job to SLURM. The workflow is as follows:
   
   Operator: log a few things & submit job & defer itself -> Trigger: check the SLURM log. When a few lines have been added, yield a TriggerEvent -> Operator: receive the new SLURM log lines in the TriggerEvent and log them in Airflow & defer itself again
   
   Therefore, all the logging happens inside the Operator, and not in the trigger.
   
   It looks like a frontend bug: when the operator is deferred (the trigger "runs point"), the website loses count of what try is happening and defaults to the last one.
   
   On Monday I'll try to write a simpler operator+triggerer (same structure but without using SLURM) to see if this keeps happening.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org