[GitHub] [airflow] ashb commented on a change in pull request #10917: Fix race conditions in task callback invocations

[GitBox <gi...@apache.org>]

ashb commented on a change in pull request #10917:
URL: https://github.com/apache/airflow/pull/10917#discussion_r537742744



##########
File path: airflow/models/taskinstance.py
##########
@@ -105,6 +107,29 @@ def set_current_context(context: Context):
             )
 
 
+def load_error_file(fd: IO[bytes]) -> Optional[Union[str, Exception]]:
+    """Load and return error from error file"""
+    fd.seek(0, 0)

Review comment:
       ```suggestion
       fd.seek(0, os.SEEK_SET)
   ```
   (Not important if we don't do this)

##########
File path: airflow/jobs/local_task_job.py
##########
@@ -98,7 +98,10 @@ def signal_handler(signum, frame):
 
             heartbeat_time_limit = conf.getint('scheduler', 'scheduler_zombie_task_threshold')
 
-            while True:
+            # task callback invocation happens either here or in
+            # self.heartbeat() instead of taskinstance._run_raw_task to
+            # avoid race conditions
+            while not self.terminating:

Review comment:
       What's this change for? How come we didn't do it before?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org