You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kosteev (via GitHub)" <gi...@apache.org> on 2024/03/07 10:50:53 UTC

[I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

kosteev opened a new issue, #37971:
URL: https://github.com/apache/airflow/issues/37971

   ### Apache Airflow version
   
   Other Airflow 2 version (please specify below)
   
   ### If "Other Airflow 2 version" selected, which one?
   
   2.6.3
   
   ### What happened?
   
   Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec.
   In Airflow 2.5.3 [scheduler]scheduler_heartbeat_sec is used correctly.
   
   The bug was introduced after refactoring of scheduler_job -> scheduler_job_runner. 
   This field is never used in the code:
   https://github.com/apache/airflow/blob/1f5960a9844ad9bcea61940a76052e0af549579f/airflow/jobs/scheduler_job_runner.py#L153
   
   ### What you think should happen instead?
   
   _No response_
   
   ### How to reproduce
   
   - run Airflow
   - set [scheduler]job_heartbeat_sec with the value greater than [scheduler]scheduler_health_check_threshold
   - see warning message in Airflow UI that scheduler is not alive
   
   ### Operating System
   
   Ubuntu
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Deployment
   
   Google Cloud Composer
   
   ### Deployment details
   
   _No response_
   
   ### Anything else?
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [ ] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985600389

   Yeah - I think the way where it is closer to heartrate property is a bit better and easier to reason about.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565404

   (just wanted to make sure how far back it goes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985546310

   Ahh.. those parameters are confusing and I am mixing them ... it is indeed way older sorry (threshold mixed with secs)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985536704

   @kosteev  - are you sure it's affecting 2.6.3 Airflow ? from what I traced #34026 has been only part of v2-8 branch. Maybe there was another reason before in 2.6.3 - but quite certainly this one was only introduced in 2.8.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985518074

   Actually not even there
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565030

   So it's there since 2.6.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985603095

   We have a bit of a mess there - because the JOB (scheduler/job_heartrabeat_sec)  has been used and abused by different jobs, so likely better to keep it in JOB.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1984702476

   Are you sure @kosteev  ? 
   
   From what I see it is used n Job's `perform_heartbeat` method::
   
   ```
   def perform_heartbeat(
       job: Job, heartbeat_callback: Callable[[Session], None], only_if_necessary: bool
   ) -> None:
       """
       Perform heartbeat for the Job passed to it,optionally checking if it is necessary.
   
       :param job: job to perform heartbeat for
       :param heartbeat_callback: callback to run by the heartbeat
       :param only_if_necessary: only heartbeat if it is necessary (i.e. if there are things to run for
           triggerer for example)
       """
       seconds_remaining: float = 0.0
       if job.latest_heartbeat and job.heartrate: # <--- HERE THE HEARTRATE IS USED FROM JOB (and one level down)
           seconds_remaining = job.heartrate - (timezone.utcnow() - job.latest_heartbeat).total_seconds()
       if seconds_remaining > 0 and only_if_necessary:
           return
       job.heartbeat(heartbeat_callback=heartbeat_callback)
   ```
   
   I believe there were some fixes after 2.6.3 that fixed bad threshold calculations for some jobs (It was confusing whether the job used 2.1* multiplier or whether you had)  but I'd expect heartrate was not affected - maybe threshold is (or rather was) calculated wrongly.
   
   Also this scenario:
   
   ```
   1. run Airflow
   2. set [scheduler]job_heartbeat_sec with the value greater than [scheduler]scheduler_health_check_threshold
   3. see warning message in Airflow UI that scheduler is not alive
   ```
   
   is expected. If heartbeat sec is greater than threshold, then yes you'd expect to see the warning (the warning is issued when the heartbeat does not arrive in the threshold time - which would be exactly what is happening if those values are set as described). 
   
   Mybe the reproduction scenario is wrong? Could you please clarify ?
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985334309

   In perform_heartbeat, job.heartrate is used.
   The issue is that for scheduler it is defined in SchedulerJobRunner (previously was SchedulerJob):
   https://github.com/apache/airflow/blob/08efc32fd13c505a3c04b97f9bc81c79d1725858/airflow/jobs/scheduler_job_runner.py#L153
   
   so, now this heartrate: int = conf.getint("scheduler", "SCHEDULER_HEARTBEAT_SEC") from SchedulerJobRunner is not used anywhere


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985337125

   About reproduction scenario:
   "If heartbeat sec is greater than threshold, then yes you'd expect to see the warning" - yes, but in the scenario you modify [scheduler]job_heartbeat_sec with property which shouldn't have to do anything with scheduler, as scheduler heartbeat should be controlled by [scheduler]scheduler_heartbeat_sec


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985513790

   I actualy took a quick look and it turned out that it was removed after adding SchedulerJobRunner. The SchedulerJob check has been removed by #34026 a bit later.
   
   I prepared a fix with test in #37992


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985472529

   > so, now this heartrate: int = conf.getint("scheduler", "SCHEDULER_HEARTBEAT_SEC") from SchedulerJobRunner is not used anywhere
   
   Indeed.
   
   > "If heartbeat sec is greater than threshold, then yes you'd expect to see the warning" - yes, but in the scenario you modify [scheduler]job_heartbeat_sec with property which shouldn't have to do anything with scheduler, as scheduler heartbeat should be controlled by [scheduler]scheduler_heartbeat_sec
   
   Ah :facepalm: -> indeed missed that it is about the other config.
   
   Would you like to fix it maybe @kosteev ? That seems like a good contribution and simple fix to make ?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985591616

   I prepared a fix like this https://github.com/apache/airflow/pull/37993
   
   But I see you have already fix (and implemented differently). Should I close my PR?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985539571

   The 2.6.3 version  is:
   
   ```
       def is_alive(self, grace_multiplier=2.1):
           """
           Is this job currently alive.
   
           We define alive as in a state of RUNNING, and having sent a heartbeat
           within a multiple of the heartrate (default of 2.1)
   
           :param grace_multiplier: multiplier of heartrate to require heart beat
               within
           """
           if self.job_type == "SchedulerJob":
               health_check_threshold: int = conf.getint("scheduler", "scheduler_health_check_threshold")
           else:
               health_check_threshold: int = self.heartrate * grace_multiplier
           return (
               self.state == State.RUNNING
               and (timezone.utcnow() - self.latest_heartbeat).total_seconds() < health_check_threshold
           )
   ```
   
   So almost for sure it wouldn't have caused the problem.
   
   (BTW. I am trying to see if we can add the fix to 2.8.3 that we vote on currently - because otherwise it would affect the whole 2.8 line)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985504904

   I will try to prepare a fix, but I do not want to commit to this, so that if anyone wants to fix it.
   I will try to work on it today, early next week.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985564821

   Tracked it down (there were quite a few refactors) and indeed it was #30255 that introduced it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37971: Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec
URL: https://github.com/apache/airflow/issues/37971


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org