You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "kosteev (via GitHub)" <gi...@apache.org> on 2024/03/07 10:50:53 UTC
[I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
kosteev opened a new issue, #37971:
URL: https://github.com/apache/airflow/issues/37971
### Apache Airflow version
Other Airflow 2 version (please specify below)
### If "Other Airflow 2 version" selected, which one?
2.6.3
### What happened?
Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec.
In Airflow 2.5.3 [scheduler]scheduler_heartbeat_sec is used correctly.
The bug was introduced after refactoring of scheduler_job -> scheduler_job_runner.
This field is never used in the code:
https://github.com/apache/airflow/blob/1f5960a9844ad9bcea61940a76052e0af549579f/airflow/jobs/scheduler_job_runner.py#L153
### What you think should happen instead?
_No response_
### How to reproduce
- run Airflow
- set [scheduler]job_heartbeat_sec with the value greater than [scheduler]scheduler_health_check_threshold
- see warning message in Airflow UI that scheduler is not alive
### Operating System
Ubuntu
### Versions of Apache Airflow Providers
_No response_
### Deployment
Google Cloud Composer
### Deployment details
_No response_
### Anything else?
_No response_
### Are you willing to submit PR?
- [ ] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985600389
Yeah - I think the way where it is closer to heartrate property is a bit better and easier to reason about.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565404
(just wanted to make sure how far back it goes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985546310
Ahh.. those parameters are confusing and I am mixing them ... it is indeed way older sorry (threshold mixed with secs)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985536704
@kosteev - are you sure it's affecting 2.6.3 Airflow ? from what I traced #34026 has been only part of v2-8 branch. Maybe there was another reason before in 2.6.3 - but quite certainly this one was only introduced in 2.8.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985518074
Actually not even there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985565030
So it's there since 2.6.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985603095
We have a bit of a mess there - because the JOB (scheduler/job_heartrabeat_sec) has been used and abused by different jobs, so likely better to keep it in JOB.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1984702476
Are you sure @kosteev ?
From what I see it is used n Job's `perform_heartbeat` method::
```
def perform_heartbeat(
job: Job, heartbeat_callback: Callable[[Session], None], only_if_necessary: bool
) -> None:
"""
Perform heartbeat for the Job passed to it,optionally checking if it is necessary.
:param job: job to perform heartbeat for
:param heartbeat_callback: callback to run by the heartbeat
:param only_if_necessary: only heartbeat if it is necessary (i.e. if there are things to run for
triggerer for example)
"""
seconds_remaining: float = 0.0
if job.latest_heartbeat and job.heartrate: # <--- HERE THE HEARTRATE IS USED FROM JOB (and one level down)
seconds_remaining = job.heartrate - (timezone.utcnow() - job.latest_heartbeat).total_seconds()
if seconds_remaining > 0 and only_if_necessary:
return
job.heartbeat(heartbeat_callback=heartbeat_callback)
```
I believe there were some fixes after 2.6.3 that fixed bad threshold calculations for some jobs (It was confusing whether the job used 2.1* multiplier or whether you had) but I'd expect heartrate was not affected - maybe threshold is (or rather was) calculated wrongly.
Also this scenario:
```
1. run Airflow
2. set [scheduler]job_heartbeat_sec with the value greater than [scheduler]scheduler_health_check_threshold
3. see warning message in Airflow UI that scheduler is not alive
```
is expected. If heartbeat sec is greater than threshold, then yes you'd expect to see the warning (the warning is issued when the heartbeat does not arrive in the threshold time - which would be exactly what is happening if those values are set as described).
Mybe the reproduction scenario is wrong? Could you please clarify ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985334309
In perform_heartbeat, job.heartrate is used.
The issue is that for scheduler it is defined in SchedulerJobRunner (previously was SchedulerJob):
https://github.com/apache/airflow/blob/08efc32fd13c505a3c04b97f9bc81c79d1725858/airflow/jobs/scheduler_job_runner.py#L153
so, now this heartrate: int = conf.getint("scheduler", "SCHEDULER_HEARTBEAT_SEC") from SchedulerJobRunner is not used anywhere
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985337125
About reproduction scenario:
"If heartbeat sec is greater than threshold, then yes you'd expect to see the warning" - yes, but in the scenario you modify [scheduler]job_heartbeat_sec with property which shouldn't have to do anything with scheduler, as scheduler heartbeat should be controlled by [scheduler]scheduler_heartbeat_sec
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985513790
I actualy took a quick look and it turned out that it was removed after adding SchedulerJobRunner. The SchedulerJob check has been removed by #34026 a bit later.
I prepared a fix with test in #37992
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985472529
> so, now this heartrate: int = conf.getint("scheduler", "SCHEDULER_HEARTBEAT_SEC") from SchedulerJobRunner is not used anywhere
Indeed.
> "If heartbeat sec is greater than threshold, then yes you'd expect to see the warning" - yes, but in the scenario you modify [scheduler]job_heartbeat_sec with property which shouldn't have to do anything with scheduler, as scheduler heartbeat should be controlled by [scheduler]scheduler_heartbeat_sec
Ah :facepalm: -> indeed missed that it is about the other config.
Would you like to fix it maybe @kosteev ? That seems like a good contribution and simple fix to make ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985591616
I prepared a fix like this https://github.com/apache/airflow/pull/37993
But I see you have already fix (and implemented differently). Should I close my PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985539571
The 2.6.3 version is:
```
def is_alive(self, grace_multiplier=2.1):
"""
Is this job currently alive.
We define alive as in a state of RUNNING, and having sent a heartbeat
within a multiple of the heartrate (default of 2.1)
:param grace_multiplier: multiplier of heartrate to require heart beat
within
"""
if self.job_type == "SchedulerJob":
health_check_threshold: int = conf.getint("scheduler", "scheduler_health_check_threshold")
else:
health_check_threshold: int = self.heartrate * grace_multiplier
return (
self.state == State.RUNNING
and (timezone.utcnow() - self.latest_heartbeat).total_seconds() < health_check_threshold
)
```
So almost for sure it wouldn't have caused the problem.
(BTW. I am trying to see if we can add the fix to 2.8.3 that we vote on currently - because otherwise it would affect the whole 2.8 line)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "kosteev (via GitHub)" <gi...@apache.org>.
kosteev commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985504904
I will try to prepare a fix, but I do not want to commit to this, so that if anyone wants to fix it.
I will try to work on it today, early next week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #37971:
URL: https://github.com/apache/airflow/issues/37971#issuecomment-1985564821
Tracked it down (there were quite a few refactors) and indeed it was #30255 that introduced it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed issue #37971: Scheduler uses [scheduler]job_heartbeat_sec property for heartbeats instead of [scheduler]scheduler_heartbeat_sec
URL: https://github.com/apache/airflow/issues/37971
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org