You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "hussein-awala (via GitHub)" <gi...@apache.org> on 2023/06/25 14:29:58 UTC
[GitHub] [airflow] hussein-awala opened a new pull request, #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
hussein-awala opened a new pull request, #32123:
URL: https://github.com/apache/airflow/pull/32123
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
closes: #32121
This PR:
- adds a new configuration for the triggerer heartbeat rate
- fixes the `assign_unassigned` method: thhe static 30-second threshold is replaced with a dynamically calculated threshold obtained by multiplying the triggerer heartbeat rate by a grace multiplier (2.1).
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241245967
##########
airflow/models/trigger.py:
##########
@@ -212,12 +213,14 @@ def assign_unassigned(cls, triggerer_id, capacity, session: Session = NEW_SESSIO
if capacity <= 0:
return
-
+ # we multiply heartrate by a grace_multiplier to give the triggerer
+ # a chance to heartbeat before we consider it dead
+ health_check_threshold = heartrate * 2.1
Review Comment:
> Airflow tries to only run triggers in one place at once, and maintains a heartbeat to all triggerers that are currently running. If a triggerer dies, or becomes partitioned from the network where Airflow’s database is running, Airflow will automatically re-schedule triggers that were on that host to run elsewhere (after waiting 30 seconds for the machine to re-appear).
https://airflow.apache.org/docs/apache-airflow/stable/authoring-and-scheduling/deferring.html#high-availability
I feel this could also be updated to reference the new configuration instead of 30 seconds. As a personal opinion multiplier can be an integer 2 or 3 than 2.1 specifically and also maybe a named constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tirkarthi commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "tirkarthi (via GitHub)" <gi...@apache.org>.
tirkarthi commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241248624
##########
airflow/models/trigger.py:
##########
@@ -212,12 +213,14 @@ def assign_unassigned(cls, triggerer_id, capacity, session: Session = NEW_SESSIO
if capacity <= 0:
return
-
+ # we multiply heartrate by a grace_multiplier to give the triggerer
+ # a chance to heartbeat before we consider it dead
+ health_check_threshold = heartrate * 2.1
Review Comment:
Got it, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #32123:
URL: https://github.com/apache/airflow/pull/32123#issuecomment-1606115474
@ephraimbuddy @pierrejeambrun I add a new Airflow configuration in this PR, but I can fix the issue without it where it fallback to scheduler heartbeat rate.
1. should I create a separate PR for the new conf?
2. should I make it fallback to scheduler heartbeat rate instead of its default (5) to avoid the breaking change?
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on PR #32123:
URL: https://github.com/apache/airflow/pull/32123#issuecomment-1615154076
> I take the thumbs-up reaction of @ephraimbuddy as agremeent :). And I think this one is worthy cherry-picking to 2.6.3 :)
Great! I asked him because of this thumbs-up.
I'll merge it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241739983
##########
airflow/config_templates/config.yml:
##########
@@ -2558,6 +2558,13 @@ triggerer:
type: string
example: ~
default: "1000"
+ job_heartbeat_sec:
+ description: |
+ How often to heartbeat the Triggerer job to ensure it hasn't been killed.
+ version_added: 2.6.3
Review Comment:
Not necessarily. I think we do not have a rule "do not add new configuration options" in patchlevel. We have the rule of not adding new features, but new configuration might be added in order to implement a bugfix (which I think is the case here).
I think it's OK (and we've done that in the past) that we introduced new configuration in patchlevel version if they served bugfix purpose. There are a few configs already that were added in non- `.0` versions (smtp ones for example).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1245908828
##########
airflow/config_templates/config.yml:
##########
@@ -2558,6 +2558,13 @@ triggerer:
type: string
example: ~
default: "1000"
+ job_heartbeat_sec:
+ description: |
+ How often to heartbeat the Triggerer job to ensure it hasn't been killed.
+ version_added: 2.6.3
Review Comment:
@ephraimbuddy Do you agree with @potiuk, or should I move it to 2.7.0?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala merged pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala merged PR #32123:
URL: https://github.com/apache/airflow/pull/32123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "ephraimbuddy (via GitHub)" <gi...@apache.org>.
ephraimbuddy commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241684995
##########
airflow/config_templates/config.yml:
##########
@@ -2558,6 +2558,13 @@ triggerer:
type: string
example: ~
default: "1000"
+ job_heartbeat_sec:
+ description: |
+ How often to heartbeat the Triggerer job to ensure it hasn't been killed.
+ version_added: 2.6.3
Review Comment:
Since we are adding a new configuration, this should be moved to 2.7.0
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] hussein-awala commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "hussein-awala (via GitHub)" <gi...@apache.org>.
hussein-awala commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241247696
##########
airflow/models/trigger.py:
##########
@@ -212,12 +213,14 @@ def assign_unassigned(cls, triggerer_id, capacity, session: Session = NEW_SESSIO
if capacity <= 0:
return
-
+ # we multiply heartrate by a grace_multiplier to give the triggerer
+ # a chance to heartbeat before we consider it dead
+ health_check_threshold = heartrate * 2.1
Review Comment:
Actually we use this value as default `grace_multiplier` in the job `is_alive` method, IMHO it's better to use the same one here to be coherent with that method:
https://github.com/apache/airflow/blob/546588a9dd02860e8bba2e47b8d87b18927c6186/airflow/jobs/job.py#L123-L140
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #32123:
URL: https://github.com/apache/airflow/pull/32123#issuecomment-1613244527
There were some "stalled" GA jobs running for that one I closed/reopened to rebuild.
I take the thumbs-up reaction of @ephraimbuddy as agremeent :). And I think this one is worthy cherry-picking to 2.6.3 :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on a diff in pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on code in PR #32123:
URL: https://github.com/apache/airflow/pull/32123#discussion_r1241741160
##########
airflow/config_templates/config.yml:
##########
@@ -2558,6 +2558,13 @@ triggerer:
type: string
example: ~
default: "1000"
+ job_heartbeat_sec:
+ description: |
+ How often to heartbeat the Triggerer job to ensure it hasn't been killed.
+ version_added: 2.6.3
Review Comment:
(and yes, that was also my initial reaction, but I thought about it and I looked at the past configuration entries we created and it's not obvious that "new configuration == new feature".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk closed pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk closed pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
URL: https://github.com/apache/airflow/pull/32123
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #32123: Fix triggerers alive check and add a new conf for triggerer heartbeat rate
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #32123:
URL: https://github.com/apache/airflow/pull/32123#issuecomment-1606133505
I think it is fine to both: add new config and fallback to 5 seconds.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org