You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/03/10 10:35:15 UTC
[GitHub] [airflow] ephraimbuddy opened a new pull request #22143: Use jobs check command for liveness probe check in airflow 2
ephraimbuddy opened a new pull request #22143:
URL: https://github.com/apache/airflow/pull/22143
This PR proposes to remove the current use of python code for liveness probe check
commands in the Scheduler & Triggerer deployments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #22143:
URL: https://github.com/apache/airflow/pull/22143#discussion_r824140526
##########
File path: chart/templates/scheduler/scheduler-deployment.yaml
##########
@@ -165,9 +165,11 @@ spec:
periodSeconds: {{ .Values.scheduler.livenessProbe.periodSeconds }}
exec:
command:
-{{- with .Values.scheduler.livenessProbe.command }}
-{{ toYaml . | indent 16 }}
-{{- end }}
+ {{- if .Values.scheduler.livenessProbe.command }}
+ {{ toYaml .Values.scheduler.livenessProbe.command | indent 16 }}
+ {{- else}}
+ {{- include "scheduler_liveness_check_command" . | indent 16 }}
Review comment:
Not sure of this one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #22143:
URL: https://github.com/apache/airflow/pull/22143#discussion_r823867006
##########
File path: chart/templates/_helpers.yaml
##########
@@ -607,6 +607,42 @@ Create the name of the cleanup service account to use
{{- end }}
{{- end }}
+{{define "scheduler_liveness_check_command"}}
+
+ {{- if semverCompare ">=2.0.0" .Values.airflowVersion }}
+ - sh
+ - -c
+ - |
+ CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint bash -c \
+ "export AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR && export AIRFLOW__CORE__LOGGING_LEVEL=ERROR && \
Review comment:
```suggestion
"export AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR && \
```
With 2.0.0+, this should be enough now.
##########
File path: chart/templates/_helpers.yaml
##########
@@ -607,6 +607,42 @@ Create the name of the cleanup service account to use
{{- end }}
{{- end }}
+{{define "scheduler_liveness_check_command"}}
+
+ {{- if semverCompare ">=2.0.0" .Values.airflowVersion }}
+ - sh
+ - -c
+ - |
+ CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint bash -c \
+ "export AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR && export AIRFLOW__CORE__LOGGING_LEVEL=ERROR && \
+ airflow jobs check --job-type SchedulerJob --hostname $(hostname)"
+ {{- else }}
+ - sh
+ - -c
+ - |
+ CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint python -Wignore -c "
+ import os
+ os.environ['AIRFLOW__CORE__LOGGING_LEVEL'] = 'ERROR'
+ os.environ['AIRFLOW__LOGGING__LOGGING_LEVEL'] = 'ERROR'
+ from airflow.jobs.scheduler_job import SchedulerJob
+ from airflow.utils.db import create_session
+ from airflow.utils.net import get_hostname
+ import sys
+ with create_session() as session:
+ job = session.query(SchedulerJob).filter_by(hostname=get_hostname()).order_by(
+ SchedulerJob.latest_heartbeat.desc()).limit(1).first()
+ sys.exit(0 if job.is_alive() else 1)"
+ {{- end }}
+{{- end }}
+{{define "triggerer_liveness_check_command"}}
Review comment:
```suggestion
{{define "triggerer_liveness_check_command"}}
```
##########
File path: chart/templates/scheduler/scheduler-deployment.yaml
##########
@@ -165,9 +165,11 @@ spec:
periodSeconds: {{ .Values.scheduler.livenessProbe.periodSeconds }}
exec:
command:
-{{- with .Values.scheduler.livenessProbe.command }}
-{{ toYaml . | indent 16 }}
-{{- end }}
+ {{- if .Values.scheduler.livenessProbe.command }}
+ {{ toYaml .Values.scheduler.livenessProbe.command | indent 16 }}
+ {{- else}}
+ {{- include "scheduler_liveness_check_command" . | indent 16 }}
Review comment:
```suggestion
{{ toYaml .Values.scheduler.livenessProbe.command | nindent 16 }}
{{- else}}
{{- include "scheduler_liveness_check_command" . | nindent 16 }}
```
We want this, no?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] github-actions[bot] commented on pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #22143:
URL: https://github.com/apache/airflow/pull/22143#issuecomment-1064101785
The PR is likely OK to be merged with just subset of tests for default Python and Database versions without running the full matrix of tests, because it does not modify the core of Airflow. If the committers decide that the full tests matrix is needed, they will add the label 'full tests needed'. Then you should rebase to the latest main or amend the last commit of the PR, and push it with --force-with-lease.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on a change in pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on a change in pull request #22143:
URL: https://github.com/apache/airflow/pull/22143#discussion_r824145422
##########
File path: chart/templates/_helpers.yaml
##########
@@ -607,6 +607,43 @@ Create the name of the cleanup service account to use
{{- end }}
{{- end }}
+{{define "scheduler_liveness_check_command"}}
+
+ {{- if semverCompare ">=2.0.0" .Values.airflowVersion }}
+ - sh
+ - -c
+ - |
+ CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint bash -c \
+ "export AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR && \
+ airflow jobs check --job-type SchedulerJob --hostname $(hostname)"
Review comment:
```suggestion
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR \
exec /entrypoint bash -c \
"airflow jobs check --job-type SchedulerJob --hostname $(hostname)"
```
This is probably better...
Also, I don't think we need nested sh/bash, does this work?:
```suggestion
CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR \
exec /entrypoint airflow jobs check --job-type SchedulerJob --hostname $(hostname)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy merged pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
ephraimbuddy merged pull request #22143:
URL: https://github.com/apache/airflow/pull/22143
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] ephraimbuddy commented on a change in pull request #22143: Use jobs check command for liveness probe check in airflow 2
Posted by GitBox <gi...@apache.org>.
ephraimbuddy commented on a change in pull request #22143:
URL: https://github.com/apache/airflow/pull/22143#discussion_r824286102
##########
File path: chart/templates/_helpers.yaml
##########
@@ -607,6 +607,43 @@ Create the name of the cleanup service account to use
{{- end }}
{{- end }}
+{{define "scheduler_liveness_check_command"}}
+
+ {{- if semverCompare ">=2.0.0" .Values.airflowVersion }}
+ - sh
+ - -c
+ - |
+ CONNECTION_CHECK_MAX_COUNT=0 exec /entrypoint bash -c \
+ "export AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR && \
+ airflow jobs check --job-type SchedulerJob --hostname $(hostname)"
Review comment:
Yeah. Worked
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org