You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/08/11 11:33:30 UTC

[GitHub] [airflow] fernhtls opened a new issue, #25667: Scheduler livenessProbe errors on new helm chart

fernhtls opened a new issue, #25667:
URL: https://github.com/apache/airflow/issues/25667

   ### Official Helm Chart version
   
   1.6.0 (latest released)
   
   ### Apache Airflow version
   
   v2.1.2
   
   ### Kubernetes Version
   
   v1.22.10 (GKE version v1.22.10-gke.600)
   
   ### Helm Chart configuration
   
   Only livenessProbe config before and during the issue:
   
   ```
   # Airflow scheduler settings
   scheduler:
     livenessProbe:
       initialDelaySeconds: 10
       timeoutSeconds: 15
       failureThreshold: 10
       periodSeconds: 60
   ```
   
   ### Docker Image customisations
   
   Here's the image we use based on the apache airflow image:
   
   ```
   ### Main official airflow image
   FROM apache/airflow:2.1.2-python3.8
   
   USER root
   
   RUN apt update
   
   RUN echo 'debconf debconf/frontend select Noninteractive' | debconf-set-selections
   
   ### Add OS packages here
   ## GCC compiler in case it's needed for installing python packages
   RUN apt install -y -q build-essential
   
   USER airflow
   
   ### Changing the default SSL / TLS mode for mysql client to work properly
   ## https://askubuntu.com/questions/1233186/ubuntu-20-04-how-to-set-lower-ssl-security-level
   ## https://bugs.launchpad.net/ubuntu/+source/mysql-8.0/+bug/1872541
   ## https://stackoverflow.com/questions/61649764/mysql-error-2026-ssl-connection-error-ubuntu-20-04
   RUN echo $'openssl_conf = default_conf\n\
   [default_conf]\n\
   ssl_conf = ssl_sect\n\
   [ssl_sect]\n\
   system_default = ssl_default_sect\n\
   [ssl_default_sect]\n\
   MinProtocol = TLSv1\n\
   CipherString = DEFAULT:@SECLEVEL=1' >> /home/airflow/.openssl.cnf
   ## OS env var to point to the new openssl.cnf file
   ENV OPENSSL_CONF=/home/airflow/.openssl.cnf
   
   ### Add airflow providers
   RUN pip install apache-airflow-providers-apache-beam
   ### End airflow providers
   
   ### Add extra python packages
   RUN pip install python-slugify==3.0.3
   ### End extra python packages
   ```
   
   ### What happened
   
   After the upgrade to the helm chart `1.6.0`, the scheduler POD was restarting as the `livenessProbe` was failing.
   
   Command for the new `livenessProbe` from helm chart `1.6.0` tested directly on our scheduler POD:
   
   ```
   airflow@yc-data-airflow-scheduler-0:/opt/airflow$ CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   > airflow jobs check --job-type SchedulerJob --hostname $(hostname)
   No alive jobs found.
   ```
   
   Removing the `--hostname` argument works and a `live job is found`:
   
   ```
   airflow@yc-data-airflow-scheduler-0:/opt/airflow$ CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   > airflow jobs check --job-type SchedulerJob
   Found one alive job.
   ```
   
   ### What you think should happen instead
   
   `livenessProbe` should not error.
   
   ### How to reproduce
   
   _No response_
   
   ### Anything else
   
   _No response_
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eduardchai commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
eduardchai commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1336700145

   I am still having this issue in:
   Airflow version: 2.4.3
   Chart version: 1.7.0
   
   Problematic config:
   ```
   "AIRFLOW__CORE__HOSTNAME_CALLABLE" = "airflow.utils.net.get_host_ip_address"
   ```
   
   `kubectl event` of the scheduler:
   ```
   LAST SEEN   TYPE      REASON      OBJECT                                       MESSAGE
   2m50s       Warning   Unhealthy   pod/airflow-main-scheduler-589ff44fd-v5c97   Liveness probe failed: No alive jobs found.
   ```
   
   I believe this is causing Kubernetes to send sigterm to scheduler pod and causing this error:
   ```
   [2022-12-05T03:31:42.219+0000] {scheduler_job.py:172} INFO - Exiting gracefully upon receiving signal 15
   [2022-12-05T03:31:43.222+0000] {process_utils.py:129} INFO - Sending Signals.SIGTERM to group 35. PIDs of all processes in the group: [35]
   [2022-12-05T03:31:43.223+0000] {process_utils.py:84} INFO - Sending the signal Signals.SIGTERM to group 35
   [2022-12-05T03:31:43.436+0000] {process_utils.py:79} INFO - Process psutil.Process(pid=35, status='terminated', exitcode=0, started='03:26:53') (35) terminated with exit code 0
   [2022-12-05T03:31:43.440+0000] {kubernetes_executor.py:823} INFO - Shutting down Kubernetes executor
   [2022-12-05T03:31:43.440+0000] {scheduler_job.py:768} ERROR - Exception when executing Executor.end
   Traceback (most recent call last):
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 745, in _execute
       self._run_scheduler_loop()
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 865, in _run_scheduler_loop
       num_queued_tis = self._do_scheduling(session)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 945, in _do_scheduling
       callback_tuples = self._schedule_all_dag_runs(guard, dag_runs, session)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 78, in wrapped_function
       for attempt in run_with_db_retries(max_retries=retries, logger=logger, **retry_kwargs):
     File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/__init__.py", line 384, in __iter__
       do = self.iter(retry_state=retry_state)
     File "/home/airflow/.local/lib/python3.9/site-packages/tenacity/__init__.py", line 351, in iter
       return fut.result()
     File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 439, in result
       return self.__get_result()
     File "/usr/local/lib/python3.9/concurrent/futures/_base.py", line 391, in __get_result
       raise self._exception
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/retries.py", line 87, in wrapped_function
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1234, in _schedule_all_dag_runs
       callback_to_run = self._schedule_dag_run(dag_run, session)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/jobs/scheduler_job.py", line 1300, in _schedule_dag_run
       schedulable_tis, callback_to_run = dag_run.update_state(session=session, execute_callbacks=False)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagrun.py", line 555, in update_state
       info = self.task_instance_scheduling_decisions(session)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagrun.py", line 673, in task_instance_scheduling_decisions
       tis = self.get_task_instances(session=session, state=State.task_states)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/utils/session.py", line 72, in wrapper
       return func(*args, **kwargs)
     File "/home/airflow/.local/lib/python3.9/site-packages/airflow/models/dagrun.py", line 455, in get_task_instances
       return tis.all()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/query.py", line 2759, in all
       return self._iter().all()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1361, in all
       return self._allrows()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 400, in _allrows
       rows = self._fetchall_impl()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1274, in _fetchall_impl
       return self._real_result._fetchall_impl()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 1686, in _fetchall_impl
       return list(self.iterator)
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/orm/loading.py", line 147, in chunks
       fetch = cursor._raw_all_rows()
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 392, in _raw_all_rows
       return [make_row(row) for row in rows]
     File "/home/airflow/.local/lib/python3.9/site-packages/sqlalchemy/engine/result.py", line 392, in <listcomp>
       return [make_row(row) for row in rows]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1537469119

   You missed that this issue is closed. If you find no answer here and you still see similar issue please open anew on with all logs and information in what circumstances you see a problem. Or even better - if you are not sure if it is an airflow issue, create a discussion and describe it there.
   
   Even if you see something similiar, the best you can do if you seek help is to describe your issue in detail so that others can help you. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] sk-ilya commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
sk-ilya commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1238076270

   `$(hostname -f)` worked for us


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1306228364

   Can any of the people here verify it? I am closing it provisionally, unless someone reports it's NOT fixed.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eduardchai commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
eduardchai commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1336730461

   @BobDu I see... Thank you for the update! I will try 2.5.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] potiuk closed issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
potiuk closed issue #25667: Scheduler livenessProbe errors on new helm chart
URL: https://github.com/apache/airflow/issues/25667


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] devopspsn commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by "devopspsn (via GitHub)" <gi...@apache.org>.
devopspsn commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1532273229

   I have same issue with chart 1.8.0 and airflow 2.5.3. What do I missed?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ilyadinaburg commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
ilyadinaburg commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1217728956

   Hi, I'm facing the same issue as @fernhtls with scheduler and triggerer pods. Running Airflow 2.3.2 on AWS, official helm chart 1.6.0.
   
   Below is the result of livenessProbe command:
   ```
   airflow@airflow2-test-us-west-2-scheduler-0:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check --job-type SchedulerJob --hostname $(hostname)"
   No alive jobs found.
   ```
   If I ran it without hostname as @fernhtls suggested:
   ```
   airflow@airflow2-test-us-west-2-scheduler-0:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check --job-type SchedulerJob"
   Found one alive job.
   ```
   However, if i run it with IP address of scheduler pod instead of hostname I got:
   
   ```
   airflow@airflow2-test-us-west-2-scheduler-0:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check --job-type SchedulerJob --hostname 172.XXX.XXX.XXX"
   Found one alive job.
   ```
   So assume that might be related to setting HOSTNAME_CALLABLE in my config:
   ```
       - name: AIRFLOW__CORE__HOSTNAME_CALLABLE
         value: 'airflow.utils.net.get_host_ip_address'
   ```
   Anyway, would be great to hear from you.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Subhashini2610 commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
Subhashini2610 commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1361045306

   @BobDu I am also facing the same issue. Even upgrading to 2.5.0 didn't help in my case.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] ilyadinaburg commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
ilyadinaburg commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1217858130

   hey @fernhtls, I think that the issue might not be directly related to AIRFLOW__CORE__HOSTNAME_CALLABLE, however, might be this is obvious but still as per your findings it is related to the fact that pod hostname not being resolved.
   So as a workaround $(hostname -i) instead of $(hostname) may be used as well in liveliness probe command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] VladimirYushkevich commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
VladimirYushkevich commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1273815310

   I have the same issue with airflow `2.4.1` to check `liveness` on `dag-processor`'s pod. Default is not working for me:
   ```
   dag-processor-ccb9f9949-7zdtv:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check --hostname $(hostname)"
   No alive jobs found.
   ``` 
   as well as with `$(hostname -i)`:
   ```
   dag-processor-ccb9f9949-7zdtv:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check --hostname $(hostname -i)"
   No alive jobs found.
   ```
   Only after removing `--hostname` flag it works:
   ```
   dag-processor-ccb9f9949-7zdtv:/opt/airflow$ sh -c "CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
   airflow jobs check"
   Found one alive job.
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] csp98 commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
csp98 commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1284073754

   @VladimirYushkevich I believe the Standalone DAG Processor issue is totally different. I have documented my theory in the following issue: https://github.com/apache/airflow/issues/27140


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alifouzizadeh commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
alifouzizadeh commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1235176361

   Hi @fernhtls, what @ilyadinaburg suggested works for me, I changed `$(hostname)`  to `$(hostname -i) `, and it is working.
   So, the code in the `values.yaml ` should be like this: 
   
   ```
   scheduler:
     livenessProbe:
       initialDelaySeconds: 10
       timeoutSeconds: 120
       failureThreshold: 20
       periodSeconds: 60
       command:
         - sh
         - -c
         - |
           CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
           airflow jobs check --job-type SchedulerJob --hostname $(hostname -i)
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] eduardchai commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
eduardchai commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1362525138

   @Subhashini2610 have you upgraded the chart to `1.7.0`? The issue is solved for me using chart 1.7.0 and airflow 2.5.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] fernhtls commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
fernhtls commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1217888723

   hi @ilyadinaburg ,thanks for info, I'll check on your proposal later on. On an HA deployment for the scheduler (statefulset with 2 pods or more), the liveness needs to check if the process is up and running only on it's own host, so what I did it's not the correct / best approach indeed. Let's wait for more info from someone from apache/airflow itself with more info.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] BobDu commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
BobDu commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1298627236

   maybe we can close this issue, fix by #24999 ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] Subhashini2610 commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
Subhashini2610 commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1362575631

   @eduardchai  I have used 1.8.0-dev version. I can see the --local change in the liveness probe command as well.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] alifouzizadeh commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
alifouzizadeh commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1233106613

   Hi  @fernhtls, we can change the liveness bash to this `airflow jobs check --job-type SchedulerJob --hostname $(hostname -i)`, in this case, we will have the hostname for HA.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] boring-cyborg[bot] commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1211868628

   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] BobDu commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
BobDu commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1336710629

   @eduardchai 
   
   use airlfow 2.5.0
   we support liveness cmd arg `--local` start in 2.5.0


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


[GitHub] [airflow] fernhtls commented on issue #25667: Scheduler livenessProbe errors on new helm chart

Posted by GitBox <gi...@apache.org>.
fernhtls commented on issue #25667:
URL: https://github.com/apache/airflow/issues/25667#issuecomment-1217770144

   hi @ilyadinaburg , I didn't have the time to check properly the whole issue on itself, in my case I'm pushing a custom `livenessProbe` to the helm chart without the `--hostname` argument as below:
   
   ```
   scheduler:
     livenessProbe:
       initialDelaySeconds: 10
       timeoutSeconds: 15
       failureThreshold: 10
       periodSeconds: 60
       command:
         - sh
         - -c
         - |
         	CONNECTION_CHECK_MAX_COUNT=0 AIRFLOW__LOGGING__LOGGING_LEVEL=ERROR exec /entrypoint \
         	airflow jobs check --job-type SchedulerJob
   ```
   
   I didn't check the env var `AIRFLOW__CORE__HOSTNAME_CALLABLE`, as I had to push a quick fix as the scheduler was restarting and breaking the execution of DAGs / tasks.
   
   Good that the issue seems to be reproducible, as you're using the same new latest helm chart version as I am (`1.6.0`). 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org