You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2021/07/23 14:14:36 UTC
[GitHub] [airflow] jedcunningham opened a new issue #17191: Healthcheck endpoint for workers
jedcunningham opened a new issue #17191:
URL: https://github.com/apache/airflow/issues/17191
**Description**
Now that the worker only responds to requests signed by the webserver (#16754), we should consider adding a health endpoint to the worker to help ensure it is up and healthy.
**Use case / motivation**
Users may want to have external monitoring on the workers logging webserver and having something that responds with a 200 instead of a 403 would be handy.
https://github.com/apache/airflow/pull/16754#issuecomment-885661439
**Related Issues**
#16574
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-888611296
@jedcunningham I am not sure. It seems to me that the main process should monitor the child process and restart it when it detects problems with it. Similarly, Scheduler monitors DagFileManager and DagFileManager monitors DagFileProcessory. The user does not need to know the internal components of the application / processes to monitor them, because the application already knows them and can monitor them. The problem is only with the process at the top of the process tree, which has no parent process and needs to be monitored externally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP? I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment. Very similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-918605664
I'm leaning toward just having a way to check both the health of the worker and log server at the same time (be it exec or http based). I feel the log server is an extra feature of the worker, rather than something deserving of being in a sidecar/separate container.
As for implementation (https vs exec), I do lean more toward http over exec (primarily since the endpoint is likely to be more easily usable outside of the k8s context too).
(Also, the LocalExecutor/SequentialExecutor scheduler should probably be brought into the mix as well, which might complicate this a bit.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] jedcunningham commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
jedcunningham commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-885707641
The inverse of that is also true though (worker is healthy, logs HTTP server is wedged)? Both probably should be checked (or at least checkable).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890598962
Yeah. I agree with @jedcunningham . I think it should be the matter of deployment (for example K8S) to take care about health of all components indepenendently from the master. The helth-check endpoint in K8S is a typical way of checking if particular endpoint is "healthy". In a way "main" process might not know how many "workers" should be available and whether there should be a specific number of those - this is something that the deployment configuration should be responsible for - in similar way as the scheduler is not responsible for "scaling" of the workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment.
Very similarly as we do with scheduler and webservers- we can manage number of schedulers and webservers independently and scaling works by adding "just another scheduler" or "just another webserver" - same with Celery, we should be able to add "just another worker". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other (specificaly how many of instances of those other workers we have) and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP? I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point. I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external d
eployment. Very similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment.
Very similarly as we do with scheduler and webservers- we can manage number of schedulers and webservers independently and scaling works by adding "just another scheduler" or "just another webserver" - same with Celery, we should be able to add "just another worker". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other (specificaly how many of instances of those other components we have) and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment.
Very similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-885698729
@green2k @jedcunningham I'm not sure it makes sense. Worker runs in a separate process from the HTTP server, so the worker may still hang and monitoring will not detect this as it will be monitoring a separate HTTP server only. In the documentation, we have described the approach to monitoring each component and good practices in this regard. In this case, we should use [CLI check for worker](https://airflow.apache.org/docs/apache-airflow/stable/logging-monitoring/check-health.html#cli-check-for-celery-workers).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
mik-laj edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890873827
@potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
Alternatively, we can move the webserver to a new container/process that will allow us to [define a new probe](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) for webserver and fully manage its lifetime in line with the Kubernetes philosophy. This will increase the complexity of deployment in environments that do not use Kubernetes and users "semi-automatically" run such processes.
Separating these processes will also allow us to control the number of web servers regardless of the number of worker, but for now we have a fairly simple 1: 1 model, i.e. we always run only one web server per worker.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment. Very similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment.
Very similarly as we do with scheduler and webservers- we can manage number of schedulers and webservers independently and scaling works by adding "just another scheduler" or "just another webserver" - same with Celery, we should be able to add "just another worker". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-918968871
Agree - http would make more sense and one healthcheck will be more than enough. I generally prefer `http` kind of healthchecks over exec, as they are more "versatile". Http healthcheck can be used in basically any execution environment and they are generally much safer. You do not need any special permissions to execute the health-check. In many enterprise environments you might simply not get permissions to execute an arbitrary command inside a running container from the monitoring services. If we already have an `http` server there, it's not even a question of adding new dependencies/opening ports etc, because it is already there, so there are no drawbacks of this solution IMHO.
Squential is lilkely not a problem, but yeah with Local we have ``--skip-serve-logs`` command - I think it would have to similar ``--skip-healthcheck` and the standalone gunicorn should be started when either logs or health-checks are enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP? I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point. I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deploy
ment. Very similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk edited a comment on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk edited a comment on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have "log http-webserver" built-in and there is one such webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP?
I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point.
I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment.
Very similarly as we do with scheduler and webservers- we can manage number of schedulers and webservers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-913034533
What do we do with this task? I think healthpoint endpoint with http check is pretty universal solution
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
potiuk commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890898149
> @potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
But since our workers already have webserver built-in and there is one webserver per worker which is involved with running the worker itself - what is the problem of each worker providing health check using HTTP? I do not see any reason why we could not use it. There are many services that provide health check over http, even if their primary task is doing something else. Such health-check end-point could internally perform more complex checks than just responding with success - it could communicate with celery master process and query it for status etc. etc. I think this would make worker a nice self-container service with its own dedicated health-check end-point. I think the basic premise of our Celery architecture is that we can simply start up as many of those self-contained workers as possible and we can manage the number of those workers in a way that is completely independent from other Airflow components - delegating deployment, scaling etc. to external deployment. Very
similarly as we do with scheduler - we can manage number of schedulers independently and scaling works by adding "just another scheduler". I think it also fits K8S philosophy very well where you can define Deployment of different components of application and put them together like "lego blocks" - in the way that different components do not have to know about each other and act independently.
Or maybe we are talking about different thing altogether ? Maybe there is something I do not understand ? I think we do not need to move webserver to a different container or anything like that, so maybe we have a misunderstanding here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] mik-laj commented on issue #17191: Healthcheck endpoint for workers
Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #17191:
URL: https://github.com/apache/airflow/issues/17191#issuecomment-890873827
@potiuk Only in our case we do not have an ideal situation that is in line with the philosophy of K8s. In our case, one component performs two roles: webserver, tasks handling service. For webserver, it is more common to use an HTTP-based probe, but for the second type of service (which does not provide an HTTP endpoint), it is more natural to use exec probe. Kubernetes also does not allow us to define of two liveness probes for one container, so we have to decide which service we want to monitor directly and which we will monitor only as a child process from the main process.
Alternatively, we can move the webserver to a new container/process that will allow us to [define a new probe](https://kubernetes.io/docs/concepts/workloads/pods/pod-lifecycle/#container-probes) for webserver and fully manage its lifetime in line with the Kubernetes philosophy. This will increase the complexity of deployment in environments that do not use Kubernetes and users "semi-automatically" run such processes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org