You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2022/11/23 17:01:47 UTC
[GitHub] [pulsar] zzzming opened a new issue, #18584: Pulsar function replicas auto scaling
zzzming opened a new issue, #18584:
URL: https://github.com/apache/pulsar/issues/18584
### Search before asking
- [X] I searched in the [issues](https://github.com/apache/pulsar/issues) and found nothing similar.
### Motivation
Pulsar function and connector in Kubernetes runtime should be able to auto scale based on resource usage or the input topic backlog size or some other metrics. The auto scale should be horizontal scaling.
### Solution
_No response_
### Alternatives
Use the existing Kubernetes HPA. But the state of replicas won't be saved because function stop removes the function statefulset. The parallelism is save in bookkeeper not Kubernetes (upon statefulset removal) in the current implementation.
### Anything else?
_No response_
### Are you willing to submit a PR?
- [ ] I'm willing to submit a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pulsar function replicas auto scaling [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat commented on issue #18584:
URL: https://github.com/apache/pulsar/issues/18584#issuecomment-6388860301
Thanks @zzzming. Autoscaling in the built-in Kubernetes runtime would add new configuration and change how a function's `parallelism` is owned (today the function metadata is the source of truth and the StatefulSet is rebuilt from it), so it needs a PIP: https://github.com/apache/pulsar/blob/master/pip/README.md. As mentioned in the thread, Function Mesh already supports HPA-based scaling of functions and connectors (https://functionmesh.io/docs/scaling). I'm closing this for now; a PIP is welcome.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hpvd commented on issue #18584: Pulsar function replicas auto scaling
Posted by GitBox <gi...@apache.org>.
hpvd commented on issue #18584:
URL: https://github.com/apache/pulsar/issues/18584#issuecomment-1325453057
maybe this is interesting
https://github.com/streamnative/function-mesh/issues/457
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] michaeljmarshall commented on issue #18584: Pulsar function replicas auto scaling
Posted by GitBox <gi...@apache.org>.
michaeljmarshall commented on issue #18584:
URL: https://github.com/apache/pulsar/issues/18584#issuecomment-1329552805
Thanks for the suggestion @hpvd. I think this issue is requesting an improvement to the kubernetes runtime that ships with pulsar.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] hpvd commented on issue #18584: Pulsar function replicas auto scaling
Posted by GitBox <gi...@apache.org>.
hpvd commented on issue #18584:
URL: https://github.com/apache/pulsar/issues/18584#issuecomment-1329708452
jep. Sorry the issue was not the right target to link (it's a preview how function scaling in function mesh should/will work in the near future -> using keda)
Wanted to point to a solution how functions could be auto-scaled today (with additional tool: function-mesh)
> With Kubernetes Horizontal Pod Autoscaler (HPA), Function Mesh supports automatically scaling the number of Pods (Pulsar instances) that are required to run Pulsar functions, sources, and sinks.
>
> Function Mesh auto-scales the number of Pods based on the CPU usage, memory usage, and custom metrics.
source:
https://functionmesh.io/docs/scaling
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] github-actions[bot] commented on issue #18584: Pulsar function replicas auto scaling
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on issue #18584:
URL: https://github.com/apache/pulsar/issues/18584#issuecomment-1367022446
The issue had no activity for 30 days, mark with Stale label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Pulsar function replicas auto scaling [pulsar]
Posted by "merlimat (via GitHub)" <gi...@apache.org>.
merlimat closed issue #18584: Pulsar function replicas auto scaling
URL: https://github.com/apache/pulsar/issues/18584
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pulsar.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org