You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pulsar.apache.org by GitBox <gi...@apache.org> on 2020/06/30 00:27:52 UTC
[GitHub] [pulsar] jerrypeng opened a new pull request #7388: WIP: Implement rebalance mechanism in Pulsar Functions
jerrypeng opened a new pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388
# Motivation
Currently there no mechanism for rebalancing functions scheduler on workers. Workload for functions may become skewed.
### Modifications
1. Add interface for rebalance strategy.
2. Allow users to trigger a rebalance on-demand
Will add mechanism to run the rebalance periodically automatically in subsequent PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #7388: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388#discussion_r448062976
##########
File path: conf/functions_worker.yml
##########
@@ -78,6 +78,7 @@ numHttpServerThreads: 8
# function assignment and scheduler
schedulerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler"
+rebalancerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler"
Review comment:
will remove
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng commented on a change in pull request #7388: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
jerrypeng commented on a change in pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388#discussion_r448063445
##########
File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/scheduler/IScheduler.java
##########
@@ -38,4 +39,12 @@
*/
List<Assignment> schedule(List<Instance> unassignedFunctionInstances, List<Assignment> currentAssignments,
Set<String> workers);
+
+ /**
+ *
+ * @param currentAssignments
+ * @param workers
+ * @return
+ */
+ List<Function.Assignment> rebalance(List<Function.Assignment> currentAssignments, Set<String> workers);
Review comment:
Sure we can do it for BC reasons. It would return a empty list because there will be no change
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] jerrypeng merged pull request #7388: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
jerrypeng merged pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] srkukarni commented on a change in pull request #7388: WIP: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
srkukarni commented on a change in pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388#discussion_r447401359
##########
File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/SchedulerManager.java
##########
@@ -93,6 +94,8 @@
private final IScheduler scheduler;
+ private final IRebalance rebalancer;
Review comment:
nit: IRebalancer?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] srkukarni commented on a change in pull request #7388: WIP: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
srkukarni commented on a change in pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388#discussion_r447401903
##########
File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/rest/api/WorkerImpl.java
##########
@@ -198,4 +198,16 @@ private boolean isSuperUser(final String clientRole) {
return this.worker().getConnectorsManager().getConnectors();
}
+
+ public void rebalance(final String clientRole) {
+ if (!isWorkerServiceAvailable()) {
+ throwUnavailableException();
+ }
+
Review comment:
This should be a super user only command
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [pulsar] merlimat commented on a change in pull request #7388: Implement rebalance mechanism in Pulsar Functions
Posted by GitBox <gi...@apache.org>.
merlimat commented on a change in pull request #7388:
URL: https://github.com/apache/pulsar/pull/7388#discussion_r448062262
##########
File path: conf/functions_worker.yml
##########
@@ -78,6 +78,7 @@ numHttpServerThreads: 8
# function assignment and scheduler
schedulerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler"
+rebalancerClassName: "org.apache.pulsar.functions.worker.scheduler.RoundRobinScheduler"
Review comment:
This shouldn't be needed anymore, no?
##########
File path: pulsar-functions/worker/src/main/java/org/apache/pulsar/functions/worker/scheduler/IScheduler.java
##########
@@ -38,4 +39,12 @@
*/
List<Assignment> schedule(List<Instance> unassignedFunctionInstances, List<Assignment> currentAssignments,
Set<String> workers);
+
+ /**
+ *
+ * @param currentAssignments
+ * @param workers
+ * @return
+ */
+ List<Function.Assignment> rebalance(List<Function.Assignment> currentAssignments, Set<String> workers);
Review comment:
Should we have a default implementation that just returns the current assignment?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org