You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "bipinprasad (via GitHub)" <gi...@apache.org> on 2023/05/03 08:24:04 UTC
[GitHub] [storm] bipinprasad opened a new pull request, #3540: [STORM-3916] Add Round Robin Scheduling stretgy with optional node li…
bipinprasad opened a new pull request, #3540:
URL: https://github.com/apache/storm/pull/3540
…mit.
## What is the purpose of the change
*Round Robin strategy will distribute components evenly across nodes. Default behavior will be to spread across all nodes. But can be limited by specifying the maximum number of nodes to use by setting Config.TOPOLOGY_ISOLATED_MACHINES*
## How was the change tested
*New and modified unit tests*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "agresch (via GitHub)" <gi...@apache.org>.
agresch commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1218307249
##########
storm-client/src/jvm/org/apache/storm/Config.java:
##########
@@ -859,9 +859,11 @@ public class Config extends HashMap<String, Object> {
@IsString
public static final String STORM_DO_AS_USER = "storm.doAsUser";
/**
- * The number of machines that should be used by this topology to isolate it from all others. Set storm.scheduler to
- * org.apache.storm.scheduler.multitenant.MultitenantScheduler
- */
+ * The number of machines that should be used by this topology to isolate it from all others.
Review Comment:
Unrelated to your change, but hopefully you can address... I don't understand why specifying a number of machines causes isolation. Your PR mentions this is a max number of machines. Can you add/modify the description to clarify this config description?
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -164,6 +168,7 @@ public SchedulingResult schedule(Cluster cluster, TopologyDetails td) {
//order executors to be scheduled
List<ExecutorDetails> orderedExecutors = execSorter.sortExecutors(unassignedExecutors);
+ isolateAckersToEnd(orderedExecutors);
Review Comment:
this seems to be a behavior change? Why was this added?
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -527,13 +538,13 @@ protected SchedulingResult scheduleExecutorsOnNodes(List<ExecutorDetails> ordere
if (execIndex == 0) {
break;
} else {
- searcherState.backtrack(execToComp, nodeForExec[execIndex - 1], workerSlotForExec[execIndex - 1]);
Review Comment:
why did this line change?
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -123,6 +123,10 @@ public BaseResourceAwareStrategy(boolean sortNodesForEachExecutor, NodeSortType
this.nodeSortType = nodeSortType;
}
+ public boolean isSortNodesForEachExecutor() {
Review Comment:
where is this used?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] agresch commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "agresch (via GitHub)" <gi...@apache.org>.
agresch commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1221809229
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -527,13 +538,13 @@ protected SchedulingResult scheduleExecutorsOnNodes(List<ExecutorDetails> ordere
if (execIndex == 0) {
break;
} else {
- searcherState.backtrack(execToComp, nodeForExec[execIndex - 1], workerSlotForExec[execIndex - 1]);
Review Comment:
Are there more missing changes that go along with this? Can we get that merged first before this PR if there are more changes?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1220291317
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -164,6 +168,7 @@ public SchedulingResult schedule(Cluster cluster, TopologyDetails td) {
//order executors to be scheduled
List<ExecutorDetails> orderedExecutors = execSorter.sortExecutors(unassignedExecutors);
+ isolateAckersToEnd(orderedExecutors);
Review Comment:
This is refactoring. This reordering was earlier being done in scheduleExecutorsOnNodes(). But it should be done when the executors are sorted. Then Round Robin strategy to can override scheduleExecutorsOnNodes() and still receive the executors sorted in right order (i.e. ackers at the end)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad merged pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad merged PR #3540:
URL: https://github.com/apache/storm/pull/3540
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1220355853
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -527,13 +538,13 @@ protected SchedulingResult scheduleExecutorsOnNodes(List<ExecutorDetails> ordere
if (execIndex == 0) {
break;
} else {
- searcherState.backtrack(execToComp, nodeForExec[execIndex - 1], workerSlotForExec[execIndex - 1]);
Review Comment:
This method signature was changed to backtrack correctly in the presence of Acker executors. I believe this is Ray's change that did not make it to community earlier.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1220322597
##########
storm-client/src/jvm/org/apache/storm/Config.java:
##########
@@ -859,9 +859,11 @@ public class Config extends HashMap<String, Object> {
@IsString
public static final String STORM_DO_AS_USER = "storm.doAsUser";
/**
- * The number of machines that should be used by this topology to isolate it from all others. Set storm.scheduler to
- * org.apache.storm.scheduler.multitenant.MultitenantScheduler
- */
+ * The number of machines that should be used by this topology to isolate it from all others.
Review Comment:
Added more detailed explanation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1220277226
##########
storm-server/src/main/java/org/apache/storm/scheduler/resource/strategies/scheduling/BaseResourceAwareStrategy.java:
##########
@@ -123,6 +123,10 @@ public BaseResourceAwareStrategy(boolean sortNodesForEachExecutor, NodeSortType
this.nodeSortType = nodeSortType;
}
+ public boolean isSortNodesForEachExecutor() {
Review Comment:
Removed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1219927539
##########
storm-client/src/jvm/org/apache/storm/Config.java:
##########
@@ -859,9 +859,11 @@ public class Config extends HashMap<String, Object> {
@IsString
public static final String STORM_DO_AS_USER = "storm.doAsUser";
/**
- * The number of machines that should be used by this topology to isolate it from all others. Set storm.scheduler to
- * org.apache.storm.scheduler.multitenant.MultitenantScheduler
- */
+ * The number of machines that should be used by this topology to isolate it from all others.
Review Comment:
I think I can add "Topology scheduling will be limited to this number of machines".
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [storm] bipinprasad commented on a diff in pull request #3540: [STORM-3916] Add Round Robin Scheduling strategy with optional node limit
Posted by "bipinprasad (via GitHub)" <gi...@apache.org>.
bipinprasad commented on code in PR #3540:
URL: https://github.com/apache/storm/pull/3540#discussion_r1219931053
##########
storm-client/src/jvm/org/apache/storm/Config.java:
##########
@@ -859,9 +859,11 @@ public class Config extends HashMap<String, Object> {
@IsString
public static final String STORM_DO_AS_USER = "storm.doAsUser";
/**
- * The number of machines that should be used by this topology to isolate it from all others. Set storm.scheduler to
- * org.apache.storm.scheduler.multitenant.MultitenantScheduler
- */
+ * The number of machines that should be used by this topology to isolate it from all others.
Review Comment:
or "Maximum limit for the number of machines on which a topology will be scheduled"
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: dev-unsubscribe@storm.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org