You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "priyen (via GitHub)" <gi...@apache.org> on 2023/04/14 16:46:53 UTC
[GitHub] [pinot] priyen opened a new issue, #10607: [bug] Rebalance doesn't correctly assign all servers
priyen opened a new issue, #10607:
URL: https://github.com/apache/pinot/issues/10607
In the following scenario, I noticed that rebalancing caused only 10/30 of the servers to be utilized:
- minimize data movement = true
- num instances per partition = 5
- num instances = 30
- num instances per replica group = 15
- num replicas = 2
- pool based = true
I had servers with id 1-30, but only 1-7 and 17-19 got assigned segments.
When I took out minimize data movement, then it correctly distributed among all servers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli closed issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli closed issue #10607: [bug] Rebalance doesn't correctly assign all servers
URL: https://github.com/apache/pinot/issues/10607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10607:
URL: https://github.com/apache/pinot/issues/10607#issuecomment-1508948736
@priyen Have you set `numPartitions` in your config? Seems that is missing
@jackjlli I don't think this is desired behavior as minimize movement causes multiple servers not assigned anything, and if you remove that it flag, all servers are assigned
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] priyen commented on issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "priyen (via GitHub)" <gi...@apache.org>.
priyen commented on issue #10607:
URL: https://github.com/apache/pinot/issues/10607#issuecomment-1508780627
@jackjlli , I dont think this is desired. Your proposal here of changing numInstancesPerPartition to 15 means every partition is on all 15 servers, which is not the desired case here.
Take segments 1-15, let's say 1-5 are partition 1, 5-10 are partition 2, 10-15 are partition 3,
instances 1-5 => get partition 1's segments
instances 6-10 => get partition 2's segments
instances 11-15 => get partition 3's segments.
If you change numInstancesPerPartition to 15, all 3 partitions' segments are on all the segments, which is NOT the desired, **we dont want to change numInstancesPerPartition to 15**
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli commented on issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli commented on issue #10607:
URL: https://github.com/apache/pinot/issues/10607#issuecomment-1509404993
Discussed with @priyen offline. We modified the existing unit test in order to print out the output of the instance partition mapping, and we don't see the scenarios where only partial of the pool are assigned. We suspected that some configs might have been messed up during that time. Based on the output of the edited unit test, we don't see the current algorithm behaves abnormally.
The below is the edited unit test:
```
@Test
public void testPoolBased() {
// 10 instances in 2 pools, each with 5 instances. Pool 0: 0-4, Pool 1: 5-9
int numInstances = 10;
int numPools = 2;
int numInstancesPerPool = numInstances / numPools;
int numReplicaGroups = 2;
List<InstanceConfig> instanceConfigs = new ArrayList<>(numInstances);
for (int i = 0; i < numInstances; i++) {
InstanceConfig instanceConfig = new InstanceConfig(SERVER_INSTANCE_ID_PREFIX + i);
instanceConfig.addTag(OFFLINE_TAG);
int pool = i / numInstancesPerPool;
instanceConfig.getRecord()
.setMapField(InstanceUtils.POOL_KEY, Collections.singletonMap(OFFLINE_TAG, Integer.toString(pool)));
instanceConfigs.add(instanceConfig);
}
InstanceReplicaGroupPartitionConfig replicaPartitionConfig = new InstanceReplicaGroupPartitionConfig(true, 10, numReplicaGroups, 5, 24, 5, true);
InstanceTagPoolConfig tagPoolConfig = new InstanceTagPoolConfig(OFFLINE_TAG, true, numPools, null);
TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(RAW_TABLE_NAME)
.setInstanceAssignmentConfigMap(Collections.singletonMap(InstancePartitionsType.OFFLINE.toString(),
new InstanceAssignmentConfig(tagPoolConfig, null, replicaPartitionConfig))).build();
tableConfig.setInstanceAssignmentConfigMap(Collections.singletonMap(InstancePartitionsType.OFFLINE.toString(),
new InstanceAssignmentConfig(tagPoolConfig, null, replicaPartitionConfig)));
InstanceAssignmentDriver driver = new InstanceAssignmentDriver(tableConfig);
InstancePartitions existingInstancePartitions = null;
InstancePartitions instancePartitions =
driver.assignInstances(InstancePartitionsType.OFFLINE, instanceConfigs, existingInstancePartitions);
assertEquals(instancePartitions.getNumReplicaGroups(), numReplicaGroups);
assertEquals(instancePartitions.getNumPartitions(), 24);
System.out.println(instancePartitions);
// Now 30 instances in 2 pools, each with 15 instances. Pool 0: 10-19, Pool 1: 20-29
numInstances = 30;
int moreNumInstancesPerPool = 10;
for (int i = 10; i < numInstances; i++) {
InstanceConfig instanceConfig = new InstanceConfig(SERVER_INSTANCE_ID_PREFIX + i);
instanceConfig.addTag(OFFLINE_TAG);
int pool = (i - 10) / moreNumInstancesPerPool;
instanceConfig.getRecord()
.setMapField(InstanceUtils.POOL_KEY, Collections.singletonMap(OFFLINE_TAG, Integer.toString(pool)));
instanceConfigs.add(instanceConfig);
}
replicaPartitionConfig = new InstanceReplicaGroupPartitionConfig(true, 30, numReplicaGroups, 15, 24, 5, true);
tableConfig.setInstanceAssignmentConfigMap(Collections.singletonMap(InstancePartitionsType.OFFLINE.toString(),
new InstanceAssignmentConfig(tagPoolConfig, null, replicaPartitionConfig)));
driver = new InstanceAssignmentDriver(tableConfig);
instancePartitions =
driver.assignInstances(InstancePartitionsType.OFFLINE, instanceConfigs, existingInstancePartitions);
assertEquals(instancePartitions.getNumReplicaGroups(), numReplicaGroups);
assertEquals(instancePartitions.getNumPartitions(), 24);
System.out.println(instancePartitions);
}
```
Since there is no issue found out from the tests, I don't think it's worthy of keeping this issue open. I'll close this issue for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on issue #10607:
URL: https://github.com/apache/pinot/issues/10607#issuecomment-1507616841
@jackjlli Can you help take a look? Seems like a bug in minimize data movement when more instances are added
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli commented on issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli commented on issue #10607:
URL: https://github.com/apache/pinot/issues/10607#issuecomment-1507699824
Discussed with @priyen offline. The issue was that `numInstancesPerPartition` was set to 5, thus only 5 instances are retained in the instance partitions mapping, thus only 10 servers are selected in total. Setting it to 15 should fix the mapping. I don't think there is any issue in the existing logic for minimize data movement in this case. We can close this issue now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] jackjlli closed issue #10607: [bug] Rebalance doesn't correctly assign all servers
Posted by "jackjlli (via GitHub)" <gi...@apache.org>.
jackjlli closed issue #10607: [bug] Rebalance doesn't correctly assign all servers
URL: https://github.com/apache/pinot/issues/10607
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org