You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "klsince (via GitHub)" <gi...@apache.org> on 2025/06/05 17:33:08 UTC
[PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
klsince opened a new pull request, #16017:
URL: https://github.com/apache/pinot/pull/16017
Like upsert table, the dedup table should use StrictRealtimeSegmentAssignment, i.e. the new segments should be put on the same hosts where the table partition's all other segments are kept, following the segment assignment as tracked in IdealState instead of the current InstancePartition which may provide a different host to hold the segment.
Meanwhile, dedup table can move its segments out of TTL to other tiers more freely, so the StrictRealtimeSegmentAssignment has been extended to support multi tiers in this PR. However, note that upsert table can not move its segments out of TTL to new tiers due to the associated validDocIds bitmaps. After we add support to move bitmaps with segments, we can allow upsert tables to use mutli tiers too, which is to be extended separately.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2165110168
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/MultiTierStrictRealtimeSegmentAssignment.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.assignment.segment;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import javax.annotation.Nullable;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.assignment.InstancePartitions;
+import org.apache.pinot.common.tier.Tier;
+import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfig;
+import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants;
+
+
+/**
+ * This segment assignment policy allows the table to use multiple tiers when rebalancing table. This can be used for
+ * dedup table, whose segments out of TTL can be moved to new tiers without messing up the dedup metadata tracked on
+ * the CONSUMING servers.
+ */
+public class MultiTierStrictRealtimeSegmentAssignment extends BaseStrictRealtimeSegmentAssignment {
+ @Override
+ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, String>> currentAssignment,
+ Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, @Nullable List<Tier> sortedTiers,
+ @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap, RebalanceConfig config) {
+ Preconditions.checkState(instancePartitionsMap.size() == 1, "One instance partition type should be provided");
+ InstancePartitions instancePartitions = instancePartitionsMap.get(InstancePartitionsType.CONSUMING);
+ Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
+ _tableNameWithType);
+ Preconditions.checkArgument(config.isIncludeConsuming(),
+ "Consuming segment must be included when rebalancing table: %s using multi-tier "
+ + "StrictRealtimeSegmentAssignment", _tableNameWithType);
+ boolean bootstrap = config.isBootstrap();
+ _logger.info("Rebalancing table: {} with instance partitions: {}, bootstrap: {}", _tableNameWithType,
+ instancePartitions, bootstrap);
+ // Rebalance tiers first. Only completed segments are moved to other tiers.
+ Pair<List<Map<String, Map<String, String>>>, Map<String, Map<String, String>>> pair =
+ rebalanceTiers(currentAssignment, sortedTiers, tierInstancePartitionsMap, bootstrap,
Review Comment:
btw, I'll try to add the checks in this PR if not much, otherwise will do a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 closed pull request #16017: Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers
URL: https://github.com/apache/pinot/pull/16017
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #16017:
URL: https://github.com/apache/pinot/pull/16017#issuecomment-2945590866
### :x: Unsupported file format
> Upload processing failed due to unsupported file format. Please review the parser error message:
> <pre><code class="language-python">Error parsing JUnit XML in /home/runner/work/pinot/pinot/pinot-query-planner/target/surefire-reports/TEST-org.apache.pinot.query.queries.ResourceBasedQueryPlansTest.xml at 142:1104
>
> Caused by:
> RuntimeError: Error converting computed name to ValidatedString
>
> Caused by:
> string is too long</code></pre>
> For more help, visit our [troubleshooting guide](https://docs.codecov.com/docs/test-analytics#troubleshooting).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2136567027
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/MultiTierStrictRealtimeSegmentAssignment.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.assignment.segment;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import javax.annotation.Nullable;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.assignment.InstancePartitions;
+import org.apache.pinot.common.tier.Tier;
+import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfig;
+import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants;
+
+
+/**
+ * This segment assignment policy allows the table to use multiple tiers when rebalancing table. This can be used for
+ * dedup table, whose segments out of TTL can be moved to new tiers without messing up the dedup metadata tracked on
+ * the CONSUMING servers.
+ */
+public class MultiTierStrictRealtimeSegmentAssignment extends BaseStrictRealtimeSegmentAssignment {
+ @Override
+ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, String>> currentAssignment,
+ Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, @Nullable List<Tier> sortedTiers,
+ @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap, RebalanceConfig config) {
+ Preconditions.checkState(instancePartitionsMap.size() == 1, "One instance partition type should be provided");
+ InstancePartitions instancePartitions = instancePartitionsMap.get(InstancePartitionsType.CONSUMING);
+ Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
+ _tableNameWithType);
+ Preconditions.checkArgument(config.isIncludeConsuming(),
+ "Consuming segment must be included when rebalancing table: %s using multi-tier "
+ + "StrictRealtimeSegmentAssignment", _tableNameWithType);
+ boolean bootstrap = config.isBootstrap();
+ _logger.info("Rebalancing table: {} with instance partitions: {}, bootstrap: {}", _tableNameWithType,
+ instancePartitions, bootstrap);
+ // Rebalance tiers first. Only completed segments are moved to other tiers.
+ Pair<List<Map<String, Map<String, String>>>, Map<String, Map<String, String>>> pair =
+ rebalanceTiers(currentAssignment, sortedTiers, tierInstancePartitionsMap, bootstrap,
Review Comment:
just for my understanding - does this handle the case where only segments past the metadataTTL can be moved for dedup? or is that a non-concern?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2136567027
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/MultiTierStrictRealtimeSegmentAssignment.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.assignment.segment;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import javax.annotation.Nullable;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.assignment.InstancePartitions;
+import org.apache.pinot.common.tier.Tier;
+import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfig;
+import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants;
+
+
+/**
+ * This segment assignment policy allows the table to use multiple tiers when rebalancing table. This can be used for
+ * dedup table, whose segments out of TTL can be moved to new tiers without messing up the dedup metadata tracked on
+ * the CONSUMING servers.
+ */
+public class MultiTierStrictRealtimeSegmentAssignment extends BaseStrictRealtimeSegmentAssignment {
+ @Override
+ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, String>> currentAssignment,
+ Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, @Nullable List<Tier> sortedTiers,
+ @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap, RebalanceConfig config) {
+ Preconditions.checkState(instancePartitionsMap.size() == 1, "One instance partition type should be provided");
+ InstancePartitions instancePartitions = instancePartitionsMap.get(InstancePartitionsType.CONSUMING);
+ Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
+ _tableNameWithType);
+ Preconditions.checkArgument(config.isIncludeConsuming(),
+ "Consuming segment must be included when rebalancing table: %s using multi-tier "
+ + "StrictRealtimeSegmentAssignment", _tableNameWithType);
+ boolean bootstrap = config.isBootstrap();
+ _logger.info("Rebalancing table: {} with instance partitions: {}, bootstrap: {}", _tableNameWithType,
+ instancePartitions, bootstrap);
+ // Rebalance tiers first. Only completed segments are moved to other tiers.
+ Pair<List<Map<String, Map<String, String>>>, Map<String, Map<String, String>>> pair =
+ rebalanceTiers(currentAssignment, sortedTiers, tierInstancePartitionsMap, bootstrap,
Review Comment:
just for my understanding - does this handle the case where only segments past the metadataTTL can be moved for dedup? or is that a non-concern?
I see in the description also, you call this out: `Meanwhile, dedup table can move its segments out of TTL to other tiers more freely` - so wanted to understand where this TTL check is happening
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2165071374
##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java:
##########
@@ -255,9 +279,74 @@ public void testAssignSegmentWithOfflineSegment() {
}
}
- @Test(expectedExceptions = IllegalStateException.class)
- public void testAssignSegmentToCompletedServers() {
- _segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
+ @Test
+ public void testRebalanceDedupTableWithTiers() {
+ SegmentAssignment segmentAssignment = createSegmentAssignment("dedup");
+ Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
+ ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
+ Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
+ Set<String> segmentsOnTier = new HashSet<>();
+ for (int segmentId = 0; segmentId < 6; segmentId++) {
+ String segmentName = _segments.get(segmentId);
+ if (segmentId < 3) {
+ segmentsOnTier.add(segmentName);
+ }
+ List<String> instancesAssigned =
+ segmentAssignment.assignSegment(segmentName, currentAssignment, _instancePartitionsMap);
+ currentAssignment.put(segmentName,
+ SegmentAssignmentUtils.getInstanceStateMap(instancesAssigned, SegmentStateModel.ONLINE));
+ }
+ String tierName = "coldTier";
+ List<Tier> sortedTiers = createSortedTiers(tierName, segmentsOnTier);
+ Map<String, InstancePartitions> tierInstancePartitionsMap = createTierInstancePartitionsMap(tierName, 3);
+ RebalanceConfig rebalanceConfig = new RebalanceConfig();
+ rebalanceConfig.setIncludeConsuming(true);
+ Map<String, Map<String, String>> newAssignment =
+ segmentAssignment.rebalanceTable(currentAssignment, onlyConsumingInstancePartitionMap, sortedTiers,
+ tierInstancePartitionsMap, rebalanceConfig);
+ assertEquals(newAssignment.size(), currentAssignment.size());
+ for (String segName : currentAssignment.keySet()) {
+ if (segmentsOnTier.contains(segName)) {
+ assertTrue(newAssignment.get(segName).keySet().stream().allMatch(s -> s.startsWith(tierName)));
+ } else {
+ assertTrue(
+ newAssignment.get(segName).keySet().stream().allMatch(s -> s.startsWith(CONSUMING_INSTANCE_NAME_PREFIX)));
+ }
+ }
+ }
+
+ @Test(expectedExceptions = IllegalStateException.class, expectedExceptionsMessageRegExp = "Tiers must not be "
+ + "specified for table.*")
+ public void testRebalanceUpsertTableWithTiers() {
+ SegmentAssignment segmentAssignment = createSegmentAssignment("upsert");
+ Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
+ ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
+ Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
+ Set<String> segmentsOnTier = new HashSet<>();
+ for (int segmentId = 0; segmentId < 6; segmentId++) {
+ String segmentName = _segments.get(segmentId);
+ if (segmentId < 3) {
+ segmentsOnTier.add(segmentName);
+ }
+ List<String> instancesAssigned =
+ segmentAssignment.assignSegment(segmentName, currentAssignment, _instancePartitionsMap);
+ currentAssignment.put(segmentName,
+ SegmentAssignmentUtils.getInstanceStateMap(instancesAssigned, SegmentStateModel.ONLINE));
+ }
+
+ String tierName = "coldTier";
+ List<Tier> sortedTiers = createSortedTiers(tierName, segmentsOnTier);
+ Map<String, InstancePartitions> tierInstancePartitionsMap = createTierInstancePartitionsMap(tierName, 3);
+ RebalanceConfig rebalanceConfig = new RebalanceConfig();
+ rebalanceConfig.setIncludeConsuming(true);
+ segmentAssignment.rebalanceTable(currentAssignment, onlyConsumingInstancePartitionMap, sortedTiers,
+ tierInstancePartitionsMap, rebalanceConfig);
+ }
+
+ @Test(expectedExceptions = IllegalStateException.class, dataProvider = "tableTypes")
+ public void testAssignSegmentToCompletedServers(String tableType) {
+ SegmentAssignment segmentAssignment = createSegmentAssignment(tableType);
+ segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
Review Comment:
correct. good catch about the input param. Not introduced in this PR, but that map should include a COMPLETED entry to validate the sanity checks. Will fix that.
Will add tests on rebalanceTable() too, to validate the sanity checks on COMPLETED tag.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2165061087
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/assignment/segment/MultiTierStrictRealtimeSegmentAssignment.java:
##########
@@ -0,0 +1,87 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.controller.helix.core.assignment.segment;
+
+import com.google.common.base.Preconditions;
+import java.util.List;
+import java.util.Map;
+import java.util.TreeMap;
+import javax.annotation.Nullable;
+import org.apache.commons.collections4.CollectionUtils;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.assignment.InstancePartitions;
+import org.apache.pinot.common.tier.Tier;
+import org.apache.pinot.controller.helix.core.rebalance.RebalanceConfig;
+import org.apache.pinot.spi.config.table.assignment.InstancePartitionsType;
+import org.apache.pinot.spi.utils.CommonConstants;
+
+
+/**
+ * This segment assignment policy allows the table to use multiple tiers when rebalancing table. This can be used for
+ * dedup table, whose segments out of TTL can be moved to new tiers without messing up the dedup metadata tracked on
+ * the CONSUMING servers.
+ */
+public class MultiTierStrictRealtimeSegmentAssignment extends BaseStrictRealtimeSegmentAssignment {
+ @Override
+ public Map<String, Map<String, String>> rebalanceTable(Map<String, Map<String, String>> currentAssignment,
+ Map<InstancePartitionsType, InstancePartitions> instancePartitionsMap, @Nullable List<Tier> sortedTiers,
+ @Nullable Map<String, InstancePartitions> tierInstancePartitionsMap, RebalanceConfig config) {
+ Preconditions.checkState(instancePartitionsMap.size() == 1, "One instance partition type should be provided");
+ InstancePartitions instancePartitions = instancePartitionsMap.get(InstancePartitionsType.CONSUMING);
+ Preconditions.checkState(instancePartitions != null, "Failed to find CONSUMING instance partitions for table: %s",
+ _tableNameWithType);
+ Preconditions.checkArgument(config.isIncludeConsuming(),
+ "Consuming segment must be included when rebalancing table: %s using multi-tier "
+ + "StrictRealtimeSegmentAssignment", _tableNameWithType);
+ boolean bootstrap = config.isBootstrap();
+ _logger.info("Rebalancing table: {} with instance partitions: {}, bootstrap: {}", _tableNameWithType,
+ instancePartitions, bootstrap);
+ // Rebalance tiers first. Only completed segments are moved to other tiers.
+ Pair<List<Map<String, Map<String, String>>>, Map<String, Map<String, String>>> pair =
+ rebalanceTiers(currentAssignment, sortedTiers, tierInstancePartitionsMap, bootstrap,
Review Comment:
Good question. The dedup TTL should be set according to tier's segmentAge (which I think should be `min[all tier's segmentAges]`), but not checked here. I think it's better to check this as part of the table/dedup config validation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #16017:
URL: https://github.com/apache/pinot/pull/16017#discussion_r2136612615
##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/assignment/segment/StrictRealtimeSegmentAssignmentTest.java:
##########
@@ -255,9 +279,74 @@ public void testAssignSegmentWithOfflineSegment() {
}
}
- @Test(expectedExceptions = IllegalStateException.class)
- public void testAssignSegmentToCompletedServers() {
- _segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
+ @Test
+ public void testRebalanceDedupTableWithTiers() {
+ SegmentAssignment segmentAssignment = createSegmentAssignment("dedup");
+ Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
+ ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
+ Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
+ Set<String> segmentsOnTier = new HashSet<>();
+ for (int segmentId = 0; segmentId < 6; segmentId++) {
+ String segmentName = _segments.get(segmentId);
+ if (segmentId < 3) {
+ segmentsOnTier.add(segmentName);
+ }
+ List<String> instancesAssigned =
+ segmentAssignment.assignSegment(segmentName, currentAssignment, _instancePartitionsMap);
+ currentAssignment.put(segmentName,
+ SegmentAssignmentUtils.getInstanceStateMap(instancesAssigned, SegmentStateModel.ONLINE));
+ }
+ String tierName = "coldTier";
+ List<Tier> sortedTiers = createSortedTiers(tierName, segmentsOnTier);
+ Map<String, InstancePartitions> tierInstancePartitionsMap = createTierInstancePartitionsMap(tierName, 3);
+ RebalanceConfig rebalanceConfig = new RebalanceConfig();
+ rebalanceConfig.setIncludeConsuming(true);
+ Map<String, Map<String, String>> newAssignment =
+ segmentAssignment.rebalanceTable(currentAssignment, onlyConsumingInstancePartitionMap, sortedTiers,
+ tierInstancePartitionsMap, rebalanceConfig);
+ assertEquals(newAssignment.size(), currentAssignment.size());
+ for (String segName : currentAssignment.keySet()) {
+ if (segmentsOnTier.contains(segName)) {
+ assertTrue(newAssignment.get(segName).keySet().stream().allMatch(s -> s.startsWith(tierName)));
+ } else {
+ assertTrue(
+ newAssignment.get(segName).keySet().stream().allMatch(s -> s.startsWith(CONSUMING_INSTANCE_NAME_PREFIX)));
+ }
+ }
+ }
+
+ @Test(expectedExceptions = IllegalStateException.class, expectedExceptionsMessageRegExp = "Tiers must not be "
+ + "specified for table.*")
+ public void testRebalanceUpsertTableWithTiers() {
+ SegmentAssignment segmentAssignment = createSegmentAssignment("upsert");
+ Map<InstancePartitionsType, InstancePartitions> onlyConsumingInstancePartitionMap =
+ ImmutableMap.of(InstancePartitionsType.CONSUMING, _instancePartitionsMap.get(InstancePartitionsType.CONSUMING));
+ Map<String, Map<String, String>> currentAssignment = new TreeMap<>();
+ Set<String> segmentsOnTier = new HashSet<>();
+ for (int segmentId = 0; segmentId < 6; segmentId++) {
+ String segmentName = _segments.get(segmentId);
+ if (segmentId < 3) {
+ segmentsOnTier.add(segmentName);
+ }
+ List<String> instancesAssigned =
+ segmentAssignment.assignSegment(segmentName, currentAssignment, _instancePartitionsMap);
+ currentAssignment.put(segmentName,
+ SegmentAssignmentUtils.getInstanceStateMap(instancesAssigned, SegmentStateModel.ONLINE));
+ }
+
+ String tierName = "coldTier";
+ List<Tier> sortedTiers = createSortedTiers(tierName, segmentsOnTier);
+ Map<String, InstancePartitions> tierInstancePartitionsMap = createTierInstancePartitionsMap(tierName, 3);
+ RebalanceConfig rebalanceConfig = new RebalanceConfig();
+ rebalanceConfig.setIncludeConsuming(true);
+ segmentAssignment.rebalanceTable(currentAssignment, onlyConsumingInstancePartitionMap, sortedTiers,
+ tierInstancePartitionsMap, rebalanceConfig);
+ }
+
+ @Test(expectedExceptions = IllegalStateException.class, dataProvider = "tableTypes")
+ public void testAssignSegmentToCompletedServers(String tableType) {
+ SegmentAssignment segmentAssignment = createSegmentAssignment(tableType);
+ segmentAssignment.assignSegment("seg01", new TreeMap<>(), new TreeMap<>());
Review Comment:
is this test ensuring that we don't move segments to "COMPLETED" instance partitions? but the passed in instancePartitionsMap is empty?
Can we also have a test to reject if instancePartitionsMap has COMPLETED?
perhaps also have similar test for rebalanceTables() API?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Make dedup table use StrictRealtimeSegmentAssignment with support of multi tiers [pinot]
Posted by "xiangfu0 (via GitHub)" <gi...@apache.org>.
xiangfu0 commented on PR #16017:
URL: https://github.com/apache/pinot/pull/16017#issuecomment-4190873270
Close stale PR, please either rebase and get this PR merged or create a new one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org