You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "somandal (via GitHub)" <gi...@apache.org> on 2025/03/13 19:21:39 UTC
[PR] Enhance rebalance progress stats [pinot]
somandal opened a new pull request, #15266:
URL: https://github.com/apache/pinot/pull/15266
This PR enhances the Table Rebalance progress stats.
It adds the following to the existing progress stats:
```
"rebalanceProgressStatsOverall" : {
"_totalSegmentsToBeAdded" : 93,
"_totalSegmentsToBeDeleted" : 93,
"_totalRemainingSegmentsToBeAdded" : 31,
"_totalRemainingSegmentsToBeDeleted" : 31,
"_percentageTotalSegmentsAddsRemaining" : 33.33333333333333,
"_percentageTotalSegmentDeletesRemaining" : 33.33333333333333,
"_estimatedTimeToCompleteAdds" : 0,
"_estimatedTimeToCompleteDeletes" : 0,
"_averageSegmentSizeInBytes" : 153608,
"_totalEstimatedDataToBeMovedInBytes" : 14285544
},
"rebalanceProgressStatsCurrentStep" : {
"_totalSegmentsToBeAdded" : 31,
"_totalSegmentsToBeDeleted" : 31,
"_totalRemainingSegmentsToBeAdded" : 0,
"_totalRemainingSegmentsToBeDeleted" : 0,
"_percentageTotalSegmentsAddsRemaining" : 0.0,
"_percentageTotalSegmentDeletesRemaining" : 0.0,
"_estimatedTimeToCompleteAdds" : -1,
"_estimatedTimeToCompleteDeletes" : -1,
"_averageSegmentSizeInBytes" : 153608,
"_totalEstimatedDataToBeMovedInBytes" : 4761848
},
```
The overall stats above are upadate at each step as well to capture the progress during IS-EV convergence as well
It also renames some of the existing stats to make them clearer
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #15266:
URL: https://github.com/apache/pinot/pull/15266#issuecomment-2735075640
Why is the json fields prefixed with `'_'`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on PR #15266:
URL: https://github.com/apache/pinot/pull/15266#issuecomment-2737369227
> Why is the json fields prefixed with `'_'`?
removed the '_'
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2027904077
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -402,9 +402,17 @@ private RebalanceResult doRebalance(TableConfig tableConfig, RebalanceConfig reb
}
}
+ List<String> segmentsToMove = SegmentAssignmentUtils.getSegmentsToMove(currentAssignment, targetAssignment);
+ Set<String> segmentsToMonitor = new HashSet<>(segmentsToMove);
+
+ long estimatedAverageSegmentSizeInBytes = summaryResult.getSegmentInfo().getEstimatedAverageSegmentSizeInBytes();
+ Set<String> allOriginalSegmentsIdealState = currentAssignment.keySet();
Review Comment:
done
this list was meant to be the list of segments prior to adding the latest newly added segments. once we process stats, then I add any new segments to this list so the next stats calculation ignores them. hope that makes sense
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +128,116 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param latestStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats lastOverallStats = getRebalanceProgressStatsOverall();
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int upperBoundOnSegmentsAdded =
Review Comment:
added comments, please take a look when you get a chance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2025459940
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +328,306 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int upperBoundOnSegmentsAdded =
+ lastStepStats._totalRemainingSegmentsToBeAdded > lastStepStats._totalSegmentsToBeAdded
+ ? lastStepStats._totalSegmentsToBeAdded : lastStepStats._totalRemainingSegmentsToBeAdded;
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int upperBoundOnSegmentsDeleted =
+ lastStepStats._totalRemainingSegmentsToBeDeleted > lastStepStats._totalSegmentsToBeDeleted
+ ? lastStepStats._totalSegmentsToBeDeleted : lastStepStats._totalRemainingSegmentsToBeDeleted;
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ if (latestStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = overallProgressStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ }
+ newOverallProgressStats._totalCarryOverSegmentsToBeAdded =
+ latestStepStats._totalCarryOverSegmentsToBeAdded;
+ if (latestStepStats._totalCarryOverSegmentsToBeDeleted > 0) {
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted =
+ overallProgressStats._totalRemainingSegmentsToBeDeleted;
+ } else {
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeDeleted + numSegmentDeletesProcessedInLastStep;
+ }
+ newOverallProgressStats._totalCarryOverSegmentsToBeDeleted =
+ latestStepStats._totalCarryOverSegmentsToBeDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToConverge = latestStepStats._totalRemainingSegmentsToConverge;
+ newOverallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ overallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + numberNewUntrackedSegmentsAdded;
+ newOverallProgressStats._percentageTotalSegmentsAddsRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeAdded,
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded
+ + newOverallProgressStats._totalCarryOverSegmentsToBeAdded);
+ newOverallProgressStats._percentageTotalSegmentDeletesRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeDeleted,
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted
+ + newOverallProgressStats._totalCarryOverSegmentsToBeDeleted);
+ // Calculate elapsed time based on start of rebalance (global)
+ newOverallProgressStats._estimatedTimeToCompleteAddsInSeconds =
+ calculateEstimatedTimeToCompleteChange(rebalanceProgressStats.getStartTimeMs(),
+ newOverallProgressStats._totalSegmentsToBeAdded, newOverallProgressStats._totalRemainingSegmentsToBeAdded);
+ newOverallProgressStats._estimatedTimeToCompleteDeletesInSeconds =
+ calculateEstimatedTimeToCompleteChange(rebalanceProgressStats.getStartTimeMs(),
+ newOverallProgressStats._totalSegmentsToBeDeleted,
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted);
+ newOverallProgressStats._averageSegmentSizeInBytes = overallProgressStats._averageSegmentSizeInBytes;
+ newOverallProgressStats._totalEstimatedDataToBeMovedInBytes =
+ overallProgressStats._totalEstimatedDataToBeMovedInBytes
+ + (numAdditionalSegmentsAdded * overallProgressStats._averageSegmentSizeInBytes);
+ newOverallProgressStats._startTimeMs = rebalanceProgressStats.getStartTimeMs();
+
+ return newOverallProgressStats;
+ }
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateOverallProgressStats(
Review Comment:
I use `rebalanceProgressStats` for the following:
- For the `IDEAL_STATE_CHANGE_TRIGGER` and `EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER`, I get the current overall or step-level stats to do the calculations based on the previous values.
- I also fetch the rebalance `startTimeMs` for the estimated time to complete calculations
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021943999
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -271,17 +315,279 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
String instanceName = instanceStateEntry.getKey();
String sourceInstanceState = sourceInstanceStateMap.get(instanceName);
if (!targetStateInstanceState.equals(sourceInstanceState)) {
- rebalanceStats._replicasToRebalance++;
+ rebalanceStats._totalSegmentsToRebalance++;
hasSegmentConverged = false;
}
}
if (!hasSegmentConverged) {
- rebalanceStats._segmentsToRebalance++;
+ rebalanceStats._uniqueSegmentsToRebalance++;
}
}
int totalSegments = targetState.size();
- rebalanceStats._percentSegmentsToRebalance =
- (totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
+ rebalanceStats._percentRemainingSegmentsToRebalance =
+ (totalSegments == 0) ? 0 : ((double) rebalanceStats._uniqueSegmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int numSegmentAddsProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentDeletesProcessedInLastStep;
Review Comment:
oops, good catch. fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2022144726
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
if `bestEfforts=false`, if there is any error in EV-IS convergence (segments in ERROR state, or timeout in convergence), the rebalance will be aborted. Similarly, if say we cancel rebalance, or other exceptions occur rebalance will be aborted. The stats will not be updated and left in the intermediate state in that case, since the rebalance job is no longer running. if `RebalanceChecker` retries the rebalance for the table, it'll start a new job and the new job will have new stats for tracking.
While thinking about this, I just realized that the stats code I have currently doesn't handle stats well when `bestEfforts=true`, since in this scenario, we move to the next rebalance step without waiting for EV-IS convergence to complete, or when we have `ERROR` state segments. In this case, the stats may not reflect correctly because the comparisons for various triggers are done between different states:
- EV-IS convergence checks EV and IS
- Other triggers compare the current assignment and target assignment, both of which are related to IS only
Due to the above, if any segments are carried over from the last step, we can start seeing -ve stats or weird stats values.
Does the above answer your question?
I'm working on a possible fix, to track these additional adds / deletes carried over from the previous step separately. Will let you know when I've updated the code to handle this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023087286
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
Thanks for the detail explanation. yes, it does explain the retry scenario i was thinking.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023159343
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -18,35 +18,28 @@
*/
package org.apache.pinot.controller.helix.core.rebalance;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
+@JsonIgnoreProperties(ignoreUnknown = true)
Review Comment:
I removed the rename changes and annotation for the older stats to remove `_` for now. I have added a TODO to clean this up later on. Also removed this `@JsonIgnoreProperties(ignoreUnknown = true)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029695528
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +126,209 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ if (numAdditionalSegmentsAdded == 0) {
+ // No new additional segments added, subtract the number of processed segments from the last step
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep;
+ } else {
+ if (currentStepStats._totalRemainingSegmentsToBeAdded <= previousStepStats._totalRemainingSegmentsToBeAdded) {
+ // Case I: currentStepStats._totalRemainingSegmentsToBeAdded
+ // <= previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 5
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 5
+ // remainingSegmentsWithoutConsideringProcessed = 5 + 10 = 15
+ // netProcessedSegments = 15 - 5 = 10 -> how much to subtract after adding additional segments to remaining
+ int remainingSegmentsWithoutConsideringProcessed =
+ previousStepStats._totalRemainingSegmentsToBeAdded + numAdditionalSegmentsAdded;
+ int netProcessedSegments = remainingSegmentsWithoutConsideringProcessed
+ - currentStepStats._totalRemainingSegmentsToBeAdded;
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded
+ + numAdditionalSegmentsAdded - netProcessedSegments;
+ } else {
+ // Case II: currentStepStats._totalRemainingSegmentsToBeAdded
+ // > previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 15
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments
+ // processed
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ }
+ }
+ }
Review Comment:
yeah makes sense, simplified it. i was calculating overall segments differently earlier (untracked as part of total) and this Math.abs() is some artifact from code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029456703
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
Also I just noticed `totalNewSegmentsNotMonitored` is always 0 here? Because you passed `null` as `segmentsToMonitored` in rebalance context while triggering the `IDEAL_STATE_CHANGE_TRIGGER`
https://github.com/apache/pinot/blob/1979601f315a945f07578c8a4aeaf714a5bb2f9d/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java#L564-L567
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2033692672
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,255 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
Review Comment:
typo? "... from the last next ..."
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021937447
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
not sure I understood your comment, what kind of errors do you mean? you mean if the stats calculation is wrong? or some exception is thrown?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021934047
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -567,6 +577,15 @@ private RebalanceResult doRebalance(TableConfig tableConfig, RebalanceConfig reb
+ "added/removed for each instance: {}", rebalanceJobId, tableNameWithType,
SegmentAssignmentUtils.getNumSegmentsToMovePerInstance(currentAssignment, nextAssignment));
+ // Record change of current ideal state and the next assignment
+ _tableRebalanceObserver.onTrigger(TableRebalanceObserver.Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER,
+ currentAssignment, nextAssignment, rebalanceContext);
+ if (_tableRebalanceObserver.isStopped()) {
Review Comment:
One example would be if someone calls the CANCEL API for rebalance to cancel the rebalance job.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023472442
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
Review Comment:
it should technically not do that. earlier it was possible for scenarios where `bestEffort=true`, but I added handling for that kind of overflow. I am also trying to track newly added segments that aren't tracked by rebalance separately etc. If we do see weird behavior it means there is some bug that needs to be fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021773303
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -18,35 +18,28 @@
*/
package org.apache.pinot.controller.helix.core.rebalance;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
+@JsonIgnoreProperties(ignoreUnknown = true)
Review Comment:
Is it safe to ignore Unknown? Should this be an error ?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -632,15 +651,15 @@ private RebalanceSummaryResult calculateDryRunSummary(Map<String, Map<String, St
for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
existingReplicationFactor = entrySet.getValue().size();
- for (String segmentKey : entrySet.getValue().keySet()) {
- existingServersToSegmentMap.computeIfAbsent(segmentKey, k -> new HashSet<>()).add(entrySet.getKey());
+ for (String instanceName : entrySet.getValue().keySet()) {
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(entrySet.getKey());
}
}
for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
newReplicationFactor = entrySet.getValue().size();
- for (String segmentKey : entrySet.getValue().keySet()) {
- newServersToSegmentMap.computeIfAbsent(segmentKey, k -> new HashSet<>()).add(entrySet.getKey());
+ for (String instanceName : entrySet.getValue().keySet()) {
Review Comment:
Why change the variable name?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
What is the impact of errors in trigger on stats calculation?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
Review Comment:
Can the stats regressed? If yes, is there a way to detetc?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -987,8 +1010,9 @@ private Pair<InstancePartitions, Boolean> getInstancePartitionsForTier(TableConf
}
}
- private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean lowDiskMode, boolean bestEfforts,
- Set<String> segmentsToMonitor, long externalViewCheckIntervalInMs, long externalViewStabilizationTimeoutInMs)
+ private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean bestEfforts,
Review Comment:
Will the changes cause any issue in rolling upgrade?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,13 +129,107 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ public static boolean progressStatsDiffer(RebalanceProgressStats base, RebalanceProgressStats compare) {
+ // Don't check for changes in the estimated time for completion as this will always be updated since it is
+ // newly calculated for each iteration based on current time and start time
+ return base._totalSegmentsToBeAdded != compare._totalSegmentsToBeAdded
+ || base._totalSegmentsToBeDeleted != compare._totalSegmentsToBeDeleted
+ || base._totalRemainingSegmentsToBeAdded != compare._totalRemainingSegmentsToBeAdded
+ || base._totalRemainingSegmentsToBeDeleted != compare._totalRemainingSegmentsToBeDeleted
+ || base._totalRemainingSegmentsToConverge != compare._totalRemainingSegmentsToConverge
+ || base._totalUniqueNewUntrackedSegmentsDuringRebalance
+ != compare._totalUniqueNewUntrackedSegmentsDuringRebalance
+ || base._percentageTotalSegmentsAddsRemaining != compare._percentageTotalSegmentsAddsRemaining
+ || base._percentageTotalSegmentDeletesRemaining != compare._percentageTotalSegmentDeletesRemaining
+ || base._averageSegmentSizeInBytes != compare._averageSegmentSizeInBytes
+ || base._totalEstimatedDataToBeMovedInBytes != compare._totalEstimatedDataToBeMovedInBytes
Review Comment:
Can estimated Data to be moved vary as well similar to estimated time?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -567,6 +577,15 @@ private RebalanceResult doRebalance(TableConfig tableConfig, RebalanceConfig reb
+ "added/removed for each instance: {}", rebalanceJobId, tableNameWithType,
SegmentAssignmentUtils.getNumSegmentsToMovePerInstance(currentAssignment, nextAssignment));
+ // Record change of current ideal state and the next assignment
+ _tableRebalanceObserver.onTrigger(TableRebalanceObserver.Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER,
+ currentAssignment, nextAssignment, rebalanceContext);
+ if (_tableRebalanceObserver.isStopped()) {
Review Comment:
In what scenario, table rebalance would stop before IS update?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on PR #15266:
URL: https://github.com/apache/pinot/pull/15266#issuecomment-2769969724
closing this as I think an interactive rebase messed up the branch
Opened the new PR with the same changes here: https://github.com/apache/pinot/pull/15427
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #15266:
URL: https://github.com/apache/pinot/pull/15266#issuecomment-2722572008
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/15266?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `91.13924%` with `21 lines` in your changes missing coverage. Please review.
> Project coverage is 34.26%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`639540b`)](https://app.codecov.io/gh/apache/pinot/commit/639540ba45eaab749995819beabfedc9041f2734?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 1843 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/pinot/pull/15266?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [.../core/rebalance/ZkBasedTableRebalanceObserver.java](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Frebalance%2FZkBasedTableRebalanceObserver.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYmFsYW5jZS9aa0Jhc2VkVGFibGVSZWJhbGFuY2VPYnNlcnZlci5qYXZh) | 90.50% | [5 Missing and 12 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ix/core/rebalance/TableRebalanceProgressStats.java](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Frebalance%2FTableRebalanceProgressStats.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYmFsYW5jZS9UYWJsZVJlYmFsYW5jZVByb2dyZXNzU3RhdHMuamF2YQ==) | 92.85% | [0 Missing and 2 partials :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...ntroller/helix/core/rebalance/TableRebalancer.java](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&filepath=pinot-controller%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fcontroller%2Fhelix%2Fcore%2Frebalance%2FTableRebalancer.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYmFsYW5jZS9UYWJsZVJlYmFsYW5jZXIuamF2YQ==) | 90.47% | [1 Missing and 1 partial :warning: ](https://app.codecov.io/gh/apache/pinot/pull/15266?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (639540b). Click for more details.
>
> <details><summary>HEAD has 53 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (639540b) |
>|------|------|------|
>|integration|7|0|
>|integration2|3|0|
>|temurin|12|1|
>|java-21|7|1|
>|skip-bytebuffers-true|3|1|
>|skip-bytebuffers-false|7|0|
>|unittests|5|1|
>|unittests1|2|0|
>|java-11|5|0|
>|unittests2|3|1|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #15266 +/- ##
=============================================
- Coverage 61.75% 34.26% -27.49%
- Complexity 207 682 +475
=============================================
Files 2436 2774 +338
Lines 133233 156460 +23227
Branches 20636 24015 +3379
=============================================
- Hits 82274 53610 -28664
- Misses 44911 98556 +53645
+ Partials 6048 4294 -1754
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.26% <91.13%> (-27.37%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.26% <91.13%> (+6.53%)` | :arrow_up: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.26% <91.13%> (-27.49%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.26% <91.13%> (-27.49%)` | :arrow_down: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/15266/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `34.26% <91.13%> (+6.53%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/15266?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary>🚀 New features to boost your workflow: </summary>
- ❄ [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- 📦 [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029525615
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on PR #15266:
URL: https://github.com/apache/pinot/pull/15266#issuecomment-2779453530
Just capturing a scenario here that I have been debugging lately:
Scenario:
- When we try to update the IdealState in the rebalance code, if the node version changed an exception is thrown and we try to redo the whole rebalance loop
- The current code updates the stats prior to the IdealState update (ideal state trigger and next assignment trigger)
- If we are unable to update the IdealState we should rollback the stats since they will be recalculated on the next iteration anyways.
- The older stats are also susceptible to this problem
Impact of not rolling back: the stats can show more progress has been made than the reality for that iteration until the IS update happens. The IS update will reset it correctly, but until then the stats will be incorrect.
Also, the above problem is more problematic if bestEfforts=true because we may not wait for EV-IS full convergence before moving to the next step. Due to this, when we loop back after IS update fails, we get into the EV-IS convergence loop again and it looks like we are making progress, but we’re actually making progress on the last step updates that hadn't completed.
I’m thinking of adding the above rollback logic in a follow up PR (my plan is to just cache the previous versions of the stats and use that for rollback) to avoid over-complicating this one. Let me know if that is fine with you folks @klsince @raghavyadav01
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029469316
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
yes for IS trigger that is intentional because we've already added the newly added segments during the EV-IS convergence step. If we re-calculate it here then it will keep adding the same segments and the stats won't be correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2013049985
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -719,23 +741,24 @@ private RebalanceSummaryResult calculateDryRunSummary(Map<String, Map<String, St
RebalanceSummaryResult.RebalanceChangeInfo numSegmentsAcrossAllReplicas
= new RebalanceSummaryResult.RebalanceChangeInfo(existingNumberSegmentsTotal, newNumberSegmentsTotal);
- int totalSegmentsToBeMoved = newNumberSegmentsTotal - segmentsNotMoved;
+ int totalSegmentsToBeAddedd = newNumberSegmentsTotal - segmentsNotMoved;
Review Comment:
Is `totalSegmentsToBeAddedd` a typo?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029595112
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
I see. Should we just remove `+ totalNewSegmentsNotMonitored` then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2027561893
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +128,116 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param latestStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats lastOverallStats = getRebalanceProgressStatsOverall();
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int upperBoundOnSegmentsAdded =
Review Comment:
I feel this method is a bit hard to follow, perhaps due to the var names are kinda similar `last..` vs. `latest... `
could you help add some comments to split the method into a few major sections, which might help read it.
btw, why we need to use `upperBoundOnSegmentsAdded`?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -402,9 +402,17 @@ private RebalanceResult doRebalance(TableConfig tableConfig, RebalanceConfig reb
}
}
+ List<String> segmentsToMove = SegmentAssignmentUtils.getSegmentsToMove(currentAssignment, targetAssignment);
+ Set<String> segmentsToMonitor = new HashSet<>(segmentsToMove);
+
+ long estimatedAverageSegmentSizeInBytes = summaryResult.getSegmentInfo().getEstimatedAverageSegmentSizeInBytes();
+ Set<String> allOriginalSegmentsIdealState = currentAssignment.keySet();
Review Comment:
nit: allOriginalSegmentsIdealState -> allSegmentsFromIdealState or is there a special reason to call it out 'original'?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
Review Comment:
hmm. sorry, i'm a bit confused on what this if-check tries to do? I see it didn't use instanceName/State so why it's inside the for-loop and why segmentsToMonitor is checked here 🤔 ? pls shed some light, or perhaps add a bit more comments for this bit method to help understand it. thanks!
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget += 1;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
Review Comment:
with the two checks on `trigger == Trigger.IDEAL_STATE_CHANGE_TRIGGER` and `trigger == Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER` , perhaps just split them into two cases
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget += 1;
Review Comment:
nit: ++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029695528
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +126,209 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ if (numAdditionalSegmentsAdded == 0) {
+ // No new additional segments added, subtract the number of processed segments from the last step
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep;
+ } else {
+ if (currentStepStats._totalRemainingSegmentsToBeAdded <= previousStepStats._totalRemainingSegmentsToBeAdded) {
+ // Case I: currentStepStats._totalRemainingSegmentsToBeAdded
+ // <= previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 5
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 5
+ // remainingSegmentsWithoutConsideringProcessed = 5 + 10 = 15
+ // netProcessedSegments = 15 - 5 = 10 -> how much to subtract after adding additional segments to remaining
+ int remainingSegmentsWithoutConsideringProcessed =
+ previousStepStats._totalRemainingSegmentsToBeAdded + numAdditionalSegmentsAdded;
+ int netProcessedSegments = remainingSegmentsWithoutConsideringProcessed
+ - currentStepStats._totalRemainingSegmentsToBeAdded;
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded
+ + numAdditionalSegmentsAdded - netProcessedSegments;
+ } else {
+ // Case II: currentStepStats._totalRemainingSegmentsToBeAdded
+ // > previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 15
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments
+ // processed
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ }
+ }
+ }
Review Comment:
yeah makes sense, simplified it. i was calculating overall segments differently earlier (untracked as part of total) and I think this Math.abs() is some artifact from code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023148850
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
i have added a fix for the `bestEfforts=true` scenario and did some testing. I added two new fields to track the carry-over adds and deletes from the previous step. I include the carry-overs in the percentage calculations since they are technically un-converged.
Please let me know your thoughts on this approach
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2027412227
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -19,33 +19,25 @@
package org.apache.pinot.controller.helix.core.rebalance;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import java.util.Objects;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
Review Comment:
Should this be updated? They're old variables
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -19,33 +19,25 @@
package org.apache.pinot.controller.helix.core.rebalance;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import java.util.Objects;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
Review Comment:
Should this be updated? They're old variables
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2013074065
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -719,23 +741,24 @@ private RebalanceSummaryResult calculateDryRunSummary(Map<String, Map<String, St
RebalanceSummaryResult.RebalanceChangeInfo numSegmentsAcrossAllReplicas
= new RebalanceSummaryResult.RebalanceChangeInfo(existingNumberSegmentsTotal, newNumberSegmentsTotal);
- int totalSegmentsToBeMoved = newNumberSegmentsTotal - segmentsNotMoved;
+ int totalSegmentsToBeAddedd = newNumberSegmentsTotal - segmentsNotMoved;
Review Comment:
oops, good catch, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021987508
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -987,8 +1010,9 @@ private Pair<InstancePartitions, Boolean> getInstancePartitionsForTier(TableConf
}
}
- private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean lowDiskMode, boolean bestEfforts,
- Set<String> segmentsToMonitor, long externalViewCheckIntervalInMs, long externalViewStabilizationTimeoutInMs)
+ private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean bestEfforts,
Review Comment:
No, I was thinking the same and I wanted to confirm. May be we can add some logs in case we run into extra convergence check cycles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029695644
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +126,209 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
Review Comment:
done, artifact from older code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029524674
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -19,33 +19,25 @@
package org.apache.pinot.controller.helix.core.rebalance;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import java.util.Objects;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
Review Comment:
done
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +128,165 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ // If additional segments were added in the previous step, these need to be added to the remaining to be added
+ // number as they are new segments that should be tracked for completion
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments processed
+ // If 'numAdditionalSegmentsAdded' = 0, that means no new segments were added in the previous step, and if there
+ // was a change in the segments processed in the last step, they completed processing
+ currentOverallStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
Review Comment:
updated, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021936600
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -987,8 +1010,9 @@ private Pair<InstancePartitions, Boolean> getInstancePartitionsForTier(TableConf
}
}
- private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean lowDiskMode, boolean bestEfforts,
- Set<String> segmentsToMonitor, long externalViewCheckIntervalInMs, long externalViewStabilizationTimeoutInMs)
+ private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean bestEfforts,
Review Comment:
I don't think so. it just forces the convergence check to ensure that we wait for the instances that exist in EV but not in IS to actually be deleted from EV as well. This can cause extra convergence check cycles to enforce this has happened rather than early return.
did you have a specific rolling upgrade scenario in mind?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal closed pull request #15266: Enhance rebalance progress stats
URL: https://github.com/apache/pinot/pull/15266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2025535193
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +328,306 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
Review Comment:
one
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +328,306 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2025372338
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +127,33 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ public static boolean progressStatsDiffer(RebalanceProgressStats base, RebalanceProgressStats compare) {
Review Comment:
add and use the equals() method for the class?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceObserver.java:
##########
@@ -49,4 +52,29 @@ void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
boolean isStopped();
RebalanceResult.Status getStopStatus();
+
+ class RebalanceContext {
+ private final long _estimatedAverageSegmentSizeInBytes;
+ private final Set<String> _uniqueSegmentList;
Review Comment:
nit: remove List suffix?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +328,306 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int upperBoundOnSegmentsAdded =
+ lastStepStats._totalRemainingSegmentsToBeAdded > lastStepStats._totalSegmentsToBeAdded
+ ? lastStepStats._totalSegmentsToBeAdded : lastStepStats._totalRemainingSegmentsToBeAdded;
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int upperBoundOnSegmentsDeleted =
+ lastStepStats._totalRemainingSegmentsToBeDeleted > lastStepStats._totalSegmentsToBeDeleted
+ ? lastStepStats._totalSegmentsToBeDeleted : lastStepStats._totalRemainingSegmentsToBeDeleted;
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ if (latestStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = overallProgressStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ }
+ newOverallProgressStats._totalCarryOverSegmentsToBeAdded =
+ latestStepStats._totalCarryOverSegmentsToBeAdded;
+ if (latestStepStats._totalCarryOverSegmentsToBeDeleted > 0) {
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted =
+ overallProgressStats._totalRemainingSegmentsToBeDeleted;
+ } else {
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeDeleted + numSegmentDeletesProcessedInLastStep;
+ }
+ newOverallProgressStats._totalCarryOverSegmentsToBeDeleted =
+ latestStepStats._totalCarryOverSegmentsToBeDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToConverge = latestStepStats._totalRemainingSegmentsToConverge;
+ newOverallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ overallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + numberNewUntrackedSegmentsAdded;
+ newOverallProgressStats._percentageTotalSegmentsAddsRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeAdded,
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded
+ + newOverallProgressStats._totalCarryOverSegmentsToBeAdded);
+ newOverallProgressStats._percentageTotalSegmentDeletesRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeDeleted,
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted
+ + newOverallProgressStats._totalCarryOverSegmentsToBeDeleted);
+ // Calculate elapsed time based on start of rebalance (global)
+ newOverallProgressStats._estimatedTimeToCompleteAddsInSeconds =
+ calculateEstimatedTimeToCompleteChange(rebalanceProgressStats.getStartTimeMs(),
+ newOverallProgressStats._totalSegmentsToBeAdded, newOverallProgressStats._totalRemainingSegmentsToBeAdded);
+ newOverallProgressStats._estimatedTimeToCompleteDeletesInSeconds =
+ calculateEstimatedTimeToCompleteChange(rebalanceProgressStats.getStartTimeMs(),
+ newOverallProgressStats._totalSegmentsToBeDeleted,
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted);
+ newOverallProgressStats._averageSegmentSizeInBytes = overallProgressStats._averageSegmentSizeInBytes;
+ newOverallProgressStats._totalEstimatedDataToBeMovedInBytes =
+ overallProgressStats._totalEstimatedDataToBeMovedInBytes
+ + (numAdditionalSegmentsAdded * overallProgressStats._averageSegmentSizeInBytes);
+ newOverallProgressStats._startTimeMs = rebalanceProgressStats.getStartTimeMs();
+
+ return newOverallProgressStats;
+ }
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateOverallProgressStats(
Review Comment:
I didn't figure out why need to pass in a `rebalanceProgressStats` to this calculate method?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -1145,8 +1174,8 @@ static boolean isExternalViewConverged(String tableNameWithType,
}
}
- // For low disk mode, check if there are extra instances in ExternalView that are not in IdealState
- if (lowDiskMode && externalViewInstanceStateMap != null) {
+ // Check if there are extra instances in ExternalView that are not in IdealState
+ if (externalViewInstanceStateMap != null) {
Review Comment:
maybe open a separate small PR for this fix for some closer reviews (and perhaps also include those simple typo fixes like segmentKey -> instanceName)
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -127,4 +163,77 @@ public static boolean statsDiffer(RebalanceStateStats base, RebalanceStateStats
}
return false;
}
+
+ // TODO: Clean this up once new stats are verified
+ public static class RebalanceStateStats {
+ public int _segmentsMissing;
+ public int _segmentsToRebalance;
+ public double _percentSegmentsToRebalance;
+ public int _replicasToRebalance;
+
+ RebalanceStateStats() {
+ _segmentsMissing = 0;
+ _segmentsToRebalance = 0;
+ _replicasToRebalance = 0;
+ _percentSegmentsToRebalance = 0.0;
+ }
+ }
+
+ // These rebalance stats specifically track the total segments added / deleted across all replicas
+ public static class RebalanceProgressStats {
+ // Total segments - across all replicas
+ @JsonProperty("totalSegmentsToBeAdded")
+ public int _totalSegmentsToBeAdded;
+ @JsonProperty("totalSegmentsToBeDeleted")
+ public int _totalSegmentsToBeDeleted;
+ // Total segments processed so far - across all replicas
+ @JsonProperty("totalRemainingSegmentsToBeAdded")
+ public int _totalRemainingSegmentsToBeAdded;
+ @JsonProperty("totalRemainingSegmentsToBeDeleted")
+ public int _totalRemainingSegmentsToBeDeleted;
+ @JsonProperty("totalRemainingSegmentsToConverge")
+ public int _totalRemainingSegmentsToConverge;
+ // Carry over stats - for when previous step's convergence doesn't complete and next step starts (bestEffort=true)
+ @JsonProperty("totalCarryOverSegmentsToBeAdded")
+ public int _totalCarryOverSegmentsToBeAdded;
+ @JsonProperty("totalCarryOverSegmentsToBeDeleted")
+ public int _totalCarryOverSegmentsToBeDeleted;
+ // Total new segments stats (not tracked by rebalance)
+ @JsonProperty("totalUniqueNewUntrackedSegmentsDuringRebalance")
+ public int _totalUniqueNewUntrackedSegmentsDuringRebalance;
+ // Derived stats
+ @JsonProperty("percentageTotalSegmentsAddsRemaining")
+ public double _percentageTotalSegmentsAddsRemaining;
Review Comment:
nit: _percentageTotalSegment`s`DeletesRemaining
maybe refine the two var names a bit, e.g. `percentRemainingSegmentsToBeAdded` to look closer with `totalRemainingSegmentsToBeAdded`
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +328,306 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
Review Comment:
make this a method of TableRebalanceProgressStats, so we can do `overall.update(last, latest)` iiuc
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021943506
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,13 +129,107 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ public static boolean progressStatsDiffer(RebalanceProgressStats base, RebalanceProgressStats compare) {
+ // Don't check for changes in the estimated time for completion as this will always be updated since it is
+ // newly calculated for each iteration based on current time and start time
+ return base._totalSegmentsToBeAdded != compare._totalSegmentsToBeAdded
+ || base._totalSegmentsToBeDeleted != compare._totalSegmentsToBeDeleted
+ || base._totalRemainingSegmentsToBeAdded != compare._totalRemainingSegmentsToBeAdded
+ || base._totalRemainingSegmentsToBeDeleted != compare._totalRemainingSegmentsToBeDeleted
+ || base._totalRemainingSegmentsToConverge != compare._totalRemainingSegmentsToConverge
+ || base._totalUniqueNewUntrackedSegmentsDuringRebalance
+ != compare._totalUniqueNewUntrackedSegmentsDuringRebalance
+ || base._percentageTotalSegmentsAddsRemaining != compare._percentageTotalSegmentsAddsRemaining
+ || base._percentageTotalSegmentDeletesRemaining != compare._percentageTotalSegmentDeletesRemaining
+ || base._averageSegmentSizeInBytes != compare._averageSegmentSizeInBytes
+ || base._totalEstimatedDataToBeMovedInBytes != compare._totalEstimatedDataToBeMovedInBytes
Review Comment:
no, this is calculated based on average segment size and number of segments moving. so if `_totalSegmentsToBeAdded` is updated, only then this should change.
We calculate average segment size prior to table rebalance start and use the same throughout (it's only an estimate)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021942607
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -18,35 +18,28 @@
*/
package org.apache.pinot.controller.helix.core.rebalance;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
+@JsonIgnoreProperties(ignoreUnknown = true)
Review Comment:
I was more thinking from upgrade perspective, since I'm renaming a bunch of the existing stats. but eventual goal is to actually remove the existing stats as well 😅
I can keep this, just that if we upgrade during ongoing rebalance, there could be errors in reading this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2033707619
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,255 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
Review Comment:
oops, fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029524839
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +128,165 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ // If additional segments were added in the previous step, these need to be added to the remaining to be added
+ // number as they are new segments that should be tracked for completion
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments processed
+ // If 'numAdditionalSegmentsAdded' = 0, that means no new segments were added in the previous step, and if there
+ // was a change in the segments processed in the last step, they completed processing
+ currentOverallStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
Review Comment:
good catch, updated, please take a look
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029454774
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
I was thinking about whether a segment could be double counted, or the case when a segment is added as an untracked segment but gets removed from the IS in the next step.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029592306
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +126,209 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ if (numAdditionalSegmentsAdded == 0) {
+ // No new additional segments added, subtract the number of processed segments from the last step
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep;
+ } else {
+ if (currentStepStats._totalRemainingSegmentsToBeAdded <= previousStepStats._totalRemainingSegmentsToBeAdded) {
+ // Case I: currentStepStats._totalRemainingSegmentsToBeAdded
+ // <= previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 5
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 5
+ // remainingSegmentsWithoutConsideringProcessed = 5 + 10 = 15
+ // netProcessedSegments = 15 - 5 = 10 -> how much to subtract after adding additional segments to remaining
+ int remainingSegmentsWithoutConsideringProcessed =
+ previousStepStats._totalRemainingSegmentsToBeAdded + numAdditionalSegmentsAdded;
+ int netProcessedSegments = remainingSegmentsWithoutConsideringProcessed
+ - currentStepStats._totalRemainingSegmentsToBeAdded;
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded
+ + numAdditionalSegmentsAdded - netProcessedSegments;
+ } else {
+ // Case II: currentStepStats._totalRemainingSegmentsToBeAdded
+ // > previousStepStats._totalRemainingSegmentsToBeAdded
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // currentStepStats: totalRemainingSegmentsToBeAdded = 15
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments
+ // processed
+ currentOverallStats._totalRemainingSegmentsToBeAdded =
+ previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ }
+ }
+ }
Review Comment:
I think this altogether equivalents to
```java
currentOverallStats._totalRemainingSegmentsToBeAdded =
previousOverallStats._totalRemainingSegmentsToBeAdded -
(upperBoundOnSegmentsAdded - currentStepStats._totalRemainingSegmentsToBeAdded);
```
or
```java
// Math.abs was removed here
int numSegmentAddsProcessedInLastStep = upperBoundOnSegmentsAdded - currentStepStats._totalRemainingSegmentsToBeAdded;
currentOverallStats._totalRemainingSegmentsToBeAdded =
previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep;
```
Please check if this makes sense, might be wrong here.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +126,209 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
Review Comment:
Is it going to happen? I think you subtract the carry over from the `remainingSegmentsToBeAdded` here?
https://github.com/apache/pinot/blob/1af5e8a832b5ef8f3dc9e8ba07af839a93ee59c0/pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java#L530-L534
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029417170
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
`segmentsToMonitor` is not a full list of all segments that were monitored by the TableRebalancer from start to end. It only captures the list of segments from the current step and the last step
What's error prone here, can you elaborate on that?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "J-HowHuang (via GitHub)" <gi...@apache.org>.
J-HowHuang commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2027413419
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -19,33 +19,25 @@
package org.apache.pinot.controller.helix.core.rebalance;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
+import java.util.Objects;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
Review Comment:
Should this be moved down to `RebalanceStateStats`?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +128,165 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ /**
+ * Updates the overall and step progress stats based on the latest calculated step's progress stats. This should
+ * be called during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param currentStepStats latest step level stats calculated in this iteration
+ */
+ public void updateOverallAndStepStatsFromLatestStepStats(
+ TableRebalanceProgressStats.RebalanceProgressStats currentStepStats) {
+ // Fetch the step level and overall stats that were calculated in the last convergence check. These will be used
+ // to calculate the overall stats in the current convergence check
+ TableRebalanceProgressStats.RebalanceProgressStats previousStepStats = getRebalanceProgressStatsCurrentStep();
+ TableRebalanceProgressStats.RebalanceProgressStats previousOverallStats = getRebalanceProgressStatsOverall();
+
+ // Calculate the new segments added / deleted since the last step to track overall change in segments. These are
+ // only new segments tracked by table rebalance (segments that aren't tracked by rebalance convergence check are
+ // ignored because the convergence check does not wait for them to complete)
+ int numAdditionalSegmentsAdded =
+ currentStepStats._totalSegmentsToBeAdded - previousStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ currentStepStats._totalSegmentsToBeDeleted - previousStepStats._totalSegmentsToBeDeleted;
+
+ // Calculate the carry-over segment adds / deletes from the previous rebalance step if any (can happen if
+ // bestEfforts=true or if there are deletions left over from the last step) and the difference in the number of
+ // segments processed since the previous stats update.
+ // Example of what carry-over will look like:
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // previousStepStats: totalRemainingSegmentsToBeAdded = 15 (this can only be larger than totalSegmentsToBeAdded
+ // if carry-over segments exist)
+ // Carry-over: 15 - 10 = 5 segments -> which were added in the last IS update by rebalance, but which didn't
+ // finish getting added before making the next assignment IS update
+ // Using the above, we calculate the 'upperBoundOnSegmentsAdded/Deleted' as the maximum segments to use when
+ // calculating the number of segments processed (number of segments that got added / deleted in EV based on expected
+ // IS) since the last step. We want to skip including the carry-over segments if any as they are tracked separately.
+ int upperBoundOnSegmentsAdded =
+ previousStepStats._totalRemainingSegmentsToBeAdded > previousStepStats._totalSegmentsToBeAdded
+ ? previousStepStats._totalSegmentsToBeAdded : previousStepStats._totalRemainingSegmentsToBeAdded;
+ int upperBoundOnSegmentsDeleted =
+ previousStepStats._totalRemainingSegmentsToBeDeleted > previousStepStats._totalSegmentsToBeDeleted
+ ? previousStepStats._totalSegmentsToBeDeleted : previousStepStats._totalRemainingSegmentsToBeDeleted;
+
+ // Number of adds / deletes processed in the last step. Take the absolute here since if new segments were added /
+ // deleted in the current step that are being tracking, we need to treat those as new segments that remain to
+ // be processed.
+ int numSegmentAddsProcessedInLastStep = Math.abs(upperBoundOnSegmentsAdded
+ - currentStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(upperBoundOnSegmentsDeleted
+ - currentStepStats._totalRemainingSegmentsToBeDeleted);
+
+ // Number of untracked segments that were added
+ int numberNewUntrackedSegmentsAdded = currentStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - previousStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats currentOverallStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ // New number of total segment adds / deletes should include any new segments added from the previous stats
+ currentOverallStats._totalSegmentsToBeAdded = previousOverallStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ currentOverallStats._totalSegmentsToBeDeleted = previousOverallStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+
+ // If carry-over segments are present, keep the upper bound of remaining segments to be added / deleted at the same
+ // level as the previous stats. This is so we track progress to handle the carry-over segments before we start
+ // tracking progress against the segments that need to be handled as part of the current rebalance step.
+ if (currentStepStats._totalCarryOverSegmentsToBeAdded > 0) {
+ currentOverallStats._totalRemainingSegmentsToBeAdded = previousOverallStats._totalRemainingSegmentsToBeAdded;
+ } else {
+ // If additional segments were added in the previous step, these need to be added to the remaining to be added
+ // number as they are new segments that should be tracked for completion
+ // Example:
+ // currentStepStats: totalSegmentsToBeAdded = 20
+ // previousStepStats: totalSegmentsToBeAdded = 10
+ // numAdditionalSegmentsAdded = 20 - 10 = 10
+ // In the above scenario, the new 'numAdditionalSegmentsAdded' also need to be tracked as
+ // 'totalRemainingSegmentsToBeAdded' for the current step, thus we need to add the number of segments processed
+ // If 'numAdditionalSegmentsAdded' = 0, that means no new segments were added in the previous step, and if there
+ // was a change in the segments processed in the last step, they completed processing
+ currentOverallStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? previousOverallStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : previousOverallStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
Review Comment:
What if the previous step has
```
{
totalSegmentsToBeAdded: 10,
totalRemainingSegmentsToBeAdded: 10
}
```
and current has
```
{
totalSegmentsToBeAdded: 15,
totalRemainingSegmentsToBeAdded: 5
}
```
Does the new overall stats get 5 more segments (`abs(upperBoundOnSegmentsAdded
- currentStepStats._totalRemainingSegmentsToBeAdded) == 5`) in `_totalRemainingSegmentsToBeAdded` than the previous overall stats? I was a bit confused because I thought we should have +5 new segment to add and -10 segments this step added so a net -5 remaining segment to added right?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
Review Comment:
Shouldn't this part be placed one scope out? Right now it skips a replica instead of the whole segment
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
Though I think it's correct accumulate this number over each IS update in rebalancer.
But we can also calculate the cardinality diff between `rebalanceContext._uniqueSegments` and `rebalanceContext._segmentsToMonitor` here? I think this way is less error prone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #15266:
URL: https://github.com/apache/pinot/pull/15266
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021940809
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -271,17 +315,279 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
String instanceName = instanceStateEntry.getKey();
String sourceInstanceState = sourceInstanceStateMap.get(instanceName);
if (!targetStateInstanceState.equals(sourceInstanceState)) {
- rebalanceStats._replicasToRebalance++;
+ rebalanceStats._totalSegmentsToRebalance++;
hasSegmentConverged = false;
}
}
if (!hasSegmentConverged) {
- rebalanceStats._segmentsToRebalance++;
+ rebalanceStats._uniqueSegmentsToRebalance++;
}
}
int totalSegments = targetState.size();
- rebalanceStats._percentSegmentsToRebalance =
- (totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
+ rebalanceStats._percentRemainingSegmentsToRebalance =
+ (totalSegments == 0) ? 0 : ((double) rebalanceStats._uniqueSegmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int numSegmentAddsProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentDeletesProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToConverge = latestStepStats._totalRemainingSegmentsToConverge;
+ newOverallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ overallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + numberNewUntrackedSegmentsAdded;
+ newOverallProgressStats._percentageTotalSegmentsAddsRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeAdded,
Review Comment:
I think it is correct:
```
private static double calculatePercentageChange(int totalSegmentsToChange, int remainingSegmentsToChange) {
return totalSegmentsToChange == 0
? 0.0 : (double) remainingSegmentsToChange / (double) totalSegmentsToChange * 100.0;
}
```
`_totalSegmentsToBeAdded` -> total segments to add
`_totalRemainingSegmentsToBeAdded` -> remaining segments to add
e.g.
Say _totalSegmentsToBeAdded = 100, and _totalRemainingSegmentsToBeAdded = 75, I want to do: (75.0 / 100.0 * 100.0 = 75%)
Or did you mean the parameters in the function should be swapped?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021942607
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -18,35 +18,28 @@
*/
package org.apache.pinot.controller.helix.core.rebalance;
+import com.fasterxml.jackson.annotation.JsonIgnoreProperties;
import com.fasterxml.jackson.annotation.JsonProperty;
+import com.fasterxml.jackson.annotation.JsonPropertyOrder;
/**
* These are rebalance stats as to how the current state is, when compared to the target state.
* Eg: If the current has 4 segments whose replicas (16) don't match the target state, _segmentsToRebalance
* is 4 and _replicasToRebalance is 16.
*/
+@JsonIgnoreProperties(ignoreUnknown = true)
Review Comment:
I was more thinking from upgrade perspective, since I'm renaming a bunch of the existing stats. but eventual goal is to actually remove the existing stats as well 😅
I can remove this, just that if we upgrade during ongoing rebalance, there could be errors in reading this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2025535644
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -1145,8 +1174,8 @@ static boolean isExternalViewConverged(String tableNameWithType,
}
}
- // For low disk mode, check if there are extra instances in ExternalView that are not in IdealState
- if (lowDiskMode && externalViewInstanceStateMap != null) {
+ // Check if there are extra instances in ExternalView that are not in IdealState
+ if (externalViewInstanceStateMap != null) {
Review Comment:
done - removed this. will open a separate PR later
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -127,4 +163,77 @@ public static boolean statsDiffer(RebalanceStateStats base, RebalanceStateStats
}
return false;
}
+
+ // TODO: Clean this up once new stats are verified
+ public static class RebalanceStateStats {
+ public int _segmentsMissing;
+ public int _segmentsToRebalance;
+ public double _percentSegmentsToRebalance;
+ public int _replicasToRebalance;
+
+ RebalanceStateStats() {
+ _segmentsMissing = 0;
+ _segmentsToRebalance = 0;
+ _replicasToRebalance = 0;
+ _percentSegmentsToRebalance = 0.0;
+ }
+ }
+
+ // These rebalance stats specifically track the total segments added / deleted across all replicas
+ public static class RebalanceProgressStats {
+ // Total segments - across all replicas
+ @JsonProperty("totalSegmentsToBeAdded")
+ public int _totalSegmentsToBeAdded;
+ @JsonProperty("totalSegmentsToBeDeleted")
+ public int _totalSegmentsToBeDeleted;
+ // Total segments processed so far - across all replicas
+ @JsonProperty("totalRemainingSegmentsToBeAdded")
+ public int _totalRemainingSegmentsToBeAdded;
+ @JsonProperty("totalRemainingSegmentsToBeDeleted")
+ public int _totalRemainingSegmentsToBeDeleted;
+ @JsonProperty("totalRemainingSegmentsToConverge")
+ public int _totalRemainingSegmentsToConverge;
+ // Carry over stats - for when previous step's convergence doesn't complete and next step starts (bestEffort=true)
+ @JsonProperty("totalCarryOverSegmentsToBeAdded")
+ public int _totalCarryOverSegmentsToBeAdded;
+ @JsonProperty("totalCarryOverSegmentsToBeDeleted")
+ public int _totalCarryOverSegmentsToBeDeleted;
+ // Total new segments stats (not tracked by rebalance)
+ @JsonProperty("totalUniqueNewUntrackedSegmentsDuringRebalance")
+ public int _totalUniqueNewUntrackedSegmentsDuringRebalance;
+ // Derived stats
+ @JsonProperty("percentageTotalSegmentsAddsRemaining")
+ public double _percentageTotalSegmentsAddsRemaining;
Review Comment:
done
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceProgressStats.java:
##########
@@ -118,6 +127,33 @@ public RebalanceStateStats getCurrentToTargetConvergence() {
return _currentToTargetConvergence;
}
+ public RebalanceProgressStats getRebalanceProgressStatsOverall() {
+ return _rebalanceProgressStatsOverall;
+ }
+
+ public RebalanceProgressStats getRebalanceProgressStatsCurrentStep() {
+ return _rebalanceProgressStatsCurrentStep;
+ }
+
+ public static boolean progressStatsDiffer(RebalanceProgressStats base, RebalanceProgressStats compare) {
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2025535969
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalanceObserver.java:
##########
@@ -49,4 +52,29 @@ void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
boolean isStopped();
RebalanceResult.Status getStopStatus();
+
+ class RebalanceContext {
+ private final long _estimatedAverageSegmentSizeInBytes;
+ private final Set<String> _uniqueSegmentList;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023466733
##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java:
##########
@@ -1264,116 +1264,95 @@ public void testIsExternalViewConverged() {
boolean[] falseAndTrue = new boolean[]{false, true};
// Empty segment states should match
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Do not check segment that does not exist in IdealState
Map<String, String> instanceStateMap = new TreeMap<>();
instanceStateMap.put("instance1", ONLINE);
externalViewSegmentStates.put("segment1", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Do not check segment that is OFFLINE in IdealState
instanceStateMap = new TreeMap<>();
instanceStateMap.put("instance1", OFFLINE);
idealStateSegmentStates.put("segment2", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when a segment has CONSUMING instance in IdealState but does not exist in ExternalView
instanceStateMap.put("instance2", CONSUMING);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when instance state does not exist
instanceStateMap = new TreeMap<>();
externalViewSegmentStates.put("segment2", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when instance state does not match
instanceStateMap.put("instance2", OFFLINE);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should pass when instance state matches
instanceStateMap.put("instance2", CONSUMING);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
- // When there are extra instances in ExternalView, should pass in regular mode but fail in low disk mode
+ // When there are extra instances in ExternalView, should fail (always wait for extra instances to be removed)
instanceStateMap.put("instance3", CONSUMING);
for (boolean bestEfforts : falseAndTrue) {
- assertTrue(
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- false, bestEfforts, null));
assertFalse(
TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- true, bestEfforts, null));
+ bestEfforts, null));
}
// When instance state is ERROR in ExternalView, should fail in regular mode but pass in best-efforts mode
instanceStateMap.put("instance2", ERROR);
instanceStateMap.remove("instance3");
- for (boolean lowDiskMode : falseAndTrue) {
- try {
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- lowDiskMode, false, null);
- fail();
- } catch (Exception e) {
- // Expected
- }
- assertTrue(
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- lowDiskMode, true, null));
+ try {
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
+ false, null);
+ fail();
+ } catch (Exception e) {
+ // Expected
}
+ assertTrue(
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
+ true, null));
- // When the extra instance is in ERROR state, should throw exception in low disk mode when best-efforts is disabled
+ // When the extra instance is in ERROR state, should throw exception when best-efforts is disabled
instanceStateMap.put("instance2", CONSUMING);
instanceStateMap.put("instance3", ERROR);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- if (lowDiskMode && !bestEfforts) {
- try {
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, true, false, null);
- fail();
- } catch (Exception e) {
- // Expected
- }
- } else {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
+ for (boolean bestEfforts : falseAndTrue) {
+ if (!bestEfforts) {
+ try {
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, false, null);
+ fail();
+ } catch (Exception e) {
+ // Expected
Review Comment:
Will there be a value adding log here? It might flood if it is too aggressive.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
Review Comment:
By regressed I mean can the stats go back or become negative.
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -271,17 +315,279 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
String instanceName = instanceStateEntry.getKey();
String sourceInstanceState = sourceInstanceStateMap.get(instanceName);
if (!targetStateInstanceState.equals(sourceInstanceState)) {
- rebalanceStats._replicasToRebalance++;
+ rebalanceStats._totalSegmentsToRebalance++;
hasSegmentConverged = false;
}
}
if (!hasSegmentConverged) {
- rebalanceStats._segmentsToRebalance++;
+ rebalanceStats._uniqueSegmentsToRebalance++;
}
}
int totalSegments = targetState.size();
- rebalanceStats._percentSegmentsToRebalance =
- (totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
+ rebalanceStats._percentRemainingSegmentsToRebalance =
+ (totalSegments == 0) ? 0 : ((double) rebalanceStats._uniqueSegmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int numSegmentAddsProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentDeletesProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToConverge = latestStepStats._totalRemainingSegmentsToConverge;
+ newOverallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ overallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + numberNewUntrackedSegmentsAdded;
+ newOverallProgressStats._percentageTotalSegmentsAddsRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeAdded,
Review Comment:
Thanks. Looks good.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021986640
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ }
+ if (TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(latestProgress);
+ }
trackStatsInZk();
updatedStatsInZk = true;
}
break;
case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER,
+ _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(
- _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
- _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)
+ || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setExternalViewToIdealStateConvergence(latest);
+ }
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats =
+ _tableRebalanceProgressStats.getRebalanceProgressStatsCurrentStep();
+ if (TableRebalanceProgressStats.progressStatsDiffer(lastStepStats, latestProgress)) {
+ _tableRebalanceProgressStats.setRebalanceProgressStatsOverall(
+ updateOverallProgressStatsFromStep(_tableRebalanceProgressStats, lastStepStats, latestProgress));
+ _tableRebalanceProgressStats.setRebalanceProgressStatsCurrentStep(latestProgress);
+ }
+ trackStatsInZk();
+ updatedStatsInZk = true;
+ }
+ break;
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ latestProgress = calculateOverallProgressStats(targetState,
Review Comment:
I mean any error which is retried during state trigger .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021936909
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -68,31 +70,67 @@ public ZkBasedTableRebalanceObserver(String tableNameWithType, String rebalanceJ
@Override
public void onTrigger(Trigger trigger, Map<String, Map<String, String>> currentState,
- Map<String, Map<String, String>> targetState) {
+ Map<String, Map<String, String>> targetState, RebalanceContext rebalanceContext) {
boolean updatedStatsInZk = false;
_controllerMetrics.setValueOfTableGauge(_tableNameWithType, ControllerGauge.TABLE_REBALANCE_IN_PROGRESS, 1);
+ TableRebalanceProgressStats.RebalanceStateStats latest;
+ TableRebalanceProgressStats.RebalanceProgressStats latestProgress;
switch (trigger) {
case START_TRIGGER:
- updateOnStart(currentState, targetState);
+ updateOnStart(currentState, targetState, rebalanceContext);
trackStatsInZk();
updatedStatsInZk = true;
break;
// Write to Zk if there's change since previous stats computation
case IDEAL_STATE_CHANGE_TRIGGER:
- TableRebalanceProgressStats.RebalanceStateStats latest =
- getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latest = getDifferenceBetweenTableRebalanceStates(targetState, currentState);
+ latestProgress = calculateOverallProgressStats(targetState,
+ currentState, rebalanceContext, Trigger.IDEAL_STATE_CHANGE_TRIGGER, _tableRebalanceProgressStats);
if (TableRebalanceProgressStats.statsDiffer(_tableRebalanceProgressStats.getCurrentToTargetConvergence(),
- latest)) {
- _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
+ latest) || TableRebalanceProgressStats.progressStatsDiffer(
+ _tableRebalanceProgressStats.getRebalanceProgressStatsOverall(), latestProgress)) {
+ if (TableRebalanceProgressStats.statsDiffer(
+ _tableRebalanceProgressStats.getExternalViewToIdealStateConvergence(), latest)) {
+ _tableRebalanceProgressStats.setCurrentToTargetConvergence(latest);
Review Comment:
what do you mean by regressed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021935742
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -632,15 +651,15 @@ private RebalanceSummaryResult calculateDryRunSummary(Map<String, Map<String, St
for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
existingReplicationFactor = entrySet.getValue().size();
- for (String segmentKey : entrySet.getValue().keySet()) {
- existingServersToSegmentMap.computeIfAbsent(segmentKey, k -> new HashSet<>()).add(entrySet.getKey());
+ for (String instanceName : entrySet.getValue().keySet()) {
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(entrySet.getKey());
}
}
for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
newReplicationFactor = entrySet.getValue().size();
- for (String segmentKey : entrySet.getValue().keySet()) {
- newServersToSegmentMap.computeIfAbsent(segmentKey, k -> new HashSet<>()).add(entrySet.getKey());
+ for (String instanceName : entrySet.getValue().keySet()) {
Review Comment:
original name was incorrect, it should be instanceName here. The format is supposed to be:
```
segment1:
instance1: ONLINE,
instance2: ONLINE,
instance3: ONLINE
segment2:
instance5: ONLINE,
instance6: ONLINE,
instance7: ONLINE
```
The for-loop is going through the inner instanceToStateMap
Just a note that if another PR gets merged before this, that might fix this up and when I rebase I won't need to make this change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "raghavyadav01 (via GitHub)" <gi...@apache.org>.
raghavyadav01 commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2021797489
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -271,17 +315,279 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
String instanceName = instanceStateEntry.getKey();
String sourceInstanceState = sourceInstanceStateMap.get(instanceName);
if (!targetStateInstanceState.equals(sourceInstanceState)) {
- rebalanceStats._replicasToRebalance++;
+ rebalanceStats._totalSegmentsToRebalance++;
hasSegmentConverged = false;
}
}
if (!hasSegmentConverged) {
- rebalanceStats._segmentsToRebalance++;
+ rebalanceStats._uniqueSegmentsToRebalance++;
}
}
int totalSegments = targetState.size();
- rebalanceStats._percentSegmentsToRebalance =
- (totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
+ rebalanceStats._percentRemainingSegmentsToRebalance =
+ (totalSegments == 0) ? 0 : ((double) rebalanceStats._uniqueSegmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int numSegmentAddsProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentDeletesProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToConverge = latestStepStats._totalRemainingSegmentsToConverge;
+ newOverallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ overallProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + numberNewUntrackedSegmentsAdded;
+ newOverallProgressStats._percentageTotalSegmentsAddsRemaining =
+ calculatePercentageChange(newOverallProgressStats._totalSegmentsToBeAdded,
Review Comment:
Should the first param to calculate method be _totalRemainingSegmentsToBeAdded ?
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -271,17 +315,279 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
String instanceName = instanceStateEntry.getKey();
String sourceInstanceState = sourceInstanceStateMap.get(instanceName);
if (!targetStateInstanceState.equals(sourceInstanceState)) {
- rebalanceStats._replicasToRebalance++;
+ rebalanceStats._totalSegmentsToRebalance++;
hasSegmentConverged = false;
}
}
if (!hasSegmentConverged) {
- rebalanceStats._segmentsToRebalance++;
+ rebalanceStats._uniqueSegmentsToRebalance++;
}
}
int totalSegments = targetState.size();
- rebalanceStats._percentSegmentsToRebalance =
- (totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
+ rebalanceStats._percentRemainingSegmentsToRebalance =
+ (totalSegments == 0) ? 0 : ((double) rebalanceStats._uniqueSegmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Updates the overall progress stats based on the current step's progress stats. This should be called
+ * during the EV-IS convergence trigger to ensure the overall stats reflect the changes as they are made.
+ * @param rebalanceProgressStats the rebalance stats
+ * @param lastStepStats step level stats from the last iteration
+ * @param latestStepStats latest step level stats calculated in this iteration
+ * @return the newly calculated overall progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats updateOverallProgressStatsFromStep(
+ TableRebalanceProgressStats rebalanceProgressStats,
+ TableRebalanceProgressStats.RebalanceProgressStats lastStepStats,
+ TableRebalanceProgressStats.RebalanceProgressStats latestStepStats) {
+ int numAdditionalSegmentsAdded =
+ latestStepStats._totalSegmentsToBeAdded - lastStepStats._totalSegmentsToBeAdded;
+ int numAdditionalSegmentsDeleted =
+ latestStepStats._totalSegmentsToBeDeleted - lastStepStats._totalSegmentsToBeDeleted;
+ int numSegmentAddsProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeAdded
+ - latestStepStats._totalRemainingSegmentsToBeAdded);
+ int numSegmentDeletesProcessedInLastStep = Math.abs(lastStepStats._totalRemainingSegmentsToBeDeleted
+ - latestStepStats._totalRemainingSegmentsToBeDeleted);
+ int numberNewUntrackedSegmentsAdded = latestStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance
+ - lastStepStats._totalUniqueNewUntrackedSegmentsDuringRebalance;
+
+ TableRebalanceProgressStats.RebalanceProgressStats overallProgressStats =
+ rebalanceProgressStats.getRebalanceProgressStatsOverall();
+
+ TableRebalanceProgressStats.RebalanceProgressStats newOverallProgressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+
+ newOverallProgressStats._totalSegmentsToBeAdded = overallProgressStats._totalSegmentsToBeAdded
+ + numAdditionalSegmentsAdded;
+ newOverallProgressStats._totalSegmentsToBeDeleted = overallProgressStats._totalSegmentsToBeDeleted
+ + numAdditionalSegmentsDeleted;
+ newOverallProgressStats._totalRemainingSegmentsToBeAdded = numAdditionalSegmentsAdded == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeAdded - numSegmentAddsProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentAddsProcessedInLastStep;
+ newOverallProgressStats._totalRemainingSegmentsToBeDeleted = numAdditionalSegmentsDeleted == 0
+ ? overallProgressStats._totalRemainingSegmentsToBeDeleted - numSegmentDeletesProcessedInLastStep
+ : overallProgressStats._totalRemainingSegmentsToBeAdded + numSegmentDeletesProcessedInLastStep;
Review Comment:
Should this be _totalRemainingSegmentsToBeDeleted?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023146433
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancer.java:
##########
@@ -987,8 +1010,9 @@ private Pair<InstancePartitions, Boolean> getInstancePartitionsForTier(TableConf
}
}
- private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean lowDiskMode, boolean bestEfforts,
- Set<String> segmentsToMonitor, long externalViewCheckIntervalInMs, long externalViewStabilizationTimeoutInMs)
+ private IdealState waitForExternalViewToConverge(String tableNameWithType, boolean bestEfforts,
Review Comment:
done, added a log for this scenario
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2023470413
##########
pinot-controller/src/test/java/org/apache/pinot/controller/helix/core/rebalance/TableRebalancerTest.java:
##########
@@ -1264,116 +1264,95 @@ public void testIsExternalViewConverged() {
boolean[] falseAndTrue = new boolean[]{false, true};
// Empty segment states should match
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Do not check segment that does not exist in IdealState
Map<String, String> instanceStateMap = new TreeMap<>();
instanceStateMap.put("instance1", ONLINE);
externalViewSegmentStates.put("segment1", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Do not check segment that is OFFLINE in IdealState
instanceStateMap = new TreeMap<>();
instanceStateMap.put("instance1", OFFLINE);
idealStateSegmentStates.put("segment2", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when a segment has CONSUMING instance in IdealState but does not exist in ExternalView
instanceStateMap.put("instance2", CONSUMING);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when instance state does not exist
instanceStateMap = new TreeMap<>();
externalViewSegmentStates.put("segment2", instanceStateMap);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should fail when instance state does not match
instanceStateMap.put("instance2", OFFLINE);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertFalse(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
// Should pass when instance state matches
instanceStateMap.put("instance2", CONSUMING);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
- }
+ for (boolean bestEfforts : falseAndTrue) {
+ assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, bestEfforts, null));
}
- // When there are extra instances in ExternalView, should pass in regular mode but fail in low disk mode
+ // When there are extra instances in ExternalView, should fail (always wait for extra instances to be removed)
instanceStateMap.put("instance3", CONSUMING);
for (boolean bestEfforts : falseAndTrue) {
- assertTrue(
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- false, bestEfforts, null));
assertFalse(
TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- true, bestEfforts, null));
+ bestEfforts, null));
}
// When instance state is ERROR in ExternalView, should fail in regular mode but pass in best-efforts mode
instanceStateMap.put("instance2", ERROR);
instanceStateMap.remove("instance3");
- for (boolean lowDiskMode : falseAndTrue) {
- try {
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- lowDiskMode, false, null);
- fail();
- } catch (Exception e) {
- // Expected
- }
- assertTrue(
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
- lowDiskMode, true, null));
+ try {
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
+ false, null);
+ fail();
+ } catch (Exception e) {
+ // Expected
}
+ assertTrue(
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates, idealStateSegmentStates,
+ true, null));
- // When the extra instance is in ERROR state, should throw exception in low disk mode when best-efforts is disabled
+ // When the extra instance is in ERROR state, should throw exception when best-efforts is disabled
instanceStateMap.put("instance2", CONSUMING);
instanceStateMap.put("instance3", ERROR);
- for (boolean lowDiskMode : falseAndTrue) {
- for (boolean bestEfforts : falseAndTrue) {
- if (lowDiskMode && !bestEfforts) {
- try {
- TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, true, false, null);
- fail();
- } catch (Exception e) {
- // Expected
- }
- } else {
- assertTrue(TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
- idealStateSegmentStates, lowDiskMode, bestEfforts, null));
+ for (boolean bestEfforts : falseAndTrue) {
+ if (!bestEfforts) {
+ try {
+ TableRebalancer.isExternalViewConverged(offlineTableName, externalViewSegmentStates,
+ idealStateSegmentStates, false, null);
+ fail();
+ } catch (Exception e) {
+ // Expected
Review Comment:
this is in the test though. why would a log be useful here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2029695759
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,253 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @param targetAssignment target assignment (either updated IS or the target end IS depending on the step)
+ * @param currentAssignment current assignment (either EV or the current IS depending on the step)
+ * @param rebalanceContext rebalance context
+ * @param trigger reason to trigger the stats update
+ * @param rebalanceProgressStats current value of the rebalance progress stats, used to calculate the next
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ // Segments to monitor is the list of segments that are being moved as part of the table rebalance that the
+ // table rebalance intends to track convergence for. This list usually includes segments from the last next
+ // assignment IS update and the current IS update. The EV-IS convergence check only tracks convergence for the
+ // segments on this list, and if any additional segments are found that haven't converged they are ignored.
+ // From the stats perspective, we also only care about tracking the convergence of actual segments that the
+ // rebalance cares about, that is why we skip any segments that are not on this list.
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list, remove them if they were added
+ // before
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget++;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ // Don't track segments that are not on the rebalance monitor list
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats existingProgressStats;
+ long startTimeMs;
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ existingProgressStats = rebalanceProgressStats.getRebalanceProgressStatsOverall();
+ progressStats._totalSegmentsToBeAdded =
+ existingProgressStats._totalSegmentsToBeAdded + newSegsAddedInThisAssignment;
+ progressStats._totalSegmentsToBeDeleted =
+ existingProgressStats._totalSegmentsToBeDeleted + newSegsDeletedInThisAssignment;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance =
+ existingProgressStats._totalUniqueNewUntrackedSegmentsDuringRebalance + totalNewSegmentsNotMonitored;
Review Comment:
done, also updated test to pass null for segmentsToMove for this trigger
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Enhance rebalance progress stats [pinot]
Posted by "somandal (via GitHub)" <gi...@apache.org>.
somandal commented on code in PR #15266:
URL: https://github.com/apache/pinot/pull/15266#discussion_r2027903591
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
Review Comment:
Have added some comments about this. Basically, the logic in Table Rebalancer (`isExternalViewConverged`) only tracks segments if they fall into the following conditions:
- Are part of the `segmentsToMonitor` set, which is basically the list of all segments that are moving as part of this rebalance step (or the previous one)
- Segment is not OFFLINE state in IS
- For `lowDiskMode` if the EV deletion isn't completed yet, etc
Here I also only monitor stats for segments that are on the `segmentsToMonitor` list as calculated by table rebalancer. That's why if I see a segment is not on the monitor list, I skip processing it for the stats. hope that makes sense
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget += 1;
+ newServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ for (Map.Entry<String, Map<String, String>> entrySet : currentAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ for (String instanceName : entrySet.getValue().keySet()) {
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ continue;
+ }
+ if (targetInstanceToOfflineSegmentsMap.containsKey(instanceName)
+ && targetInstanceToOfflineSegmentsMap.get(instanceName).contains(segmentName)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ continue;
+ }
+ existingServersToSegmentMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ }
+ }
+
+ int segmentsNotMoved = 0;
+ int totalSegmentsToBeDeleted = 0;
+ int segmentsUnchangedYetNotConverged = 0;
+ for (Map.Entry<String, Set<String>> entry : newServersToSegmentMap.entrySet()) {
+ String server = entry.getKey();
+ Set<String> segmentSet = entry.getValue();
+
+ Set<String> newSegmentSet = new HashSet<>(segmentSet);
+ Set<String> existingSegmentSet = new HashSet<>();
+ int segmentsUnchanged = 0;
+ if (existingServersToSegmentMap.containsKey(server)) {
+ Set<String> segmentSetForServer = existingServersToSegmentMap.get(server);
+ existingSegmentSet.addAll(segmentSetForServer);
+ Set<String> intersection = new HashSet<>(segmentSetForServer);
+ intersection.retainAll(newSegmentSet);
+ segmentsUnchanged = intersection.size();
+ segmentsNotMoved += segmentsUnchanged;
+
+ for (String segmentName : intersection) {
+ String currentInstanceState = currentAssignment.get(segmentName).get(server);
+ String targetInstanceState = targetAssignment.get(segmentName).get(server);
+ if (!currentInstanceState.equals(targetInstanceState)) {
+ segmentsUnchangedYetNotConverged++;
+ }
+ }
+ }
+ newSegmentSet.removeAll(existingSegmentSet);
+ totalSegmentsToBeDeleted += existingSegmentSet.size() - segmentsUnchanged;
+ }
+
+ for (Map.Entry<String, Set<String>> entry : existingServersToSegmentMap.entrySet()) {
+ if (!newServersToSegmentMap.containsKey(entry.getKey())) {
+ totalSegmentsToBeDeleted += entry.getValue().size();
+ }
+ }
+
+ int newSegsAddedInThisAssignment = 0;
+ int newSegsDeletedInThisAssignment = 0;
+ for (String segment : newSegmentsNotExistingBefore) {
+ Set<String> currentSegmentAssign = currentAssignment.get(segment) != null
+ ? currentAssignment.get(segment).keySet() : new HashSet<>();
+ Set<String> targetSegmentAssign = targetAssignment.get(segment) != null
+ ? targetAssignment.get(segment).keySet() : new HashSet<>();
+
+ Set<String> segmentsAdded = new HashSet<>(targetSegmentAssign);
+ segmentsAdded.removeAll(currentSegmentAssign);
+ newSegsAddedInThisAssignment += segmentsAdded.size();
+
+ Set<String> segmentsDeleted = new HashSet<>(currentSegmentAssign);
+ segmentsDeleted.removeAll(targetSegmentAssign);
+ newSegsDeletedInThisAssignment += segmentsDeleted.size();
+ }
+
+ int newNumberSegmentsTotal = totalSegmentsTarget;
+ int totalSegmentsToBeAdded = newNumberSegmentsTotal - segmentsNotMoved;
+
+ TableRebalanceProgressStats.RebalanceProgressStats progressStats =
+ new TableRebalanceProgressStats.RebalanceProgressStats();
+ switch (trigger) {
+ case START_TRIGGER:
+ case NEXT_ASSINGMENT_CALCULATION_TRIGGER:
+ // These are initialization steps for global / step progress stats
+ progressStats._totalSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalRemainingSegmentsToBeAdded = totalSegmentsToBeAdded;
+ progressStats._totalRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted;
+ progressStats._totalCarryOverSegmentsToBeAdded = 0;
+ progressStats._totalCarryOverSegmentsToBeDeleted = 0;
+ progressStats._totalRemainingSegmentsToConverge = segmentsUnchangedYetNotConverged;
+ progressStats._totalUniqueNewUntrackedSegmentsDuringRebalance = totalNewSegmentsNotMonitored;
+ progressStats._percentageRemainingSegmentsToBeAdded = totalSegmentsToBeAdded == 0 ? 0.0 : 100.0;
+ progressStats._percentageRemainingSegmentsToBeDeleted = totalSegmentsToBeDeleted == 0 ? 0.0 : 100.0;
+ progressStats._estimatedTimeToCompleteAddsInSeconds = totalSegmentsToBeAdded == 0 ? 0.0 : -1.0;
+ progressStats._estimatedTimeToCompleteDeletesInSeconds = totalSegmentsToBeDeleted == 0 ? 0.0 : -1.0;
+ progressStats._averageSegmentSizeInBytes = rebalanceContext.getEstimatedAverageSegmentSizeInBytes();
+ progressStats._totalEstimatedDataToBeMovedInBytes =
+ TableRebalanceProgressStats.calculateNewEstimatedDataToBeMovedInBytes(0,
+ rebalanceContext.getEstimatedAverageSegmentSizeInBytes(), totalSegmentsToBeAdded);
+ progressStats._startTimeMs = trigger == Trigger.NEXT_ASSINGMENT_CALCULATION_TRIGGER
+ ? System.currentTimeMillis() : rebalanceProgressStats.getStartTimeMs();
+ break;
+ case IDEAL_STATE_CHANGE_TRIGGER:
+ case EXTERNAL_VIEW_TO_IDEAL_STATE_CONVERGENCE_TRIGGER:
Review Comment:
done
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/rebalance/ZkBasedTableRebalanceObserver.java:
##########
@@ -284,4 +319,204 @@ public static TableRebalanceProgressStats.RebalanceStateStats getDifferenceBetwe
(totalSegments == 0) ? 0 : ((double) rebalanceStats._segmentsToRebalance / totalSegments) * 100.0;
return rebalanceStats;
}
+
+ /**
+ * Calculates the progress stats for the given step or for the overall based on the trigger type
+ * @return the calculated step or progress stats
+ */
+ @VisibleForTesting
+ static TableRebalanceProgressStats.RebalanceProgressStats calculateUpdatedProgressStats(
+ Map<String, Map<String, String>> targetAssignment, Map<String, Map<String, String>> currentAssignment,
+ RebalanceContext rebalanceContext, Trigger trigger, TableRebalanceProgressStats rebalanceProgressStats) {
+ Map<String, Set<String>> existingServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> newServersToSegmentMap = new HashMap<>();
+ Map<String, Set<String>> targetInstanceToOfflineSegmentsMap = new HashMap<>();
+ Set<String> newSegmentsNotExistingBefore = new HashSet<>();
+
+ Set<String> segmentsToMonitor = rebalanceContext.getSegmentsToMonitor();
+ int totalNewSegmentsNotMonitored = 0;
+ int totalSegmentsTarget = 0;
+ for (Map.Entry<String, Map<String, String>> entrySet : targetAssignment.entrySet()) {
+ String segmentName = entrySet.getKey();
+ if (!rebalanceContext.getUniqueSegments().contains(segmentName)) {
+ newSegmentsNotExistingBefore.add(segmentName);
+ }
+ for (Map.Entry<String, String> entry : entrySet.getValue().entrySet()) {
+ String instanceName = entry.getKey();
+ String instanceState = entry.getValue();
+ if (segmentsToMonitor != null && !segmentsToMonitor.contains(segmentName)) {
+ if (newSegmentsNotExistingBefore.contains(segmentName)) {
+ // Don't track newly added segments unless they're on the monitor list
+ totalNewSegmentsNotMonitored++;
+ newSegmentsNotExistingBefore.remove(segmentName);
+ }
+ continue;
+ }
+ if (instanceState.equals(CommonConstants.Helix.StateModel.SegmentStateModel.OFFLINE)) {
+ // Skip tracking segments that are in OFFLINE state in the target assignment
+ targetInstanceToOfflineSegmentsMap.computeIfAbsent(instanceName, k -> new HashSet<>()).add(segmentName);
+ continue;
+ }
+ totalSegmentsTarget += 1;
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org