You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "zhtaoxiang (via GitHub)" <gi...@apache.org> on 2023/02/07 08:34:11 UTC
[GitHub] [pinot] zhtaoxiang opened a new pull request, #10239: add an API to reset segments with Error EV
zhtaoxiang opened a new pull request, #10239:
URL: https://github.com/apache/pinot/pull/10239
Similar to `segments/{tableNameWithType}/{segmentName}/reset` and `segments/{tableNameWithType}/resetErrorEvSegments`, this PR adds an API to reset segments with Error EV.
We add it because in practice,
1. users want to reset only segment with Error EV, which is faster than resetting all segments.
2. currently, users need to use `segments/{tableNameWithType}/{segmentName}/reset` to reset segments one by one, which is not convinient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] zhtaoxiang commented on a diff in pull request #10239: add an API to reset segments with Error EV
Posted by "zhtaoxiang (via GitHub)" <gi...@apache.org>.
zhtaoxiang commented on code in PR #10239:
URL: https://github.com/apache/pinot/pull/10239#discussion_r1099739928
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -541,6 +541,41 @@ public SuccessResponse resetSegment(
}
}
+ /**
+ * Resets segments with Error EV of the given table
+ * This API will take segments to OFFLINE state, wait for External View to stabilize, and then back to
+ * ONLINE/CONSUMING state,
+ * thus effective in resetting segments or consumers in error states.
+ */
+ @POST
+ @Path("segments/{tableNameWithType}/resetErrorEvSegments")
Review Comment:
make sense, updated
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2430,11 +2429,41 @@ public void resetSegment(String tableNameWithType, String segmentName, @Nullable
}
}
+ /**
+ * Resets segments with Error EV of a table. This operation invoke resetPartition via state transition message.
+ */
+ public void resetErrorEvSegments(String tableNameWithType, @Nullable String targetInstance) {
Review Comment:
make sense, updated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10239: update an existing API to reset only segments with Error EV
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10239:
URL: https://github.com/apache/pinot/pull/10239#discussion_r1100594101
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -552,18 +552,21 @@ public SuccessResponse resetSegment(
@Produces(MediaType.APPLICATION_JSON)
@Authenticate(AccessType.UPDATE)
@ApiOperation(
- value = "Resets all segments of the table, by first disabling them, waiting for external view to stabilize, and"
- + " finally enabling the segments", notes = "Resets a segment by disabling and then enabling a segment")
+ value = "Resets all segments (when errorEVSegmentsOnly = false) or segments with Error external view (when "
+ + "errorEVSegmentsOnly = true) of the table, by first disabling them, waiting for external view to stabilize,"
+ + " and finally enabling them", notes = "Resets segments by disabling and then enabling them")
public SuccessResponse resetAllSegments(
@ApiParam(value = "Name of the table with type", required = true) @PathParam("tableNameWithType")
String tableNameWithType,
@ApiParam(value = "Name of the target instance to reset") @QueryParam("targetInstance") @Nullable
- String targetInstance) {
+ String targetInstance,
+ @ApiParam(value = "Whether to reset only segments with error external view") @QueryParam("errorEVSegmentsOnly")
+ @DefaultValue("false") boolean errorEVSegmentsOnly) {
Review Comment:
(optional) Suggest renaming it to `errorSegmentsOnly` since it is implicit that ERROR is in EV, same for javadoc and variable names
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2447,21 +2446,34 @@ public void resetAllSegments(String tableNameWithType, @Nullable String targetIn
Set<String> instanceSet = parseInstanceSet(idealState, segmentName, targetInstance);
Map<String, String> externalViewStateMap = externalView.getStateMap(segmentName);
for (String instance : instanceSet) {
- if (externalViewStateMap == null || SegmentStateModel.OFFLINE.equals(externalViewStateMap.get(instance))) {
- instanceToSkippedSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ if (errorEVSegmentsOnly) {
+ if (externalViewStateMap != null && SegmentStateModel.ERROR.equals(externalViewStateMap.get(instance))) {
+ instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ }
} else {
- instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ if (externalViewStateMap == null || SegmentStateModel.OFFLINE.equals(externalViewStateMap.get(instance))) {
+ instanceToSkippedSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ } else {
+ instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ }
}
}
}
- LOGGER.info("Resetting all segments of table: {}", tableNameWithType);
+ if (instanceToResetSegmentsMap.isEmpty()) {
+ LOGGER.info("No segments to reset for table: {}", tableNameWithType);
+ return;
+ }
+
+ LOGGER.info("Resetting segments of table: {}", tableNameWithType);
Review Comment:
Let's log all the segments to be reset
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2447,21 +2446,34 @@ public void resetAllSegments(String tableNameWithType, @Nullable String targetIn
Set<String> instanceSet = parseInstanceSet(idealState, segmentName, targetInstance);
Map<String, String> externalViewStateMap = externalView.getStateMap(segmentName);
for (String instance : instanceSet) {
- if (externalViewStateMap == null || SegmentStateModel.OFFLINE.equals(externalViewStateMap.get(instance))) {
- instanceToSkippedSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ if (errorEVSegmentsOnly) {
+ if (externalViewStateMap != null && SegmentStateModel.ERROR.equals(externalViewStateMap.get(instance))) {
+ instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ }
} else {
- instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ if (externalViewStateMap == null || SegmentStateModel.OFFLINE.equals(externalViewStateMap.get(instance))) {
+ instanceToSkippedSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ } else {
+ instanceToResetSegmentsMap.computeIfAbsent(instance, i -> new HashSet<>()).add(segmentName);
+ }
}
}
}
- LOGGER.info("Resetting all segments of table: {}", tableNameWithType);
+ if (instanceToResetSegmentsMap.isEmpty()) {
+ LOGGER.info("No segments to reset for table: {}", tableNameWithType);
+ return;
+ }
+
+ LOGGER.info("Resetting segments of table: {}", tableNameWithType);
for (Map.Entry<String, Set<String>> entry : instanceToResetSegmentsMap.entrySet()) {
resetPartitionAllState(entry.getKey(), tableNameWithType, entry.getValue());
}
- LOGGER.info("Reset segments for table {} finished. With the following segments skipped: {}", tableNameWithType,
- instanceToSkippedSegmentsMap);
+ if (!instanceToSkippedSegmentsMap.isEmpty()) {
Review Comment:
Don't add this if check because we want to log when reset is done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #10239: update an existing API to reset only segments with Error EV
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #10239:
URL: https://github.com/apache/pinot/pull/10239
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10239: add an API to reset segments with Error EV
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10239:
URL: https://github.com/apache/pinot/pull/10239#discussion_r1099349671
##########
pinot-controller/src/main/java/org/apache/pinot/controller/api/resources/PinotSegmentRestletResource.java:
##########
@@ -541,6 +541,41 @@ public SuccessResponse resetSegment(
}
}
+ /**
+ * Resets segments with Error EV of the given table
+ * This API will take segments to OFFLINE state, wait for External View to stabilize, and then back to
+ * ONLINE/CONSUMING state,
+ * thus effective in resetting segments or consumers in error states.
+ */
+ @POST
+ @Path("segments/{tableNameWithType}/resetErrorEvSegments")
Review Comment:
Suggest modifying `segments/{tableNameWithType}/reset` and include a query parameter `errorSegmentsOnly` instead of introducing a new API since their purpose is very similar
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -2430,11 +2429,41 @@ public void resetSegment(String tableNameWithType, String segmentName, @Nullable
}
}
+ /**
+ * Resets segments with Error EV of a table. This operation invoke resetPartition via state transition message.
+ */
+ public void resetErrorEvSegments(String tableNameWithType, @Nullable String targetInstance) {
Review Comment:
Same here, we may introduce a boolean `errorSegmentsOnly` to `resetAllSegments`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10239: add an API to reset segments with Error EV
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10239:
URL: https://github.com/apache/pinot/pull/10239#issuecomment-1420436886
# [Codecov](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10239](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f8f5276) into [master](https://codecov.io/gh/apache/pinot/commit/3d6163a73bde38b0efc086d3f6486047c27a9bfb?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (3d6163a) will **decrease** coverage by `50.54%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #10239 +/- ##
=============================================
- Coverage 64.23% 13.70% -50.54%
+ Complexity 5732 178 -5554
=============================================
Files 1961 1961
Lines 106677 106708 +31
Branches 16291 16299 +8
=============================================
- Hits 68526 14620 -53906
- Misses 33179 90936 +57757
+ Partials 4972 1152 -3820
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests1 | `?` | |
| unittests2 | `13.70% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ler/api/resources/PinotSegmentRestletResource.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2VnbWVudFJlc3RsZXRSZXNvdXJjZS5qYXZh) | `9.97% <0.00%> (-0.32%)` | :arrow_down: |
| [...ntroller/helix/core/PinotHelixResourceManager.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL1Bpbm90SGVsaXhSZXNvdXJjZU1hbmFnZXIuamF2YQ==) | `61.75% <0.00%> (-0.72%)` | :arrow_down: |
| [...src/main/java/org/apache/pinot/sql/FilterKind.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvRmlsdGVyS2luZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...in/java/org/apache/pinot/spi/utils/BytesUtils.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQnl0ZXNVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ain/java/org/apache/pinot/common/CustomObject.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vQ3VzdG9tT2JqZWN0LmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...n/java/org/apache/pinot/core/data/table/Table.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1RhYmxlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/spi/trace/BaseRecording.java](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdHJhY2UvQmFzZVJlY29yZGluZy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1328 more](https://codecov.io/gh/apache/pinot/pull/10239?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org