You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "snleee (via GitHub)" <gi...@apache.org> on 2023/10/18 21:43:36 UTC
[PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
snleee opened a new pull request, #11832:
URL: https://github.com/apache/pinot/pull/11832
Currently, we fetch the list of segment zk metadata directly from the property store. This can cause the issue when the segment deletion process fails in the middle (deleted the segment from idealstate while the zk metadata is not cleaned up).
In order to avoid scheduling dangling segment, we should compute the intersection of segment zk metadata and idealstate.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1364819270
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -805,21 +805,26 @@ public List<String> getSegmentsFor(String tableNameWithType, boolean shouldExclu
IdealState idealState = getTableIdealState(tableNameWithType);
Preconditions.checkState(idealState != null, "Failed to find ideal state for table: %s", tableNameWithType);
List<String> segments = new ArrayList<>(idealState.getPartitionSet());
- if (startTimestamp == Long.MIN_VALUE && endTimestamp == Long.MAX_VALUE) {
- return shouldExcludeReplacedSegments ? excludeReplacedSegments(tableNameWithType, segments) : segments;
- } else {
- List<String> selectedSegments = new ArrayList<>();
- List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
- for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
- String segmentName = segmentZKMetadata.getSegmentName();
- if (segments.contains(segmentName) && isSegmentWithinTimeStamps(segmentZKMetadata, startTimestamp, endTimestamp,
- excludeOverlapping)) {
+ List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
+ List<String> selectedSegments = new ArrayList<>();
+ for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
+ String segmentName = segmentZKMetadata.getSegmentName();
+ // Compute the interesction of segmentZK metadata and idealstate for valid segmnets
+ if (!segments.contains(segmentName)) {
Review Comment:
I thought about it again. In our production incident, this diff were >10k. This can easily mess up the log. I added log to provide # of filtered segments and up to 100 names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1365809430
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -805,21 +805,26 @@ public List<String> getSegmentsFor(String tableNameWithType, boolean shouldExclu
IdealState idealState = getTableIdealState(tableNameWithType);
Preconditions.checkState(idealState != null, "Failed to find ideal state for table: %s", tableNameWithType);
List<String> segments = new ArrayList<>(idealState.getPartitionSet());
- if (startTimestamp == Long.MIN_VALUE && endTimestamp == Long.MAX_VALUE) {
- return shouldExcludeReplacedSegments ? excludeReplacedSegments(tableNameWithType, segments) : segments;
- } else {
- List<String> selectedSegments = new ArrayList<>();
- List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
- for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
- String segmentName = segmentZKMetadata.getSegmentName();
- if (segments.contains(segmentName) && isSegmentWithinTimeStamps(segmentZKMetadata, startTimestamp, endTimestamp,
- excludeOverlapping)) {
+ List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
+ List<String> selectedSegments = new ArrayList<>();
+ for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
+ String segmentName = segmentZKMetadata.getSegmentName();
+ // Compute the interesction of segmentZK metadata and idealstate for valid segmnets
+ if (!segments.contains(segmentName)) {
Review Comment:
Samples are good. More problematic would be when something is in ideal state but not in ZK metadata, because we will querying those inconsistent segments?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11832:
URL: https://github.com/apache/pinot/pull/11832#issuecomment-1769581035
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11832](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (2104f72) into [master](https://app.codecov.io/gh/apache/pinot/commit/d7d504b7f6ec10ed2b96a9442420d5967bfd2f83?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d7d504b) will **decrease** coverage by `13.03%`.
> Report is 18 commits behind head on master.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #11832 +/- ##
=============================================
- Coverage 63.03% 50.01% -13.03%
+ Complexity 1140 6 -1134
=============================================
Files 2343 2274 -69
Lines 126328 123530 -2798
Branches 19425 19039 -386
=============================================
- Hits 79628 61779 -17849
- Misses 41020 57608 +16588
+ Partials 5680 4143 -1537
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-0.01%)` | :arrow_down: |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.99% <0.00%> (+35.58%)` | :arrow_up: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.86% <0.00%> (-13.12%)` | :arrow_down: |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (-62.99%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `50.01% <0.00%> (-13.03%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.14% <ø> (+4.11%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.14% <ø> (+0.02%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11832/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [...troller/helix/core/minion/ClusterInfoAccessor.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9DbHVzdGVySW5mb0FjY2Vzc29yLmphdmE=) | `0.00% <0.00%> (-43.75%)` | :arrow_down: |
| [...on/tasks/mergerollup/MergeRollupTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvbWVyZ2Vyb2xsdXAvTWVyZ2VSb2xsdXBUYXNrR2VuZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-63.80%)` | :arrow_down: |
| [...gments/RealtimeToOfflineSegmentsTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvcmVhbHRpbWV0b29mZmxpbmVzZWdtZW50cy9SZWFsdGltZVRvT2ZmbGluZVNlZ21lbnRzVGFza0dlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (-95.42%)` | :arrow_down: |
| [...andpush/SegmentGenerationAndPushTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3Mvc2VnbWVudGdlbmVyYXRpb25hbmRwdXNoL1NlZ21lbnRHZW5lcmF0aW9uQW5kUHVzaFRhc2tHZW5lcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-2.14%)` | :arrow_down: |
| [...psertcompaction/UpsertCompactionTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvdXBzZXJ0Y29tcGFjdGlvbi9VcHNlcnRDb21wYWN0aW9uVGFza0dlbmVyYXRvci5qYXZh) | `0.00% <0.00%> (-63.98%)` | :arrow_down: |
| [.../tasks/BaseMultipleSegmentsConversionExecutor.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvQmFzZU11bHRpcGxlU2VnbWVudHNDb252ZXJzaW9uRXhlY3V0b3IuamF2YQ==) | `0.00% <0.00%> (-2.92%)` | :arrow_down: |
| [.../plugin/minion/tasks/purge/PurgeTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvcHVyZ2UvUHVyZ2VUYXNrR2VuZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-3.95%)` | :arrow_down: |
| [...che/pinot/plugin/minion/tasks/MinionTaskUtils.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtcGx1Z2lucy9waW5vdC1taW5pb24tdGFza3MvcGlub3QtbWluaW9uLWJ1aWx0aW4tdGFza3Mvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3BsdWdpbi9taW5pb24vdGFza3MvTWluaW9uVGFza1V0aWxzLmphdmE=) | `0.00% <0.00%> (-22.73%)` | :arrow_down: |
| [...helix/core/minion/generator/BaseTaskGenerator.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9nZW5lcmF0b3IvQmFzZVRhc2tHZW5lcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-41.38%)` | :arrow_down: |
| [...ntroller/helix/core/PinotHelixResourceManager.java](https://app.codecov.io/gh/apache/pinot/pull/11832?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL1Bpbm90SGVsaXhSZXNvdXJjZU1hbmFnZXIuamF2YQ==) | `0.00% <0.00%> (-62.97%)` | :arrow_down: |
... and [537 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11832/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee merged PR #11832:
URL: https://github.com/apache/pinot/pull/11832
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "jtao15 (via GitHub)" <gi...@apache.org>.
jtao15 commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1364580069
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/BaseTaskGenerator.java:
##########
@@ -89,6 +93,28 @@ public int getMaxAttemptsPerTask() {
return MinionConstants.DEFAULT_MAX_ATTEMPTS_PER_TASK;
}
+ /**
+ * Returns the list of segment zk metadata for available segments in the table. The list does NOT filter out active
Review Comment:
(nit) inactive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "swaminathanmanish (via GitHub)" <gi...@apache.org>.
swaminathanmanish commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1364669982
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -805,21 +805,26 @@ public List<String> getSegmentsFor(String tableNameWithType, boolean shouldExclu
IdealState idealState = getTableIdealState(tableNameWithType);
Preconditions.checkState(idealState != null, "Failed to find ideal state for table: %s", tableNameWithType);
List<String> segments = new ArrayList<>(idealState.getPartitionSet());
- if (startTimestamp == Long.MIN_VALUE && endTimestamp == Long.MAX_VALUE) {
- return shouldExcludeReplacedSegments ? excludeReplacedSegments(tableNameWithType, segments) : segments;
- } else {
- List<String> selectedSegments = new ArrayList<>();
- List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
- for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
- String segmentName = segmentZKMetadata.getSegmentName();
- if (segments.contains(segmentName) && isSegmentWithinTimeStamps(segmentZKMetadata, startTimestamp, endTimestamp,
- excludeOverlapping)) {
+ List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
+ List<String> selectedSegments = new ArrayList<>();
+ for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
+ String segmentName = segmentZKMetadata.getSegmentName();
+ // Compute the interesction of segmentZK metadata and idealstate for valid segmnets
+ if (!segments.contains(segmentName)) {
Review Comment:
Could you add log indicating that ideal state is different from Segment ZK metadata and which segments they are ? This will have to be fixed manually? LGTM otherwise. Thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1364698744
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/PinotHelixResourceManager.java:
##########
@@ -805,21 +805,26 @@ public List<String> getSegmentsFor(String tableNameWithType, boolean shouldExclu
IdealState idealState = getTableIdealState(tableNameWithType);
Preconditions.checkState(idealState != null, "Failed to find ideal state for table: %s", tableNameWithType);
List<String> segments = new ArrayList<>(idealState.getPartitionSet());
- if (startTimestamp == Long.MIN_VALUE && endTimestamp == Long.MAX_VALUE) {
- return shouldExcludeReplacedSegments ? excludeReplacedSegments(tableNameWithType, segments) : segments;
- } else {
- List<String> selectedSegments = new ArrayList<>();
- List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
- for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
- String segmentName = segmentZKMetadata.getSegmentName();
- if (segments.contains(segmentName) && isSegmentWithinTimeStamps(segmentZKMetadata, startTimestamp, endTimestamp,
- excludeOverlapping)) {
+ List<SegmentZKMetadata> segmentZKMetadataList = getSegmentsZKMetadata(tableNameWithType);
+ List<String> selectedSegments = new ArrayList<>();
+ for (SegmentZKMetadata segmentZKMetadata : segmentZKMetadataList) {
+ String segmentName = segmentZKMetadata.getSegmentName();
+ // Compute the interesction of segmentZK metadata and idealstate for valid segmnets
+ if (!segments.contains(segmentName)) {
Review Comment:
Good point. I will add it to the log. We should clean these dangling segments automatically in the retention manager to avoid the manual intervention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Fix the way of fetching the segment zk metadata for task generators [pinot]
Posted by "snleee (via GitHub)" <gi...@apache.org>.
snleee commented on code in PR #11832:
URL: https://github.com/apache/pinot/pull/11832#discussion_r1364602985
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/minion/generator/BaseTaskGenerator.java:
##########
@@ -89,6 +93,28 @@ public int getMaxAttemptsPerTask() {
return MinionConstants.DEFAULT_MAX_ATTEMPTS_PER_TASK;
}
+ /**
+ * Returns the list of segment zk metadata for available segments in the table. The list does NOT filter out active
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org