You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/09/25 20:39:00 UTC

[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11679: Remove the global lock in SegmentCompletionManager

Jackie-Jiang opened a new pull request, #11679:
URL: https://github.com/apache/pinot/pull/11679

   Use `computeIfAbsent()` to achieve the same guarantee but better parallelism


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11679: Remove the global lock in SegmentCompletionManager

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11679:
URL: https://github.com/apache/pinot/pull/11679#discussion_r1337612141


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java:
##########
@@ -122,51 +115,30 @@ protected StreamPartitionMsgOffsetFactory getStreamPartitionMsgOffsetFactory(LLC
     return StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
   }
 
-  // We need to make sure that we never create multiple FSMs for the same segment
-  // Obtain locks based on segment name, so as to disallow same segment names entering together

Review Comment:
   Yes, we should use `ConcurrentHashMap.computeIfAbsent()` to achieve this instead of adding lock. The old code uses `get()` which doesn't provide the same guarantee



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #11679: Remove the global lock in SegmentCompletionManager

Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11679:
URL: https://github.com/apache/pinot/pull/11679


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #11679: Remove the global lock in SegmentCompletionManager

Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11679:
URL: https://github.com/apache/pinot/pull/11679#issuecomment-1734513789

   ## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#11679](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (66a746a) into [master](https://app.codecov.io/gh/apache/pinot/commit/46b0c468ef1fa994acd7ea160891b427c4ea4dae?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (46b0c46) will **increase** coverage by `0.04%`.
   > Report is 1 commits behind head on master.
   > The diff coverage is `100.00%`.
   
   ```diff
   @@             Coverage Diff              @@
   ##             master   #11679      +/-   ##
   ============================================
   + Coverage     63.05%   63.10%   +0.04%     
   - Complexity     1120     1121       +1     
   ============================================
     Files          2343     2343              
     Lines        125698   125685      -13     
     Branches      19310    19309       -1     
   ============================================
   + Hits          79256    79309      +53     
   + Misses        40789    40723      -66     
     Partials       5653     5653              
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [integration](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration1](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
   | [integration2](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
   | [java-11](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `50.09% <0.00%> (-12.93%)` | :arrow_down: |
   | [java-17](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.94% <100.00%> (+48.49%)` | :arrow_up: |
   | [java-20](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.96% <0.00%> (-12.95%)` | :arrow_down: |
   | [temurin](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.10% <100.00%> (+0.04%)` | :arrow_up: |
   | [unittests](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.09% <100.00%> (+0.04%)` | :arrow_up: |
   | [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.25% <ø> (+0.05%)` | :arrow_up: |
   | [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.46% <100.00%> (+<0.01%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [.../helix/core/realtime/SegmentCompletionManager.java](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `69.83% <100.00%> (-0.81%)` | :arrow_down: |
   
   ... and [18 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11679/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #11679: Remove the global lock in SegmentCompletionManager

Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on code in PR #11679:
URL: https://github.com/apache/pinot/pull/11679#discussion_r1337546809


##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java:
##########
@@ -122,51 +115,30 @@ protected StreamPartitionMsgOffsetFactory getStreamPartitionMsgOffsetFactory(LLC
     return StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
   }
 
-  // We need to make sure that we never create multiple FSMs for the same segment
-  // Obtain locks based on segment name, so as to disallow same segment names entering together

Review Comment:
   The locks were added in this PR: https://github.com/apache/pinot/pull/4159
   At that time, fsmMap was also available. Am I missing a point here or could've we done the same thing in the PR#4159?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org