You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "Jackie-Jiang (via GitHub)" <gi...@apache.org> on 2023/09/25 20:39:00 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request, #11679: Remove the global lock in SegmentCompletionManager
Jackie-Jiang opened a new pull request, #11679:
URL: https://github.com/apache/pinot/pull/11679
Use `computeIfAbsent()` to achieve the same guarantee but better parallelism
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #11679: Remove the global lock in SegmentCompletionManager
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #11679:
URL: https://github.com/apache/pinot/pull/11679#discussion_r1337612141
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java:
##########
@@ -122,51 +115,30 @@ protected StreamPartitionMsgOffsetFactory getStreamPartitionMsgOffsetFactory(LLC
return StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
}
- // We need to make sure that we never create multiple FSMs for the same segment
- // Obtain locks based on segment name, so as to disallow same segment names entering together
Review Comment:
Yes, we should use `ConcurrentHashMap.computeIfAbsent()` to achieve this instead of adding lock. The old code uses `get()` which doesn't provide the same guarantee
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #11679: Remove the global lock in SegmentCompletionManager
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #11679:
URL: https://github.com/apache/pinot/pull/11679
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #11679: Remove the global lock in SegmentCompletionManager
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #11679:
URL: https://github.com/apache/pinot/pull/11679#issuecomment-1734513789
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
> Merging [#11679](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (66a746a) into [master](https://app.codecov.io/gh/apache/pinot/commit/46b0c468ef1fa994acd7ea160891b427c4ea4dae?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (46b0c46) will **increase** coverage by `0.04%`.
> Report is 1 commits behind head on master.
> The diff coverage is `100.00%`.
```diff
@@ Coverage Diff @@
## master #11679 +/- ##
============================================
+ Coverage 63.05% 63.10% +0.04%
- Complexity 1120 1121 +1
============================================
Files 2343 2343
Lines 125698 125685 -13
Branches 19310 19309 -1
============================================
+ Hits 79256 79309 +53
+ Misses 40789 40723 -66
Partials 5653 5653
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [integration](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `<0.01% <0.00%> (ø)` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `50.09% <0.00%> (-12.93%)` | :arrow_down: |
| [java-17](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `62.94% <100.00%> (+48.49%)` | :arrow_up: |
| [java-20](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `49.96% <0.00%> (-12.95%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.10% <100.00%> (+0.04%)` | :arrow_up: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `63.09% <100.00%> (+0.04%)` | :arrow_up: |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `67.25% <ø> (+0.05%)` | :arrow_up: |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/11679/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `14.46% <100.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [.../helix/core/realtime/SegmentCompletionManager.java](https://app.codecov.io/gh/apache/pinot/pull/11679?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1NlZ21lbnRDb21wbGV0aW9uTWFuYWdlci5qYXZh) | `69.83% <100.00%> (-0.81%)` | :arrow_down: |
... and [18 files with indirect coverage changes](https://app.codecov.io/gh/apache/pinot/pull/11679/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] sajjad-moradi commented on a diff in pull request #11679: Remove the global lock in SegmentCompletionManager
Posted by "sajjad-moradi (via GitHub)" <gi...@apache.org>.
sajjad-moradi commented on code in PR #11679:
URL: https://github.com/apache/pinot/pull/11679#discussion_r1337546809
##########
pinot-controller/src/main/java/org/apache/pinot/controller/helix/core/realtime/SegmentCompletionManager.java:
##########
@@ -122,51 +115,30 @@ protected StreamPartitionMsgOffsetFactory getStreamPartitionMsgOffsetFactory(LLC
return StreamConsumerFactoryProvider.create(streamConfig).createStreamMsgOffsetFactory();
}
- // We need to make sure that we never create multiple FSMs for the same segment
- // Obtain locks based on segment name, so as to disallow same segment names entering together
Review Comment:
The locks were added in this PR: https://github.com/apache/pinot/pull/4159
At that time, fsmMap was also available. Am I missing a point here or could've we done the same thing in the PR#4159?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org