You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/09/29 00:34:21 UTC

[GitHub] [pinot] npawar opened a new pull request #7493: Move acquire/release to right before/after execution call

npawar opened a new pull request #7493:
URL: https://github.com/apache/pinot/pull/7493


   1) Add more info to FetchContext 
   2) Make AcquireReleaseOperator take the plan instead of the operator for the child, to ensure acquire can happen before buffers accessed in planning
   3) Move acquire release to BaseCombineOperator level, to better ensure release only happens after buffer access are done
   
   To ensure clean acquire/release semantics, dataBuffers can no longer cache DirectByteBuffers. Removing it from BitmapInvertedIndexReader, RangeInvertedIndexReader and NullValueVectorReaderImpl.
   Included jmh benchmarks in the change, which shows
   1. while cached bitmaps outperform in a setup with large heap and high repeatability, it performs significantly worse in a setup with tight heap and low repeatability, compared to always building the bitmap.
   2. in the case where alwaysBuild performed slower, the overhead we're talking about here is <1 microseconds per read. In the case where alwaysBuild performed better, it outdid by 4.6 microseconds per read.
   3. These results remain consistent regardless of low/high cardinality
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar commented on a change in pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
npawar commented on a change in pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#discussion_r718839226



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       following up to our offline discussion about it not following lock/unlock semantics, will stick to this for now and revisit if need be, since functionality wise it's the same




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter commented on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7493](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1e2489) into [master](https://codecov.io/gh/apache/pinot/commit/240bcb80546454d5ab449d3d3b13867f4943706d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (240bcb8) will **increase** coverage by `37.67%`.
   > The diff coverage is `59.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7493/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7493       +/-   ##
   =============================================
   + Coverage     32.19%   69.87%   +37.67%     
   - Complexity        0     3327     +3327     
   =============================================
     Files          1514     1129      -385     
     Lines         75167    53390    -21777     
     Branches      10966     8037     -2929     
   =============================================
   + Hits          24203    37308    +13105     
   + Misses        48870    13441    -35429     
   - Partials       2094     2641      +547     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.87% <59.64%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ore/plan/AcquireReleaseColumnsSegmentPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FjcXVpcmVSZWxlYXNlQ29sdW1uc1NlZ21lbnRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | `72.17% <0.00%> (+5.21%)` | :arrow_up: |
   | [...gment/index/readers/BitmapInvertedIndexReader.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `100.00% <ø> (+100.00%)` | :arrow_up: |
   | [...al/segment/index/readers/RangeIndexReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvUmFuZ2VJbmRleFJlYWRlckltcGwuamF2YQ==) | `89.02% <ø> (+89.02%)` | :arrow_up: |
   | [...ava/org/apache/pinot/segment/spi/FetchContext.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0ZldGNoQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `79.48% <33.33%> (-9.56%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `80.95% <79.41%> (+17.66%)` | :arrow_up: |
   | [...gment/index/readers/NullValueVectorReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvTnVsbFZhbHVlVmVjdG9yUmVhZGVySW1wbC5qYXZh) | `100.00% <100.00%> (+100.00%)` | :arrow_up: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1311 more](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [240bcb8...a1e2489](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7493](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1e2489) into [master](https://codecov.io/gh/apache/pinot/commit/240bcb80546454d5ab449d3d3b13867f4943706d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (240bcb8) will **increase** coverage by `39.82%`.
   > The diff coverage is `59.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7493/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7493       +/-   ##
   =============================================
   + Coverage     32.19%   72.02%   +39.82%     
   - Complexity        0     3407     +3407     
   =============================================
     Files          1514     1523        +9     
     Lines         75167    75495      +328     
     Branches      10966    10998       +32     
   =============================================
   + Hits          24203    54373    +30170     
   + Misses        48870    17468    -31402     
   - Partials       2094     3654     +1560     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.44% <38.59%> (+0.04%)` | :arrow_up: |
   | integration2 | `29.14% <38.59%> (+0.12%)` | :arrow_up: |
   | unittests1 | `69.87% <59.64%> (?)` | |
   | unittests2 | `14.49% <0.00%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ore/plan/AcquireReleaseColumnsSegmentPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FjcXVpcmVSZWxlYXNlQ29sdW1uc1NlZ21lbnRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | `82.60% <0.00%> (+15.65%)` | :arrow_up: |
   | [...gment/index/readers/BitmapInvertedIndexReader.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `100.00% <ø> (+100.00%)` | :arrow_up: |
   | [...al/segment/index/readers/RangeIndexReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvUmFuZ2VJbmRleFJlYWRlckltcGwuamF2YQ==) | `89.02% <ø> (+89.02%)` | :arrow_up: |
   | [...ava/org/apache/pinot/segment/spi/FetchContext.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0ZldGNoQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `88.46% <33.33%> (-0.58%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `80.95% <79.41%> (+17.66%)` | :arrow_up: |
   | [...gment/index/readers/NullValueVectorReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvTnVsbFZhbHVlVmVjdG9yUmVhZGVySW1wbC5qYXZh) | `100.00% <100.00%> (+100.00%)` | :arrow_up: |
   | [...a/manager/realtime/RealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvUmVhbHRpbWVTZWdtZW50RGF0YU1hbmFnZXIuamF2YQ==) | `50.00% <0.00%> (-25.00%)` | :arrow_down: |
   | ... and [1006 more](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [240bcb8...a1e2489](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar commented on a change in pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
npawar commented on a change in pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#discussion_r718780263



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       actually i intentionally put it inside the try, so that we release no matter what happens in acquire or execution

##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       following up to our offline discussion about it not following lock/unlock semantics, will stick to this for now and revisit if need be, since functionality wise it's the same




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar merged pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
npawar merged pull request #7493:
URL: https://github.com/apache/pinot/pull/7493


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar merged pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
npawar merged pull request #7493:
URL: https://github.com/apache/pinot/pull/7493


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#discussion_r718768587



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       I think this should be outside of the try block, so that we don't release the operator if acquire fails. Same for other places

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/NullValueVectorReaderImpl.java
##########
@@ -25,18 +25,20 @@
 
 public class NullValueVectorReaderImpl implements NullValueVectorReader {
 
-  private final ImmutableRoaringBitmap _nullBitmap;
+  private final PinotDataBuffer _dataBuffer;
 
   public NullValueVectorReaderImpl(PinotDataBuffer dataBuffer) {
-    _nullBitmap = new ImmutableRoaringBitmap(dataBuffer.toDirectByteBuffer(0, (int) dataBuffer.size()));
+    _dataBuffer = dataBuffer;
   }
 
   public boolean isNull(int docId) {
-    return _nullBitmap.contains(docId);
+    ImmutableRoaringBitmap nullBitmap =
+        new ImmutableRoaringBitmap(_dataBuffer.toDirectByteBuffer(0, (int) _dataBuffer.size()));
+    return nullBitmap.contains(docId);

Review comment:
       (nit)
   ```suggestion
     @Override
     public boolean isNull(int docId) {
       return getNullBitmap().contains(docId);
     }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7493](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f09d35a) into [master](https://codecov.io/gh/apache/pinot/commit/debb132575ffe3485ba3f61b19fdaa6ce33da425?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (debb132) will **decrease** coverage by `2.14%`.
   > The diff coverage is `56.60%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7493/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7493      +/-   ##
   ============================================
   - Coverage     72.01%   69.87%   -2.15%     
   + Complexity     3405     3327      -78     
   ============================================
     Files          1523     1129     -394     
     Lines         75531    53402   -22129     
     Branches      11010     8043    -2967     
   ============================================
   - Hits          54396    37314   -17082     
   + Misses        17468    13444    -4024     
   + Partials       3667     2644    -1023     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.87% <56.60%> (+0.05%)` | :arrow_up: |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ore/plan/AcquireReleaseColumnsSegmentPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FjcXVpcmVSZWxlYXNlQ29sdW1uc1NlZ21lbnRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | `72.22% <0.00%> (-10.32%)` | :arrow_down: |
   | [...gment/index/readers/BitmapInvertedIndexReader.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `100.00% <ø> (+7.14%)` | :arrow_up: |
   | [...al/segment/index/readers/RangeIndexReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvUmFuZ2VJbmRleFJlYWRlckltcGwuamF2YQ==) | `89.02% <ø> (+3.73%)` | :arrow_up: |
   | [...ava/org/apache/pinot/segment/spi/FetchContext.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0ZldGNoQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `78.37% <33.33%> (-14.38%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `81.17% <78.12%> (-3.83%)` | :arrow_down: |
   | [...gment/index/readers/NullValueVectorReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvTnVsbFZhbHVlVmVjdG9yUmVhZGVySW1wbC5qYXZh) | `100.00% <100.00%> (ø)` | |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [603 more](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [debb132...f09d35a](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter commented on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7493](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a1e2489) into [master](https://codecov.io/gh/apache/pinot/commit/240bcb80546454d5ab449d3d3b13867f4943706d?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (240bcb8) will **increase** coverage by `37.67%`.
   > The diff coverage is `59.64%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7493/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7493       +/-   ##
   =============================================
   + Coverage     32.19%   69.87%   +37.67%     
   - Complexity        0     3327     +3327     
   =============================================
     Files          1514     1129      -385     
     Lines         75167    53390    -21777     
     Branches      10966     8037     -2929     
   =============================================
   + Hits          24203    37308    +13105     
   + Misses        48870    13441    -35429     
   - Partials       2094     2641      +547     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `69.87% <59.64%> (?)` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ore/plan/AcquireReleaseColumnsSegmentPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FjcXVpcmVSZWxlYXNlQ29sdW1uc1NlZ21lbnRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | `72.17% <0.00%> (+5.21%)` | :arrow_up: |
   | [...gment/index/readers/BitmapInvertedIndexReader.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `100.00% <ø> (+100.00%)` | :arrow_up: |
   | [...al/segment/index/readers/RangeIndexReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvUmFuZ2VJbmRleFJlYWRlckltcGwuamF2YQ==) | `89.02% <ø> (+89.02%)` | :arrow_up: |
   | [...ava/org/apache/pinot/segment/spi/FetchContext.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0ZldGNoQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `79.48% <33.33%> (-9.56%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `80.95% <79.41%> (+17.66%)` | :arrow_up: |
   | [...gment/index/readers/NullValueVectorReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvTnVsbFZhbHVlVmVjdG9yUmVhZGVySW1wbC5qYXZh) | `100.00% <100.00%> (+100.00%)` | :arrow_up: |
   | [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1311 more](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [240bcb8...a1e2489](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] npawar commented on a change in pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
npawar commented on a change in pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#discussion_r718780263



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       actually i intentionally put it inside the try, so that we release no matter what happens in acquire or execution




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7493](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (f09d35a) into [master](https://codecov.io/gh/apache/pinot/commit/debb132575ffe3485ba3f61b19fdaa6ce33da425?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (debb132) will **decrease** coverage by `8.34%`.
   > The diff coverage is `56.60%`.
   
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7493/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7493      +/-   ##
   ============================================
   - Coverage     72.01%   63.67%   -8.35%     
   + Complexity     3405     3327      -78     
   ============================================
     Files          1523     1514       -9     
     Lines         75531    75161     -370     
     Branches      11010    10966      -44     
   ============================================
   - Hits          54396    47861    -6535     
   - Misses        17468    23847    +6379     
   + Partials       3667     3453     -214     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `30.46% <41.50%> (-0.16%)` | :arrow_down: |
   | integration2 | `?` | |
   | unittests1 | `69.87% <56.60%> (+0.05%)` | :arrow_up: |
   | unittests2 | `?` | |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...ore/plan/AcquireReleaseColumnsSegmentPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FjcXVpcmVSZWxlYXNlQ29sdW1uc1NlZ21lbnRQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...pinot/core/plan/maker/InstancePlanMakerImplV2.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL21ha2VyL0luc3RhbmNlUGxhbk1ha2VySW1wbFYyLmphdmE=) | `76.19% <0.00%> (-6.35%)` | :arrow_down: |
   | [...gment/index/readers/BitmapInvertedIndexReader.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvQml0bWFwSW52ZXJ0ZWRJbmRleFJlYWRlci5qYXZh) | `100.00% <ø> (+7.14%)` | :arrow_up: |
   | [...al/segment/index/readers/RangeIndexReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvUmFuZ2VJbmRleFJlYWRlckltcGwuamF2YQ==) | `89.02% <ø> (+3.73%)` | :arrow_up: |
   | [...ava/org/apache/pinot/segment/spi/FetchContext.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL0ZldGNoQ29udGV4dC5qYXZh) | `0.00% <0.00%> (ø)` | |
   | [...not/core/operator/combine/BaseCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0Jhc2VDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `87.83% <33.33%> (-4.92%)` | :arrow_down: |
   | [...perator/combine/GroupByOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlPcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `81.17% <78.12%> (-3.83%)` | :arrow_down: |
   | [...gment/index/readers/NullValueVectorReaderImpl.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L3JlYWRlcnMvTnVsbFZhbHVlVmVjdG9yUmVhZGVySW1wbC5qYXZh) | `100.00% <100.00%> (ø)` | |
   | [.../pinot/controller/util/SegmentCompletionUtils.java](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci91dGlsL1NlZ21lbnRDb21wbGV0aW9uVXRpbHMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [264 more](https://codecov.io/gh/apache/pinot/pull/7493/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [debb132...f09d35a](https://codecov.io/gh/apache/pinot/pull/7493?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] codecov-commenter edited a comment on pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#issuecomment-929741943






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org

[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7493: Move acquire/release to right before/after execution call

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7493:
URL: https://github.com/apache/pinot/pull/7493#discussion_r718768587



##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/combine/BaseCombineOperator.java
##########
@@ -153,7 +154,18 @@ public void runJob() {
    */
   protected void processSegments(int threadIndex) {
     for (int operatorIndex = threadIndex; operatorIndex < _numOperators; operatorIndex += _numThreads) {
-      IntermediateResultsBlock resultsBlock = (IntermediateResultsBlock) _operators.get(operatorIndex).nextBlock();
+      Operator operator = _operators.get(operatorIndex);
+      IntermediateResultsBlock resultsBlock;
+      try {
+        if (operator instanceof AcquireReleaseColumnsSegmentOperator) {
+          ((AcquireReleaseColumnsSegmentOperator) operator).acquire();
+        }

Review comment:
       I think this should be outside of the try block, so that we don't release the operator if acquire fails. Same for other places

##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/segment/index/readers/NullValueVectorReaderImpl.java
##########
@@ -25,18 +25,20 @@
 
 public class NullValueVectorReaderImpl implements NullValueVectorReader {
 
-  private final ImmutableRoaringBitmap _nullBitmap;
+  private final PinotDataBuffer _dataBuffer;
 
   public NullValueVectorReaderImpl(PinotDataBuffer dataBuffer) {
-    _nullBitmap = new ImmutableRoaringBitmap(dataBuffer.toDirectByteBuffer(0, (int) dataBuffer.size()));
+    _dataBuffer = dataBuffer;
   }
 
   public boolean isNull(int docId) {
-    return _nullBitmap.contains(docId);
+    ImmutableRoaringBitmap nullBitmap =
+        new ImmutableRoaringBitmap(_dataBuffer.toDirectByteBuffer(0, (int) _dataBuffer.size()));
+    return nullBitmap.contains(docId);

Review comment:
       (nit)
   ```suggestion
     @Override
     public boolean isNull(int docId) {
       return getNullBitmap().contains(docId);
     }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org