You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/30 22:07:44 UTC

[GitHub] [pinot] Jackie-Jiang opened a new pull request #7844: Make upsert inner segment update atomic

Jackie-Jiang opened a new pull request #7844:
URL: https://github.com/apache/pinot/pull/7844


   When updating the upsert metadata, if the update (invalidate an old record, and validate a new record) applies to the same segment, make the update atomic to reduce the temporary inconsistency.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (183c30c) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `6.40%`.
   > The diff coverage is `68.87%`.
   
   > :exclamation: Current head 183c30c differs from pull request most recent head 82aab9e. Consider uploading reports for the commit 82aab9e to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7844/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@             Coverage Diff              @@
   ##             master    #7844      +/-   ##
   ============================================
   - Coverage     71.68%   65.28%   -6.41%     
   - Complexity     4089     4091       +2     
   ============================================
     Files          1579     1535      -44     
     Lines         80819    79223    -1596     
     Branches      12017    11865     -152     
   ============================================
   - Hits          57932    51717    -6215     
   - Misses        18982    23833    +4851     
   + Partials       3905     3673     -232     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `68.76% <70.96%> (+0.08%)` | :arrow_up: |
   | unittests2 | `14.57% <2.20%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `68.20% <ø> (-10.85%)` | :arrow_down: |
   | [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `76.96% <ø> (-3.87%)` | :arrow_down: |
   | [...ot/controller/helix/core/util/ZKMetadataUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3V0aWwvWktNZXRhZGF0YVV0aWxzLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...in/java/org/apache/pinot/core/common/Operator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <ø> (-82.95%)` | :arrow_down: |
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | [...he/pinot/core/operator/BitmapDocIdSetOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9CaXRtYXBEb2NJZFNldE9wZXJhdG9yLmphdmE=) | `68.18% <0.00%> (-6.82%)` | :arrow_down: |
   | [.../core/operator/combine/GroupByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `73.91% <0.00%> (-1.09%)` | :arrow_down: |
   | [...nMaxValueBasedSelectionOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL01pbk1heFZhbHVlQmFzZWRTZWxlY3Rpb25PcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `60.60% <0.00%> (-11.15%)` | :arrow_down: |
   | [...or/dociditerators/ExpressionScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9FeHByZXNzaW9uU2NhbkRvY0lkSXRlcmF0b3IuamF2YQ==) | `47.18% <0.00%> (-19.25%)` | :arrow_down: |
   | ... and [421 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...82aab9e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang merged pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7844:
URL: https://github.com/apache/pinot/pull/7844


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759747251



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       Also, don’t be concerned about adding null checks, they aren’t compiled the way you see them - they get speculatively optimised away based on profiling data collected in lower tiers. So this is not more efficient than a null check which never fails, it’s exactly as efficient but with more bytecode weight. This is a good related read on the topic (it’s about implicit null checks which raise NPEs, but the same mechanism applies to branches not taken): https://shipilev.net/jvm/anatomy-quarks/25-implicit-null-checks/




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759734412



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       `assert` is used to suppress the IDE warning on potential NPE without adding overhead. It generally means it is not possible to have `null` validDocIds. I don't think it is worth commenting though




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759780655



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;
-              currentSegment.getValidDocIds().remove(currentRecordLocation.getDocId());
-              assert segment.getValidDocIds() != null;
-              segment.getValidDocIds().add(recordInfo._docId);
+              if (segment == currentSegment) {
+                validDocIds.replace(currentDocId, recordInfo._docId);
+              } else {
+                assert currentSegment.getValidDocIds() != null;
+                currentSegment.getValidDocIds().remove(currentDocId);

Review comment:
       nit, use validDocIds?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759760648



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       @yupeng9 That shouldn't matter as it just reads a member variable from the segment into a local variable. I simply extracted that common logic ahead to avoid making multiple assertions.
   
   @richardstartin Great article, thanks for sharing. In that case should we prefer `Objects.requireNonNull(currentSegment.getValidDocIds()).remove(currentDocId);` as IDE suggested to suppress the warning?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759743586



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       Assert statements generally do add some overhead because they add about 10 bytes of bytecode each, which interferes with hotspot’s  bytecode size based inlining heuristics and may result in a method containing asserts not being inlined. I absolutely wouldn’t suggest removing it, but it’s important to know they aren’t free.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759711050



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       why do we need to move this up? can you add some comment?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759738827



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       I meant `segment.getValidDocIds()` is called earlier




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759747251



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       Also, don’t be concerned about adding null checks, they aren’t compiled the way you see them - they get speculatively optimised away based on profiling data collected in lower tiers. So this is not more efficient than a null check which never fails, it’s exactly as efficient but with more bytecode weight. This is a good read on the topic: https://shipilev.net/jvm/anatomy-quarks/25-implicit-null-checks/




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r760534077



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       @richardstartin The method can return `null` for non-upsert case, but in the current context it can never be `null`. `Objects.requireNonNull()` is basically the concise version null check without putting the if/else block




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter edited a comment on pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (183c30c) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `57.10%`.
   > The diff coverage is `2.20%`.
   
   > :exclamation: Current head 183c30c differs from pull request most recent head 82aab9e. Consider uploading reports for the commit 82aab9e to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7844/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7844       +/-   ##
   =============================================
   - Coverage     71.68%   14.57%   -57.11%     
   + Complexity     4089       80     -4009     
   =============================================
     Files          1579     1535       -44     
     Lines         80819    79223     -1596     
     Branches      12017    11865      -152     
   =============================================
   - Hits          57932    11550    -46382     
   - Misses        18982    66821    +47839     
   + Partials       3905      852     -3053     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.57% <2.20%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...not/common/metadata/segment/SegmentZKMetadata.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvc2VnbWVudC9TZWdtZW50WktNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (-74.56%)` | :arrow_down: |
   | [...t/common/response/broker/BrokerResponseNative.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlLmphdmE=) | `0.00% <0.00%> (-95.49%)` | :arrow_down: |
   | [...java/org/apache/pinot/common/utils/DataSchema.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVNjaGVtYS5qYXZh) | `0.00% <0.00%> (-79.24%)` | :arrow_down: |
   | [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `0.00% <0.00%> (-91.89%)` | :arrow_down: |
   | [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `68.20% <ø> (-10.85%)` | :arrow_down: |
   | [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `76.96% <ø> (-3.87%)` | :arrow_down: |
   | [...ot/controller/helix/core/util/ZKMetadataUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3V0aWwvWktNZXRhZGF0YVV0aWxzLmphdmE=) | `100.00% <ø> (ø)` | |
   | [...in/java/org/apache/pinot/core/common/Operator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | |
   | [...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <ø> (-82.95%)` | :arrow_down: |
   | [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
   | ... and [1314 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...82aab9e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759986534



##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
       }
     }
 
+    ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+    assert validDocIds != null;
     _result = record;
-    _primaryKeyToRecordLocationMap
-        .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+    _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+        (primaryKey, currentRecordLocation) -> {
           if (currentRecordLocation != null) {
             // Existing primary key
 
-            // Update the record location when the new comparison value is greater than or equal to the current value
-            // . Update
-            // the record location when there is a tie to keep the newer record.
+            // Update the record location when the new comparison value is greater than or equal to the current value.
+            // Update the record location when there is a tie to keep the newer record.
             if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
               IndexSegment currentSegment = currentRecordLocation.getSegment();
+              int currentDocId = currentRecordLocation.getDocId();
               if (_partialUpsertHandler != null) {
                 // Partial upsert
                 _reuse.clear();
-                GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+                GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
                 _result = _partialUpsertHandler.merge(previousRecord, record);
               }
-              assert currentSegment.getValidDocIds() != null;

Review comment:
       @Jackie-Jiang what about just a null check? If it really can't be null then it doesn't change the logic. Also, if it _really_ can't be null, why not annotate the method `@Nonnull`?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org


[GitHub] [pinot] codecov-commenter commented on pull request #7844: Make upsert inner segment update atomic

Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519


   # [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
   > Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a9bfa2b) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `57.05%`.
   > The diff coverage is `0.00%`.
   
   > :exclamation: Current head a9bfa2b differs from pull request most recent head 7f1602e. Consider uploading reports for the commit 7f1602e to get more accurate results
   [![Impacted file tree graph](https://codecov.io/gh/apache/pinot/pull/7844/graphs/tree.svg?width=650&height=150&src=pr&token=4ibza2ugkz&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   
   ```diff
   @@              Coverage Diff              @@
   ##             master    #7844       +/-   ##
   =============================================
   - Coverage     71.68%   14.62%   -57.06%     
   + Complexity     4089       80     -4009     
   =============================================
     Files          1579     1534       -45     
     Lines         80819    78950     -1869     
     Branches      12017    11814      -203     
   =============================================
   - Hits          57932    11548    -46384     
   - Misses        18982    66545    +47563     
   + Partials       3905      857     -3048     
   ```
   
   | Flag | Coverage Δ | |
   |---|---|---|
   | integration1 | `?` | |
   | integration2 | `?` | |
   | unittests1 | `?` | |
   | unittests2 | `14.62% <0.00%> (+0.03%)` | :arrow_up: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
   |---|---|---|
   | [...t/local/upsert/PartitionUpsertMetadataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvUGFydGl0aW9uVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-74.04%)` | :arrow_down: |
   | [...ment/spi/index/ThreadSafeMutableRoaringBitmap.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L1RocmVhZFNhZmVNdXRhYmxlUm9hcmluZ0JpdG1hcC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
   | ... and [1252 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
   
   ------
   
   [Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
   > `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
   > Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...7f1602e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org