You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2021/11/30 22:07:44 UTC
[GitHub] [pinot] Jackie-Jiang opened a new pull request #7844: Make upsert inner segment update atomic
Jackie-Jiang opened a new pull request #7844:
URL: https://github.com/apache/pinot/pull/7844
When updating the upsert metadata, if the update (invalidate an old record, and validate a new record) applies to the same segment, make the update atomic to reduce the temporary inconsistency.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (183c30c) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `6.40%`.
> The diff coverage is `68.87%`.
> :exclamation: Current head 183c30c differs from pull request most recent head 82aab9e. Consider uploading reports for the commit 82aab9e to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7844 +/- ##
============================================
- Coverage 71.68% 65.28% -6.41%
- Complexity 4089 4091 +2
============================================
Files 1579 1535 -44
Lines 80819 79223 -1596
Branches 12017 11865 -152
============================================
- Hits 57932 51717 -6215
- Misses 18982 23833 +4851
+ Partials 3905 3673 -232
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.76% <70.96%> (+0.08%)` | :arrow_up: |
| unittests2 | `14.57% <2.20%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `68.20% <ø> (-10.85%)` | :arrow_down: |
| [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `76.96% <ø> (-3.87%)` | :arrow_down: |
| [...ot/controller/helix/core/util/ZKMetadataUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3V0aWwvWktNZXRhZGF0YVV0aWxzLmphdmE=) | `100.00% <ø> (ø)` | |
| [...in/java/org/apache/pinot/core/common/Operator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <ø> (-82.95%)` | :arrow_down: |
| [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...he/pinot/core/operator/BitmapDocIdSetOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9CaXRtYXBEb2NJZFNldE9wZXJhdG9yLmphdmE=) | `68.18% <0.00%> (-6.82%)` | :arrow_down: |
| [.../core/operator/combine/GroupByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL0dyb3VwQnlDb21iaW5lT3BlcmF0b3IuamF2YQ==) | `73.91% <0.00%> (-1.09%)` | :arrow_down: |
| [...nMaxValueBasedSelectionOrderByCombineOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9jb21iaW5lL01pbk1heFZhbHVlQmFzZWRTZWxlY3Rpb25PcmRlckJ5Q29tYmluZU9wZXJhdG9yLmphdmE=) | `60.60% <0.00%> (-11.15%)` | :arrow_down: |
| [...or/dociditerators/ExpressionScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9FeHByZXNzaW9uU2NhbkRvY0lkSXRlcmF0b3IuamF2YQ==) | `47.18% <0.00%> (-19.25%)` | :arrow_down: |
| ... and [421 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...82aab9e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang merged pull request #7844:
URL: https://github.com/apache/pinot/pull/7844
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759747251
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
Also, don’t be concerned about adding null checks, they aren’t compiled the way you see them - they get speculatively optimised away based on profiling data collected in lower tiers. So this is not more efficient than a null check which never fails, it’s exactly as efficient but with more bytecode weight. This is a good related read on the topic (it’s about implicit null checks which raise NPEs, but the same mechanism applies to branches not taken): https://shipilev.net/jvm/anatomy-quarks/25-implicit-null-checks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759734412
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
`assert` is used to suppress the IDE warning on potential NPE without adding overhead. It generally means it is not possible to have `null` validDocIds. I don't think it is worth commenting though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759780655
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
- currentSegment.getValidDocIds().remove(currentRecordLocation.getDocId());
- assert segment.getValidDocIds() != null;
- segment.getValidDocIds().add(recordInfo._docId);
+ if (segment == currentSegment) {
+ validDocIds.replace(currentDocId, recordInfo._docId);
+ } else {
+ assert currentSegment.getValidDocIds() != null;
+ currentSegment.getValidDocIds().remove(currentDocId);
Review comment:
nit, use validDocIds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759760648
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
@yupeng9 That shouldn't matter as it just reads a member variable from the segment into a local variable. I simply extracted that common logic ahead to avoid making multiple assertions.
@richardstartin Great article, thanks for sharing. In that case should we prefer `Objects.requireNonNull(currentSegment.getValidDocIds()).remove(currentDocId);` as IDE suggested to suppress the warning?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759743586
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
Assert statements generally do add some overhead because they add about 10 bytes of bytecode each, which interferes with hotspot’s bytecode size based inlining heuristics and may result in a method containing asserts not being inlined. I absolutely wouldn’t suggest removing it, but it’s important to know they aren’t free.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759711050
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
why do we need to move this up? can you add some comment?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] yupeng9 commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
yupeng9 commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759738827
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
I meant `segment.getValidDocIds()` is called earlier
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759747251
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
Also, don’t be concerned about adding null checks, they aren’t compiled the way you see them - they get speculatively optimised away based on profiling data collected in lower tiers. So this is not more efficient than a null check which never fails, it’s exactly as efficient but with more bytecode weight. This is a good read on the topic: https://shipilev.net/jvm/anatomy-quarks/25-implicit-null-checks/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r760534077
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
@richardstartin The method can return `null` for non-upsert case, but in the current context it can never be `null`. `Objects.requireNonNull()` is basically the concise version null check without putting the if/else block
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (183c30c) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `57.10%`.
> The diff coverage is `2.20%`.
> :exclamation: Current head 183c30c differs from pull request most recent head 82aab9e. Consider uploading reports for the commit 82aab9e to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7844 +/- ##
=============================================
- Coverage 71.68% 14.57% -57.11%
+ Complexity 4089 80 -4009
=============================================
Files 1579 1535 -44
Lines 80819 79223 -1596
Branches 12017 11865 -152
=============================================
- Hits 57932 11550 -46382
- Misses 18982 66821 +47839
+ Partials 3905 852 -3053
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.57% <2.20%> (-0.02%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...not/common/metadata/segment/SegmentZKMetadata.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0YWRhdGEvc2VnbWVudC9TZWdtZW50WktNZXRhZGF0YS5qYXZh) | `0.00% <0.00%> (-74.56%)` | :arrow_down: |
| [...t/common/response/broker/BrokerResponseNative.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vcmVzcG9uc2UvYnJva2VyL0Jyb2tlclJlc3BvbnNlTmF0aXZlLmphdmE=) | `0.00% <0.00%> (-95.49%)` | :arrow_down: |
| [...java/org/apache/pinot/common/utils/DataSchema.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVNjaGVtYS5qYXZh) | `0.00% <0.00%> (-79.24%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `0.00% <0.00%> (-91.89%)` | :arrow_down: |
| [.../core/realtime/PinotLLCRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90TExDUmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `68.20% <ø> (-10.85%)` | :arrow_down: |
| [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `76.96% <ø> (-3.87%)` | :arrow_down: |
| [...ot/controller/helix/core/util/ZKMetadataUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3V0aWwvWktNZXRhZGF0YVV0aWxzLmphdmE=) | `100.00% <ø> (ø)` | |
| [...in/java/org/apache/pinot/core/common/Operator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...manager/realtime/HLRealtimeSegmentDataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL21hbmFnZXIvcmVhbHRpbWUvSExSZWFsdGltZVNlZ21lbnREYXRhTWFuYWdlci5qYXZh) | `0.00% <ø> (-82.95%)` | :arrow_down: |
| [...operator/AcquireReleaseColumnsSegmentOperator.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9BY3F1aXJlUmVsZWFzZUNvbHVtbnNTZWdtZW50T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| ... and [1314 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...82aab9e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#discussion_r759986534
##########
File path: pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/PartitionUpsertMetadataManager.java
##########
@@ -186,27 +182,32 @@ public GenericRow updateRecord(IndexSegment segment, RecordInfo recordInfo, Gene
}
}
+ ThreadSafeMutableRoaringBitmap validDocIds = segment.getValidDocIds();
+ assert validDocIds != null;
_result = record;
- _primaryKeyToRecordLocationMap
- .compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction), (primaryKey, currentRecordLocation) -> {
+ _primaryKeyToRecordLocationMap.compute(hashPrimaryKey(recordInfo._primaryKey, _hashFunction),
+ (primaryKey, currentRecordLocation) -> {
if (currentRecordLocation != null) {
// Existing primary key
- // Update the record location when the new comparison value is greater than or equal to the current value
- // . Update
- // the record location when there is a tie to keep the newer record.
+ // Update the record location when the new comparison value is greater than or equal to the current value.
+ // Update the record location when there is a tie to keep the newer record.
if (recordInfo._comparisonValue.compareTo(currentRecordLocation.getComparisonValue()) >= 0) {
IndexSegment currentSegment = currentRecordLocation.getSegment();
+ int currentDocId = currentRecordLocation.getDocId();
if (_partialUpsertHandler != null) {
// Partial upsert
_reuse.clear();
- GenericRow previousRecord = currentSegment.getRecord(currentRecordLocation.getDocId(), _reuse);
+ GenericRow previousRecord = currentSegment.getRecord(currentDocId, _reuse);
_result = _partialUpsertHandler.merge(previousRecord, record);
}
- assert currentSegment.getValidDocIds() != null;
Review comment:
@Jackie-Jiang what about just a null check? If it really can't be null then it doesn't change the logic. Also, if it _really_ can't be null, why not annotate the method `@Nonnull`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #7844: Make upsert inner segment update atomic
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7844:
URL: https://github.com/apache/pinot/pull/7844#issuecomment-983082519
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7844](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a9bfa2b) into [master](https://codecov.io/gh/apache/pinot/commit/eff570b251107e5045102b66b5fe7fb97de68c36?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (eff570b) will **decrease** coverage by `57.05%`.
> The diff coverage is `0.00%`.
> :exclamation: Current head a9bfa2b differs from pull request most recent head 7f1602e. Consider uploading reports for the commit 7f1602e to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7844 +/- ##
=============================================
- Coverage 71.68% 14.62% -57.06%
+ Complexity 4089 80 -4009
=============================================
Files 1579 1534 -45
Lines 80819 78950 -1869
Branches 12017 11814 -203
=============================================
- Hits 57932 11548 -46384
- Misses 18982 66545 +47563
+ Partials 3905 857 -3048
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.62% <0.00%> (+0.03%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/local/upsert/PartitionUpsertMetadataManager.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvUGFydGl0aW9uVXBzZXJ0TWV0YWRhdGFNYW5hZ2VyLmphdmE=) | `0.00% <0.00%> (-74.04%)` | :arrow_down: |
| [...ment/spi/index/ThreadSafeMutableRoaringBitmap.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2luZGV4L1RocmVhZFNhZmVNdXRhYmxlUm9hcmluZ0JpdG1hcC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ain/java/org/apache/pinot/core/data/table/Key.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL0tleS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/data/table/Record.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9kYXRhL3RhYmxlL1JlY29yZC5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../java/org/apache/pinot/core/util/GroupByUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL0dyb3VwQnlVdGlscy5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [1252 more](https://codecov.io/gh/apache/pinot/pull/7844/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [eff570b...7f1602e](https://codecov.io/gh/apache/pinot/pull/7844?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org