You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "tibrewalpratik17 (via GitHub)" <gi...@apache.org> on 2024/07/10 22:03:37 UTC
[PR] Remove preloading-type nature of enableSnapshot config [pinot]
tibrewalpratik17 opened a new pull request, #13579:
URL: https://github.com/apache/pinot/pull/13579
Addresses #13466.
When `enableSnapshot` is enabled, we only iterate through the validDocIds during segment-addition. This is counter-intuitive as we have `enablePreload` for the same.
In order to make any upsert config change we have to do 2 server restarts currently (first by disabling enableSnapshot and then to enable it again). We can make this restart behaviour only limited to `enablePreload` method rather than tying it with `enableSnapshot` config as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1675492201
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -432,30 +432,11 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
String segmentName = segment.getSegmentName();
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
-
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
Review Comment:
@deemoliu this method is not called at all in case the segment is out of metadataTTL window. Ref: https://github.com/apache/pinot/blob/dacc6d06907c44e83721454f1090e5f00c824f15/pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java#L389-L406
We are returning in the caller method above for out of metadataTTL segments.
This flow affects the segments which are in metadataTTL window, in that case it is okay to read the whole segment and not just from snapshot.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1676165564
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -433,29 +433,10 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
- if (validDocIds != null && validDocIds.isEmpty()) {
- _logger.info("Skip adding segment: {} without valid doc, current primary key count: {}",
- segment.getSegmentName(), getNumPrimaryKeys());
- segment.enableUpsert(this, new ThreadSafeMutableRoaringBitmap(), null);
- return;
- }
- } else {
- validDocIds = null;
- segment.deleteValidDocIdsSnapshot();
- }
-
try (UpsertUtils.RecordInfoReader recordInfoReader = new UpsertUtils.RecordInfoReader(segment, _primaryKeyColumns,
_comparisonColumns, _deleteRecordColumn)) {
- Iterator<RecordInfo> recordInfoIterator;
- if (validDocIds != null) {
- recordInfoIterator = UpsertUtils.getRecordInfoIterator(recordInfoReader, validDocIds);
- } else {
- recordInfoIterator =
- UpsertUtils.getRecordInfoIterator(recordInfoReader, segment.getSegmentMetadata().getTotalDocs());
- }
+ Iterator<RecordInfo> recordInfoIterator =
Review Comment:
Done! I can think of a scenario where this would make sense. Say, a within-TTL segment gets snapshot removed (without this condition) but it restarts after moving out-of-TTL but before the next segment commit. This would help in that case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1674344090
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -433,29 +433,10 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
- if (validDocIds != null && validDocIds.isEmpty()) {
- _logger.info("Skip adding segment: {} without valid doc, current primary key count: {}",
- segment.getSegmentName(), getNumPrimaryKeys());
- segment.enableUpsert(this, new ThreadSafeMutableRoaringBitmap(), null);
- return;
- }
- } else {
- validDocIds = null;
- segment.deleteValidDocIdsSnapshot();
- }
-
try (UpsertUtils.RecordInfoReader recordInfoReader = new UpsertUtils.RecordInfoReader(segment, _primaryKeyColumns,
_comparisonColumns, _deleteRecordColumn)) {
- Iterator<RecordInfo> recordInfoIterator;
- if (validDocIds != null) {
- recordInfoIterator = UpsertUtils.getRecordInfoIterator(recordInfoReader, validDocIds);
- } else {
- recordInfoIterator =
- UpsertUtils.getRecordInfoIterator(recordInfoReader, segment.getSegmentMetadata().getTotalDocs());
- }
+ Iterator<RecordInfo> recordInfoIterator =
Review Comment:
Do we really need to remove it 🤔 ? Can there be a scenario where valid doc ids of snapshot differs from the loading segment? I am fine with removing it as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "tibrewalpratik17 (via GitHub)" <gi...@apache.org>.
tibrewalpratik17 commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1674344090
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -433,29 +433,10 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
- if (validDocIds != null && validDocIds.isEmpty()) {
- _logger.info("Skip adding segment: {} without valid doc, current primary key count: {}",
- segment.getSegmentName(), getNumPrimaryKeys());
- segment.enableUpsert(this, new ThreadSafeMutableRoaringBitmap(), null);
- return;
- }
- } else {
- validDocIds = null;
- segment.deleteValidDocIdsSnapshot();
- }
-
try (UpsertUtils.RecordInfoReader recordInfoReader = new UpsertUtils.RecordInfoReader(segment, _primaryKeyColumns,
_comparisonColumns, _deleteRecordColumn)) {
- Iterator<RecordInfo> recordInfoIterator;
- if (validDocIds != null) {
- recordInfoIterator = UpsertUtils.getRecordInfoIterator(recordInfoReader, validDocIds);
- } else {
- recordInfoIterator =
- UpsertUtils.getRecordInfoIterator(recordInfoReader, segment.getSegmentMetadata().getTotalDocs());
- }
+ Iterator<RecordInfo> recordInfoIterator =
Review Comment:
Do we really need to remove it 🤔 ? Can there be a scenario where valid doc ids of snapshot differs from the loading segment? I am fine with removing it as well as for partial-upsert we will load it upfront again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1676147972
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -433,29 +433,10 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
- if (validDocIds != null && validDocIds.isEmpty()) {
- _logger.info("Skip adding segment: {} without valid doc, current primary key count: {}",
- segment.getSegmentName(), getNumPrimaryKeys());
- segment.enableUpsert(this, new ThreadSafeMutableRoaringBitmap(), null);
- return;
- }
- } else {
- validDocIds = null;
- segment.deleteValidDocIdsSnapshot();
- }
-
try (UpsertUtils.RecordInfoReader recordInfoReader = new UpsertUtils.RecordInfoReader(segment, _primaryKeyColumns,
_comparisonColumns, _deleteRecordColumn)) {
- Iterator<RecordInfo> recordInfoIterator;
- if (validDocIds != null) {
- recordInfoIterator = UpsertUtils.getRecordInfoIterator(recordInfoReader, validDocIds);
- } else {
- recordInfoIterator =
- UpsertUtils.getRecordInfoIterator(recordInfoReader, segment.getSegmentMetadata().getTotalDocs());
- }
+ Iterator<RecordInfo> recordInfoIterator =
Review Comment:
maybe let’s wrap this removeSnapshot method in a ‘if(!enableSnapshot) {}’ (as reminded by Qiaochu’s question on metadataTTL, although that shouldn’t be affected as you analyzed)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "deemoliu (via GitHub)" <gi...@apache.org>.
deemoliu commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1674703351
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -432,30 +432,11 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
String segmentName = segment.getSegmentName();
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
-
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
Review Comment:
can you please check if metadata TTL is enabled. i don't think we can remove it if metadata TTL is enabled.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1678236440
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -432,30 +432,11 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
String segmentName = segment.getSegmentName();
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
-
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
Review Comment:
yeah, I think calling removeSnapshot inside ‘if(!enableSnapshot) {}’ would help avoid this, and thanks for your early questions for the reminder.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince merged PR #13579:
URL: https://github.com/apache/pinot/pull/13579
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "deemoliu (via GitHub)" <gi...@apache.org>.
deemoliu commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1678200981
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -432,30 +432,11 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
String segmentName = segment.getSegmentName();
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
-
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
Review Comment:
I recall the prior discussions with @klsince and @Jackie-Jiang, upsert TTL (metadataTTL) snapshotting supposed to be atomic at partition level instead of segment level. We need to avoid keeping some snapshot meanwhile delete the other snapshots for the same partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #13579:
URL: https://github.com/apache/pinot/pull/13579#issuecomment-2221645999
## [Codecov](https://app.codecov.io/gh/apache/pinot/pull/13579?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
Attention: Patch coverage is `0%` with `2 lines` in your changes missing coverage. Please review.
> Project coverage is 0.00%. Comparing base [(`59551e4`)](https://app.codecov.io/gh/apache/pinot/commit/59551e45224f1535c4863fd577622b37366ccc97?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) to head [(`b8a2025`)](https://app.codecov.io/gh/apache/pinot/commit/b8a2025fb0c16d9a271912bd939b23f28e7d7348?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
> Report is 740 commits behind head on master.
| [Files](https://app.codecov.io/gh/apache/pinot/pull/13579?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...cal/upsert/BasePartitionUpsertMetadataManager.java](https://app.codecov.io/gh/apache/pinot/pull/13579?src=pr&el=tree&filepath=pinot-segment-local%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fpinot%2Fsegment%2Flocal%2Fupsert%2FBasePartitionUpsertMetadataManager.java&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91cHNlcnQvQmFzZVBhcnRpdGlvblVwc2VydE1ldGFkYXRhTWFuYWdlci5qYXZh) | 0.00% | [2 Missing :warning: ](https://app.codecov.io/gh/apache/pinot/pull/13579?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
> :exclamation: There is a different number of reports uploaded between BASE (59551e4) and HEAD (b8a2025). Click for more details.
>
> <details><summary>HEAD has 53 uploads less than BASE</summary>
>
>| Flag | BASE (59551e4) | HEAD (b8a2025) |
>|------|------|------|
>|integration|7|1|
>|integration2|3|1|
>|temurin|12|1|
>|java-21|7|1|
>|skip-bytebuffers-true|3|1|
>|skip-bytebuffers-false|7|0|
>|unittests|5|0|
>|unittests1|2|0|
>|java-11|5|0|
>|unittests2|3|0|
>|integration1|2|0|
>|custom-integration1|2|0|
></details>
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #13579 +/- ##
=============================================
- Coverage 61.75% 0.00% -61.76%
=============================================
Files 2436 2483 +47
Lines 133233 137261 +4028
Branches 20636 21307 +671
=============================================
- Hits 82274 0 -82274
- Misses 44911 137261 +92350
+ Partials 6048 0 -6048
```
| [Flag](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [custom-integration1](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-0.01%)` | :arrow_down: |
| [integration1](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [integration2](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (ø)` | |
| [java-11](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [java-21](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.63%)` | :arrow_down: |
| [skip-bytebuffers-false](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [skip-bytebuffers-true](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-27.73%)` | :arrow_down: |
| [temurin](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `0.00% <0.00%> (-61.76%)` | :arrow_down: |
| [unittests](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests1](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
| [unittests2](https://app.codecov.io/gh/apache/pinot/pull/13579/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/pinot/pull/13579?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "deemoliu (via GitHub)" <gi...@apache.org>.
deemoliu commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1674703351
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -432,30 +432,11 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
String segmentName = segment.getSegmentName();
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
-
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
Review Comment:
can you please check if metadata TTL is enabled. i don't think we can remove it if metadata TTL is enabled.
by removing these lines, when metadataTTL is enabled but preloading is not enabled, we will have missing valid docIDs for historical segments
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
Re: [PR] Remove preloading-type nature of enableSnapshot config [pinot]
Posted by "klsince (via GitHub)" <gi...@apache.org>.
klsince commented on code in PR #13579:
URL: https://github.com/apache/pinot/pull/13579#discussion_r1674337556
##########
pinot-segment-local/src/main/java/org/apache/pinot/segment/local/upsert/BasePartitionUpsertMetadataManager.java:
##########
@@ -433,29 +433,10 @@ protected void doAddSegment(ImmutableSegmentImpl segment) {
_logger.info("Adding segment: {}, current primary key count: {}", segmentName, getNumPrimaryKeys());
long startTimeMs = System.currentTimeMillis();
- MutableRoaringBitmap validDocIds;
- if (_enableSnapshot) {
- validDocIds = segment.loadValidDocIdsFromSnapshot();
- if (validDocIds != null && validDocIds.isEmpty()) {
- _logger.info("Skip adding segment: {} without valid doc, current primary key count: {}",
- segment.getSegmentName(), getNumPrimaryKeys());
- segment.enableUpsert(this, new ThreadSafeMutableRoaringBitmap(), null);
- return;
- }
- } else {
- validDocIds = null;
- segment.deleteValidDocIdsSnapshot();
- }
-
try (UpsertUtils.RecordInfoReader recordInfoReader = new UpsertUtils.RecordInfoReader(segment, _primaryKeyColumns,
_comparisonColumns, _deleteRecordColumn)) {
- Iterator<RecordInfo> recordInfoIterator;
- if (validDocIds != null) {
- recordInfoIterator = UpsertUtils.getRecordInfoIterator(recordInfoReader, validDocIds);
- } else {
- recordInfoIterator =
- UpsertUtils.getRecordInfoIterator(recordInfoReader, segment.getSegmentMetadata().getTotalDocs());
- }
+ Iterator<RecordInfo> recordInfoIterator =
Review Comment:
Should we try to remove snapshot as in the else branch above
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org