You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/02 16:29:17 UTC
[GitHub] [pinot] tuor713 opened a new pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
tuor713 opened a new pull request #7971:
URL: https://github.com/apache/pinot/pull/7971
## Description
Even with a single segment, count queries occasionally return too few results (off by one or two). Reason appears to be that in the `FilterPlanNode` the number of documents is snapshotted before the validity bitmap, which means in between documents can be upsert with the new valid documents beyond the snapshotted `_numDocs`.
This PR snapshots the bitmap first, then number of documents during the construction of the `FilterPlanNode`.
## Upgrade Notes
Does this PR prevent a zero down-time upgrade?
No
Does this PR fix a zero-downtime upgrade introduced earlier?
No
Does this PR otherwise need attention when creating release notes?
No
## Testing
- One unit test added to expose the snapshot inconsistency of `FilterNodePlan`.
- Tested manually in constructed example case (1 controller, 1 broker, 1 server, 1 RT table)
- 500k unique records initially inserted
- After that ingest 5mm updates after initial population is loaded, all on existing keys (segment size is 10mm so no segment roll)
- During ingestion continuously query `select count(*) from <table>`, which is expected to stay constant at 500k
- After this fix any inconsistencies of counts less than 500k disappear.
- The same held true in a larger real-life installation (4 servers, 10mm unique records): outside of segment roll queries consistently return the correct count of unique records
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r785424027
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -52,19 +52,30 @@
import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
public class FilterPlanNode implements PlanNode {
private final IndexSegment _indexSegment;
private final QueryContext _queryContext;
private final int _numDocs;
+ private final MutableRoaringBitmap _validDocIds;
// Cache the predicate evaluators
private final Map<Predicate, PredicateEvaluator> _predicateEvaluatorMap = new HashMap<>();
public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
_indexSegment = indexSegment;
_queryContext = queryContext;
+
+ // Snapshot validDocIds *before* _numDocs since otherwise newest upserts could get lost
Review comment:
In any case I know how to make snapshots faster in the library given that the bitmap is only ever appended to, so we could do that work if it ever showed up in profiles.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
richardstartin commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1022186514
Hi @tuor713 I have rerun the tests, but this is something which needs to be investigated.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1004468532
Formatting fixed for both files.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1015777935
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7971](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (fc6147f) into [master](https://codecov.io/gh/apache/pinot/commit/0bf408615cb8dc96defa80f4b8a0c302741dd40c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0bf4086) will **decrease** coverage by `3.10%`.
> The diff coverage is `75.96%`.
> :exclamation: Current head fc6147f differs from pull request most recent head 36b70aa. Consider uploading reports for the commit 36b70aa to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7971 +/- ##
============================================
- Coverage 71.32% 68.22% -3.11%
+ Complexity 4196 4119 -77
============================================
Files 1594 1201 -393
Lines 82507 59841 -22666
Branches 12304 9230 -3074
============================================
- Hits 58851 40828 -18023
+ Misses 19683 16143 -3540
+ Partials 3973 2870 -1103
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.22% <75.96%> (-0.02%)` | :arrow_down: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...a/org/apache/pinot/common/metrics/BrokerMeter.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJNZXRlci5qYXZh) | `92.85% <ø> (-7.15%)` | :arrow_down: |
| [...pache/pinot/core/util/SegmentRefreshSemaphore.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS91dGlsL1NlZ21lbnRSZWZyZXNoU2VtYXBob3JlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `87.12% <83.33%> (+0.26%)` | :arrow_up: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `86.79% <90.90%> (-0.50%)` | :arrow_down: |
| [...he/pinot/segment/local/utils/TableConfigUtils.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC91dGlscy9UYWJsZUNvbmZpZ1V0aWxzLmphdmE=) | `70.66% <95.00%> (+0.55%)` | :arrow_up: |
| [...ent/spi/loader/SegmentDirectoryLoaderRegistry.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1zcGkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL3Bpbm90L3NlZ21lbnQvc3BpL2xvYWRlci9TZWdtZW50RGlyZWN0b3J5TG9hZGVyUmVnaXN0cnkuamF2YQ==) | `79.31% <100.00%> (+8.93%)` | :arrow_up: |
| [...a/org/apache/pinot/common/metrics/MinionMeter.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25NZXRlci5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...g/apache/pinot/common/metrics/ControllerMeter.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Db250cm9sbGVyTWV0ZXIuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/BrokerQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJRdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../apache/pinot/common/metrics/MinionQueryPhase.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9NaW5pb25RdWVyeVBoYXNlLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [619 more](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0bf4086...36b70aa](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1086758174
Seems like the prerequisite PR https://github.com/apache/pinot/pull/8392 has been merged. @tuor713 , would you like to rebase and merge this one?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1076502363
@Jackie-Jiang Makes sense - in terms of procedure is it ok to take the commit 8018 then add the test on top as a separate commit (so authorship stays clear) and rebase or should I do it a different way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r787046585
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -52,19 +52,30 @@
import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
public class FilterPlanNode implements PlanNode {
private final IndexSegment _indexSegment;
private final QueryContext _queryContext;
private final int _numDocs;
+ private final MutableRoaringBitmap _validDocIds;
// Cache the predicate evaluators
private final Map<Predicate, PredicateEvaluator> _predicateEvaluatorMap = new HashMap<>();
public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
_indexSegment = indexSegment;
_queryContext = queryContext;
+
+ // Snapshot validDocIds *before* _numDocs since otherwise newest upserts could get lost
Review comment:
@richardstartin It is not only appended to, but also modified. We can probably keep the writer in the `ThreadSafeMutableRoaringBitmap` and only create the bitmap during the snapshot if that gives better performance
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r784333878
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -52,19 +52,30 @@
import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
public class FilterPlanNode implements PlanNode {
private final IndexSegment _indexSegment;
private final QueryContext _queryContext;
private final int _numDocs;
+ private final MutableRoaringBitmap _validDocIds;
// Cache the predicate evaluators
private final Map<Predicate, PredicateEvaluator> _predicateEvaluatorMap = new HashMap<>();
public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
_indexSegment = indexSegment;
_queryContext = queryContext;
+
+ // Snapshot validDocIds *before* _numDocs since otherwise newest upserts could get lost
Review comment:
Taking snapshot when creating the plan nodes can potentially cause performance issue since it is done single-threaded
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r777780785
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -73,20 +84,19 @@ public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
@Override
public BaseFilterOperator run() {
FilterContext filter = _queryContext.getFilter();
- ThreadSafeMutableRoaringBitmap validDocIds = _indexSegment.getValidDocIds();
- boolean applyValidDocIds = validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
+ boolean applyValidDocIds = _validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
if (filter != null) {
BaseFilterOperator filterOperator = constructPhysicalOperator(filter);
if (applyValidDocIds) {
BaseFilterOperator validDocFilter =
- new BitmapBasedFilterOperator(validDocIds.getMutableRoaringBitmap(), false, _numDocs);
+ new BitmapBasedFilterOperator(_validDocIds, false, _numDocs);
Review comment:
With regard to first question: what I am trying to fix is known ids disappearing temporarily from the results. Say there are 1000 positions with ids 1-1000 and they are all published at the start of the day and then updated during the day. Whenever I query regardless of what upsets are in place I would like to get exactly 1000 items back - one for each id.
What is currently happening is that numDocs is snapshotted - say 50,000 during the middle of the day - then concurrently some id, is updated and the new update is in row 50,001 and the bitmap is updated - lastly the validDocId bitmap is snapshotted. That means when the query runs row 50,001 is excluded due to the numDoc snapshot and the result is only 999 docs - one missing.
This kind of case - missing one or two records that were previously available - happens fairly reliable and is the result of snapshotting the total docs first while concurrent operations can still happen on the validity bitmap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1076801630
@tuor713 When merging, the commits will be squashed into one. Let me reopen 8018, and we can rebase this PR to add the test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1074537202
Please also take a look at the approach in #8018. Even though it is practically the same, snapshotting in `run()` is cleaner IMO
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1015777935
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7971](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (36b70aa) into [master](https://codecov.io/gh/apache/pinot/commit/0bf408615cb8dc96defa80f4b8a0c302741dd40c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0bf4086) will **decrease** coverage by `33.19%`.
> The diff coverage is `83.33%`.
[](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7971 +/- ##
=============================================
- Coverage 71.32% 38.13% -33.20%
+ Complexity 4196 80 -4116
=============================================
Files 1594 1595 +1
Lines 82507 82597 +90
Branches 12304 12327 +23
=============================================
- Hits 58851 31500 -27351
- Misses 19683 48672 +28989
+ Partials 3973 2425 -1548
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `29.06% <33.33%> (+0.07%)` | :arrow_up: |
| integration2 | `27.67% <83.33%> (-0.05%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `14.32% <0.00%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `64.35% <83.33%> (-22.52%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/spi/config/table/TableType.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1RhYmxlVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/spi/data/DimensionFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9EaW1lbnNpb25GaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/spi/data/readers/FileFormat.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0ZpbGVGb3JtYXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...org/apache/pinot/spi/config/table/QuotaConfig.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1F1b3RhQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [896 more](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0bf4086...36b70aa](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r785381347
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -52,19 +52,30 @@
import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
public class FilterPlanNode implements PlanNode {
private final IndexSegment _indexSegment;
private final QueryContext _queryContext;
private final int _numDocs;
+ private final MutableRoaringBitmap _validDocIds;
// Cache the predicate evaluators
private final Map<Predicate, PredicateEvaluator> _predicateEvaluatorMap = new HashMap<>();
public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
_indexSegment = indexSegment;
_queryContext = queryContext;
+
+ // Snapshot validDocIds *before* _numDocs since otherwise newest upserts could get lost
Review comment:
Oh you are right. We recently changed the logic of `FilterPlanNode`, and it is not constructed in the plan maker, so there is no practical difference now
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r785308831
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -52,19 +52,30 @@
import org.apache.pinot.segment.spi.index.reader.JsonIndexReader;
import org.apache.pinot.segment.spi.index.reader.NullValueVectorReader;
import org.apache.pinot.spi.exception.BadQueryRequestException;
+import org.roaringbitmap.buffer.MutableRoaringBitmap;
public class FilterPlanNode implements PlanNode {
private final IndexSegment _indexSegment;
private final QueryContext _queryContext;
private final int _numDocs;
+ private final MutableRoaringBitmap _validDocIds;
// Cache the predicate evaluators
private final Map<Predicate, PredicateEvaluator> _predicateEvaluatorMap = new HashMap<>();
public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
_indexSegment = indexSegment;
_queryContext = queryContext;
+
+ // Snapshot validDocIds *before* _numDocs since otherwise newest upserts could get lost
Review comment:
Good point. From what I could see the FilterPlanNode constructor call site is currently always adjacent to the run method invocation (DocSetIdPlanNode#run, AggregationPlanNode#run, AggregationGroupByPlanNode#run, AggregationGroupByOrderByPlanNode#run) so I don't think that it could currently make a difference whether snapshotting happens in constructor or run. However, the intent of other operators seems definitely to be that run happens during parallelized execution. From that perspective the approach in #8018 to only snapshot in the run method is definitely preferable.
Only question I had was what the original reason was behind the checkpointing in the constructor (given the same argument as above it seems to make no practical difference at the moment :)?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1022445843
@richardstartin Thanks a lot - the new build fails with different failures :) I will take a look to see what might be happening.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1022149600
Build failures may be transient :( What I saw in the JDK 11 Quickstart test is that the build ran successfully and the test seemed to complete but the post job cleanup failed to upload to some (Github?) cache. What's the usual approach here - try to rerun, rebase on master and retry, dig into the failures in this PR?
```
2022-01-18T19:26:57.6881659Z Post job cleanup.
2022-01-18T19:26:57.8637035Z [command]/usr/bin/tar -cz -f /home/runner/work/_temp/3051cf70-7cdf-4c5f-9732-07cffd83b882/cache.tgz -C /home/runner/.m2/repository .
2022-01-18T19:28:52.2887996Z [warning]Cache service responded with 400 during chunk upload.
2022-01-18T19:28:52.2921708Z events.js:187
2022-01-18T19:28:52.2923145Z throw er; // Unhandled 'error' event
2022-01-18T19:28:52.2923523Z ^
2022-01-18T19:28:52.2923727Z
2022-01-18T19:28:52.2923978Z Error: EBADF: bad file descriptor, read
2022-01-18T19:28:52.2924355Z Emitted 'error' event on ReadStream instance at:
2022-01-18T19:28:52.2924852Z at internal/fs/streams.js:167:12
2022-01-18T19:28:52.2925172Z at FSReqCallback.wrapper [as oncomplete] (fs.js:470:5) {
2022-01-18T19:28:52.2925961Z errno: -9,
2022-01-18T19:28:52.2926187Z code: 'EBADF',
2022-01-18T19:28:52.2926532Z syscall: 'read'
2022-01-18T19:28:52.2926727Z }
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] snleee commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
snleee commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r777736934
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -73,20 +84,19 @@ public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
@Override
public BaseFilterOperator run() {
FilterContext filter = _queryContext.getFilter();
- ThreadSafeMutableRoaringBitmap validDocIds = _indexSegment.getValidDocIds();
- boolean applyValidDocIds = validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
+ boolean applyValidDocIds = _validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
if (filter != null) {
BaseFilterOperator filterOperator = constructPhysicalOperator(filter);
if (applyValidDocIds) {
BaseFilterOperator validDocFilter =
- new BitmapBasedFilterOperator(validDocIds.getMutableRoaringBitmap(), false, _numDocs);
+ new BitmapBasedFilterOperator(_validDocIds, false, _numDocs);
Review comment:
Are you using the pinot code style?
https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#setup-ide
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] mayankshriv commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
mayankshriv commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1074535087
Seems like we didn't conclude on this PR. @tuor713 may I request to rebase, resolve the conflicts and address any open questions/comments? We can then get this merged. cc: @Jackie-Jiang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1015777935
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7971](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (36b70aa) into [master](https://codecov.io/gh/apache/pinot/commit/0bf408615cb8dc96defa80f4b8a0c302741dd40c?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0bf4086) will **decrease** coverage by `35.10%`.
> The diff coverage is `83.33%`.
[](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7971 +/- ##
=============================================
- Coverage 71.32% 36.21% -35.11%
+ Complexity 4196 80 -4116
=============================================
Files 1594 1595 +1
Lines 82507 82597 +90
Branches 12304 12327 +23
=============================================
- Hits 58851 29916 -28935
- Misses 19683 50295 +30612
+ Partials 3973 2386 -1587
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.67% <83.33%> (-0.05%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `14.32% <0.00%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `61.38% <83.33%> (-25.49%)` | :arrow_down: |
| [.../java/org/apache/pinot/spi/utils/BooleanUtils.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQm9vbGVhblV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/config/table/FSTType.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL0ZTVFR5cGUuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...ava/org/apache/pinot/spi/data/MetricFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9NZXRyaWNGaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...va/org/apache/pinot/spi/utils/BigDecimalUtils.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvdXRpbHMvQmlnRGVjaW1hbFV0aWxzLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...java/org/apache/pinot/common/tier/TierFactory.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdGllci9UaWVyRmFjdG9yeS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...a/org/apache/pinot/spi/config/table/TableType.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1RhYmxlVHlwZS5qYXZh) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/spi/data/DimensionFieldSpec.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9EaW1lbnNpb25GaWVsZFNwZWMuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [.../org/apache/pinot/spi/data/readers/FileFormat.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvZGF0YS9yZWFkZXJzL0ZpbGVGb3JtYXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...org/apache/pinot/spi/config/table/QuotaConfig.java](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc3BpL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcGkvY29uZmlnL3RhYmxlL1F1b3RhQ29uZmlnLmphdmE=) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| ... and [971 more](https://codecov.io/gh/apache/pinot/pull/7971/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0bf4086...36b70aa](https://codecov.io/gh/apache/pinot/pull/7971?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] tuor713 commented on a change in pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
tuor713 commented on a change in pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#discussion_r777779783
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/FilterPlanNode.java
##########
@@ -73,20 +84,19 @@ public FilterPlanNode(IndexSegment indexSegment, QueryContext queryContext) {
@Override
public BaseFilterOperator run() {
FilterContext filter = _queryContext.getFilter();
- ThreadSafeMutableRoaringBitmap validDocIds = _indexSegment.getValidDocIds();
- boolean applyValidDocIds = validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
+ boolean applyValidDocIds = _validDocIds != null && !QueryOptionsUtils.isSkipUpsert(_queryContext.getQueryOptions());
if (filter != null) {
BaseFilterOperator filterOperator = constructPhysicalOperator(filter);
if (applyValidDocIds) {
BaseFilterOperator validDocFilter =
- new BitmapBasedFilterOperator(validDocIds.getMutableRoaringBitmap(), false, _numDocs);
+ new BitmapBasedFilterOperator(_validDocIds, false, _numDocs);
Review comment:
Ah my bad let me fix that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #7971: Fix to query inconsistencies under heavy upsert load (resolves #7958)
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #7971:
URL: https://github.com/apache/pinot/pull/7971#issuecomment-1076865340
@tuor713 Just merged #8392. Can you please rebase the current master to add the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org