You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by "gortiz (via GitHub)" <gi...@apache.org> on 2023/03/02 16:09:52 UTC
[GitHub] [pinot] gortiz opened a new pull request, #10372: Let applyAnd to be applied using different window sizes
gortiz opened a new pull request, #10372:
URL: https://github.com/apache/pinot/pull/10372
Before this PR, `ScanBasedDocIdIterator.applyAnd` was always applied using windows of OPTIMAL_ITERATOR_BATCH_SIZE, which is defined as 256.
At StarTree we found some inefficiencies when fetching data from slower backends (for example when a network request is needed). Therefore we would like to be able to decide the window size depending on the situation.
This PR is a little bit larger than expected because we also needed to modify `ValueMatcher` in order to do not contain a buffer of predefined size. Instead is the caller the one that creates and supply the buffer that has to be used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang merged pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang merged PR #10372:
URL: https://github.com/apache/pinot/pull/10372
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1455758280
> IIUC, the intention is to batch more reads from ForwardIndexReader to reduce the total times of accessing the reader? If that is the case, I think we should add the batch size into the SVScanDocIdIterator constructor without changing the applyAnd() interface
That would make the change quite more complex. There are two buffers in this algorithm: the one used by `WhateverMatcher`
and the one used by `applyAnd`. Let's call `buffer` to the one in `WhateverMatcher` and `docIds` to the one in `applyAnd`.
The code assumes that:
- `length` <= `docIds.length` <= `buffer.length`. The reality is that `docIds.length` == `buffer.length`
- `length` entries are going to be read into `buffer`.
I'm not sure on what you suggest, but I'm going to assume that the idea is to increase `buffer.length` and break the second invariant.
I think that would make the code quite more complex. `buffer` would need to be an actual cache over the data that is read and must read in some batch size defined at construction time. Then we would need a cursor to indicate which entry was the last that was consumed by the caller. Finally each time we call `readValuesSV` (I'm assuming with MV would be the same) we would need to check whether there are enough elements in `buffer` which may be resolved as:
1. There are more documents to read than `length`, so directly read from `buffer` and move the cursor.
2. `buffer` is exhausted, so read more from the backend, set the cursor to 0 and repeat.
3. `buffer` is not exhausted but there are not enough elements to read. First consume the elements on `buffer`, then read more and then read from 0 to the remaining number of elements.
We can avoid the 3rd case if we assume that:
1. `docIds.length = k * buffer.length` where `k` is some natural positive number
2. In all calls but the last, `length = docIds.length`
3. In the last call `length` = remaining docs in the segment and therefore `buffer` will contain exactly `length` elements.
This algorithm introduce several not so trivial to analyze costs in Pinot's hotpath and contrary to my proposal, these changes will affect Pinot even when this customization is not used. In case we add something like that, I would be with Richard and I would require to add some benchmarks in order to hint whether the change may affect the performance negatively in some cases.
Contrary to this approach, mine is simpler (it doesn't change the current algorithm or invariants at all), it doesn't require to add the new concept of fetch size into the reader, it doesn't affect the performance on OSS Pinot and it doesn't close the door to the possibility of adding the fetch size to the reader in a future in case we find other interesting cases.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "codecov-commenter (via GitHub)" <gi...@apache.org>.
codecov-commenter commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1452508047
# [Codecov](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#10372](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6b4cfc2) into [master](https://codecov.io/gh/apache/pinot/commit/9c7e771d22a0cbd3391593e9976669fff4d4b442?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (9c7e771) will **increase** coverage by `0.01%`.
> The diff coverage is `0.00%`.
```diff
@@ Coverage Diff @@
## master #10372 +/- ##
============================================
+ Coverage 13.76% 13.77% +0.01%
Complexity 231 231
============================================
Files 1979 1979
Lines 107729 107754 +25
Branches 16457 16459 +2
============================================
+ Hits 14831 14847 +16
- Misses 91717 91726 +9
Partials 1181 1181
```
| Flag | Coverage Δ | |
|---|---|---|
| unittests2 | `13.77% <0.00%> (+0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...he/pinot/core/operator/BitmapDocIdSetOperator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9CaXRtYXBEb2NJZFNldE9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...or/dociditerators/ExpressionScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9FeHByZXNzaW9uU2NhbkRvY0lkSXRlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...e/operator/dociditerators/MVScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9NVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...e/operator/dociditerators/SVScanDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TVlNjYW5Eb2NJZEl0ZXJhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...perator/dociditerators/ScanBasedDocIdIterator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZGl0ZXJhdG9ycy9TY2FuQmFzZWREb2NJZEl0ZXJhdG9yLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../core/operator/filter/ScanBasedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvU2NhbkJhc2VkRmlsdGVyT3BlcmF0b3IuamF2YQ==) | `0.00% <ø> (ø)` | |
| [...ocal/segment/index/loader/ForwardIndexHandler.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3Qtc2VnbWVudC1sb2NhbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3Qvc2VnbWVudC9sb2NhbC9zZWdtZW50L2luZGV4L2xvYWRlci9Gb3J3YXJkSW5kZXhIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ces/PinotSegmentUploadDownloadRestletResource.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2VnbWVudFVwbG9hZERvd25sb2FkUmVzdGxldFJlc291cmNlLmphdmE=) | `39.83% <0.00%> (+0.82%)` | :arrow_up: |
| [...controller/helix/core/minion/PinotTaskManager.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdFRhc2tNYW5hZ2VyLmphdmE=) | `38.72% <0.00%> (+1.44%)` | :arrow_up: |
| [...lix/core/realtime/PinotRealtimeSegmentManager.java](https://codecov.io/gh/apache/pinot/pull/10372?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL3JlYWx0aW1lL1Bpbm90UmVhbHRpbWVTZWdtZW50TWFuYWdlci5qYXZh) | `79.00% <0.00%> (+4.50%)` | :arrow_up: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on a diff in pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #10372:
URL: https://github.com/apache/pinot/pull/10372#discussion_r1131137892
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java:
##########
@@ -47,15 +47,17 @@ public final class MVScanDocIdIterator implements ScanBasedDocIdIterator {
private int _nextDocId = 0;
private long _numEntriesScanned = 0L;
+ private final int _batchSize;
- public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs) {
+ public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs, int batchSize) {
Review Comment:
You are right, although I guess we will need to modify `MVScanDocIdIterator` in future in order to optimize the access in this case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a diff in pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on code in PR #10372:
URL: https://github.com/apache/pinot/pull/10372#discussion_r1129923530
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/ScanBasedDocIdIterator.java:
##########
@@ -34,10 +36,17 @@
*/
public interface ScanBasedDocIdIterator extends BlockDocIdIterator {
+ MutableRoaringBitmap applyAnd(BatchIterator batchIterator, OptionalInt firstDoc, OptionalInt lastDoc);
Review Comment:
Trying to understand why do we need to add this API. The `BatchIterator` should always be returned from the roaring bitmap. If you need to pass `RoaringBitmap` here, you may modify the existing `applyAnd()` to take an `ImmutableBitmapDataProvider`.
In order to apply a different start/end to the bitmap, we may do an AND operation to the bitmap with the range.
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/MVScanDocIdIterator.java:
##########
@@ -47,15 +47,17 @@ public final class MVScanDocIdIterator implements ScanBasedDocIdIterator {
private int _nextDocId = 0;
private long _numEntriesScanned = 0L;
+ private final int _batchSize;
- public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs) {
+ public MVScanDocIdIterator(PredicateEvaluator predicateEvaluator, DataSource dataSource, int numDocs, int batchSize) {
Review Comment:
Custom batch size might not apply here since this batch is only used when iterating over the bitmap, where 256 might be the optimal one. It won't reduce the time of index access
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java:
##########
@@ -44,16 +46,19 @@ public final class SVScanDocIdIterator implements ScanBasedDocIdIterator {
private final ForwardIndexReaderContext _readerContext;
private final int _numDocs;
private final ValueMatcher _valueMatcher;
- private final int[] _batch = new int[OPTIMAL_ITERATOR_BATCH_SIZE];
+ private final int[] _batch;
private int _firstMismatch;
private int _cursor;
private final int _cardinality;
private int _nextDocId = 0;
private long _numEntriesScanned = 0L;
+ private final int _batchSize;
Review Comment:
(nit) Keep the final variables together for readability
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java:
##########
@@ -87,7 +94,7 @@ public int next() {
int limit;
int batchSize = 0;
do {
- limit = Math.min(_numDocs - _nextDocId, OPTIMAL_ITERATOR_BATCH_SIZE);
+ limit = Math.min(_numDocs - _nextDocId, _batch.length);
Review Comment:
(nit)
```suggestion
limit = Math.min(_numDocs - _nextDocId, _batchSize);
```
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java:
##########
@@ -280,11 +295,12 @@ public int matchValues(int limit, int[] docIds) {
private class DictIdMatcherAndNullHandler implements ValueMatcher {
- private final int[] _buffer = new int[OPTIMAL_ITERATOR_BATCH_SIZE];
+ private final int[] _buffer;
private final ImmutableRoaringBitmap _nullBitmap;
- public DictIdMatcherAndNullHandler(ImmutableRoaringBitmap nullBitmap) {
+ public DictIdMatcherAndNullHandler(ImmutableRoaringBitmap nullBitmap, int batchSize) {
Review Comment:
(minor) We don't need to pass it since it is a local variable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1458498923
Ok, but that is not achieving that much from what I've got. I mean, yes, with your option the window size can be changed in other cases. I still think that the buffer should be managed by the caller, but your option require less changes, so lets use that. In my specific case I know the custom window size when the Operator is created, so it doesn't hurt me.
But in my use case I would also need to be able to `applyAnd` using an iterator. I have this `applyAnd` that receives ` BatchIterator` but does not include the batch size
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "Jackie-Jiang (via GitHub)" <gi...@apache.org>.
Jackie-Jiang commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1457396761
This is what I meant: https://github.com/apache/pinot/commit/345cbc6c5f27a5117e974c57413335e5cd3b3eea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1453113881
I know you are good, but I wouldn't request you to review the performance impact on an external project without showing you the code. But I'm sure you can review these changes and try to imagine the impact on Apache Pinot. The code is mostly the same but:
1. We now create two OptionaInt objects.
2. The buffer is stored as a local variable instead of a final attribute.
3. `ScanBasedDocIdIterator.applyAnd(ImmutableRoaringBitmap)` is implemented as a call to the new overloaded method that receives an iterator.
I would say that 2 should have a positive but negligible impact and given the length of `ScanBasedDocIdIterator.applyAnd(ImmutableRoaringBitmap)` it should always be inlined. About 1, it may have a very low impact, but we can find a way to do not create them if we think it is problematic.
About the StarTree optimization: our plan is not to provide a better general window size and keep it in the proprietary code but to use optimize an inefficient case we only have in our proprietary plugins. The default 256 value may be great to improve the CPU cache utilization when the data is in main memory and as said, we don't want to change that. But when data has to be fetched using the network, the cost network cost is larger than anything else. In that specific case, larger blocks reduce the number of network calls and significantly increase the performance. Ideally we would like to continue applying windows of size 256 in the calculation part but fetch (and cache) larger windows, but that would require to communicate the planning layer with the storage layer. That dependency totally make sense, but it would need to do a larger refactor, so we are happy with this intermediate solution right now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1452129459
@Jackie-Jiang @KKcorps @saurabhd336 please review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "richardstartin (via GitHub)" <gi...@apache.org>.
richardstartin commented on PR #10372:
URL: https://github.com/apache/pinot/pull/10372#issuecomment-1453095098
I don't think I can review this because
* a performance improvement is reported, but not in Apache Pinot itself, so it's impossible to review based on the sources alone
* no benchmarks to demonstrate improvement in the external component this change accelerates are provided with the change
* benchmarks to quantify the degradation to users who aren't using the external component this change accelerates haven't been run (do they regress? Do they improve?)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] gortiz commented on a diff in pull request #10372: Let applyAnd to be applied using different window sizes
Posted by "gortiz (via GitHub)" <gi...@apache.org>.
gortiz commented on code in PR #10372:
URL: https://github.com/apache/pinot/pull/10372#discussion_r1131155553
##########
pinot-core/src/main/java/org/apache/pinot/core/operator/dociditerators/SVScanDocIdIterator.java:
##########
@@ -87,7 +94,7 @@ public int next() {
int limit;
int batchSize = 0;
do {
- limit = Math.min(_numDocs - _nextDocId, OPTIMAL_ITERATOR_BATCH_SIZE);
+ limit = Math.min(_numDocs - _nextDocId, _batch.length);
Review Comment:
I've removed `_batchSize` attribute and substitute its usages with `_batchSize.length`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org