You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@pinot.apache.org by GitBox <gi...@apache.org> on 2022/01/04 07:12:26 UTC
[GitHub] [pinot] atris opened a new pull request #7916: FILTER Clauses for Aggregates
atris opened a new pull request #7916:
URL: https://github.com/apache/pinot/pull/7916
This PR implements support for FILTER clauses in aggregations:
SELECT SUM(COL1) FILTER(WHERE COL2 > 300), AVG(COL2) FILTER (WHERE COL2 < 50) FROM MyTable WHERE COL1 > 50;
The approach implements the swim lane design highlighted in the design document by splitting at the filter operator. The implementation gets the filter block for main predicate and each filter predicate, ANDs them together and returns a combined filter operator.
The main predicate is scanned only once and reused for all filter clauses.
The implementation allows each filter swim lane to use any available indices independently.
If two or more filter clauses have the same predicate, the result will be computed only once and fed to each of the aggregates.
https://docs.google.com/document/d/1ZM-2c0jJkbeJ61m8sJF0qj19t5UYLhnTFvIAz-HCJmk/edit?usp=sharing
Performance benchmark:
3 warm up iterations per run, 5 runs in total. Data set size -- 1.5 million documents. Apple M1 Pro, 32GB RAM
X axis represents number of iterations and Y axis represents latency in MS.
FILTER query, compared to its equivalent CASE query, is 120-140% faster on average.
<img width="400" alt="image" src="https://user-images.githubusercontent.com/1724131/146372915-6cda2f19-7303-4d12-a91b-42a15b9285e0.png">
GROUP BY is not supported yet and will be done in a follow up PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770879446
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(Map<ExpressionContext, BaseFilterOperator> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return new ArrayList<>(_filterOperators.values());
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected FilterBlock getNextBlock() {
+ if (_resultBlock != null) {
+ return _resultBlock;
+ }
+
+ FilterBlock mainFilterBlock = _mainFilterOperator.nextBlock();
+
+ Map<ExpressionContext, FilterBlock> filterBlockMap = new HashMap<>();
+ Iterator<Map.Entry<ExpressionContext, BaseFilterOperator>> iterator = _filterOperators.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, BaseFilterOperator> entry = iterator.next();
+ FilterBlock subFilterBlock = entry.getValue().nextBlock();
+
+ filterBlockMap.put(entry.getKey(),
Review comment:
Sure, but I don't think `Map` is the right data structure for such a small collection of elements.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6d0d3ad) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `57.02%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.24% 14.22% -57.03%
+ Complexity 4262 81 -4181
=============================================
Files 1607 1566 -41
Lines 83409 81679 -1730
Branches 12458 12273 -185
=============================================
- Hits 59426 11616 -47810
- Misses 19941 69204 +49263
+ Partials 4042 859 -3183
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.22% <0.00%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `0.00% <0.00%> (-57.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-92.99%)` | :arrow_down: |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `0.00% <0.00%> (-86.87%)` | :arrow_down: |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `0.00% <0.00%> (-97.91%)` | :arrow_down: |
| ... and [1288 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...6d0d3ad](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c8d9ad5) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `56.99%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.24% 14.25% -57.00%
+ Complexity 4262 81 -4181
=============================================
Files 1607 1566 -41
Lines 83409 81674 -1735
Branches 12458 12273 -185
=============================================
- Hits 59426 11641 -47785
- Misses 19941 69166 +49225
+ Partials 4042 867 -3175
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.25% <0.00%> (+<0.01%)` | :arrow_up: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `0.00% <0.00%> (-57.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-92.99%)` | :arrow_down: |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `0.00% <0.00%> (-86.87%)` | :arrow_down: |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `0.00% <0.00%> (-97.91%)` | :arrow_down: |
| ... and [1287 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...c8d9ad5](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771094592
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/SwimLaneDocIdSetOperator.java
##########
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator;
+
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.DocIdSetBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
+import org.apache.pinot.core.operator.filter.CombinedFilterOperator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+
+/**
+ * DocIdSetOperator for a swimlane query plan.
+ */
+public class SwimLaneDocIdSetOperator extends DocIdSetOperator {
+ private static final String OPERATOR_NAME = "SwimLaneDocIdSetOperator";
+
+ private final CombinedFilterOperator _filterOperator;
+ private final ExpressionContext _expressionContext;
+ private final int _maxSizeOfDocIdSet;
+
+ private FilterBlockDocIdSet _filterBlockDocIdSet;
+ private BlockDocIdIterator _blockDocIdIterator;
+ private int _currentDocId = 0;
+
+ public SwimLaneDocIdSetOperator(CombinedFilterOperator filterOperator, ExpressionContext expressionContext,
+ int maxSizeOfDocIdSet) {
+ super(filterOperator, maxSizeOfDocIdSet);
+
+ Preconditions.checkArgument(maxSizeOfDocIdSet > 0 && maxSizeOfDocIdSet <= DocIdSetPlanNode.MAX_DOC_PER_CALL);
+ _filterOperator = filterOperator;
+ _expressionContext = expressionContext;
+ _maxSizeOfDocIdSet = maxSizeOfDocIdSet;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return Arrays.asList(_filterOperator);
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected DocIdSetBlock getNextBlock() {
+ if (_currentDocId == Constants.EOF) {
+ return null;
+ }
+
+ // Initialize filter block document Id set
+ if (_filterBlockDocIdSet == null) {
+ CombinedFilterBlock combinedFilterBlock = (CombinedFilterBlock) _filterOperator.nextBlock();
+ FilterBlock filterBlock = combinedFilterBlock.getFilterBlock(_expressionContext);
+
+ if (filterBlock == null) {
+ throw new IllegalStateException("Null block seen");
+ }
+
+ _filterBlockDocIdSet = filterBlock.getBlockDocIdSet();
+ _blockDocIdIterator = _filterBlockDocIdSet.iterator();
+ }
+
+ int pos = 0;
+ int[] docIds = new int[DocIdSetPlanNode.MAX_DOC_PER_CALL];
+
+ for (int i = 0; i < _maxSizeOfDocIdSet; i++) {
+ _currentDocId = _blockDocIdIterator.next();
+ if (_currentDocId == Constants.EOF) {
+ break;
+ }
+ docIds[pos++] = _currentDocId;
+ }
+ if (pos > 0) {
+ return new DocIdSetBlock(docIds, pos);
+ } else {
+ return null;
+ }
Review comment:
This is common code between DocIdSetIterator and this, and we should do the refactoring in a separate PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771095193
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -52,7 +61,8 @@ public BlockValSet getBlockValueSet(String column) {
@Override
public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
+ return _docIdSetBlock != null ? _docIdSetBlock.getBlockDocIdSet()
Review comment:
Sure, I just like to be pedantic :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771100510
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ List<Pair<ExpressionContext, BaseFilterOperator>> baseFilterOperatorList =
+ new ArrayList<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorList.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight()));
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
+ }
+ }
+
+ CombinedFilterOperator combinedFilterOperator = new CombinedFilterOperator(baseFilterOperatorList,
+ mainPredicateFilterOperator);
+
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ for (Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>> pair
+ : filterPredicatesAndMetadata) {
+ Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> innerPair =
+ buildOperators(combinedFilterOperator, pair.getRight().getLeft(),
+ true, pair.getLeft());
+
+ transformOperatorMap.put(pair.getLeft(), innerPair.getLeft());
+ }
+
+ // Add the main predicate filter operator to the map
+ transformOperatorMap.put(_queryContext.getFilterExpression(), nonFilteredTransformOperator);
+
+ return new CombinedTransformOperator(transformOperatorMap, _queryContext.getFilterExpression(),
+ expressionsToTransform);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @param isSwimlane Is this plan a swim lane?
Review comment:
Please refer to the document as to what a swim lane represents. Update the javadocs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r788376924
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/startree/plan/StarTreeProjectionPlanNode.java
##########
@@ -41,6 +41,7 @@ public StarTreeProjectionPlanNode(StarTreeV2 starTreeV2, Set<String> projectionC
for (String projectionColumn : projectionColumns) {
_dataSourceMap.put(projectionColumn, starTreeV2.getDataSource(projectionColumn));
}
+
Review comment:
Let's revert this file since it is not relevant
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,57 +68,15 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
- int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
- AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
-
- FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext);
- BaseFilterOperator filterOperator = filterPlanNode.run();
-
- // Use metadata/dictionary to solve the query if possible
- // TODO: Use the same operator for both of them so that COUNT(*), MAX(col) can be optimized
- if (filterOperator.isResultMatchingAll()) {
- if (isFitForMetadataBasedPlan(aggregationFunctions)) {
- return new MetadataBasedAggregationOperator(aggregationFunctions, _indexSegment.getSegmentMetadata(),
- Collections.emptyMap());
- } else if (isFitForDictionaryBasedPlan(aggregationFunctions, _indexSegment)) {
- Map<String, Dictionary> dictionaryMap = new HashMap<>();
- for (AggregationFunction aggregationFunction : aggregationFunctions) {
- String column = ((ExpressionContext) aggregationFunction.getInputExpressions().get(0)).getIdentifier();
- dictionaryMap.computeIfAbsent(column, k -> _indexSegment.getDataSource(k).getDictionary());
- }
- return new DictionaryBasedAggregationOperator(aggregationFunctions, dictionaryMap, numTotalDocs);
- }
- }
-
- // Use star-tree to solve the query if possible
- List<StarTreeV2> starTrees = _indexSegment.getStarTrees();
- if (starTrees != null && !StarTreeUtils.isStarTreeDisabled(_queryContext)) {
- AggregationFunctionColumnPair[] aggregationFunctionColumnPairs =
- StarTreeUtils.extractAggregationFunctionPairs(aggregationFunctions);
- if (aggregationFunctionColumnPairs != null) {
- Map<String, List<CompositePredicateEvaluator>> predicateEvaluatorsMap =
- StarTreeUtils.extractPredicateEvaluatorsMap(_indexSegment, _queryContext.getFilter(),
- filterPlanNode.getPredicateEvaluatorMap());
- if (predicateEvaluatorsMap != null) {
- for (StarTreeV2 starTreeV2 : starTrees) {
- if (StarTreeUtils.isFitForStarTree(starTreeV2.getMetadata(), aggregationFunctionColumnPairs, null,
- predicateEvaluatorsMap.keySet())) {
- TransformOperator transformOperator =
- new StarTreeTransformPlanNode(starTreeV2, aggregationFunctionColumnPairs, null,
- predicateEvaluatorsMap, _queryContext.getDebugOptions()).run();
- return new AggregationOperator(aggregationFunctions, transformOperator, numTotalDocs, true);
- }
- }
- }
- }
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+ BaseOperator<IntermediateResultsBlock> aggOperator;
+ if (hasFilteredPredicates) {
+ aggOperator = buildFilteredAggOperator();
Review comment:
(minor) this part can be more concise by directly return instead of putting the operator in an local variable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -119,11 +122,13 @@ private QueryContext(String tableName, List<ExpressionContext> selectExpressions
@Nullable FilterContext filter, @Nullable List<ExpressionContext> groupByExpressions,
@Nullable FilterContext havingFilter, @Nullable List<OrderByExpressionContext> orderByExpressions, int limit,
int offset, Map<String, String> queryOptions, @Nullable Map<String, String> debugOptions,
- BrokerRequest brokerRequest) {
+ BrokerRequest brokerRequest, boolean hasFilteredAggregations,
Review comment:
`hasFilteredAggregations` should not be set through the constructor. It is updated in `generateAggregationFunctions()`
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -350,6 +381,7 @@ public String toString() {
private List<ExpressionContext> _selectExpressions;
private List<String> _aliasList;
private FilterContext _filter;
+ private ExpressionContext _filterExpression;
Review comment:
The change in the `Builder` is not necessary
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/utils/BrokerRequestToQueryContextConverter.java
##########
@@ -144,6 +146,7 @@ private static QueryContext convertSQL(BrokerRequest brokerRequest) {
return new QueryContext.Builder().setTableName(pinotQuery.getDataSource().getTableName())
.setSelectExpressions(selectExpressions).setAliasList(aliasList).setFilter(filter)
+ .setFilterExpression(filterExpressionContext)
Review comment:
This seems not used in `QueryContext`, and this file can be reverted
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -74,6 +74,7 @@
private final List<ExpressionContext> _selectExpressions;
private final List<String> _aliasList;
private final FilterContext _filter;
+ private final ExpressionContext _filterExpression;
Review comment:
This field is not used, let's remove it. Filter shouldn't be stored as `ExpressionContext`
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +118,181 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator, int numTotalDocs) {
+ Map<FilterContext, Pair<List<AggregationFunction>, TransformOperator>> filterContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+ List<Pair<FilterContext, AggregationFunction>> aggregationFunctions = _queryContext
+ .getAggregationsWithFilters();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (Pair<FilterContext, AggregationFunction> inputPair : aggregationFunctions) {
+ if (inputPair.getLeft() != null) {
+ FilterContext currentFilterExpression = inputPair.getLeft();
+ if (filterContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ filterContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(inputPair.getRight());
+ continue;
+ }
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(currentFilterExpression);
+ BaseFilterOperator wrappedFilterOperator = new CombinedFilterOperator(mainPredicateFilterOperator,
+ pair.getRight());
+ Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> innerPair =
+ buildOperators(wrappedFilterOperator, pair.getLeft());
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ List aggFunctionList = new ArrayList<>();
+ aggFunctionList.add(inputPair.getRight());
+ filterContextToAggFuncsMap.put(currentFilterExpression,
+ Pair.of(aggFunctionList, innerPair.getLeft()));
+ } else {
+ nonFilteredAggregationFunctions.add(inputPair.getRight());
+ }
+ }
+ List<Pair<AggregationFunction[], TransformOperator>> aggToTransformOpList =
+ new ArrayList<>();
+ // Convert to array since FilteredAggregationOperator expects it
+ for (Pair<List<AggregationFunction>, TransformOperator> pair
+ : filterContextToAggFuncsMap.values()) {
+ List<AggregationFunction> aggregationFunctionList = pair.getLeft();
+ if (aggregationFunctionList == null) {
+ throw new IllegalStateException("Null aggregation list seen");
+ }
+ aggToTransformOpList.add(Pair.of(aggregationFunctionList.toArray(new AggregationFunction[0]),
+ pair.getRight()));
+ }
+ aggToTransformOpList.add(Pair.of(nonFilteredAggregationFunctions.toArray(new AggregationFunction[0]),
+ mainTransformOperator));
+
+ return new FilteredAggregationOperator(_queryContext.getAggregationFunctions(), aggToTransformOpList,
+ numTotalDocs);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @return Pair, consisting of the built TransformOperator and Aggregation operator for chain
+ */
+ private Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> buildOperators(BaseFilterOperator filterOperator,
+ FilterPlanNode filterPlanNode) {
+ assert _queryContext.getAggregationFunctions() != null;
+
+ int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
+ AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+
+ Set<ExpressionContext> expressionsToTransform =
+ AggregationFunctionUtils.collectExpressionsToTransform(aggregationFunctions, null);
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+
+ List<StarTreeV2> starTrees = _indexSegment.getStarTrees();
+
+ // Use metadata/dictionary to solve the query if possible
+ // TODO: Use the same operator for both of them so that COUNT(*), MAX(col) can be optimized
+ if (filterOperator.isResultMatchingAll() && !hasFilteredPredicates) {
+ if (isFitForMetadataBasedPlan(aggregationFunctions)) {
+ return Pair.of(null, new MetadataBasedAggregationOperator(aggregationFunctions,
+ _indexSegment.getSegmentMetadata(), Collections.emptyMap()));
+ } else if (isFitForDictionaryBasedPlan(aggregationFunctions, _indexSegment)) {
+ Map<String, Dictionary> dictionaryMap = new HashMap<>();
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ String column = ((ExpressionContext) aggregationFunction.getInputExpressions().get(0)).getIdentifier();
+ dictionaryMap.computeIfAbsent(column, k -> _indexSegment.getDataSource(k).getDictionary());
+ }
+ return Pair.of(null,
+ new DictionaryBasedAggregationOperator(aggregationFunctions, dictionaryMap,
+ numTotalDocs));
+ }
+ }
+
+ if (starTrees != null && !StarTreeUtils.isStarTreeDisabled(_queryContext)) {
+ // Use star-tree to solve the query if possible
+
+ AggregationFunctionColumnPair[] aggregationFunctionColumnPairs =
+ StarTreeUtils.extractAggregationFunctionPairs(aggregationFunctions);
+ if (aggregationFunctionColumnPairs != null) {
+ Map<String, List<CompositePredicateEvaluator>> predicateEvaluatorsMap =
+ StarTreeUtils.extractPredicateEvaluatorsMap(_indexSegment, _queryContext.getFilter(),
+ filterPlanNode.getPredicateEvaluatorMap());
+ if (predicateEvaluatorsMap != null) {
+ for (StarTreeV2 starTreeV2 : starTrees) {
+ if (StarTreeUtils.isFitForStarTree(starTreeV2.getMetadata(), aggregationFunctionColumnPairs, null,
+ predicateEvaluatorsMap.keySet())) {
+
+ TransformOperator transformOperator = new StarTreeTransformPlanNode(starTreeV2,
+ aggregationFunctionColumnPairs, null,
+ predicateEvaluatorsMap, _queryContext.getDebugOptions()).run();
+ AggregationOperator aggregationOperator = new AggregationOperator(aggregationFunctions,
+ transformOperator, numTotalDocs, true);
+
+ return Pair.of(transformOperator, aggregationOperator);
+ }
+ }
+ }
+ }
+ }
+
+ TransformOperator transformOperator = new TransformPlanNode(_indexSegment, _queryContext,
+ expressionsToTransform, DocIdSetPlanNode.MAX_DOC_PER_CALL, filterOperator).run();
+ AggregationOperator aggregationOperator = new AggregationOperator(aggregationFunctions,
+ transformOperator, numTotalDocs, false);
+
+ return Pair.of(transformOperator, aggregationOperator);
+ }
+
+ /**
+ * Builds the operator to be used for non filtered aggregations
+ */
+ private BaseOperator<IntermediateResultsBlock> buildNonFilteredAggOperator() {
Review comment:
What I meant is to move the current code into this method, and implement `buildFilteredAggOperator()` separately. The reason being:
1. The metadata/dictionary based operator and star-tree does not apply to the filtered aggregation
2. Sharing `buildOperators()` method can bring extra overhead to non-filtered aggregations
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/request/context/utils/BrokerRequestToQueryContextConverterTest.java
##########
@@ -663,4 +649,18 @@ public void testPqlAndSqlCompatible()
assertNull(sqlReader.readLine());
}
}
+
Review comment:
Seems the only change is moving the test, can we revert it?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/startree/plan/StarTreeTransformPlanNode.java
##########
@@ -57,6 +57,7 @@ public StarTreeTransformPlanNode(StarTreeV2 starTreeV2,
_groupByExpressions = Collections.emptyList();
groupByColumns = null;
}
+
Review comment:
Let's revert this file since it is not relevant
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +91,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
- // TODO: Use Pair<FunctionContext, FilterContext> as key to support filtered aggregations in order-by and post
- // aggregation
+
+ private List<Pair<FilterContext, AggregationFunction>> _aggregationFunctionsWithMetadata;
Review comment:
Let's rename it to `_filteredAggregations` (it has nothing to do with metadata)? Also suggest putting `AggregationFunction` as the first argument of the pair to be consistent with other fields
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,76 +480,106 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
+ List<Pair<FilterContext, AggregationFunction>> aggregationFunctionsWithMetadata = new ArrayList<>();
Map<FunctionContext, Integer> aggregationFunctionIndexMap = new HashMap<>();
+ Map<Pair<FunctionContext, FilterContext>, Integer> filterExpressionIndexMap = new HashMap<>();
// Add aggregation functions in the SELECT clause
// NOTE: DO NOT deduplicate the aggregation functions in the SELECT clause because that involves protocol change.
- List<FunctionContext> aggregationsInSelect = new ArrayList<>();
- List<Pair<FunctionContext, FilterContext>> filteredAggregations = new ArrayList<>();
+ List<Pair<Pair<FilterContext, ExpressionContext>, FunctionContext>> aggregationsInSelect = new ArrayList<>();
Review comment:
I don't think we need to keep `ExpressionContext` here. `List<FunctionContext, FilterContext>` should be enough for the following computations
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,76 +480,106 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
+ List<Pair<FilterContext, AggregationFunction>> aggregationFunctionsWithMetadata = new ArrayList<>();
Review comment:
Please rename the variable
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
Review comment:
The format still doesn't align with the pinot style
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r787464846
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
Review comment:
Ok, removed FilterableAggregationFunction and replaced with a pair list. However, I am not removing _aggregationFunctions since it is widely used.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r787520413
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator,
+ AggregationFunction[] aggregationFunctions, int numTotalDocs) {
+ Map<ExpressionContext, Pair<List<AggregationFunction>, TransformOperator>> expressionContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+
+ ExpressionContext currentFilterExpression = filterableAggregationFunction
+ .getAssociatedExpressionContext();
+
+ if (expressionContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ expressionContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(aggregationFunction);
Review comment:
There are existing tests that test multiple aggregates for the same filter, and have added more. If we do not share TransformOperators, would that not mean a separate chain per aggregation operator, even if they had the exact same predicate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r787207706
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -52,7 +52,7 @@ public BlockValSet getBlockValueSet(String column) {
@Override
public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
+ return null;
Review comment:
Revert this since it is no longer relevant
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
Review comment:
(code format) Can you apply the latest [code format](https://docs.pinot.apache.org/developers/developers-and-contributors/code-setup#intellij) and reformat this file? Several places does not follow the code format. Also, can we reduce some empty lines in this method?
##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/FilteredAggregationsTest.java
##########
@@ -0,0 +1,512 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.queries;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.common.response.broker.BrokerResponseNative;
+import org.apache.pinot.common.response.broker.ResultTable;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.testng.Assert;
+import org.testng.annotations.AfterClass;
+import org.testng.annotations.BeforeClass;
+import org.testng.annotations.Test;
+
+
+public class FilteredAggregationsTest extends BaseQueriesTest {
+ private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(), "FilteredAggregationsTest");
+ private static final String TABLE_NAME = "MyTable";
+ private static final String FIRST_SEGMENT_NAME = "firstTestSegment";
+ private static final String SECOND_SEGMENT_NAME = "secondTestSegment";
+ private static final String INT_COL_NAME = "INT_COL";
+ private static final String NO_INDEX_INT_COL_NAME = "NO_INDEX_COL";
+ private static final String STATIC_INT_COL_NAME = "STATIC_INT_COL";
+ private static final Integer INT_BASE_VALUE = 0;
+ private static final Integer NUM_ROWS = 30000;
+
+
+ private IndexSegment _indexSegment;
+ private List<IndexSegment> _indexSegments;
+
+ @Override
+ protected String getFilter() {
+ return "";
+ }
+
+ @Override
+ protected IndexSegment getIndexSegment() {
+ return _indexSegment;
+ }
+
+ @Override
+ protected List<IndexSegment> getIndexSegments() {
+ return _indexSegments;
+ }
+
+ @BeforeClass
+ public void setUp()
+ throws Exception {
+ FileUtils.deleteQuietly(INDEX_DIR);
+
+ buildSegment(FIRST_SEGMENT_NAME);
+ buildSegment(SECOND_SEGMENT_NAME);
+ IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+
+ Set<String> invertedIndexCols = new HashSet<>();
+ invertedIndexCols.add(INT_COL_NAME);
+
+ indexLoadingConfig.setInvertedIndexColumns(invertedIndexCols);
+ ImmutableSegment firstImmutableSegment =
+ ImmutableSegmentLoader.load(new File(INDEX_DIR, FIRST_SEGMENT_NAME), indexLoadingConfig);
+ ImmutableSegment secondImmutableSegment =
+ ImmutableSegmentLoader.load(new File(INDEX_DIR, SECOND_SEGMENT_NAME), indexLoadingConfig);
+ _indexSegment = firstImmutableSegment;
+ _indexSegments = Arrays.asList(firstImmutableSegment, secondImmutableSegment);
+ }
+
+ @AfterClass
+ public void tearDown() {
+ _indexSegment.destroy();
+ FileUtils.deleteQuietly(INDEX_DIR);
+ }
+
+ private List<GenericRow> createTestData(int numRows) {
+ List<GenericRow> rows = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
+ GenericRow row = new GenericRow();
+ row.putField(INT_COL_NAME, INT_BASE_VALUE + i);
+ row.putField(NO_INDEX_INT_COL_NAME, i);
+ row.putField(STATIC_INT_COL_NAME, 10);
+
+ rows.add(row);
+ }
+ return rows;
+ }
+
+ private void buildSegment(String segmentName)
+ throws Exception {
+ List<GenericRow> rows = createTestData(NUM_ROWS);
+ List<FieldConfig> fieldConfigs = new ArrayList<>();
+
+ TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(Arrays.asList(INT_COL_NAME)).setFieldConfigList(fieldConfigs).build();
+ Schema schema = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension(NO_INDEX_INT_COL_NAME, FieldSpec.DataType.INT)
+ .addSingleValueDimension(STATIC_INT_COL_NAME, FieldSpec.DataType.INT)
+ .addMetric(INT_COL_NAME, FieldSpec.DataType.INT).build();
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(tableConfig, schema);
+ config.setOutDir(INDEX_DIR.getPath());
+ config.setTableName(TABLE_NAME);
+ config.setSegmentName(segmentName);
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ try (RecordReader recordReader = new GenericRowRecordReader(rows)) {
+ driver.init(config, recordReader);
+ driver.build();
+ }
+ }
+
+ private void testInterSegmentAggregationQueryHelper(String firstQuery, String secondQuery) {
+ // SQL
+ BrokerResponseNative firstBrokerResponseNative = getBrokerResponseForSqlQuery(firstQuery);
+ BrokerResponseNative secondBrokerResponseNative = getBrokerResponseForSqlQuery(secondQuery);
+ ResultTable firstResultTable = firstBrokerResponseNative.getResultTable();
+ ResultTable secondResultTable = secondBrokerResponseNative.getResultTable();
+ DataSchema firstDataSchema = firstResultTable.getDataSchema();
+ DataSchema secondDataSchema = secondResultTable.getDataSchema();
+
+ Assert.assertEquals(firstDataSchema.size(), secondDataSchema.size());
+
+ List<Object[]> firstSetOfRows = firstResultTable.getRows();
+ List<Object[]> secondSetOfRows = secondResultTable.getRows();
+
+ Assert.assertEquals(firstSetOfRows.size(), secondSetOfRows.size());
+
+ for (int i = 0; i < firstSetOfRows.size(); i++) {
+ Object[] firstSetRow = firstSetOfRows.get(i);
+ Object[] secondSetRow = secondSetOfRows.get(i);
+
+ Assert.assertEquals(firstSetRow.length, secondSetRow.length);
+
+ for (int j = 0; j < firstSetRow.length; j++) {
+ //System.out.println("FIRST " + firstSetRow[j] + " SECOND " + secondSetRow[j] + " j " + j);
+ Assert.assertEquals(firstSetRow[j], secondSetRow[j]);
+ }
+ }
+ }
+
+ @Test
+ public void testInterSegment() {
+
+ String query =
+ "SELECT SUM(INT_COL) FILTER(WHERE INT_COL > 9999)"
+ + "FROM MyTable WHERE INT_COL < 1000000";
+
+ String nonFilterQuery =
+ "SELECT SUM(INT_COL)"
+ + "FROM MyTable WHERE INT_COL > 9999 AND INT_COL < 1000000";
+
+ testInterSegmentAggregationQueryHelper(query, nonFilterQuery);
+
+ query = "SELECT SUM(INT_COL) FILTER(WHERE INT_COL > 1234 AND INT_COL < 22000)"
+ + "FROM MyTable";
+
+ nonFilterQuery = "SELECT SUM("
Review comment:
(code style) Suggest reformatting the queries in this test to be more compact
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,57 +69,25 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
- int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
- AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
- FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext);
- BaseFilterOperator filterOperator = filterPlanNode.run();
+ Pair<FilterPlanNode, BaseFilterOperator> filterOperatorPair =
Review comment:
What I meant is that we can branch these 2 cases earlier because the optimizations for regular aggregation don't apply to filtered aggregation (e.g. extra check on line 246, also star-tree should not be used for filtered aggregation which is not checked properly in the current code). It is more readable if we totally split these 2 cases:
```
if (hasFilteredPredicates) {
return buildOperatorForFilteredAggregations();
} else {
return buildOperatorForNonFilteredAggregations();
}
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/TransformBlock.java
##########
@@ -43,6 +43,11 @@ public TransformBlock(ProjectionBlock projectionBlock,
_transformFunctionMap = transformFunctionMap;
}
+ protected TransformBlock(TransformBlock transformBlock) {
Review comment:
Revert this file
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
Review comment:
I see your point, but my concern is that aggregation function and filter are logically two independent concept, and embedding filter into an aggregation function could cause confusion. If we need to associate some extra attributes to an aggregation function, I'd suggest adding a wrapper class instead of implementing a special `AggregationFunction`.
Based on the current implementation, I feel `Pair` itself should be enough (only need to associate the `FilterContext` with the `AggregationFunction`). To maintain the order of the aggregations, we may add pairs with `null` `FilterContext`
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,34 +471,54 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
Map<FunctionContext, Integer> aggregationFunctionIndexMap = new HashMap<>();
+ Map<Pair<FunctionContext, FilterContext>, Integer> filterExpressionIndexMap = new HashMap<>();
// Add aggregation functions in the SELECT clause
// NOTE: DO NOT deduplicate the aggregation functions in the SELECT clause because that involves protocol change.
- List<FunctionContext> aggregationsInSelect = new ArrayList<>();
- List<Pair<FunctionContext, FilterContext>> filteredAggregations = new ArrayList<>();
+ List<Pair<Pair<FilterContext, ExpressionContext>, FunctionContext>> aggregationsInSelect = new ArrayList<>();
for (ExpressionContext selectExpression : queryContext._selectExpressions) {
- getAggregations(selectExpression, aggregationsInSelect, filteredAggregations);
+ getAggregations(selectExpression, aggregationsInSelect);
}
- for (FunctionContext function : aggregationsInSelect) {
+ for (Pair<Pair<FilterContext, ExpressionContext>, FunctionContext> pair : aggregationsInSelect) {
+ FunctionContext function = pair.getRight();
int functionIndex = aggregationFunctions.size();
AggregationFunction aggregationFunction =
AggregationFunctionFactory.getAggregationFunction(function, queryContext);
+
+ // Hack: If the left pair is not null, implies a filtered aggregation
Review comment:
Revise this comment? We should not have hack in production code
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator,
+ AggregationFunction[] aggregationFunctions, int numTotalDocs) {
+ Map<ExpressionContext, Pair<List<AggregationFunction>, TransformOperator>> expressionContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+
+ ExpressionContext currentFilterExpression = filterableAggregationFunction
Review comment:
The `currentFilterExpression` seems redundant. You may directly use `filterContext` as the key
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FilterableAggregationFunction.java
##########
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.BlockValSet;
+import org.apache.pinot.core.query.aggregation.AggregationResultHolder;
+import org.apache.pinot.core.query.aggregation.groupby.GroupByResultHolder;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+
+/**
+ * Represents a filtered aggregation
+ */
+public class FilterableAggregationFunction implements
+ AggregationFunction<Object, Comparable> {
+ private AggregationFunction<Object, Comparable> _innerAggregationFunction;
+ private ExpressionContext _associatedExpressionContext;
+ private FilterContext _filterContext;
+
+ public FilterableAggregationFunction(AggregationFunction aggregationFunction,
+ ExpressionContext associatedExpressionContext, FilterContext filterContext) {
+ _innerAggregationFunction = aggregationFunction;
+ _associatedExpressionContext = associatedExpressionContext;
+ _filterContext = filterContext;
+ }
+
+ @Override
+ public AggregationFunctionType getType() {
+ return _innerAggregationFunction.getType();
+ }
+
+ @Override
+ public String getColumnName() {
+ return _innerAggregationFunction.getColumnName();
+ }
+
+ @Override
+ public String getResultColumnName() {
+ return _innerAggregationFunction.getResultColumnName();
+ }
+
+ @Override
+ public List<ExpressionContext> getInputExpressions() {
+ return _innerAggregationFunction.getInputExpressions();
+ }
+
+ @Override
+ public AggregationResultHolder createAggregationResultHolder() {
+ return _innerAggregationFunction.createAggregationResultHolder();
+ }
+
+ @Override
+ public GroupByResultHolder createGroupByResultHolder(int initialCapacity, int maxCapacity) {
+ return _innerAggregationFunction.createGroupByResultHolder(initialCapacity, maxCapacity);
+ }
+
+ @Override
+ public void aggregate(int length, AggregationResultHolder aggregationResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregate(length, aggregationResultHolder, blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupBySV(int length, int[] groupKeyArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupBySV(length, groupKeyArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupByMV(int length, int[][] groupKeysArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupByMV(length, groupKeysArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public Object extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
+ return _innerAggregationFunction.extractAggregationResult(aggregationResultHolder);
+ }
+
+ @Override
+ public Object extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
+ return _innerAggregationFunction.extractGroupByResult(groupByResultHolder, groupKey);
+ }
+
+ @Override
+ public Object merge(Object intermediateResult1, Object intermediateResult2) {
+ return _innerAggregationFunction.merge(intermediateResult1, intermediateResult2);
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getIntermediateResultColumnType() {
+ return _innerAggregationFunction.getIntermediateResultColumnType();
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getFinalResultColumnType() {
+ return _innerAggregationFunction.getFinalResultColumnType();
+ }
+
+ @Override
+ public Comparable extractFinalResult(Object o) {
+ return _innerAggregationFunction.extractFinalResult(o);
+ }
+
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
^^
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator,
+ AggregationFunction[] aggregationFunctions, int numTotalDocs) {
+ Map<ExpressionContext, Pair<List<AggregationFunction>, TransformOperator>> expressionContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+
+ ExpressionContext currentFilterExpression = filterableAggregationFunction
+ .getAssociatedExpressionContext();
+
+ if (expressionContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ expressionContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(aggregationFunction);
Review comment:
(Major) I think the `TransformOperator` cannot be shared among multiple aggregations. Once it iterates over a block, it won't process the same block again for the next aggregation. Let's add some test queries with multiple aggregations on the same filter.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785772717
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected List<Pair<ExpressionContext, BaseFilterOperator>> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(List<Pair<ExpressionContext, BaseFilterOperator>> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ List<Operator> operators = new ArrayList<>();
+
+ for (Pair<ExpressionContext, BaseFilterOperator> pair : _filterOperators) {
+ operators.add(pair.getRight());
+ }
+
+ return operators;
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
This is an outdated version of the class -- this class has a different structure in the latest version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a8e02d7) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `43.51%`.
> The diff coverage is `37.38%`.
> :exclamation: Current head a8e02d7 differs from pull request most recent head ec5b19c. Consider uploading reports for the commit ec5b19c to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.24% 27.73% -43.52%
=============================================
Files 1607 1602 -5
Lines 83409 83211 -198
Branches 12458 12439 -19
=============================================
- Hits 59426 23075 -36351
- Misses 19941 58027 +38086
+ Partials 4042 2109 -1933
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.73% <37.38%> (+0.06%)` | :arrow_up: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `30.00% <0.00%> (-27.15%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `84.88% <12.50%> (-7.43%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `53.21% <43.20%> (-39.78%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `87.20% <68.00%> (-10.71%)` | :arrow_down: |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `60.78% <100.00%> (-26.09%)` | :arrow_down: |
| ... and [1160 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...ec5b19c](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794304693
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,76 +480,106 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
+ List<Pair<FilterContext, AggregationFunction>> aggregationFunctionsWithMetadata = new ArrayList<>();
Map<FunctionContext, Integer> aggregationFunctionIndexMap = new HashMap<>();
+ Map<Pair<FunctionContext, FilterContext>, Integer> filterExpressionIndexMap = new HashMap<>();
// Add aggregation functions in the SELECT clause
// NOTE: DO NOT deduplicate the aggregation functions in the SELECT clause because that involves protocol change.
- List<FunctionContext> aggregationsInSelect = new ArrayList<>();
- List<Pair<FunctionContext, FilterContext>> filteredAggregations = new ArrayList<>();
+ List<Pair<Pair<FilterContext, ExpressionContext>, FunctionContext>> aggregationsInSelect = new ArrayList<>();
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bd19945) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `7.09%`.
> The diff coverage is `88.16%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 64.15% -7.10%
+ Complexity 4262 4180 -82
============================================
Files 1607 1602 -5
Lines 83409 83206 -203
Branches 12458 12441 -17
============================================
- Hits 59426 53380 -6046
- Misses 19941 25979 +6038
+ Partials 4042 3847 -195
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.97% <21.30%> (+<0.01%)` | :arrow_up: |
| integration2 | `27.63% <21.30%> (-0.04%)` | :arrow_down: |
| unittests1 | `67.97% <88.16%> (+0.10%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `90.90% <88.09%> (-2.08%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.58% <97.77%> (-0.33%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [225 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...bd19945](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785071519
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FilterableAggregationFunction.java
##########
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.BlockValSet;
+import org.apache.pinot.core.query.aggregation.AggregationResultHolder;
+import org.apache.pinot.core.query.aggregation.groupby.GroupByResultHolder;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+
+/**
+ * Represents a filtered aggregation
+ */
+public class FilterableAggregationFunction implements
+ AggregationFunction<Object, Comparable> {
+ private AggregationFunction<Object, Comparable> _innerAggregationFunction;
+ private ExpressionContext _associatedExpressionContext;
+ private FilterContext _filterContext;
+
+ public FilterableAggregationFunction(AggregationFunction aggregationFunction,
+ ExpressionContext associatedExpressionContext, FilterContext filterContext) {
+ _innerAggregationFunction = aggregationFunction;
+ _associatedExpressionContext = associatedExpressionContext;
+ _filterContext = filterContext;
+ }
+
+ @Override
+ public AggregationFunctionType getType() {
+ return _innerAggregationFunction.getType();
+ }
+
+ @Override
+ public String getColumnName() {
+ return _innerAggregationFunction.getColumnName();
+ }
+
+ @Override
+ public String getResultColumnName() {
+ return _innerAggregationFunction.getResultColumnName();
+ }
+
+ @Override
+ public List<ExpressionContext> getInputExpressions() {
+ return _innerAggregationFunction.getInputExpressions();
+ }
+
+ @Override
+ public AggregationResultHolder createAggregationResultHolder() {
+ return _innerAggregationFunction.createAggregationResultHolder();
+ }
+
+ @Override
+ public GroupByResultHolder createGroupByResultHolder(int initialCapacity, int maxCapacity) {
+ return _innerAggregationFunction.createGroupByResultHolder(initialCapacity, maxCapacity);
+ }
+
+ @Override
+ public void aggregate(int length, AggregationResultHolder aggregationResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregate(length, aggregationResultHolder, blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupBySV(int length, int[] groupKeyArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupBySV(length, groupKeyArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupByMV(int length, int[][] groupKeysArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupByMV(length, groupKeysArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public Object extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
+ return _innerAggregationFunction.extractAggregationResult(aggregationResultHolder);
+ }
+
+ @Override
+ public Object extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
+ return _innerAggregationFunction.extractGroupByResult(groupByResultHolder, groupKey);
+ }
+
+ @Override
+ public Object merge(Object intermediateResult1, Object intermediateResult2) {
+ return _innerAggregationFunction.merge(intermediateResult1, intermediateResult2);
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getIntermediateResultColumnType() {
+ return _innerAggregationFunction.getIntermediateResultColumnType();
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getFinalResultColumnType() {
+ return _innerAggregationFunction.getFinalResultColumnType();
+ }
+
+ @Override
+ public Comparable extractFinalResult(Object o) {
+ return _innerAggregationFunction.extractFinalResult(o);
+ }
+
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
This hasn't bee resolved yet.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785070934
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,94 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected List<Pair<ExpressionContext, BaseFilterOperator>> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(List<Pair<ExpressionContext, BaseFilterOperator>> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ List<Operator> operators = new ArrayList<>();
+
+ for (Pair<ExpressionContext, BaseFilterOperator> pair : _filterOperators) {
+ operators.add(pair.getRight());
+ }
+
+ return operators;
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
Would suggest implementing all the toExplainString() functions, so that EXPLAIN PLAN FOR statements work correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b124b54) into [master](https://codecov.io/gh/apache/pinot/commit/b1186001ef637c5719c76716e5ba76c6ce963f4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b118600) will **decrease** coverage by `2.15%`.
> The diff coverage is `84.23%`.
> :exclamation: Current head b124b54 differs from pull request most recent head 7d85659. Consider uploading reports for the commit 7d85659 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 70.43% 68.28% -2.16%
+ Complexity 4224 4145 -79
============================================
Files 1599 1207 -392
Lines 82965 60284 -22681
Branches 12377 9288 -3089
============================================
- Hits 58438 41165 -17273
+ Misses 20515 16231 -4284
+ Partials 4012 2888 -1124
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| unittests1 | `68.28% <84.23%> (+0.15%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...core/startree/plan/StarTreeProjectionPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlUHJvamVjdGlvblBsYW5Ob2RlLmphdmE=) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [.../core/startree/plan/StarTreeTransformPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlVHJhbnNmb3JtUGxhbk5vZGUuamF2YQ==) | `0.00% <ø> (-100.00%)` | :arrow_down: |
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `79.09% <78.40%> (-13.90%)` | :arrow_down: |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.16% <96.07%> (-0.74%)` | :arrow_down: |
| ... and [596 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b118600...7d85659](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770692340
##########
File path: pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkFilteredAggregations.java
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.queries.BaseQueriesTest;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@Fork(1)
+@Warmup(iterations = 3, time = 10)
+@Measurement(iterations = 5, time = 10)
+@State(Scope.Benchmark)
+public class BenchmarkFilteredAggregations extends BaseQueriesTest {
+
+ private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(), "FilteredAggregationsTest");
+ private static final String TABLE_NAME = "MyTable";
+ private static final String FIRST_SEGMENT_NAME = "firstTestSegment";
+ private static final String SECOND_SEGMENT_NAME = "secondTestSegment";
+ private static final String INT_COL_NAME = "INT_COL";
+ private static final String NO_INDEX_INT_COL_NAME = "NO_INDEX_INT_COL";
+ private static final Integer INT_BASE_VALUE = 0;
+ private static final Integer NUM_ROWS = 1500000;
+
+ private IndexSegment _indexSegment;
+ private List<IndexSegment> _indexSegments;
+
+ public String _filteredQuery = "SELECT SUM(INT_COL) FILTER(WHERE INT_COL > 123 AND INT_COL < 599999),"
+ + "MAX(INT_COL) FILTER(WHERE INT_COL > 123 AND INT_COL < 599999) "
+ + "FROM MyTable WHERE NO_INDEX_INT_COL > 5 AND NO_INDEX_INT_COL < 1499999";
+
+ public String _nonFilteredQuery = "SELECT SUM("
+ + "CASE "
+ + "WHEN (INT_COL > 123 AND INT_COL < 599999) THEN INT_COL "
+ + "ELSE 0 "
+ + "END) AS total_sum,"
+ + "MAX("
+ + "CASE "
+ + "WHEN (INT_COL > 123 AND INT_COL < 599999) THEN INT_COL "
+ + "ELSE 0 "
+ + "END) AS total_avg "
+ + "FROM MyTable WHERE NO_INDEX_INT_COL > 5 AND NO_INDEX_INT_COL < 1499999";
+
+ @Setup
+ public void setUp()
+ throws Exception {
+ FileUtils.deleteQuietly(INDEX_DIR);
+
+ buildSegment(FIRST_SEGMENT_NAME);
+ buildSegment(SECOND_SEGMENT_NAME);
+ IndexLoadingConfig indexLoadingConfig = new IndexLoadingConfig();
+
+ Set<String> invertedIndexCols = new HashSet<>();
+ invertedIndexCols.add(INT_COL_NAME);
+
+ indexLoadingConfig.setRangeIndexColumns(invertedIndexCols);
+ indexLoadingConfig.setInvertedIndexColumns(invertedIndexCols);
+
+ ImmutableSegment firstImmutableSegment =
+ ImmutableSegmentLoader.load(new File(INDEX_DIR, FIRST_SEGMENT_NAME), indexLoadingConfig);
+ ImmutableSegment secondImmutableSegment =
+ ImmutableSegmentLoader.load(new File(INDEX_DIR, SECOND_SEGMENT_NAME), indexLoadingConfig);
+ _indexSegment = firstImmutableSegment;
+ _indexSegments = Arrays.asList(firstImmutableSegment, secondImmutableSegment);
+ }
+
+ @TearDown
+ public void tearDown() {
+ for (IndexSegment indexSegment : _indexSegments) {
+ indexSegment.destroy();
+ }
+
+ FileUtils.deleteQuietly(INDEX_DIR);
+ }
+
+ private List<GenericRow> createTestData(int numRows) {
+ List<GenericRow> rows = new ArrayList<>();
+
+ for (int i = 0; i < numRows; i++) {
+ GenericRow row = new GenericRow();
+ row.putField(INT_COL_NAME, INT_BASE_VALUE + i);
+ row.putField(NO_INDEX_INT_COL_NAME, INT_BASE_VALUE + i);
+
+ rows.add(row);
+ }
+ return rows;
+ }
+
+ private void buildSegment(String segmentName)
+ throws Exception {
+ List<GenericRow> rows = createTestData(NUM_ROWS);
+ List<FieldConfig> fieldConfigs = new ArrayList<>();
+
+ TableConfig tableConfig = new TableConfigBuilder(TableType.OFFLINE).setTableName(TABLE_NAME)
+ .setInvertedIndexColumns(Arrays.asList(INT_COL_NAME)).setFieldConfigList(fieldConfigs).build();
+ Schema schema = new Schema.SchemaBuilder().setSchemaName(TABLE_NAME)
+ .addSingleValueDimension(NO_INDEX_INT_COL_NAME, FieldSpec.DataType.INT)
+ .addSingleValueDimension(INT_COL_NAME, FieldSpec.DataType.INT).build();
+ SegmentGeneratorConfig config = new SegmentGeneratorConfig(tableConfig, schema);
+ config.setOutDir(INDEX_DIR.getPath());
+ config.setTableName(TABLE_NAME);
+ config.setSegmentName(segmentName);
+
+ SegmentIndexCreationDriverImpl driver = new SegmentIndexCreationDriverImpl();
+ try (RecordReader recordReader = new GenericRowRecordReader(rows)) {
+ driver.init(config, recordReader);
+ driver.build();
+ }
+ }
+
+ @Benchmark
+ public void testFilteredAggregations(Blackhole blackhole) {
+ blackhole.consume(getBrokerResponseForSqlQuery(_filteredQuery));
+ }
+
+ @Benchmark
+ public void testNonFilteredAggregations(Blackhole blackhole) {
+ blackhole.consume(getBrokerResponseForSqlQuery(_nonFilteredQuery));
+ }
Review comment:
Just FYI, it's ok and encouraged to just return the value here, no need to blackhole the value yourself.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770683666
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedTransformBlock.java
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdSet;
+import org.apache.pinot.core.common.BlockDocIdValueSet;
+import org.apache.pinot.core.common.BlockMetadata;
+import org.apache.pinot.core.common.BlockValSet;
+
+/**
+ * Represents a combination of multiple TransformBlock instances
+ */
+public class CombinedTransformBlock extends TransformBlock {
+ protected Map<ExpressionContext, TransformBlock> _transformBlockMap;
+ protected ExpressionContext _mainPredicateExpressionContext;
+
+ public CombinedTransformBlock(Map<ExpressionContext, TransformBlock> transformBlockMap,
+ ExpressionContext mainPredicateExpressionContext) {
+ super(transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._projectionBlock,
+ transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._transformFunctionMap);
Review comment:
I mean add:
```java
protected TransformBlock(TransformBlock transformBlock) {
_projectionBlock = transformBlock == null ? null : transformBlock._projectionBlock;
_transformFunctionMap = transformBlock == null ? null : transformBlock. _transformFunctionMap;
}
```
to the class `TransformBlock`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794305426
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
I haven't seen a coding convention mentioning the same, yet. Is this documented somewhere?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ec5b19c) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `6.44%`.
> The diff coverage is `87.85%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 64.80% -6.45%
- Complexity 4262 4263 +1
============================================
Files 1607 1566 -41
Lines 83409 81679 -1730
Branches 12458 12273 -185
============================================
- Hits 59426 52933 -6493
- Misses 19941 24980 +5039
+ Partials 4042 3766 -276
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.02% <87.85%> (+0.15%)` | :arrow_up: |
| unittests2 | `14.23% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `77.06% <87.65%> (-15.92%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.15% <96.00%> (-0.75%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [382 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...ec5b19c](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770696101
##########
File path: pinot-perf/src/main/java/org/apache/pinot/perf/BenchmarkFilteredAggregations.java
##########
@@ -0,0 +1,192 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.perf;
+
+import java.io.File;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashSet;
+import java.util.List;
+import java.util.Set;
+import java.util.concurrent.TimeUnit;
+import org.apache.commons.io.FileUtils;
+import org.apache.pinot.queries.BaseQueriesTest;
+import org.apache.pinot.segment.local.indexsegment.immutable.ImmutableSegmentLoader;
+import org.apache.pinot.segment.local.segment.creator.impl.SegmentIndexCreationDriverImpl;
+import org.apache.pinot.segment.local.segment.index.loader.IndexLoadingConfig;
+import org.apache.pinot.segment.local.segment.readers.GenericRowRecordReader;
+import org.apache.pinot.segment.spi.ImmutableSegment;
+import org.apache.pinot.segment.spi.IndexSegment;
+import org.apache.pinot.segment.spi.creator.SegmentGeneratorConfig;
+import org.apache.pinot.spi.config.table.FieldConfig;
+import org.apache.pinot.spi.config.table.TableConfig;
+import org.apache.pinot.spi.config.table.TableType;
+import org.apache.pinot.spi.data.FieldSpec;
+import org.apache.pinot.spi.data.Schema;
+import org.apache.pinot.spi.data.readers.GenericRow;
+import org.apache.pinot.spi.data.readers.RecordReader;
+import org.apache.pinot.spi.utils.builder.TableConfigBuilder;
+import org.openjdk.jmh.annotations.Benchmark;
+import org.openjdk.jmh.annotations.BenchmarkMode;
+import org.openjdk.jmh.annotations.Fork;
+import org.openjdk.jmh.annotations.Measurement;
+import org.openjdk.jmh.annotations.Mode;
+import org.openjdk.jmh.annotations.OutputTimeUnit;
+import org.openjdk.jmh.annotations.Scope;
+import org.openjdk.jmh.annotations.Setup;
+import org.openjdk.jmh.annotations.State;
+import org.openjdk.jmh.annotations.TearDown;
+import org.openjdk.jmh.annotations.Warmup;
+import org.openjdk.jmh.infra.Blackhole;
+import org.openjdk.jmh.runner.Runner;
+import org.openjdk.jmh.runner.options.OptionsBuilder;
+
+@BenchmarkMode(Mode.AverageTime)
+@OutputTimeUnit(TimeUnit.MILLISECONDS)
+@Fork(1)
+@Warmup(iterations = 3, time = 10)
+@Measurement(iterations = 5, time = 10)
+@State(Scope.Benchmark)
+public class BenchmarkFilteredAggregations extends BaseQueriesTest {
+
+ private static final File INDEX_DIR = new File(FileUtils.getTempDirectory(), "FilteredAggregationsTest");
+ private static final String TABLE_NAME = "MyTable";
+ private static final String FIRST_SEGMENT_NAME = "firstTestSegment";
+ private static final String SECOND_SEGMENT_NAME = "secondTestSegment";
+ private static final String INT_COL_NAME = "INT_COL";
+ private static final String NO_INDEX_INT_COL_NAME = "NO_INDEX_INT_COL";
+ private static final Integer INT_BASE_VALUE = 0;
+ private static final Integer NUM_ROWS = 1500000;
Review comment:
It's not a good practice to use `static final` values for benchmark parameters because there's a risk of constant folding. I can't say how likely it is in this particular case without inspecting the JIT'd code but if 1500000 propagates to a loop it will alter the way the loop gets compiled, unrealistically. It's better to do:
```java
@Param("1500000")
int _numRows;
@Param("0")
int _intBaseValue;
```
even if you only intend to use one value, and it also allows the size of the benchmark to be overridden by the JMH CLI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770668157
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/SwimLaneDocIdSetOperator.java
##########
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator;
+
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.DocIdSetBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
+import org.apache.pinot.core.operator.filter.CombinedFilterOperator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+
+/**
+ * DocIdSetOperator for a swimlane query plan.
+ */
+public class SwimLaneDocIdSetOperator extends DocIdSetOperator {
+ private static final String OPERATOR_NAME = "SwimLaneDocIdSetOperator";
+
+ private final CombinedFilterOperator _filterOperator;
+ private final ExpressionContext _expressionContext;
+ private final int _maxSizeOfDocIdSet;
+
+ private FilterBlockDocIdSet _filterBlockDocIdSet;
+ private BlockDocIdIterator _blockDocIdIterator;
+ private int _currentDocId = 0;
+
+ public SwimLaneDocIdSetOperator(CombinedFilterOperator filterOperator, ExpressionContext expressionContext,
+ int maxSizeOfDocIdSet) {
+ super(filterOperator, maxSizeOfDocIdSet);
+
+ Preconditions.checkArgument(maxSizeOfDocIdSet > 0 && maxSizeOfDocIdSet <= DocIdSetPlanNode.MAX_DOC_PER_CALL);
+ _filterOperator = filterOperator;
+ _expressionContext = expressionContext;
+ _maxSizeOfDocIdSet = maxSizeOfDocIdSet;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return Arrays.asList(_filterOperator);
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected DocIdSetBlock getNextBlock() {
+ if (_currentDocId == Constants.EOF) {
+ return null;
+ }
+
+ // Initialize filter block document Id set
+ if (_filterBlockDocIdSet == null) {
+ CombinedFilterBlock combinedFilterBlock = (CombinedFilterBlock) _filterOperator.nextBlock();
+ FilterBlock filterBlock = combinedFilterBlock.getFilterBlock(_expressionContext);
+
+ if (filterBlock == null) {
+ throw new IllegalStateException("Null block seen");
+ }
+
+ _filterBlockDocIdSet = filterBlock.getBlockDocIdSet();
+ _blockDocIdIterator = _filterBlockDocIdSet.iterator();
+ }
+
+ int pos = 0;
+ int[] docIds = new int[DocIdSetPlanNode.MAX_DOC_PER_CALL];
+
+ for (int i = 0; i < _maxSizeOfDocIdSet; i++) {
+ _currentDocId = _blockDocIdIterator.next();
+ if (_currentDocId == Constants.EOF) {
+ break;
+ }
+ docIds[pos++] = _currentDocId;
+ }
+ if (pos > 0) {
+ return new DocIdSetBlock(docIds, pos);
+ } else {
+ return null;
+ }
Review comment:
This can be reworked slightly to improve this for empty blocks.
```java
_currentDocId = _blockDocIdIterator.next();
if (_currentDocId == Constants.EOF) {
return null;
}
int[] docIds = new int[_maxSizeOfDocIdSet];
int pos = 0;
do {
docIds[pos++] = _currentDocId;
_currentDocId = _blockDocIdIterator.next();
} while (pos < docIds.length && _currentDocId != Constants.EOF);
return new DocIdSetBlock(docIds, pos);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770677607
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ Map<ExpressionContext, BaseFilterOperator> baseFilterOperatorMap =
+ new HashMap<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorMap.put(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight());
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
Review comment:
I'm not very familiar with this code, but I've noticed there are lots of `Map`s used where it looks like `List<Pair<K, V>>` is the right choice. Is `filterPredicatesAndMetadata` ever expected to contain duplicate `FilterPlanNode` keys?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] MrNeocore commented on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
MrNeocore commented on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996973701
Thanks you @atris !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770868089
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(Map<ExpressionContext, BaseFilterOperator> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return new ArrayList<>(_filterOperators.values());
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected FilterBlock getNextBlock() {
+ if (_resultBlock != null) {
+ return _resultBlock;
+ }
+
+ FilterBlock mainFilterBlock = _mainFilterOperator.nextBlock();
+
+ Map<ExpressionContext, FilterBlock> filterBlockMap = new HashMap<>();
+ Iterator<Map.Entry<ExpressionContext, BaseFilterOperator>> iterator = _filterOperators.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, BaseFilterOperator> entry = iterator.next();
+ FilterBlock subFilterBlock = entry.getValue().nextBlock();
+
+ filterBlockMap.put(entry.getKey(),
Review comment:
This isn't meant for deduplicating (FYI, there will be no duplicates in _filterOperators) but for letting each swim lane find its relevant filter block, since CombinedFilterOperator is shared amongst all swim lanes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris closed pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris closed pull request #7916:
URL: https://github.com/apache/pinot/pull/7916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794293449
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +118,181 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator, int numTotalDocs) {
+ Map<FilterContext, Pair<List<AggregationFunction>, TransformOperator>> filterContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+ List<Pair<FilterContext, AggregationFunction>> aggregationFunctions = _queryContext
+ .getAggregationsWithFilters();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (Pair<FilterContext, AggregationFunction> inputPair : aggregationFunctions) {
+ if (inputPair.getLeft() != null) {
+ FilterContext currentFilterExpression = inputPair.getLeft();
+ if (filterContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ filterContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(inputPair.getRight());
+ continue;
+ }
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(currentFilterExpression);
+ BaseFilterOperator wrappedFilterOperator = new CombinedFilterOperator(mainPredicateFilterOperator,
+ pair.getRight());
+ Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> innerPair =
+ buildOperators(wrappedFilterOperator, pair.getLeft());
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ List aggFunctionList = new ArrayList<>();
+ aggFunctionList.add(inputPair.getRight());
+ filterContextToAggFuncsMap.put(currentFilterExpression,
+ Pair.of(aggFunctionList, innerPair.getLeft()));
+ } else {
+ nonFilteredAggregationFunctions.add(inputPair.getRight());
+ }
+ }
+ List<Pair<AggregationFunction[], TransformOperator>> aggToTransformOpList =
+ new ArrayList<>();
+ // Convert to array since FilteredAggregationOperator expects it
+ for (Pair<List<AggregationFunction>, TransformOperator> pair
+ : filterContextToAggFuncsMap.values()) {
+ List<AggregationFunction> aggregationFunctionList = pair.getLeft();
+ if (aggregationFunctionList == null) {
+ throw new IllegalStateException("Null aggregation list seen");
+ }
+ aggToTransformOpList.add(Pair.of(aggregationFunctionList.toArray(new AggregationFunction[0]),
+ pair.getRight()));
+ }
+ aggToTransformOpList.add(Pair.of(nonFilteredAggregationFunctions.toArray(new AggregationFunction[0]),
+ mainTransformOperator));
+
+ return new FilteredAggregationOperator(_queryContext.getAggregationFunctions(), aggToTransformOpList,
+ numTotalDocs);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @return Pair, consisting of the built TransformOperator and Aggregation operator for chain
+ */
+ private Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> buildOperators(BaseFilterOperator filterOperator,
+ FilterPlanNode filterPlanNode) {
+ assert _queryContext.getAggregationFunctions() != null;
+
+ int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
+ AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+
+ Set<ExpressionContext> expressionsToTransform =
+ AggregationFunctionUtils.collectExpressionsToTransform(aggregationFunctions, null);
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+
+ List<StarTreeV2> starTrees = _indexSegment.getStarTrees();
+
+ // Use metadata/dictionary to solve the query if possible
+ // TODO: Use the same operator for both of them so that COUNT(*), MAX(col) can be optimized
+ if (filterOperator.isResultMatchingAll() && !hasFilteredPredicates) {
+ if (isFitForMetadataBasedPlan(aggregationFunctions)) {
+ return Pair.of(null, new MetadataBasedAggregationOperator(aggregationFunctions,
+ _indexSegment.getSegmentMetadata(), Collections.emptyMap()));
+ } else if (isFitForDictionaryBasedPlan(aggregationFunctions, _indexSegment)) {
+ Map<String, Dictionary> dictionaryMap = new HashMap<>();
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ String column = ((ExpressionContext) aggregationFunction.getInputExpressions().get(0)).getIdentifier();
+ dictionaryMap.computeIfAbsent(column, k -> _indexSegment.getDataSource(k).getDictionary());
+ }
+ return Pair.of(null,
+ new DictionaryBasedAggregationOperator(aggregationFunctions, dictionaryMap,
+ numTotalDocs));
+ }
+ }
+
+ if (starTrees != null && !StarTreeUtils.isStarTreeDisabled(_queryContext)) {
+ // Use star-tree to solve the query if possible
+
+ AggregationFunctionColumnPair[] aggregationFunctionColumnPairs =
+ StarTreeUtils.extractAggregationFunctionPairs(aggregationFunctions);
+ if (aggregationFunctionColumnPairs != null) {
+ Map<String, List<CompositePredicateEvaluator>> predicateEvaluatorsMap =
+ StarTreeUtils.extractPredicateEvaluatorsMap(_indexSegment, _queryContext.getFilter(),
+ filterPlanNode.getPredicateEvaluatorMap());
+ if (predicateEvaluatorsMap != null) {
+ for (StarTreeV2 starTreeV2 : starTrees) {
+ if (StarTreeUtils.isFitForStarTree(starTreeV2.getMetadata(), aggregationFunctionColumnPairs, null,
+ predicateEvaluatorsMap.keySet())) {
+
+ TransformOperator transformOperator = new StarTreeTransformPlanNode(starTreeV2,
+ aggregationFunctionColumnPairs, null,
+ predicateEvaluatorsMap, _queryContext.getDebugOptions()).run();
+ AggregationOperator aggregationOperator = new AggregationOperator(aggregationFunctions,
+ transformOperator, numTotalDocs, true);
+
+ return Pair.of(transformOperator, aggregationOperator);
+ }
+ }
+ }
+ }
+ }
+
+ TransformOperator transformOperator = new TransformPlanNode(_indexSegment, _queryContext,
+ expressionsToTransform, DocIdSetPlanNode.MAX_DOC_PER_CALL, filterOperator).run();
+ AggregationOperator aggregationOperator = new AggregationOperator(aggregationFunctions,
+ transformOperator, numTotalDocs, false);
+
+ return Pair.of(transformOperator, aggregationOperator);
+ }
+
+ /**
+ * Builds the operator to be used for non filtered aggregations
+ */
+ private BaseOperator<IntermediateResultsBlock> buildNonFilteredAggOperator() {
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771099403
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/startree/plan/StarTreeDocIdSetPlanNode.java
##########
@@ -31,16 +32,29 @@
public class StarTreeDocIdSetPlanNode implements PlanNode {
Review comment:
This class is pretty brief, and I would honestly avoid adding new plan nodes unless there is a major functionality that is different.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771145891
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/SwimLaneDocIdSetOperator.java
##########
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator;
+
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.DocIdSetBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
+import org.apache.pinot.core.operator.filter.CombinedFilterOperator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+
+/**
+ * DocIdSetOperator for a swimlane query plan.
+ */
+public class SwimLaneDocIdSetOperator extends DocIdSetOperator {
+ private static final String OPERATOR_NAME = "SwimLaneDocIdSetOperator";
+
+ private final CombinedFilterOperator _filterOperator;
+ private final ExpressionContext _expressionContext;
+ private final int _maxSizeOfDocIdSet;
+
+ private FilterBlockDocIdSet _filterBlockDocIdSet;
+ private BlockDocIdIterator _blockDocIdIterator;
+ private int _currentDocId = 0;
+
+ public SwimLaneDocIdSetOperator(CombinedFilterOperator filterOperator, ExpressionContext expressionContext,
+ int maxSizeOfDocIdSet) {
+ super(filterOperator, maxSizeOfDocIdSet);
+
+ Preconditions.checkArgument(maxSizeOfDocIdSet > 0 && maxSizeOfDocIdSet <= DocIdSetPlanNode.MAX_DOC_PER_CALL);
+ _filterOperator = filterOperator;
+ _expressionContext = expressionContext;
+ _maxSizeOfDocIdSet = maxSizeOfDocIdSet;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return Arrays.asList(_filterOperator);
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
I think this should be implemented so that this and any other operators being added can be properly displayed in EXPLAIN PLAN output. Also it would be great to add an aggregate filter query to `ExplainPlanQueriesTest.java`.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/aggregation/function/FilterableAggregationFunction.java
##########
@@ -0,0 +1,139 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.query.aggregation.function;
+
+import java.util.List;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.common.request.context.FilterContext;
+import org.apache.pinot.common.utils.DataSchema;
+import org.apache.pinot.core.common.BlockValSet;
+import org.apache.pinot.core.query.aggregation.AggregationResultHolder;
+import org.apache.pinot.core.query.aggregation.groupby.GroupByResultHolder;
+import org.apache.pinot.segment.spi.AggregationFunctionType;
+
+/**
+ * Represents a filtered aggregation
+ */
+public class FilterableAggregationFunction implements
+ AggregationFunction<Object, Comparable> {
+ private AggregationFunction<Object, Comparable> _innerAggregationFunction;
+ private ExpressionContext _associatedExpressionContext;
+ private FilterContext _filterContext;
+
+ public FilterableAggregationFunction(AggregationFunction aggregationFunction,
+ ExpressionContext associatedExpressionContext, FilterContext filterContext) {
+ _innerAggregationFunction = aggregationFunction;
+ _associatedExpressionContext = associatedExpressionContext;
+ _filterContext = filterContext;
+ }
+
+ @Override
+ public AggregationFunctionType getType() {
+ return _innerAggregationFunction.getType();
+ }
+
+ @Override
+ public String getColumnName() {
+ return _innerAggregationFunction.getColumnName();
+ }
+
+ @Override
+ public String getResultColumnName() {
+ return _innerAggregationFunction.getResultColumnName();
+ }
+
+ @Override
+ public List<ExpressionContext> getInputExpressions() {
+ return _innerAggregationFunction.getInputExpressions();
+ }
+
+ @Override
+ public AggregationResultHolder createAggregationResultHolder() {
+ return _innerAggregationFunction.createAggregationResultHolder();
+ }
+
+ @Override
+ public GroupByResultHolder createGroupByResultHolder(int initialCapacity, int maxCapacity) {
+ return _innerAggregationFunction.createGroupByResultHolder(initialCapacity, maxCapacity);
+ }
+
+ @Override
+ public void aggregate(int length, AggregationResultHolder aggregationResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregate(length, aggregationResultHolder, blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupBySV(int length, int[] groupKeyArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupBySV(length, groupKeyArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public void aggregateGroupByMV(int length, int[][] groupKeysArray, GroupByResultHolder groupByResultHolder,
+ Map<ExpressionContext, BlockValSet> blockValSetMap) {
+ _innerAggregationFunction.aggregateGroupByMV(length, groupKeysArray, groupByResultHolder,
+ blockValSetMap);
+ }
+
+ @Override
+ public Object extractAggregationResult(AggregationResultHolder aggregationResultHolder) {
+ return _innerAggregationFunction.extractAggregationResult(aggregationResultHolder);
+ }
+
+ @Override
+ public Object extractGroupByResult(GroupByResultHolder groupByResultHolder, int groupKey) {
+ return _innerAggregationFunction.extractGroupByResult(groupByResultHolder, groupKey);
+ }
+
+ @Override
+ public Object merge(Object intermediateResult1, Object intermediateResult2) {
+ return _innerAggregationFunction.merge(intermediateResult1, intermediateResult2);
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getIntermediateResultColumnType() {
+ return _innerAggregationFunction.getIntermediateResultColumnType();
+ }
+
+ @Override
+ public DataSchema.ColumnDataType getFinalResultColumnType() {
+ return _innerAggregationFunction.getFinalResultColumnType();
+ }
+
+ @Override
+ public Comparable extractFinalResult(Object o) {
+ return _innerAggregationFunction.extractFinalResult(o);
+ }
+
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
This one too.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794301294
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -119,11 +122,13 @@ private QueryContext(String tableName, List<ExpressionContext> selectExpressions
@Nullable FilterContext filter, @Nullable List<ExpressionContext> groupByExpressions,
@Nullable FilterContext havingFilter, @Nullable List<OrderByExpressionContext> orderByExpressions, int limit,
int offset, Map<String, String> queryOptions, @Nullable Map<String, String> debugOptions,
- BrokerRequest brokerRequest) {
+ BrokerRequest brokerRequest, boolean hasFilteredAggregations,
Review comment:
Good catch, fixed.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -350,6 +381,7 @@ public String toString() {
private List<ExpressionContext> _selectExpressions;
private List<String> _aliasList;
private FilterContext _filter;
+ private ExpressionContext _filterExpression;
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794900915
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
We should avoid using stream api for performance critical operations. This one is at query path, but might not be that performance critical (only called once). Using regular api could give slightly better performance, but IMO both way is okay
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r784428201
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/ProjectionOperator.java
##########
@@ -63,7 +63,7 @@ protected ProjectionBlock getNextBlock() {
return null;
} else {
_dataBlockCache.initNewBlock(docIdSetBlock.getDocIdSet(), docIdSetBlock.getSearchableLength());
- return new ProjectionBlock(_dataSourceMap, _dataBlockCache);
+ return new ProjectionBlock(_dataSourceMap, _dataBlockCache, docIdSetBlock);
Review comment:
I don't think this change is required anymore. Same for the change in `ProjectionBlock` and `TransformBlock`
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationGroupByOrderByPlanNode.java
##########
@@ -77,7 +77,7 @@ public AggregationGroupByOrderByOperator run() {
groupByExpressions, predicateEvaluatorsMap.keySet())) {
TransformOperator transformOperator =
new StarTreeTransformPlanNode(starTreeV2, aggregationFunctionColumnPairs, groupByExpressions,
- predicateEvaluatorsMap, _queryContext.getDebugOptions()).run();
+ predicateEvaluatorsMap, null, _queryContext.getDebugOptions()).run();
Review comment:
This change should not be required anymore. We should only need to change the `AggregationPlanNode`
##########
File path: pinot-core/src/test/java/org/apache/pinot/core/query/request/context/utils/BrokerRequestToQueryContextConverterTest.java
##########
@@ -577,20 +575,6 @@ public void testHardcodedQueries() {
};
}
- @Test
- public void testFilteredAggregations() {
Review comment:
We should keep this test
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
+ private boolean _hasFilteredAggregations;
// TODO: Use Pair<FunctionContext, FilterContext> as key to support filtered aggregations in order-by and post
// aggregation
+ private Map<Pair<FunctionContext, FilterContext>, Integer> _filteredAgggregationsIndexMap;
Review comment:
```suggestion
private Map<Pair<FunctionContext, FilterContext>, Integer> _filteredAggregationsIndexMap;
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/FilterBlock.java
##########
@@ -54,4 +65,4 @@ public BlockDocIdValueSet getBlockDocIdValueSet() {
public BlockMetadata getMetadata() {
throw new UnsupportedOperationException();
}
-}
+}
Review comment:
（minor) revert
##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/InterSegmentAggregationMultiValueQueriesTest.java
##########
@@ -651,6 +651,13 @@ public void testNumGroupsLimit() {
assertTrue(brokerResponse.isNumGroupsLimitReached());
}
+ @Test
+ public void testFilteredAggregations() {
+ String query = "SELECT COUNT(*) FILTER(WHERE column1 > 5) FROM testTable WHERE column3 > 0";
+ BrokerResponseNative brokerResponse = getBrokerResponseForSqlQuery(query);
+ assertEquals(brokerResponse.getResultTable().getRows().size(), 1);
Review comment:
Also verify the result?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
+ private boolean _hasFilteredAggregations;
// TODO: Use Pair<FunctionContext, FilterContext> as key to support filtered aggregations in order-by and post
Review comment:
This TODO can be removed
##########
File path: pinot-core/src/test/java/org/apache/pinot/queries/InnerSegmentAggregationSingleValueQueriesTest.java
##########
@@ -49,27 +49,49 @@
" GROUP BY column1, column3, column6, column7, column9, column11, column12, column17, column18";
@Test
- public void testAggregationOnly() {
Review comment:
Don't remove the existing `testAggregationOnly()`. Since we added a new `FilteredAggregationsTest`, not sure if we still want to add this test. If so, let's keep them as 2 separate tests
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,57 +69,25 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
- int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
- AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
- FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext);
- BaseFilterOperator filterOperator = filterPlanNode.run();
+ Pair<FilterPlanNode, BaseFilterOperator> filterOperatorPair =
Review comment:
Suggest splitting the logic of filtered aggregation and regular aggregation. Currently the logic is mixed in `buildOperators()` which is hard to read or debug.
We can keep the regular aggregation handling unchanged, and add the code path for the filtered aggregations. The metadata/dictionary/startree based operator does not apply to the filtered aggregation.
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
Review comment:
Can we keep this list and remove the `FilterableAggregationFunction`? We should avoid adding this special aggregation function if possible because the filter is not part of the aggregation
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (7d85659) into [master](https://codecov.io/gh/apache/pinot/commit/b1186001ef637c5719c76716e5ba76c6ce963f4e?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (b118600) will **decrease** coverage by `56.13%`.
> The diff coverage is `0.00%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 70.43% 14.30% -56.14%
+ Complexity 4224 81 -4143
=============================================
Files 1599 1558 -41
Lines 82965 81240 -1725
Branches 12377 12192 -185
=============================================
- Hits 58438 11622 -46816
- Misses 20515 68758 +48243
+ Partials 4012 860 -3152
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.30% <0.00%> (-0.03%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `0.00% <0.00%> (-57.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `0.00% <0.00%> (-92.99%)` | :arrow_down: |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `0.00% <0.00%> (-86.87%)` | :arrow_down: |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `0.00% <0.00%> (-92.31%)` | :arrow_down: |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `0.00% <0.00%> (-97.91%)` | :arrow_down: |
| ... and [1260 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [b118600...7d85659](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r787518816
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,57 +69,25 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
- int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
- AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
- FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext);
- BaseFilterOperator filterOperator = filterPlanNode.run();
+ Pair<FilterPlanNode, BaseFilterOperator> filterOperatorPair =
Review comment:
I am not sure if I understand -- why should star-tree be not used for a filtered aggregation? If a swim lane is able to use star tree to process the underlying predicate, shouldn't we allow that? Or is there a technical limitation to that?
Regarding the forking, I agree the code readability was an issue. I have fixed the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bd19945) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `3.26%`.
> The diff coverage is `88.16%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 67.97% -3.27%
+ Complexity 4262 4180 -82
============================================
Files 1607 1215 -392
Lines 83409 60703 -22706
Branches 12458 9368 -3090
============================================
- Hits 59426 41264 -18162
+ Misses 19941 16535 -3406
+ Partials 4042 2904 -1138
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.97% <88.16%> (+0.10%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `75.75% <88.09%> (-17.23%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.58% <97.77%> (-0.33%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [624 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...bd19945](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris merged pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris merged pull request #7916:
URL: https://github.com/apache/pinot/pull/7916
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770635998
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedTransformBlock.java
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdSet;
+import org.apache.pinot.core.common.BlockDocIdValueSet;
+import org.apache.pinot.core.common.BlockMetadata;
+import org.apache.pinot.core.common.BlockValSet;
+
+/**
+ * Represents a combination of multiple TransformBlock instances
+ */
+public class CombinedTransformBlock extends TransformBlock {
+ protected Map<ExpressionContext, TransformBlock> _transformBlockMap;
+ protected ExpressionContext _mainPredicateExpressionContext;
+
+ public CombinedTransformBlock(Map<ExpressionContext, TransformBlock> transformBlockMap,
+ ExpressionContext mainPredicateExpressionContext) {
+ super(transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._projectionBlock,
+ transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._transformFunctionMap);
Review comment:
This looks messy and is wasteful, I suggest adding a copy constructor to `TransformBlock`:
```java
protected TransformBlock(TransformBlock transformBlock) {
_projectionBlock = transformBlock == null ? null : transformBlock._projectionBlock;
_transformFunctionMap = transformBlock == null ? null : transformBlock. _transformFunctionMap;
}
...
super(transformBlockMap.get(mainPredicateExpressionContext));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770637128
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedFilterBlock.java
##########
@@ -0,0 +1,47 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+
+
+/**
+ * Contains a mapping between expressions and filter blocks
+ */
+public class CombinedFilterBlock extends FilterBlock {
+ private final Map<ExpressionContext, FilterBlock> _filterBlockMap;
+
+ public CombinedFilterBlock(Map<ExpressionContext, FilterBlock> filterBlockMap,
+ FilterBlock mainFilterBlock) {
+ // Initialize with main predicate's filter block to allow main predicate
+ // chain to operate normally
+ super(mainFilterBlock.getBlockDocIdSet());
+
+ _filterBlockMap = filterBlockMap;
+ }
+
+ public FilterBlock getFilterBlock(ExpressionContext expressionContext) {
+ if (expressionContext == null) {
+ throw new IllegalStateException("ExpressionContext is null");
+ }
+
+ return _filterBlockMap.get(expressionContext);
Review comment:
```java
return _filterBlockMap.get(Objects.requireNonNull(expressionContext, "ExpressionContext is null"));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d87c14) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `6.58%`.
> The diff coverage is `54.50%`.
> :exclamation: Current head 5d87c14 differs from pull request most recent head a089a68. Consider uploading reports for the commit a089a68 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.63% 65.04% -6.59%
- Complexity 4088 4090 +2
============================================
Files 1580 1551 -29
Lines 81100 80523 -577
Branches 12068 12099 +31
============================================
- Hits 58093 52374 -5719
- Misses 19092 24417 +5325
+ Partials 3915 3732 -183
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `68.43% <56.39%> (-0.33%)` | :arrow_down: |
| unittests2 | `14.34% <0.94%> (-0.22%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `21.51% <ø> (-49.79%)` | :arrow_down: |
| [...roker/requesthandler/GrpcBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvR3JwY0Jyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `89.47% <ø> (ø)` | |
| [...ller/api/resources/PinotControllerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90Q29udHJvbGxlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...e/pinot/core/common/datatable/DataTableImplV2.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMi5qYXZh) | `74.24% <0.00%> (-3.54%)` | :arrow_down: |
| [...e/pinot/core/common/datatable/DataTableImplV3.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMy5qYXZh) | `92.39% <0.00%> (-3.66%)` | :arrow_down: |
| [...re/operator/StreamingInstanceResponseOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9TdHJlYW1pbmdJbnN0YW5jZVJlc3BvbnNlT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...ot/core/operator/blocks/InstanceResponseBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvSW5zdGFuY2VSZXNwb25zZUJsb2NrLmphdmE=) | `37.50% <0.00%> (-8.66%)` | :arrow_down: |
| [...r/transform/function/TransformFunctionFactory.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vZnVuY3Rpb24vVHJhbnNmb3JtRnVuY3Rpb25GYWN0b3J5LmphdmE=) | `79.43% <0.00%> (-5.18%)` | :arrow_down: |
| ... and [466 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...a089a68](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770796515
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ Map<ExpressionContext, BaseFilterOperator> baseFilterOperatorMap =
+ new HashMap<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorMap.put(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight());
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
Review comment:
Sorry, I meant baseOperatorMap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785075210
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ List<Pair<ExpressionContext, BaseFilterOperator>> baseFilterOperatorList =
+ new ArrayList<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorList.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight()));
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
+ }
+ }
+
+ CombinedFilterOperator combinedFilterOperator = new CombinedFilterOperator(baseFilterOperatorList,
+ mainPredicateFilterOperator);
+
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ for (Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>> pair
+ : filterPredicatesAndMetadata) {
+ Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> innerPair =
+ buildOperators(combinedFilterOperator, pair.getRight().getLeft(),
+ true, pair.getLeft());
+
+ transformOperatorMap.put(pair.getLeft(), innerPair.getLeft());
+ }
+
+ // Add the main predicate filter operator to the map
+ transformOperatorMap.put(_queryContext.getFilterExpression(), nonFilteredTransformOperator);
+
+ return new CombinedTransformOperator(transformOperatorMap, _queryContext.getFilterExpression(),
+ expressionsToTransform);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @param isSwimlane Is this plan a swim lane?
Review comment:
This doesn't appear to be resolved. I understand that you have a design doc, but it will be useful to describe "SwimLane" in code as well to make the code more readable since references to design doc tend to get lost over time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r788373396
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator,
+ AggregationFunction[] aggregationFunctions, int numTotalDocs) {
+ Map<ExpressionContext, Pair<List<AggregationFunction>, TransformOperator>> expressionContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+
+ ExpressionContext currentFilterExpression = filterableAggregationFunction
+ .getAssociatedExpressionContext();
+
+ if (expressionContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ expressionContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(aggregationFunction);
Review comment:
Oh, actually it is already handled in `FilteredAggregationOperator` where aggregations with the same filter is grouped together, and processed on each transform block. nvm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770699770
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
Review comment:
I'm curious if there is a reason this couldn't be `List<Pair<ExpressionContext, BaseFilterOperator>>` - is the map really serving a purpose of deduplicating expression contexts?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770648030
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/CombinedTransformOperator.java
##########
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.operator.blocks.CombinedTransformBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+
+
+/**
+ * Used for processing a set of TransformOperators, fed by an underlying
+ * main predicate transform operator.
+ *
+ * This class returns a CombinedTransformBlock, with blocks ordered in
+ * the order in which their parent filter clauses appear in the query
+ */
+public class CombinedTransformOperator extends TransformOperator {
+ private static final String OPERATOR_NAME = "CombinedTransformOperator";
+
+ protected final Map<ExpressionContext, TransformOperator> _transformOperatorMap;
+ protected final ExpressionContext _mainPredicateExpression;
+
+ /**
+ * Constructor for the class
+ */
+ public CombinedTransformOperator(Map<ExpressionContext, TransformOperator> transformOperatorMap,
+ ExpressionContext mainPredicateExpression,
+ Collection<ExpressionContext> expressions) {
+ super(null, transformOperatorMap.entrySet().iterator().next().getValue()._projectionOperator,
+ expressions);
+
+ _mainPredicateExpression = mainPredicateExpression;
+ _transformOperatorMap = transformOperatorMap;
+ }
+
+ @Override
+ protected TransformBlock getNextBlock() {
+ Map<ExpressionContext, TransformBlock> expressionContextTransformBlockMap = new HashMap<>();
+ boolean hasBlock = false;
+
+ Iterator<Map.Entry<ExpressionContext, TransformOperator>> iterator = _transformOperatorMap.entrySet().iterator();
+ // Get next block from all underlying transform operators
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, TransformOperator> entry = iterator.next();
+
+ TransformBlock transformBlock = entry.getValue().getNextBlock();
+
+ if (transformBlock != null) {
+ hasBlock = true;
+ }
+
+ expressionContextTransformBlockMap.put(entry.getKey(), transformBlock);
+ }
+
+ if (!hasBlock) {
+ return null;
+ }
+
+ return new
+ CombinedTransformBlock(expressionContextTransformBlockMap,
+ _mainPredicateExpression);
Review comment:
Is there a good reason to put null values in the map? Otherwise:
```java
Map<ExpressionContext, TransformBlock> expressionContextTransformBlockMap = new HashMap<>();
// Get next block from all underlying transform operators
for (Map.Entry<ExpressionContext, TransformOperator> entry: _transformOperatorMap.entrySet()) {
TransformBlock transformBlock = entry.getValue().getNextBlock();
if (transformBlock != null) {
expressionContextTransformBlockMap.put(entry.getKey(), transformBlock);
}
}
return expressionContextTransformBlockMap.isEmpty() ? null : new CombinedTransformBlock(expressionContextTransformBlockMap, _mainPredicateExpression);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785078730
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+ int numFilteredAggregations = aggregationFunctions.length;
Review comment:
There is no benefit to stashing an array’s length in a local variable, it’s not used anywhere else so I suggest moving this in to the loop termination condition to reduce verbosity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (839069c) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `43.89%`.
> The diff coverage is `29.81%`.
> :exclamation: Current head 839069c differs from pull request most recent head 6150f87. Consider uploading reports for the commit 6150f87 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.63% 27.73% -43.90%
=============================================
Files 1580 1595 +15
Lines 81100 82778 +1678
Branches 12068 12356 +288
=============================================
- Hits 58093 22957 -35136
- Misses 19092 57731 +38639
+ Partials 3915 2090 -1825
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.73% <29.81%> (-0.17%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `63.11% <ø> (-8.20%)` | :arrow_down: |
| [...a/org/apache/pinot/common/metrics/BrokerMeter.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vbWV0cmljcy9Ccm9rZXJNZXRlci5qYXZh) | `100.00% <ø> (ø)` | |
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `89.47% <ø> (ø)` | |
| [...ller/api/resources/PinotControllerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90Q29udHJvbGxlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...ces/PinotSegmentUploadDownloadRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90U2VnbWVudFVwbG9hZERvd25sb2FkUmVzdGxldFJlc291cmNlLmphdmE=) | `49.13% <0.00%> (-9.14%)` | :arrow_down: |
| [...ler/api/resources/TableConfigsRestletResource.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1RhYmxlQ29uZmlnc1Jlc3RsZXRSZXNvdXJjZS5qYXZh) | `0.00% <0.00%> (-73.13%)` | :arrow_down: |
| [...ot/controller/api/resources/ZookeeperResource.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1pvb2tlZXBlclJlc291cmNlLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../controller/helix/ControllerRequestURLBuilder.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9Db250cm9sbGVyUmVxdWVzdFVSTEJ1aWxkZXIuamF2YQ==) | `16.66% <0.00%> (-66.54%)` | :arrow_down: |
| [...controller/helix/core/minion/PinotTaskManager.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9oZWxpeC9jb3JlL21pbmlvbi9QaW5vdFRhc2tNYW5hZ2VyLmphdmE=) | `22.13% <0.00%> (-43.38%)` | :arrow_down: |
| ... and [1290 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...6150f87](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785070053
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/SwimLaneDocIdSetOperator.java
##########
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator;
+
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.DocIdSetBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
+import org.apache.pinot.core.operator.filter.CombinedFilterOperator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+
+/**
+ * DocIdSetOperator for a swimlane query plan.
+ */
+public class SwimLaneDocIdSetOperator extends DocIdSetOperator {
+ private static final String OPERATOR_NAME = "SwimLaneDocIdSetOperator";
+
+ private final CombinedFilterOperator _filterOperator;
+ private final ExpressionContext _expressionContext;
+ private final int _maxSizeOfDocIdSet;
+
+ private FilterBlockDocIdSet _filterBlockDocIdSet;
+ private BlockDocIdIterator _blockDocIdIterator;
+ private int _currentDocId = 0;
+
+ public SwimLaneDocIdSetOperator(CombinedFilterOperator filterOperator, ExpressionContext expressionContext,
+ int maxSizeOfDocIdSet) {
+ super(filterOperator, maxSizeOfDocIdSet);
+
+ Preconditions.checkArgument(maxSizeOfDocIdSet > 0 && maxSizeOfDocIdSet <= DocIdSetPlanNode.MAX_DOC_PER_CALL);
+ _filterOperator = filterOperator;
+ _expressionContext = expressionContext;
+ _maxSizeOfDocIdSet = maxSizeOfDocIdSet;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return Arrays.asList(_filterOperator);
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
Review comment:
This hasn't been resolved yet. It is still returning null.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter commented on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter commented on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (a898d99) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `57.30%`.
> The diff coverage is `0.29%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.63% 14.32% -57.31%
+ Complexity 4088 80 -4008
=============================================
Files 1580 1551 -29
Lines 81100 80528 -572
Branches 12068 12098 +30
=============================================
- Hits 58093 11537 -46556
- Misses 19092 68135 +49043
+ Partials 3915 856 -3059
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.32% <0.29%> (-0.24%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/ProjectionOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9Qcm9qZWN0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.62%)` | :arrow_down: |
| [.../pinot/core/operator/SwimLaneDocIdSetOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9Td2ltTGFuZURvY0lkU2V0T3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...inot/core/operator/blocks/CombinedFilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvQ29tYmluZWRGaWx0ZXJCbG9jay5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/blocks/CombinedTransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvQ29tYmluZWRUcmFuc2Zvcm1CbG9jay5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
| [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...pinot/core/operator/query/AggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9BZ2dyZWdhdGlvbk9wZXJhdG9yLmphdmE=) | `0.00% <0.00%> (-92.00%)` | :arrow_down: |
| [.../operator/transform/CombinedTransformOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci90cmFuc2Zvcm0vQ29tYmluZWRUcmFuc2Zvcm1PcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...t/core/plan/AggregationGroupByOrderByPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uR3JvdXBCeU9yZGVyQnlQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-45.46%)` | :arrow_down: |
| ... and [1292 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...a898d99](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770648030
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/CombinedTransformOperator.java
##########
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.operator.blocks.CombinedTransformBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+
+
+/**
+ * Used for processing a set of TransformOperators, fed by an underlying
+ * main predicate transform operator.
+ *
+ * This class returns a CombinedTransformBlock, with blocks ordered in
+ * the order in which their parent filter clauses appear in the query
+ */
+public class CombinedTransformOperator extends TransformOperator {
+ private static final String OPERATOR_NAME = "CombinedTransformOperator";
+
+ protected final Map<ExpressionContext, TransformOperator> _transformOperatorMap;
+ protected final ExpressionContext _mainPredicateExpression;
+
+ /**
+ * Constructor for the class
+ */
+ public CombinedTransformOperator(Map<ExpressionContext, TransformOperator> transformOperatorMap,
+ ExpressionContext mainPredicateExpression,
+ Collection<ExpressionContext> expressions) {
+ super(null, transformOperatorMap.entrySet().iterator().next().getValue()._projectionOperator,
+ expressions);
+
+ _mainPredicateExpression = mainPredicateExpression;
+ _transformOperatorMap = transformOperatorMap;
+ }
+
+ @Override
+ protected TransformBlock getNextBlock() {
+ Map<ExpressionContext, TransformBlock> expressionContextTransformBlockMap = new HashMap<>();
+ boolean hasBlock = false;
+
+ Iterator<Map.Entry<ExpressionContext, TransformOperator>> iterator = _transformOperatorMap.entrySet().iterator();
+ // Get next block from all underlying transform operators
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, TransformOperator> entry = iterator.next();
+
+ TransformBlock transformBlock = entry.getValue().getNextBlock();
+
+ if (transformBlock != null) {
+ hasBlock = true;
+ }
+
+ expressionContextTransformBlockMap.put(entry.getKey(), transformBlock);
+ }
+
+ if (!hasBlock) {
+ return null;
+ }
+
+ return new
+ CombinedTransformBlock(expressionContextTransformBlockMap,
+ _mainPredicateExpression);
Review comment:
Is there a good reason to put null values in the map? Otherwise:
```java
Map<ExpressionContext, TransformBlock> expressionContextTransformBlockMap = new HashMap<>();
// Get next block from all underlying transform operators
for (Map.Entry<ExpressionContext, TransformOperator> entry: _transformOperatorMap.entrySet()) {
TransformBlock transformBlock = entry.getValue().getNextBlock();
if (transformBlock != null) {
expressionContextTransformBlockMap.put(entry.getKey(), entry.getValue().getNextBlock());
}
}
return expressionContextTransformBlockMap.isEmpty() ? null : new CombinedTransformBlock(expressionContextTransformBlockMap, _mainPredicateExpression);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770668157
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/SwimLaneDocIdSetOperator.java
##########
@@ -0,0 +1,111 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator;
+
+import com.google.common.base.Preconditions;
+import java.util.Arrays;
+import java.util.List;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdIterator;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.DocIdSetBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.FilterBlockDocIdSet;
+import org.apache.pinot.core.operator.filter.CombinedFilterOperator;
+import org.apache.pinot.core.plan.DocIdSetPlanNode;
+import org.apache.pinot.segment.spi.Constants;
+
+/**
+ * DocIdSetOperator for a swimlane query plan.
+ */
+public class SwimLaneDocIdSetOperator extends DocIdSetOperator {
+ private static final String OPERATOR_NAME = "SwimLaneDocIdSetOperator";
+
+ private final CombinedFilterOperator _filterOperator;
+ private final ExpressionContext _expressionContext;
+ private final int _maxSizeOfDocIdSet;
+
+ private FilterBlockDocIdSet _filterBlockDocIdSet;
+ private BlockDocIdIterator _blockDocIdIterator;
+ private int _currentDocId = 0;
+
+ public SwimLaneDocIdSetOperator(CombinedFilterOperator filterOperator, ExpressionContext expressionContext,
+ int maxSizeOfDocIdSet) {
+ super(filterOperator, maxSizeOfDocIdSet);
+
+ Preconditions.checkArgument(maxSizeOfDocIdSet > 0 && maxSizeOfDocIdSet <= DocIdSetPlanNode.MAX_DOC_PER_CALL);
+ _filterOperator = filterOperator;
+ _expressionContext = expressionContext;
+ _maxSizeOfDocIdSet = maxSizeOfDocIdSet;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return Arrays.asList(_filterOperator);
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected DocIdSetBlock getNextBlock() {
+ if (_currentDocId == Constants.EOF) {
+ return null;
+ }
+
+ // Initialize filter block document Id set
+ if (_filterBlockDocIdSet == null) {
+ CombinedFilterBlock combinedFilterBlock = (CombinedFilterBlock) _filterOperator.nextBlock();
+ FilterBlock filterBlock = combinedFilterBlock.getFilterBlock(_expressionContext);
+
+ if (filterBlock == null) {
+ throw new IllegalStateException("Null block seen");
+ }
+
+ _filterBlockDocIdSet = filterBlock.getBlockDocIdSet();
+ _blockDocIdIterator = _filterBlockDocIdSet.iterator();
+ }
+
+ int pos = 0;
+ int[] docIds = new int[DocIdSetPlanNode.MAX_DOC_PER_CALL];
+
+ for (int i = 0; i < _maxSizeOfDocIdSet; i++) {
+ _currentDocId = _blockDocIdIterator.next();
+ if (_currentDocId == Constants.EOF) {
+ break;
+ }
+ docIds[pos++] = _currentDocId;
+ }
+ if (pos > 0) {
+ return new DocIdSetBlock(docIds, pos);
+ } else {
+ return null;
+ }
Review comment:
This can be reworked slightly to improve this for empty blocks.
```java
_currentDocId = _blockDocIdIterator.next();
if (_currentDocId == Constants.EOF) {
return null;
}
// precondition: _maxSizeOfDocIdSet > 0
int[] docIds = new int[_maxSizeOfDocIdSet];
int pos = 0;
do {
docIds[pos++] = _currentDocId;
_currentDocId = _blockDocIdIterator.next();
} while (pos < docIds.length && _currentDocId != Constants.EOF);
return new DocIdSetBlock(docIds, pos);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5d87c14) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `57.28%`.
> The diff coverage is `0.94%`.
> :exclamation: Current head 5d87c14 differs from pull request most recent head a089a68. Consider uploading reports for the commit a089a68 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.63% 14.34% -57.29%
+ Complexity 4088 80 -4008
=============================================
Files 1580 1551 -29
Lines 81100 80523 -577
Branches 12068 12099 +31
=============================================
- Hits 58093 11552 -46541
- Misses 19092 68116 +49024
+ Partials 3915 855 -3060
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.34% <0.94%> (-0.22%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...t/broker/api/resources/PinotBrokerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvYXBpL3Jlc291cmNlcy9QaW5vdEJyb2tlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...roker/requesthandler/BaseBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvQmFzZUJyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `21.51% <ø> (-49.79%)` | :arrow_down: |
| [...roker/requesthandler/GrpcBrokerRequestHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtYnJva2VyL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9icm9rZXIvcmVxdWVzdGhhbmRsZXIvR3JwY0Jyb2tlclJlcXVlc3RIYW5kbGVyLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [.../java/org/apache/pinot/common/utils/DataTable.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvRGF0YVRhYmxlLmphdmE=) | `0.00% <ø> (-89.48%)` | :arrow_down: |
| [...ache/pinot/common/utils/TarGzCompressionUtils.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9jb21tb24vdXRpbHMvVGFyR3pDb21wcmVzc2lvblV0aWxzLmphdmE=) | `0.00% <0.00%> (-85.30%)` | :arrow_down: |
| [...org/apache/pinot/sql/parsers/CalciteSqlParser.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29tbW9uL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9waW5vdC9zcWwvcGFyc2Vycy9DYWxjaXRlU3FsUGFyc2VyLmphdmE=) | `0.00% <0.00%> (-91.96%)` | :arrow_down: |
| [...ller/api/resources/PinotControllerHealthCheck.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci9hcGkvcmVzb3VyY2VzL1Bpbm90Q29udHJvbGxlckhlYWx0aENoZWNrLmphdmE=) | `0.00% <0.00%> (ø)` | |
| [...a/org/apache/pinot/core/common/DataBlockCache.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUJsb2NrQ2FjaGUuamF2YQ==) | `0.00% <0.00%> (-96.22%)` | :arrow_down: |
| [...java/org/apache/pinot/core/common/DataFetcher.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vRGF0YUZldGNoZXIuamF2YQ==) | `0.00% <0.00%> (-89.96%)` | :arrow_down: |
| [...e/pinot/core/common/datatable/DataTableImplV2.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9jb21tb24vZGF0YXRhYmxlL0RhdGFUYWJsZUltcGxWMi5qYXZh) | `0.00% <0.00%> (-77.78%)` | :arrow_down: |
| ... and [1335 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...a089a68](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-1004574477
Discussed with @Jackie-Jiang and we will be convening on this PR itself, so reviving it. Sorry for the confusion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (c0661d5) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `57.37%`.
> The diff coverage is `0.38%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.63% 14.26% -57.38%
+ Complexity 4088 80 -4008
=============================================
Files 1580 1555 -25
Lines 81100 80940 -160
Branches 12068 12150 +82
=============================================
- Hits 58093 11543 -46550
- Misses 19092 68538 +49446
+ Partials 3915 859 -3056
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `?` | |
| unittests2 | `14.26% <0.38%> (-0.30%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/ProjectionOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9Qcm9qZWN0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (-84.62%)` | :arrow_down: |
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `0.00% <0.00%> (-57.15%)` | :arrow_down: |
| [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `0.00% <0.00%> (-60.00%)` | :arrow_down: |
| [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `0.00% <0.00%> (-69.24%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (-100.00%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/plan/AggregationGroupByOrderByPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uR3JvdXBCeU9yZGVyQnlQbGFuTm9kZS5qYXZh) | `0.00% <0.00%> (-45.46%)` | :arrow_down: |
| ... and [1296 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...c0661d5](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r795398047
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,31 +68,164 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+ if (hasFilteredPredicates) {
+ return buildFilteredAggOperator();
+ }
+
+ return buildNonFilteredAggOperator();
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r795404103
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,76 +460,105 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
+ List<Pair<AggregationFunction, FilterContext>> filteredAggregations = new ArrayList<>();
Map<FunctionContext, Integer> aggregationFunctionIndexMap = new HashMap<>();
+ Map<Pair<FunctionContext, FilterContext>, Integer> filterExpressionIndexMap = new HashMap<>();
// Add aggregation functions in the SELECT clause
// NOTE: DO NOT deduplicate the aggregation functions in the SELECT clause because that involves protocol change.
- List<FunctionContext> aggregationsInSelect = new ArrayList<>();
- List<Pair<FunctionContext, FilterContext>> filteredAggregations = new ArrayList<>();
+ List<Pair<FilterContext, FunctionContext>> aggregationsInSelect = new ArrayList<>();
for (ExpressionContext selectExpression : queryContext._selectExpressions) {
- getAggregations(selectExpression, aggregationsInSelect, filteredAggregations);
+ getAggregations(selectExpression, aggregationsInSelect);
}
- for (FunctionContext function : aggregationsInSelect) {
- int functionIndex = aggregationFunctions.size();
+ for (Pair<FilterContext, FunctionContext> pair : aggregationsInSelect) {
+ FunctionContext function = pair.getRight();
+ int functionIndex = filteredAggregations.size();
AggregationFunction aggregationFunction =
AggregationFunctionFactory.getAggregationFunction(function, queryContext);
- aggregationFunctions.add(aggregationFunction);
+
+ FilterContext filterContext = null;
+ // If the left pair is not null, implies a filtered aggregation
+ if (pair.getLeft() != null) {
+ if (_groupByExpressions != null) {
+ throw new IllegalStateException("GROUP BY with FILTER clauses is not supported");
+ }
+
+ queryContext._hasFilteredAggregations = true;
+
+ filterContext = pair.getLeft();
+
+ Pair<FunctionContext, FilterContext> filterContextPair =
+ Pair.of(function, filterContext);
+
+ if (!filterExpressionIndexMap.containsKey(filterContextPair)) {
+ int filterMapIndex = filterExpressionIndexMap.size();
+
+ filterExpressionIndexMap.put(filterContextPair, filterMapIndex);
+ }
+ }
+ filteredAggregations.add(Pair.of(aggregationFunction, filterContext));
+
Review comment:
Fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r793459198
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/startree/plan/StarTreeTransformPlanNode.java
##########
@@ -57,6 +57,7 @@ public StarTreeTransformPlanNode(StarTreeV2 starTreeV2,
_groupByExpressions = Collections.emptyList();
groupByColumns = null;
}
+
Review comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785074311
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -52,7 +61,8 @@ public BlockValSet getBlockValueSet(String column) {
@Override
public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
+ return _docIdSetBlock != null ? _docIdSetBlock.getBlockDocIdSet()
Review comment:
I don’t think this null check improves the readability of the code. Whenever I see a null check, I think about when and how the reference can be null, so adding tautological checks adds cognitive burden.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785233754
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
Review comment:
There can be duplicate aggregation functions, so we need to check the reference instead of the hash code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785767852
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -90,9 +92,11 @@
// Pre-calculate the aggregation functions and columns for the query so that it can be shared across all the segments
private AggregationFunction[] _aggregationFunctions;
- private List<Pair<AggregationFunction, FilterContext>> _filteredAggregationFunctions;
+
Review comment:
FilterableAggregationFunction provides a nice abstraction for defining the sub aggregation, filter and associated context (for collecting same predicate aggregation functions together) without requiring a lot of code ripping in QueryContext. It also helps maintain the order aggregations as present in the original SELECT list.
FilterableAggregationFunction is limited to the execution of filtered aggregates and is not evident anywhere else. I would advocate keeping this class for simplicity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (bd19945) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `9.57%`.
> The diff coverage is `88.16%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 61.67% -9.58%
+ Complexity 4262 4180 -82
============================================
Files 1607 1602 -5
Lines 83409 83206 -203
Branches 12458 12441 -17
============================================
- Hits 59426 51314 -8112
- Misses 19941 28170 +8229
+ Partials 4042 3722 -320
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `27.63% <21.30%> (-0.04%)` | :arrow_down: |
| unittests1 | `67.97% <88.16%> (+0.10%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `89.89% <88.09%> (-3.09%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.58% <97.77%> (-0.33%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [300 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...bd19945](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r795407089
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
I agree with @richardstartin -- we only need to worry about streams when the code is invoked in a tight loop for multiple iterations -- none of which is applicable in this specific case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785073260
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -52,7 +61,8 @@ public BlockValSet getBlockValueSet(String column) {
@Override
public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
+ return _docIdSetBlock != null ? _docIdSetBlock.getBlockDocIdSet()
Review comment:
This appears to be unresolved.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785080162
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator,
+ AggregationFunction[] aggregationFunctions, int numTotalDocs) {
+ Map<ExpressionContext, Pair<List<AggregationFunction>, TransformOperator>> expressionContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+
+ ExpressionContext currentFilterExpression = filterableAggregationFunction
+ .getAssociatedExpressionContext();
+
+ if (expressionContextToAggFuncsMap.containsKey(currentFilterExpression)) {
Review comment:
It’s better to get then null check, otherwise the key gets hashed and equality-checked twice.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785770857
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,57 +69,25 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
- int numTotalDocs = _indexSegment.getSegmentMetadata().getTotalDocs();
- AggregationFunction[] aggregationFunctions = _queryContext.getAggregationFunctions();
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
- FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext);
- BaseFilterOperator filterOperator = filterPlanNode.run();
+ Pair<FilterPlanNode, BaseFilterOperator> filterOperatorPair =
Review comment:
buildOperators has no code specific to filtered aggregations. It builds a generic pipeline of operators, and buildOperatorForFilteredAggregations does the actual specific code if filtered aggregations are present.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770787541
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/transform/CombinedTransformOperator.java
##########
@@ -0,0 +1,88 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.transform;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.operator.blocks.CombinedTransformBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+
+
+/**
+ * Used for processing a set of TransformOperators, fed by an underlying
+ * main predicate transform operator.
+ *
+ * This class returns a CombinedTransformBlock, with blocks ordered in
+ * the order in which their parent filter clauses appear in the query
+ */
+public class CombinedTransformOperator extends TransformOperator {
+ private static final String OPERATOR_NAME = "CombinedTransformOperator";
+
+ protected final Map<ExpressionContext, TransformOperator> _transformOperatorMap;
+ protected final ExpressionContext _mainPredicateExpression;
+
+ /**
+ * Constructor for the class
+ */
+ public CombinedTransformOperator(Map<ExpressionContext, TransformOperator> transformOperatorMap,
+ ExpressionContext mainPredicateExpression,
+ Collection<ExpressionContext> expressions) {
+ super(null, transformOperatorMap.entrySet().iterator().next().getValue()._projectionOperator,
+ expressions);
+
+ _mainPredicateExpression = mainPredicateExpression;
+ _transformOperatorMap = transformOperatorMap;
+ }
+
+ @Override
+ protected TransformBlock getNextBlock() {
+ Map<ExpressionContext, TransformBlock> expressionContextTransformBlockMap = new HashMap<>();
+ boolean hasBlock = false;
+
+ Iterator<Map.Entry<ExpressionContext, TransformOperator>> iterator = _transformOperatorMap.entrySet().iterator();
+ // Get next block from all underlying transform operators
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, TransformOperator> entry = iterator.next();
+
+ TransformBlock transformBlock = entry.getValue().getNextBlock();
+
+ if (transformBlock != null) {
+ hasBlock = true;
+ }
+
+ expressionContextTransformBlockMap.put(entry.getKey(), transformBlock);
+ }
+
+ if (!hasBlock) {
+ return null;
+ }
+
+ return new
+ CombinedTransformBlock(expressionContextTransformBlockMap,
+ _mainPredicateExpression);
Review comment:
Ideally, each swim lane has representation of its own block -- so one swim lane can be exhausted while others still have blocks. Hence the idea of null (to represent an empty block). I guess since we use a map, a non existence of key will also represent the same thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r771022492
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/startree/plan/StarTreeDocIdSetPlanNode.java
##########
@@ -31,16 +32,29 @@
public class StarTreeDocIdSetPlanNode implements PlanNode {
Review comment:
From what I am seeing if the `filterOperator` is present, the existing implementation is completely overridden (new constructor and if statement in `run` method) to the point that the old code isn't being touched at all. I am wondering if it will be better to create a new class (for example `StarTreeFilteredDocIdSetPlanNode`) and doing that will also avoid the null checks?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ List<Pair<ExpressionContext, BaseFilterOperator>> baseFilterOperatorList =
+ new ArrayList<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorList.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight()));
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
+ }
+ }
+
+ CombinedFilterOperator combinedFilterOperator = new CombinedFilterOperator(baseFilterOperatorList,
+ mainPredicateFilterOperator);
+
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ for (Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>> pair
+ : filterPredicatesAndMetadata) {
+ Pair<TransformOperator,
+ BaseOperator<IntermediateResultsBlock>> innerPair =
+ buildOperators(combinedFilterOperator, pair.getRight().getLeft(),
+ true, pair.getLeft());
+
+ transformOperatorMap.put(pair.getLeft(), innerPair.getLeft());
+ }
+
+ // Add the main predicate filter operator to the map
+ transformOperatorMap.put(_queryContext.getFilterExpression(), nonFilteredTransformOperator);
+
+ return new CombinedTransformOperator(transformOperatorMap, _queryContext.getFilterExpression(),
+ expressionsToTransform);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @param isSwimlane Is this plan a swim lane?
Review comment:
What does `isSwimlane` mean? I am wondering if there is a more functional name that can be used here along with a good description?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/ProjectionBlock.java
##########
@@ -52,7 +61,8 @@ public BlockValSet getBlockValueSet(String column) {
@Override
public BlockDocIdSet getBlockDocIdSet() {
- throw new UnsupportedOperationException();
+ return _docIdSetBlock != null ? _docIdSetBlock.getBlockDocIdSet()
Review comment:
Is this null check really needed? The constructor that sets `_docIdSetBlock` is being called after a null check in `ProjectionOperator.java`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770630116
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedTransformBlock.java
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdSet;
+import org.apache.pinot.core.common.BlockDocIdValueSet;
+import org.apache.pinot.core.common.BlockMetadata;
+import org.apache.pinot.core.common.BlockValSet;
+
+/**
+ * Represents a combination of multiple TransformBlock instances
+ */
+public class CombinedTransformBlock extends TransformBlock {
+ protected Map<ExpressionContext, TransformBlock> _transformBlockMap;
+ protected ExpressionContext _mainPredicateExpressionContext;
+
+ public CombinedTransformBlock(Map<ExpressionContext, TransformBlock> transformBlockMap,
+ ExpressionContext mainPredicateExpressionContext) {
+ super(transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._projectionBlock,
+ transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._transformFunctionMap);
+
+ _transformBlockMap = transformBlockMap;
+ _mainPredicateExpressionContext = mainPredicateExpressionContext;
+ }
+
+ public int getNumDocs() {
+ int numDocs = 0;
+
+ Iterator<Map.Entry<ExpressionContext, TransformBlock>> iterator = _transformBlockMap.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, TransformBlock> entry = iterator.next();
+ TransformBlock transformBlock = entry.getValue();
+
+ if (transformBlock != null) {
+ numDocs = numDocs + transformBlock._projectionBlock.getNumDocs();
+ }
+ }
+
+ return numDocs;
Review comment:
This is hard to read, what's below would be better:
```java
int numDocs = 0;
for (TransformBlock transformBlock : _transformBlockMap.values()) {
// null check can be avoided by not putting null values in the map
numDocs += transformBlock._projectionBlock.getNumDocs();
}
return numDocs;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770688683
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedTransformBlock.java
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdSet;
+import org.apache.pinot.core.common.BlockDocIdValueSet;
+import org.apache.pinot.core.common.BlockMetadata;
+import org.apache.pinot.core.common.BlockValSet;
+
+/**
+ * Represents a combination of multiple TransformBlock instances
+ */
+public class CombinedTransformBlock extends TransformBlock {
+ protected Map<ExpressionContext, TransformBlock> _transformBlockMap;
+ protected ExpressionContext _mainPredicateExpressionContext;
+
+ public CombinedTransformBlock(Map<ExpressionContext, TransformBlock> transformBlockMap,
+ ExpressionContext mainPredicateExpressionContext) {
+ super(transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._projectionBlock,
+ transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._transformFunctionMap);
Review comment:
Or even
```java
protected TransformBlock(TransformBlock transformBlock) {
this(transformBlock == null ? null : transformBlock._projectionBlock,
transformBlock == null ? null : transformBlock._transformFunctionMap);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770793263
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +138,151 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a CombinedTransformOperator given the main predicate filter operator and the corresponding
+ * aggregation functions.
+ * @param nonFilteredTransformOperator Transform operator corresponding to the main predicate
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param expressionsToTransform Expressions to transform
+ * @param aggregationFunctions Aggregation functions in the query
+ */
+ private TransformOperator buildOperatorForFilteredAggregations(TransformOperator nonFilteredTransformOperator,
+ BaseFilterOperator mainPredicateFilterOperator,
+ Set<ExpressionContext> expressionsToTransform,
+ AggregationFunction[] aggregationFunctions) {
+ Map<ExpressionContext, TransformOperator> transformOperatorMap = new HashMap<>();
+ Map<ExpressionContext, BaseFilterOperator> baseFilterOperatorMap =
+ new HashMap<>();
+ List<Pair<ExpressionContext, Pair<FilterPlanNode, BaseFilterOperator>>> filterPredicatesAndMetadata =
+ new ArrayList<>();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and metadata
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction instanceof FilterableAggregationFunction) {
+ FilterableAggregationFunction filterableAggregationFunction =
+ (FilterableAggregationFunction) aggregationFunction;
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(filterableAggregationFunction.getFilterContext());
+
+ baseFilterOperatorMap.put(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair.getRight());
+ filterPredicatesAndMetadata.add(Pair.of(filterableAggregationFunction.getAssociatedExpressionContext(),
+ pair));
Review comment:
I am confused. Isn't filterPredicatesAndMetadata a List already?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770807676
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
Review comment:
Yes. This allows reusing filter operators sharing the same predicate, ensuring that redundant swim lanes are not invoked.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r788444873
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
Review comment:
To match the naming pattern in AggregateGroupByOperator (and other Aggregation*Operator classes), the EXPLAIN_NAME should be set to AGGREGATE_FILTERED.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794904539
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,31 +68,164 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+ if (hasFilteredPredicates) {
+ return buildFilteredAggOperator();
+ }
+
+ return buildNonFilteredAggOperator();
Review comment:
(minor) Can be simplified for better readability
```suggestion
return _queryContext.isHasFilteredAggregations() ? buildFilteredAggOperator() : buildNonFilteredAggOperator();
```
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,31 +68,164 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+ if (hasFilteredPredicates) {
+ return buildFilteredAggOperator();
+ }
+
+ return buildNonFilteredAggOperator();
+ }
+
+ /**
+ * Returns {@code true} if the given aggregations can be solved with segment metadata, {@code false} otherwise.
+ * <p>Aggregations supported: COUNT
+ */
+ private static boolean isFitForMetadataBasedPlan(AggregationFunction[] aggregationFunctions) {
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction.getType() != AggregationFunctionType.COUNT) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Returns {@code true} if the given aggregations can be solved with dictionary, {@code false} otherwise.
+ * <p>Aggregations supported: MIN, MAX, MIN_MAX_RANGE, DISTINCT_COUNT, SEGMENT_PARTITIONED_DISTINCT_COUNT
+ */
+ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggregationFunctions,
+ IndexSegment indexSegment) {
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ AggregationFunctionType functionType = aggregationFunction.getType();
+ if (functionType != AggregationFunctionType.MIN && functionType != AggregationFunctionType.MAX
+ && functionType != AggregationFunctionType.MINMAXRANGE
+ && functionType != AggregationFunctionType.DISTINCTCOUNT
+ && functionType != AggregationFunctionType.SEGMENTPARTITIONEDDISTINCTCOUNT) {
+ return false;
+ }
+ ExpressionContext argument = (ExpressionContext) aggregationFunction.getInputExpressions().get(0);
+ if (argument.getType() != ExpressionContext.Type.IDENTIFIER) {
+ return false;
+ }
+ String column = argument.getIdentifier();
+ Dictionary dictionary = indexSegment.getDataSource(column).getDictionary();
+ if (dictionary == null) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
+ TransformOperator mainTransformOperator, int numTotalDocs) {
+ Map<FilterContext, Pair<List<AggregationFunction>, TransformOperator>> filterContextToAggFuncsMap =
+ new HashMap<>();
+ List<AggregationFunction> nonFilteredAggregationFunctions = new ArrayList<>();
+ List<Pair<AggregationFunction, FilterContext>> aggregationFunctions = _queryContext
+ .getFilteredAggregations();
+
+ // For each aggregation function, check if the aggregation function is a filtered agg.
+ // If it is, populate the corresponding filter operator and corresponding transform operator
+ for (Pair<AggregationFunction, FilterContext> inputPair : aggregationFunctions) {
+ if (inputPair.getLeft() != null) {
+ FilterContext currentFilterExpression = inputPair.getRight();
+ if (filterContextToAggFuncsMap.get(currentFilterExpression) != null) {
+ filterContextToAggFuncsMap.get(currentFilterExpression).getLeft().add(inputPair.getLeft());
+ continue;
+ }
+ Pair<FilterPlanNode, BaseFilterOperator> pair =
+ buildFilterOperator(currentFilterExpression);
+ BaseFilterOperator wrappedFilterOperator = new CombinedFilterOperator(mainPredicateFilterOperator,
+ pair.getRight());
+ TransformOperator newTransformOperator = buildTransformOperatorForFilteredAggregates(wrappedFilterOperator);
+ // For each transform operator, associate it with the underlying expression. This allows
+ // fetching the relevant TransformOperator when resolving blocks during aggregation
+ // execution
+ List<AggregationFunction> aggFunctionList = new ArrayList<>();
+ aggFunctionList.add(inputPair.getLeft());
+ filterContextToAggFuncsMap.put(currentFilterExpression,
+ Pair.of(aggFunctionList, newTransformOperator));
+ } else {
+ nonFilteredAggregationFunctions.add(inputPair.getLeft());
+ }
+ }
+ List<Pair<AggregationFunction[], TransformOperator>> aggToTransformOpList =
+ new ArrayList<>();
+ // Convert to array since FilteredAggregationOperator expects it
+ for (Pair<List<AggregationFunction>, TransformOperator> pair
+ : filterContextToAggFuncsMap.values()) {
+ List<AggregationFunction> aggregationFunctionList = pair.getLeft();
+ if (aggregationFunctionList == null) {
+ throw new IllegalStateException("Null aggregation list seen");
+ }
+ aggToTransformOpList.add(Pair.of(aggregationFunctionList.toArray(new AggregationFunction[0]),
+ pair.getRight()));
+ }
+ aggToTransformOpList.add(Pair.of(nonFilteredAggregationFunctions.toArray(new AggregationFunction[0]),
+ mainTransformOperator));
+
+ return new FilteredAggregationOperator(_queryContext.getAggregationFunctions(), aggToTransformOpList,
+ numTotalDocs);
+ }
+
+ /**
+ * Build a filter operator from the given FilterContext.
+ *
+ * It returns the FilterPlanNode to allow reusing plan level components such as predicate
+ * evaluator map
+ */
+ private Pair<FilterPlanNode, BaseFilterOperator> buildFilterOperator(FilterContext filterContext) {
+ FilterPlanNode filterPlanNode = new FilterPlanNode(_indexSegment, _queryContext, filterContext);
+
+ return Pair.of(filterPlanNode, filterPlanNode.run());
+ }
+
+ /**
+ * Build transform and aggregation operators for the given bottom level plan.
+ * Note that this method operates only for non filtered aggregates.
+ * @param filterOperator Filter operator to be used in the corresponding chain
+ * @param filterPlanNode Plan node associated with the filter operator
+ * @return Pair, consisting of the built TransformOperator and Aggregation operator for chain
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorsForNonFilteredAggs(BaseFilterOperator filterOperator,
Review comment:
Shall we revert the changes within this method? The changes in this method should be irrelevant to the FILTER feature
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -62,31 +68,164 @@ public AggregationPlanNode(IndexSegment indexSegment, QueryContext queryContext)
public Operator<IntermediateResultsBlock> run() {
assert _queryContext.getAggregationFunctions() != null;
+ boolean hasFilteredPredicates = _queryContext.isHasFilteredAggregations();
+ if (hasFilteredPredicates) {
+ return buildFilteredAggOperator();
+ }
+
+ return buildNonFilteredAggOperator();
+ }
+
+ /**
+ * Returns {@code true} if the given aggregations can be solved with segment metadata, {@code false} otherwise.
+ * <p>Aggregations supported: COUNT
+ */
+ private static boolean isFitForMetadataBasedPlan(AggregationFunction[] aggregationFunctions) {
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ if (aggregationFunction.getType() != AggregationFunctionType.COUNT) {
+ return false;
+ }
+ }
+ return true;
+ }
+
+ /**
+ * Returns {@code true} if the given aggregations can be solved with dictionary, {@code false} otherwise.
+ * <p>Aggregations supported: MIN, MAX, MIN_MAX_RANGE, DISTINCT_COUNT, SEGMENT_PARTITIONED_DISTINCT_COUNT
+ */
+ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggregationFunctions,
+ IndexSegment indexSegment) {
+ for (AggregationFunction aggregationFunction : aggregationFunctions) {
+ AggregationFunctionType functionType = aggregationFunction.getType();
+ if (functionType != AggregationFunctionType.MIN && functionType != AggregationFunctionType.MAX
+ && functionType != AggregationFunctionType.MINMAXRANGE
+ && functionType != AggregationFunctionType.DISTINCTCOUNT
+ && functionType != AggregationFunctionType.SEGMENTPARTITIONEDDISTINCTCOUNT) {
+ return false;
+ }
+ ExpressionContext argument = (ExpressionContext) aggregationFunction.getInputExpressions().get(0);
+ if (argument.getType() != ExpressionContext.Type.IDENTIFIER) {
+ return false;
+ }
+ String column = argument.getIdentifier();
+ Dictionary dictionary = indexSegment.getDataSource(column).getDictionary();
+ if (dictionary == null) {
+ return false;
+ }
+ }
+ return true;
+ }
Review comment:
Can we move these 2 methods back? They are not changed, but make code review and future tracking harder
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/query/request/context/QueryContext.java
##########
@@ -441,76 +460,105 @@ public QueryContext build() {
*/
private void generateAggregationFunctions(QueryContext queryContext) {
List<AggregationFunction> aggregationFunctions = new ArrayList<>();
- List<Pair<AggregationFunction, FilterContext>> filteredAggregationFunctions = new ArrayList<>();
+ List<Pair<AggregationFunction, FilterContext>> filteredAggregations = new ArrayList<>();
Map<FunctionContext, Integer> aggregationFunctionIndexMap = new HashMap<>();
+ Map<Pair<FunctionContext, FilterContext>, Integer> filterExpressionIndexMap = new HashMap<>();
// Add aggregation functions in the SELECT clause
// NOTE: DO NOT deduplicate the aggregation functions in the SELECT clause because that involves protocol change.
- List<FunctionContext> aggregationsInSelect = new ArrayList<>();
- List<Pair<FunctionContext, FilterContext>> filteredAggregations = new ArrayList<>();
+ List<Pair<FilterContext, FunctionContext>> aggregationsInSelect = new ArrayList<>();
for (ExpressionContext selectExpression : queryContext._selectExpressions) {
- getAggregations(selectExpression, aggregationsInSelect, filteredAggregations);
+ getAggregations(selectExpression, aggregationsInSelect);
}
- for (FunctionContext function : aggregationsInSelect) {
- int functionIndex = aggregationFunctions.size();
+ for (Pair<FilterContext, FunctionContext> pair : aggregationsInSelect) {
+ FunctionContext function = pair.getRight();
+ int functionIndex = filteredAggregations.size();
AggregationFunction aggregationFunction =
AggregationFunctionFactory.getAggregationFunction(function, queryContext);
- aggregationFunctions.add(aggregationFunction);
+
+ FilterContext filterContext = null;
+ // If the left pair is not null, implies a filtered aggregation
+ if (pair.getLeft() != null) {
+ if (_groupByExpressions != null) {
+ throw new IllegalStateException("GROUP BY with FILTER clauses is not supported");
+ }
+
+ queryContext._hasFilteredAggregations = true;
+
+ filterContext = pair.getLeft();
+
+ Pair<FunctionContext, FilterContext> filterContextPair =
+ Pair.of(function, filterContext);
+
+ if (!filterExpressionIndexMap.containsKey(filterContextPair)) {
+ int filterMapIndex = filterExpressionIndexMap.size();
+
+ filterExpressionIndexMap.put(filterContextPair, filterMapIndex);
+ }
+ }
+ filteredAggregations.add(Pair.of(aggregationFunction, filterContext));
+
Review comment:
Shall we reduce some empty lines in this part of the code?
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/plan/AggregationPlanNode.java
##########
@@ -154,4 +129,168 @@ private static boolean isFitForDictionaryBasedPlan(AggregationFunction[] aggrega
}
return true;
}
+
+ /**
+ * Build a FilteredAggregationOperator given the parameters.
+ * @param mainPredicateFilterOperator Filter operator corresponding to the main predicate
+ * @param mainTransformOperator Transform operator corresponding to the main predicate
+ * @param aggregationFunctions Aggregation functions in the query
+ * @param numTotalDocs Number of total docs
+ */
+ private BaseOperator<IntermediateResultsBlock> buildOperatorForFilteredAggregations(
+ BaseFilterOperator mainPredicateFilterOperator,
Review comment:
(code format) Please apply the pinot code format and use it to auto-reformat this file. Several changes do not comply with the format
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794381262
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
I'm unaware of such a convention and have seen plenty of code using the streams API for performance non-critical operations (like this one) recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770671437
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/blocks/CombinedTransformBlock.java
##########
@@ -0,0 +1,91 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.blocks;
+
+import java.util.Iterator;
+import java.util.Map;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.BlockDocIdSet;
+import org.apache.pinot.core.common.BlockDocIdValueSet;
+import org.apache.pinot.core.common.BlockMetadata;
+import org.apache.pinot.core.common.BlockValSet;
+
+/**
+ * Represents a combination of multiple TransformBlock instances
+ */
+public class CombinedTransformBlock extends TransformBlock {
+ protected Map<ExpressionContext, TransformBlock> _transformBlockMap;
+ protected ExpressionContext _mainPredicateExpressionContext;
+
+ public CombinedTransformBlock(Map<ExpressionContext, TransformBlock> transformBlockMap,
+ ExpressionContext mainPredicateExpressionContext) {
+ super(transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._projectionBlock,
+ transformBlockMap.get(mainPredicateExpressionContext) == null ? null
+ : transformBlockMap.get(mainPredicateExpressionContext)._transformFunctionMap);
Review comment:
I dont think I quite understand this -- the copy constructor won't work since there is no default constructor for TransformBlock?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770702505
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(Map<ExpressionContext, BaseFilterOperator> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return new ArrayList<>(_filterOperators.values());
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected FilterBlock getNextBlock() {
+ if (_resultBlock != null) {
+ return _resultBlock;
+ }
+
+ FilterBlock mainFilterBlock = _mainFilterOperator.nextBlock();
+
+ Map<ExpressionContext, FilterBlock> filterBlockMap = new HashMap<>();
+ Iterator<Map.Entry<ExpressionContext, BaseFilterOperator>> iterator = _filterOperators.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, BaseFilterOperator> entry = iterator.next();
+ FilterBlock subFilterBlock = entry.getValue().nextBlock();
+
+ filterBlockMap.put(entry.getKey(),
Review comment:
This map can't be deduplicating because the previous `subFilterBlock` for a duplicate `ExpressionContext` would be lost here, which would either be wasteful or incorrect. I think this should be `List<Pair<ExpressionContext, BaseFilterOperator>>` instead.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] atris commented on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
atris commented on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-1000694234
@Jackie-Jiang is working on a parallel implementation, so closing this PR to avoid conflict
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r770643525
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/filter/CombinedFilterOperator.java
##########
@@ -0,0 +1,92 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.filter;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.HashMap;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Map;
+import javax.annotation.Nullable;
+import org.apache.pinot.common.request.context.ExpressionContext;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.blocks.CombinedFilterBlock;
+import org.apache.pinot.core.operator.blocks.FilterBlock;
+import org.apache.pinot.core.operator.docidsets.AndDocIdSet;
+
+/**
+ * A filter operator consisting of one main predicate block and multiple
+ * sub blocks. The main predicate block and sub blocks are ANDed before
+ * returning.
+ */
+public class CombinedFilterOperator extends BaseFilterOperator {
+ private static final String OPERATOR_NAME = "CombinedFilterOperator";
+
+ protected Map<ExpressionContext, BaseFilterOperator> _filterOperators;
+ protected BaseFilterOperator _mainFilterOperator;
+ protected CombinedFilterBlock _resultBlock;
+
+ public CombinedFilterOperator(Map<ExpressionContext, BaseFilterOperator> filterOperators,
+ BaseFilterOperator mainFilterOperator) {
+ _filterOperators = filterOperators;
+ _mainFilterOperator = mainFilterOperator;
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return new ArrayList<>(_filterOperators.values());
+ }
+
+ @Nullable
+ @Override
+ public String toExplainString() {
+ return null;
+ }
+
+ @Override
+ protected FilterBlock getNextBlock() {
+ if (_resultBlock != null) {
+ return _resultBlock;
+ }
+
+ FilterBlock mainFilterBlock = _mainFilterOperator.nextBlock();
+
+ Map<ExpressionContext, FilterBlock> filterBlockMap = new HashMap<>();
+ Iterator<Map.Entry<ExpressionContext, BaseFilterOperator>> iterator = _filterOperators.entrySet().iterator();
+
+ while (iterator.hasNext()) {
+ Map.Entry<ExpressionContext, BaseFilterOperator> entry = iterator.next();
+ FilterBlock subFilterBlock = entry.getValue().nextBlock();
+
+ filterBlockMap.put(entry.getKey(),
+ new FilterBlock(new AndDocIdSet(Arrays.asList(subFilterBlock.getBlockDocIdSet(),
+ mainFilterBlock.getBlockDocIdSet()))));
+ }
+
+ _resultBlock = new CombinedFilterBlock(filterBlockMap, mainFilterBlock);
+
+ return _resultBlock;
Review comment:
This can take up a lot less space:
```java
Map<ExpressionContext, FilterBlock> filterBlockMap = new HashMap<>();
for (Map.Entry<ExpressionContext, BaseFilterOperator> entry : _filterOperators.entrySet()) {
FilterBlock subFilterBlock = entry.getValue().nextBlock();
filterBlockMap.put(entry.getKey(),
new FilterBlock(new AndDocIdSet(Arrays.asList(subFilterBlock.getBlockDocIdSet(),
mainFilterBlock.getBlockDocIdSet()))));
}
_resultBlock = new CombinedFilterBlock(filterBlockMap, mainFilterBlock);
return _resultBlock;
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (404372f) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `33.62%`.
> The diff coverage is `33.46%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
=============================================
- Coverage 71.63% 38.00% -33.63%
+ Complexity 4088 80 -4008
=============================================
Files 1580 1600 +20
Lines 81100 82799 +1699
Branches 12068 12350 +282
=============================================
- Hits 58093 31466 -26627
- Misses 19092 48893 +29801
+ Partials 3915 2440 -1475
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.93% <33.46%> (-0.47%)` | :arrow_down: |
| integration2 | `27.68% <32.69%> (-0.22%)` | :arrow_down: |
| unittests1 | `?` | |
| unittests2 | `14.26% <0.38%> (-0.30%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `30.00% <0.00%> (-27.15%)` | :arrow_down: |
| [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `56.25% <0.00%> (-12.99%)` | :arrow_down: |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `0.00% <0.00%> (ø)` | |
| [...t/core/plan/AggregationGroupByOrderByPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uR3JvdXBCeU9yZGVyQnlQbGFuTm9kZS5qYXZh) | `45.45% <0.00%> (ø)` | |
| [...gation/function/FilterableAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9GaWx0ZXJhYmxlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `0.00% <0.00%> (ø)` | |
| [...not/controller/validation/StorageQuotaChecker.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1N0b3JhZ2VRdW90YUNoZWNrZXIuamF2YQ==) | `71.60% <16.66%> (-4.72%)` | :arrow_down: |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `84.88% <28.57%> (-7.33%)` | :arrow_down: |
| ... and [951 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...404372f](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (6150f87) into [master](https://codecov.io/gh/apache/pinot/commit/47e49ecd6e11aebe74f8868cdac22051b175d4c5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (47e49ec) will **decrease** coverage by `1.19%`.
> The diff coverage is `84.45%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.63% 70.44% -1.20%
- Complexity 4088 4225 +137
============================================
Files 1580 1604 +24
Lines 81100 83126 +2026
Branches 12068 12394 +326
============================================
+ Hits 58093 58554 +461
- Misses 19092 20563 +1471
- Partials 3915 4009 +94
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `29.04% <32.35%> (-0.36%)` | :arrow_down: |
| integration2 | `?` | |
| unittests1 | `68.24% <80.17%> (-0.52%)` | :arrow_down: |
| unittests2 | `14.33% <0.42%> (-0.22%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...he/pinot/core/operator/blocks/ProjectionBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvUHJvamVjdGlvbkJsb2NrLmphdmE=) | `60.00% <ø> (ø)` | |
| [...che/pinot/core/operator/blocks/TransformBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvVHJhbnNmb3JtQmxvY2suamF2YQ==) | `56.25% <0.00%> (-12.99%)` | :arrow_down: |
| [...core/startree/plan/StarTreeProjectionPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlUHJvamVjdGlvblBsYW5Ob2RlLmphdmE=) | `100.00% <ø> (ø)` | |
| [.../core/startree/plan/StarTreeTransformPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9zdGFydHJlZS9wbGFuL1N0YXJUcmVlVHJhbnNmb3JtUGxhbk5vZGUuamF2YQ==) | `100.00% <ø> (ø)` | |
| [...not/controller/validation/StorageQuotaChecker.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29udHJvbGxlci9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29udHJvbGxlci92YWxpZGF0aW9uL1N0b3JhZ2VRdW90YUNoZWNrZXIuamF2YQ==) | `71.60% <16.66%> (-4.72%)` | :arrow_down: |
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...gation/function/FilterableAggregationFunction.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9hZ2dyZWdhdGlvbi9mdW5jdGlvbi9GaWx0ZXJhYmxlQWdncmVnYXRpb25GdW5jdGlvbi5qYXZh) | `74.07% <74.07%> (ø)` | |
| ... and [251 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [41c98d7...6150f87](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] richardstartin commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
richardstartin commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r785077871
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
Review comment:
Why IdentityHashMap? Computing the identity hash code has far reaching consequences and shouldn’t be the first port of call. Is this because AggregationFunction’s hashCode isn’t trusted or for performance reasons? If the latter I would like to see some evidence that this is a bottleneck.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r788446605
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
Unless this has recently changed, I think stream api usage is not consistent with Pinot coding convention.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (8a2dbd9) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `3.26%`.
> The diff coverage is `87.98%`.
> :exclamation: Current head 8a2dbd9 differs from pull request most recent head c8d9ad5. Consider uploading reports for the commit c8d9ad5 to get more accurate results
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 67.98% -3.27%
+ Complexity 4262 4180 -82
============================================
Files 1607 1215 -392
Lines 83409 60713 -22696
Branches 12458 9368 -3090
============================================
- Hits 59426 41274 -18152
+ Misses 19941 16535 -3406
+ Partials 4042 2904 -1138
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `?` | |
| integration2 | `?` | |
| unittests1 | `67.98% <87.98%> (+0.11%)` | :arrow_up: |
| unittests2 | `?` | |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `77.06% <87.65%> (-15.92%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.58% <97.77%> (-0.33%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [625 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...c8d9ad5](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] codecov-commenter edited a comment on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
codecov-commenter edited a comment on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-996061423
# [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#7916](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (ec5b19c) into [master](https://codecov.io/gh/apache/pinot/commit/0fe7ef89127b9e920ef369cd9adad9c8b817dde9?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (0fe7ef8) will **decrease** coverage by `0.93%`.
> The diff coverage is `90.18%`.
[](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
```diff
@@ Coverage Diff @@
## master #7916 +/- ##
============================================
- Coverage 71.24% 70.31% -0.94%
- Complexity 4262 4263 +1
============================================
Files 1607 1611 +4
Lines 83409 83559 +150
Branches 12458 12477 +19
============================================
- Hits 59426 58753 -673
- Misses 19941 20756 +815
- Partials 4042 4050 +8
```
| Flag | Coverage Δ | |
|---|---|---|
| integration1 | `28.93% <37.38%> (-0.05%)` | :arrow_down: |
| integration2 | `?` | |
| unittests1 | `68.02% <87.85%> (+0.15%)` | :arrow_up: |
| unittests2 | `14.23% <0.00%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#carryforward-flags-in-the-pull-request-comment) to find out more.
| [Impacted Files](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [...apache/pinot/core/operator/blocks/FilterBlock.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9ibG9ja3MvRmlsdGVyQmxvY2suamF2YQ==) | `50.00% <50.00%> (-7.15%)` | :arrow_down: |
| [.../pinot/core/operator/docidsets/BitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvQml0bWFwRG9jSWRTZXQuamF2YQ==) | `62.50% <60.00%> (-37.50%)` | :arrow_down: |
| [...t/core/operator/filter/CombinedFilterOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9maWx0ZXIvQ29tYmluZWRGaWx0ZXJPcGVyYXRvci5qYXZh) | `70.00% <70.00%> (ø)` | |
| [...t/core/operator/docidsets/FilterBlockDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvRmlsdGVyQmxvY2tEb2NJZFNldC5qYXZh) | `72.72% <72.72%> (ø)` | |
| [...inot/core/query/reduce/PostAggregationHandler.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZWR1Y2UvUG9zdEFnZ3JlZ2F0aW9uSGFuZGxlci5qYXZh) | `91.86% <87.50%> (-0.45%)` | :arrow_down: |
| [...re/operator/query/FilteredAggregationOperator.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9xdWVyeS9GaWx0ZXJlZEFnZ3JlZ2F0aW9uT3BlcmF0b3IuamF2YQ==) | `90.00% <90.00%> (ø)` | |
| [...rg/apache/pinot/core/plan/AggregationPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0FnZ3JlZ2F0aW9uUGxhbk5vZGUuamF2YQ==) | `91.74% <93.82%> (-1.24%)` | :arrow_down: |
| [...pinot/core/query/request/context/QueryContext.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9xdWVyeS9yZXF1ZXN0L2NvbnRleHQvUXVlcnlDb250ZXh0LmphdmE=) | `97.15% <96.00%> (-0.75%)` | :arrow_down: |
| [...re/operator/docidsets/RangelessBitmapDocIdSet.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9vcGVyYXRvci9kb2NpZHNldHMvUmFuZ2VsZXNzQml0bWFwRG9jSWRTZXQuamF2YQ==) | `100.00% <100.00%> (ø)` | |
| [...ava/org/apache/pinot/core/plan/FilterPlanNode.java](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-cGlub3QtY29yZS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUvcGlub3QvY29yZS9wbGFuL0ZpbHRlclBsYW5Ob2RlLmphdmE=) | `89.21% <100.00%> (+2.34%)` | :arrow_up: |
| ... and [102 more](https://codecov.io/gh/apache/pinot/pull/7916/diff?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | |
------
[Continue to review full report at Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
> **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
> `Δ = absolute <relative> (impact)`, `ø = not affected`, `? = missing data`
> Powered by [Codecov](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=footer&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Last update [0fe7ef8...ec5b19c](https://codecov.io/gh/apache/pinot/pull/7916?src=pr&el=lastupdated&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] amrishlal commented on a change in pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
amrishlal commented on a change in pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#discussion_r794846839
##########
File path: pinot-core/src/main/java/org/apache/pinot/core/operator/query/FilteredAggregationOperator.java
##########
@@ -0,0 +1,116 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+package org.apache.pinot.core.operator.query;
+
+import java.util.Arrays;
+import java.util.IdentityHashMap;
+import java.util.List;
+import java.util.stream.Collectors;
+import org.apache.commons.lang3.tuple.Pair;
+import org.apache.pinot.core.common.Operator;
+import org.apache.pinot.core.operator.BaseOperator;
+import org.apache.pinot.core.operator.ExecutionStatistics;
+import org.apache.pinot.core.operator.blocks.IntermediateResultsBlock;
+import org.apache.pinot.core.operator.blocks.TransformBlock;
+import org.apache.pinot.core.operator.transform.TransformOperator;
+import org.apache.pinot.core.query.aggregation.AggregationExecutor;
+import org.apache.pinot.core.query.aggregation.DefaultAggregationExecutor;
+import org.apache.pinot.core.query.aggregation.function.AggregationFunction;
+
+
+/**
+ * This operator processes a collection of filtered (and potentially non filtered) aggregations.
+ *
+ * For a query with either all aggregations being filtered or a mix of filtered and non filtered aggregations,
+ * FilteredAggregationOperator will come into execution.
+ */
+@SuppressWarnings("rawtypes")
+public class FilteredAggregationOperator extends BaseOperator<IntermediateResultsBlock> {
+ private static final String OPERATOR_NAME = "FilteredAggregationOperator";
+ private static final String EXPLAIN_NAME = "FILTERED_AGGREGATE";
+
+ private final AggregationFunction[] _aggregationFunctions;
+ private final List<Pair<AggregationFunction[], TransformOperator>> _aggFunctionsWithTransformOperator;
+ private final long _numTotalDocs;
+
+ private long _numDocsScanned;
+ private long _numEntriesScannedInFilter;
+ private long _numEntriesScannedPostFilter;
+
+ // We can potentially do away with aggregationFunctions parameter, but its cleaner to pass it in than to construct
+ // it from aggFunctionsWithTransformOperator
+ public FilteredAggregationOperator(AggregationFunction[] aggregationFunctions,
+ List<Pair<AggregationFunction[], TransformOperator>> aggFunctionsWithTransformOperator, long numTotalDocs) {
+ _aggregationFunctions = aggregationFunctions;
+ _aggFunctionsWithTransformOperator = aggFunctionsWithTransformOperator;
+ _numTotalDocs = numTotalDocs;
+ }
+
+ @Override
+ protected IntermediateResultsBlock getNextBlock() {
+ int numAggregations = _aggregationFunctions.length;
+ Object[] result = new Object[numAggregations];
+ IdentityHashMap<AggregationFunction, Integer> resultIndexMap = new IdentityHashMap<>(numAggregations);
+ for (int i = 0; i < numAggregations; i++) {
+ resultIndexMap.put(_aggregationFunctions[i], i);
+ }
+
+ for (Pair<AggregationFunction[], TransformOperator> filteredAggregation : _aggFunctionsWithTransformOperator) {
+ AggregationFunction[] aggregationFunctions = filteredAggregation.getLeft();
+ AggregationExecutor aggregationExecutor = new DefaultAggregationExecutor(aggregationFunctions);
+ TransformOperator transformOperator = filteredAggregation.getRight();
+ TransformBlock transformBlock;
+ int numDocsScanned = 0;
+ while ((transformBlock = transformOperator.nextBlock()) != null) {
+ aggregationExecutor.aggregate(transformBlock);
+ numDocsScanned += transformBlock.getNumDocs();
+ }
+ List<Object> filteredResult = aggregationExecutor.getResult();
+
+ for (int i = 0; i < aggregationFunctions.length; i++) {
+ result[resultIndexMap.get(aggregationFunctions[i])] = filteredResult.get(i);
+ }
+ _numDocsScanned += numDocsScanned;
+ _numEntriesScannedInFilter += transformOperator.getExecutionStatistics().getNumEntriesScannedInFilter();
+ _numEntriesScannedPostFilter += (long) numDocsScanned * transformOperator.getNumColumnsProjected();
+ }
+ return new IntermediateResultsBlock(_aggregationFunctions, Arrays.asList(result), false);
+ }
+
+ @Override
+ public String getOperatorName() {
+ return OPERATOR_NAME;
+ }
+
+ @Override
+ public List<Operator> getChildOperators() {
+ return _aggFunctionsWithTransformOperator.stream().map(Pair::getRight).collect(Collectors.toList());
Review comment:
@Jackie-Jiang Can you please clarify if stream api usage applies?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org
[GitHub] [pinot] Jackie-Jiang commented on pull request #7916: FILTER Clauses for Aggregates
Posted by GitBox <gi...@apache.org>.
Jackie-Jiang commented on pull request #7916:
URL: https://github.com/apache/pinot/pull/7916#issuecomment-1063534413
@atris This is a great feature. Could you please add some release notes to the PR description which we can refer to when cutting the next release?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@pinot.apache.org
For additional commands, e-mail: commits-help@pinot.apache.org