You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2022/09/14 03:35:26 UTC
[GitHub] [doris] zhengshiJ opened a new pull request, #12583: [feature](Nereids) add grouping sets parse
zhengshiJ opened a new pull request, #12583:
URL: https://github.com/apache/doris/pull/12583
# Proposed changes
Issue Number: close #xxx
## Problem summary
Added parsing of grouping sets, rollup and cube
## Checklist(Required)
1. Does it affect the original behavior:
- [ ] Yes
- [ ] No
- [ ] I don't know
2. Has unit tests been added:
- [ ] Yes
- [ ] No
- [ ] No Need
3. Has document been added or modified:
- [ ] Yes
- [ ] No
- [ ] No Need
4. Does it need to update dependencies:
- [ ] Yes
- [ ] No
5. Are there any changes that cannot be rolled back:
- [ ] Yes (If Yes, please explain WHY)
- [ ] No
## Further comments
If this is a relatively large or complex change, kick off the discussion at [dev@doris.apache.org](mailto:dev@doris.apache.org) by explaining why you chose the solution you did and what alternatives you considered, etc...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of times, so you can collect all used slot firstly, then collect slots according to your needs.
```java
Set<Slot> usedSlots = collectUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlots);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlots);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151519
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
Review Comment:
Again, can you rename method name to explain **what slot** should be set to nullable?
And, IMO, the slot int GroupingScalarFunction is VirtualSlotReference, is this method set nullable to non virtual slots and the virtual grouping set id slot?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020159143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -198,6 +220,10 @@ public PlanFragment visitPhysicalAggregate(
slotList.addAll(groupSlotList);
slotList.addAll(aggFunctionOutput);
outputTupleDesc = generateTupleDesc(slotList, null, context);
+
+ if (needSetSlotToNullable(outputExpressionList)) {
+ setSlotNullable(outputTupleDesc, slotList);
+ }
Review Comment:
if the method is simple enough, please let the code reader understand the core logic without enter the method.
```suggestion
if (containsGroupingScalar(outputExpressionList)
|| containsVirtualSlot(outputExpressionList)) {
setNullableToNonVirtualSlotReference(outputTupleDesc, slotList);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019915205
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +140,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
+ List<NamedExpression> outputExpressionList) {
+ List<SlotReference> groupSlotList = Lists.newArrayList();
+ Set<VirtualSlotReference> virtualSlotReferences = groupByExpressionList.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toSet());
+ for (Expression e : groupByExpressionList) {
+ if (e instanceof SlotReference && outputExpressionList.stream().anyMatch(o -> o.anyMatch(e::equals))) {
+ groupSlotList.add((SlotReference) e);
Review Comment:
```suggestion
if (virtualSlotReferences.contains(e))) {
groupSlotList.add((VirtualSlotReference) e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020162030
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +142,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
Review Comment:
rename to `collectGroupBySlots`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] zhengshiJ commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
zhengshiJ commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1003209193
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/ChildOutputPropertyDeriver.java:
##########
@@ -87,6 +89,18 @@ public PhysicalProperties visitPhysicalAggregate(PhysicalAggregate<? extends Pla
}
}
+ @Override
+ public PhysicalProperties visitPhysicalGroupBy(PhysicalGroupBy<? extends Plan> groupBy, PlanContext context) {
Review Comment:
@morrySnow Please help to see if this is correct
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/properties/RequestPropertyDeriver.java:
##########
@@ -113,6 +114,12 @@ public Void visitPhysicalAggregate(PhysicalAggregate<? extends Plan> agg, PlanCo
// TODO: add other phase logical when we support distinct aggregate
}
+ @Override
+ public Void visitPhysicalGroupBy(PhysicalGroupBy<? extends Plan> groupBy, PlanContext context) {
Review Comment:
@morrySnow Please help to see if this is correct
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of times, so you can extract all used slot, and extract slots according to your needs.
```java
Set<Slot> usedSlots = collectUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlot);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlot);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019915205
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +140,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
+ List<NamedExpression> outputExpressionList) {
+ List<SlotReference> groupSlotList = Lists.newArrayList();
+ Set<VirtualSlotReference> virtualSlotReferences = groupByExpressionList.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toSet());
+ for (Expression e : groupByExpressionList) {
+ if (e instanceof SlotReference && outputExpressionList.stream().anyMatch(o -> o.anyMatch(e::equals))) {
+ groupSlotList.add((SlotReference) e);
Review Comment:
all private method shoud move to the class's tailing
```suggestion
if (virtualSlotReferences.contains(e))) {
groupSlotList.add((VirtualSlotReference) e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019972143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,193 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so `getUsedSortedSlots` is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
.addAll(outputExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
).build();
Set<Slot> nonVirtualSlots = allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference)
.collect(ImmutableSet.toImmutableSet());
Set<Slot> virtualSlots = allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference)
.collect(ImmutableSet.toImmutableSet());
return ImmutableList.Builder()
.addAll(nonVirtualSlots)
.addAll(virtualSlots);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019934930
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +142,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
Review Comment:
all private method should move to tailing of the class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow commented on pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
morrySnow commented on PR #12583:
URL: https://github.com/apache/doris/pull/12583#issuecomment-1321576102
duplicated with #14430
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020159143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -198,6 +220,10 @@ public PlanFragment visitPhysicalAggregate(
slotList.addAll(groupSlotList);
slotList.addAll(aggFunctionOutput);
outputTupleDesc = generateTupleDesc(slotList, null, context);
+
+ if (needSetSlotToNullable(outputExpressionList)) {
+ setSlotNullable(outputTupleDesc, slotList);
+ }
Review Comment:
if the method is simple enough, please let the code reader understand the core logic without enter the method.
```suggestion
if (containsGroupingScalarFunction(outputExpressionList)
|| containsVirtualSlot(outputExpressionList)) {
setNullableToNonVirtualSlotReference(outputTupleDesc, slotList);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not rename method name to `containsGroupingScalarFunctioncOrVirtualSlot `?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -198,6 +220,10 @@ public PlanFragment visitPhysicalAggregate(
slotList.addAll(groupSlotList);
slotList.addAll(aggFunctionOutput);
outputTupleDesc = generateTupleDesc(slotList, null, context);
+
+ if (needSetSlotToNullable(outputExpressionList)) {
+ setSlotNullable(outputTupleDesc, slotList);
+ }
Review Comment:
if the invoke method is enough simple, please let the code reader understand the core logic without enter the method.
```suggestion
if (containsGroupingScalarFunctioncOrVirtualSlot(outputExpressionList)) {
setNullableToNonVirtualSlotReference(outputTupleDesc, slotList);
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
Review Comment:
this copy is useless.
```java
List<Slot> virtualSlotList = virtualGroupByExpressions.stream()
.filter(VirtualSlotReference.class::isInstance)
.map(Slot.class::cast)
.collect(Collectors.toList());
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
Review Comment:
BitSet is not semantically stronger than GroupingSetIdSlot, and In my refactor code, a GroupingSetIdSlot is a grouping set, not all grouping sets: https://github.com/924060929/incubator-doris/commit/4cb59e624881945ad40b4845ba9a6226b3938721#diff-ed44fafa8ec05af2eb892ba63a7d3040eb25f4ea71057a23d257f82da5d4d1ceR49
So I suggest rename GroupingSetIdSlot to GroupingSetShape, and represented at the shape of one grouping set and change to this:
```java
// Can you feel strong semantics???
private List<Set<Integer>> collectSlotIdsByGroupingSetsShape(
List<GroupingSetShape> groupingSetsShape, List<SlotDescriptor> groupingSlotDescs) {
ImmutableList.Builder<Set<Integer>> groupingIdList = ImmutableList.Builder();
for (GroupingSetId oneGroupingSetShape : groupingSetsShape) {
ImmutableSet.Builder<Integer> slotIdSet = ImmutableSet.Builder();
for (int i = 0; i < groupingSetsShape.size(); i++) {
GroupingSetShape oneGroupinSetShape = groupingSetsShape.get(i);
// Can you feel strong semantics???
if (!oneGroupinSetShape.shouldBeErasedToNull(i)) {
slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
}
}
groupingIdList.add(slotIdSet.build());
}
return groupingIdList.build();
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +142,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
Review Comment:
rename to `collectGroupBySlots`
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
Review Comment:
Again, can you rename method name to explain **what slot** should be set to nullable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020020010
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so collectUsedSortedSlots is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> collectUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupExpr -> groupExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(outputExpr -> outputExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
Map<Boolean, Set<Slot>> whetherIsVirtualSlots = groupBySlot.stream()
.collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
LinkedHashMap::new, ImmutableSet.toImmutableSet()));
return ImmutableList.Builder()
.addAll(whetherIsVirtualSlots.get(false)) // non virtual slots
.addAll(whetherIsVirtualSlots.get(true)) // virtual slots
.build();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019915205
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +140,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
+ List<NamedExpression> outputExpressionList) {
+ List<SlotReference> groupSlotList = Lists.newArrayList();
+ Set<VirtualSlotReference> virtualSlotReferences = groupByExpressionList.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toSet());
+ for (Expression e : groupByExpressionList) {
+ if (e instanceof SlotReference && outputExpressionList.stream().anyMatch(o -> o.anyMatch(e::equals))) {
+ groupSlotList.add((SlotReference) e);
Review Comment:
```suggestion
if (virtualSlotReferences.contains(e))) {
groupSlotList.add((VirtualSlotReference) e);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/ExpressionTranslator.java:
##########
@@ -380,4 +409,12 @@ private static org.apache.doris.analysis.AssertNumRowsElement.Assertion translat
throw new AnalysisException("UnSupported type: " + assertion);
}
}
+
+ public static GroupByClause.GroupingType translateGroupingType(
+ PhysicalRepeat repeat) {
+ if (repeat instanceof PhysicalGroupingSets) {
+ return GroupingType.GROUPING_SETS;
+ }
+ throw new AnalysisException("UnSupported type");
+ }
Review Comment:
PhysicalRepeat is a Plan, don't translate in ExpressionTranslator.
You can move to PhysicalPlanTranslator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013695443
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalRepeat.java:
##########
@@ -0,0 +1,212 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableSet;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Logical Repeat.
+ */
+public abstract class LogicalRepeat<CHILD_TYPE extends Plan>
+ extends LogicalUnary<CHILD_TYPE> implements Repeat {
+
+ public static final String COL_GROUPING_ID = "GROUPING_ID";
+ public static final String GROUPING_PREFIX = "GROUPING_PREFIX_";
+ // max num of distinct sets in grouping sets clause
+ public static final int MAX_GROUPING_SETS_NUM = 64;
+ protected final List<Expression> groupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final Set<VirtualSlotReference> virtualSlotRefs;
+ protected final List<BitSet> groupingIdList;
+ protected final List<Expression> virtualGroupingExprs;
+ protected final boolean changedOutput;
+ protected final boolean isNormalized;
Review Comment:
We should have some comments for these members.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat<CHILD_TYPE extends Plan>
+ extends PhysicalUnary<CHILD_TYPE> implements Repeat {
+ protected final List<Expression> groupByExpressions;
+ protected final List<Expression> originGroupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final List<BitSet> groupingIdList;
+ protected final List<Expression> virtualGroupByExpressions;
Review Comment:
We should add some comments for these members. In particular, `groupByExpressions` and `originGroupByExpressions` and confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019972143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,193 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so `getUsedSortedSlots` is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
Set<Slot> nonVirtualSlots = allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference)
.collect(ImmutableSet.toSet());
Set<Slot> virtualSlots = allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference)
.collect(ImmutableSet.toSet());
return ImmutableList.Builder()
.addAll(nonVirtualSlots)
.addAll(virtualSlots);
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +142,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
Review Comment:
all private method should move to tailing of the class
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020020320
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
ditto
```java
private List<Expression> collectUsedNonVirtualSlots(
List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupExpr -> groupExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(outputExpr -> outputExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
return allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference))
.collect(ImmutableList.toImmutableList());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013698522
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat<CHILD_TYPE extends Plan>
+ extends PhysicalUnary<CHILD_TYPE> implements Repeat {
+ protected final List<Expression> groupByExpressions;
+ protected final List<Expression> originGroupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final List<BitSet> groupingIdList;
+ protected final List<Expression> virtualGroupByExpressions;
Review Comment:
We should add some comments for these members. In particular, `groupByExpressions` and `originGroupByExpressions` and are confusing.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##########
@@ -0,0 +1,148 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat<CHILD_TYPE extends Plan>
+ extends PhysicalUnary<CHILD_TYPE> implements Repeat {
+ protected final List<Expression> groupByExpressions;
+ protected final List<Expression> originGroupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final List<BitSet> groupingIdList;
+ protected final List<Expression> virtualGroupByExpressions;
Review Comment:
We should add some comments for these members. In particular, `groupByExpressions` and `originGroupByExpressions` are confusing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] hello-stephen commented on pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
hello-stephen commented on PR #12583:
URL: https://github.com/apache/doris/pull/12583#issuecomment-1293355501
TeamCity pipeline, clickbench performance test result:
the sum of best hot time: 38.53 seconds
load time: 574 seconds
storage size: 17154655310 Bytes
https://doris-community-test-1308700295.cos.ap-hongkong.myqcloud.com/tmp/20221027110141_clickbench_pr_34751.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1009378540
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/GroupByResolve.java:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.analysis.OneAnalysisRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.grouping.GroupingSetsFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalGroupBy;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.planner.PlannerContext;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Parse the grouping sets operator,
+ * for the groupingFunc function, create a virtualSlotRefrance for a period of time.
+ *
+ * At the same time, synchronize the groupBy in groupingSets to aggregate.
+ */
+public class GroupByResolve extends OneAnalysisRuleFactory {
+ @Override
+ public Rule build() {
+ return logicalAggregate(logicalGroupBy().whenNot(LogicalGroupBy::isResolved)).thenApply(ctx -> {
+ LogicalAggregate<LogicalGroupBy<GroupPlan>> aggregate = ctx.root;
+ LogicalGroupBy<GroupPlan> groupBy = aggregate.child();
+
+ List<Expression> originGroupingExprs = groupBy.getOriginalGroupByExpressions();
+ List<List<Expression>> groupingSets = groupBy.getGroupingSets();
+ Set<VirtualSlotReference> newVirtualSlotRefs = new LinkedHashSet<>();
+ newVirtualSlotRefs.addAll(groupBy.getVirtualSlotRefs());
+
+ // resolve grouping func in groupBy
+ List<NamedExpression> groupByNewOutput =
+ groupBy.getOutputExpressions().stream()
+ .map(e -> resolve(e, newVirtualSlotRefs))
+ .map(NamedExpression.class::cast)
+ .collect(Collectors.toList());
+
+ List<BitSet> groupingIdList = groupBy.genGroupingIdList(originGroupingExprs, groupingSets);
+ List<Expression> virtualGroupingExprs = Lists.newArrayList();
+ virtualGroupingExprs.addAll(newVirtualSlotRefs);
+ List<List<Long>> groupingList = groupBy.genGroupingList(originGroupingExprs,
+ virtualGroupingExprs, groupingIdList, newVirtualSlotRefs);
+ List<Expression> aggGroupByExpressions = new ArrayList<>(originGroupingExprs);
+ aggGroupByExpressions.addAll(virtualGroupingExprs);
+
+ return new LogicalAggregate<>(aggGroupByExpressions, groupByNewOutput,
+ groupBy.replace(groupingSets, originGroupingExprs,
+ groupByNewOutput,
+ groupingIdList, newVirtualSlotRefs, virtualGroupingExprs,
+ groupingList, true, groupBy.hasChangedOutput(), groupBy.isNormalized()));
+ }).toRule(RuleType.GROUP_BY_RESOLVE);
+ }
+
+ private Expression resolve(Expression expr, Set<VirtualSlotReference> virtualSlotRefs) {
+ return new ResolveGroupingFun(virtualSlotRefs).resolve(expr);
+ }
+
+ private class ResolveGroupingFun extends DefaultExpressionRewriter<PlannerContext> {
Review Comment:
`ResolveGroupingFun` look like create new VirtualSlotReference for the parameter which is not VirtualSlotReference.
So why not choose a meaningful name? e.g. `WrapParamOfGroupingFunctionToVirtualSlot`.
And then you should comment it why you do this wrap.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/functions/grouping/GroupingSetsFunction.java:
##########
@@ -0,0 +1,63 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions.functions.grouping;
+
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.BoundFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.types.DataType;
+
+import java.util.List;
+import java.util.Optional;
+
+/**
+ * Functions used by all groupingSets.
+ */
+public abstract class GroupingSetsFunction extends BoundFunction {
+ protected final Optional<List<Expression>> realChildren;
+
+ public GroupingSetsFunction(String name, Expression... arguments) {
+ super(name, arguments);
+ this.realChildren = Optional.empty();
+ }
+
+ public GroupingSetsFunction(String name, Optional<List<Expression>> realChildren, Expression... arguments) {
+ super(name, arguments);
+ this.realChildren = realChildren;
Review Comment:
you should ensure realChildren can not be null
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/parser/LogicalPlanBuilder.java:
##########
@@ -906,12 +909,28 @@ private LogicalPlan withFilter(LogicalPlan input, Optional<WhereClauseContext> w
private LogicalPlan withAggregate(LogicalPlan input, SelectClauseContext selectCtx,
Optional<AggClauseContext> aggCtx) {
return input.optionalMap(aggCtx, () -> {
- List<Expression> groupByExpressions = visit(aggCtx.get().groupByItem().expression(), Expression.class);
+ List<Expression> groupByExpressions = visit(
+ aggCtx.get().groupByItem().groupingElement().expression(), Expression.class);
List<NamedExpression> namedExpressions = getNamedExpressions(selectCtx.namedExpressionSeq());
return new LogicalAggregate<>(groupByExpressions, namedExpressions, input);
});
}
+ private LogicalPlan withGroupBy(LogicalPlan input, SelectClauseContext selectCtx,
+ Optional<AggClauseContext> aggCtx) {
+ return input.optionalMap(aggCtx, () -> {
+ GroupingElementContext groupingElementCtx = aggCtx.get().groupByItem().groupingElement();
+ List<NamedExpression> namedExpressions = getNamedExpressions(selectCtx.namedExpressionSeq());
+ if (groupingElementCtx.GROUPING() != null) {
+ List<List<Expression>> sets = groupingElementCtx.groupingSet().stream()
+ .map(groupingSet -> visit(groupingSet.expression(), Expression.class))
+ .collect(Collectors.toList());
+ return new LogicalGroupingSets<>(sets, namedExpressions, input);
+ }
+ return null;
Review Comment:
you need throw a ParseException
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/VirtualSlotReference.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * it is not a real column exist in table.
+ */
+public class VirtualSlotReference extends SlotReference {
+ private final List<Expression> realSlots;
+ private final boolean hasCast;
+
+ public VirtualSlotReference(String name, DataType dataType, List<Expression> realSlots, boolean hasCast) {
+ super(name, dataType, false);
+ this.realSlots = Objects.requireNonNull(realSlots, "realSlots can not null");
+ this.hasCast = hasCast;
+ }
+
+ public VirtualSlotReference(ExprId exprId, String name, DataType dataType,
+ boolean nullable, List<String> qualifier,
+ List<Expression> realSlots, boolean hasCast) {
+ super(exprId, name, dataType, nullable, qualifier);
+ this.realSlots = Objects.requireNonNull(realSlots, "realSlots can not null");
+ this.hasCast = hasCast;
+ }
+
+ public List<Expression> getRealSlots() {
+ return realSlots;
+ }
+
+ public boolean isHasCast() {
Review Comment:
Do not name isHas
```suggestion
public boolean hasCast() {
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/expression/rewrite/ExpressionRewrite.java:
##########
@@ -130,6 +133,39 @@ public Rule build() {
}
}
+ private class GroupByExpressionRewrite extends OneRewriteRuleFactory {
+ @Override
+ public Rule build() {
+ return logicalGroupBy().then(groupBy -> {
+ List<List<Expression>> groupingSetList = groupBy.getGroupingSets();
+ List<List<Expression>> newGroupingSetList = groupingSetList.stream().map(rewriter::rewrite)
+ .collect(Collectors.toList());
+
+ List<Expression> groupByExpressions = groupBy.getOriginalGroupByExpressions();
+ List<Expression> newGroupByExpressions = rewriter.rewrite(groupByExpressions);
+
+ List<Expression> virtualGroupingExpressions = groupBy.getVirtualGroupingExprs();
+ List<Expression> newVirtualGroupingExpressions = rewriter.rewrite(virtualGroupingExpressions);
+
+ Set<VirtualSlotReference> newVirtualRefs =
+ newVirtualGroupingExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toSet());
Review Comment:
This code should have the same indent
```java
Set<VirtualSlotReference> newVirtualRefs = newVirtualGroupingExpressions.stream()
.filter(VirtualSlotReference.class::isInstance)
.map(VirtualSlotReference.class::cast)
.collect(Collectors.toSet());
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/GroupByResolve.java:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.analysis.OneAnalysisRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.grouping.GroupingSetsFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalGroupBy;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.planner.PlannerContext;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Parse the grouping sets operator,
+ * for the groupingFunc function, create a virtualSlotRefrance for a period of time.
+ *
+ * At the same time, synchronize the groupBy in groupingSets to aggregate.
+ */
+public class GroupByResolve extends OneAnalysisRuleFactory {
+ @Override
+ public Rule build() {
+ return logicalAggregate(logicalGroupBy().whenNot(LogicalGroupBy::isResolved)).thenApply(ctx -> {
Review Comment:
The isResolved state is a common state for all plan, we already support the Plan::canBind to detect whether the plan is resolved. So why not use Plan::canBind as the condition? You should remove the isResolved field.
```java
return logicalAggregate(logicalGroupBy().when(Plan::canBind)).thenApply(ctx -> {
...
});
```
But the Plan::canBind should refine to Plan::canBindFunction and Plan::canBindSlot in the future.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/expressions/VirtualSlotReference.java:
##########
@@ -0,0 +1,97 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.expressions;
+
+import org.apache.doris.nereids.trees.expressions.visitor.ExpressionVisitor;
+import org.apache.doris.nereids.types.DataType;
+
+import com.google.common.base.Preconditions;
+
+import java.util.List;
+import java.util.Objects;
+
+/**
+ * it is not a real column exist in table.
+ */
+public class VirtualSlotReference extends SlotReference {
+ private final List<Expression> realSlots;
+ private final boolean hasCast;
+
+ public VirtualSlotReference(String name, DataType dataType, List<Expression> realSlots, boolean hasCast) {
+ super(name, dataType, false);
+ this.realSlots = Objects.requireNonNull(realSlots, "realSlots can not null");
Review Comment:
You should copyOf the realSlots, the realSlots list maybe mutable inside.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupBy.java:
##########
@@ -0,0 +1,287 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.GroupBy;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Logical Grouping sets.
+ */
+public abstract class LogicalGroupBy<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements GroupBy {
+
+ public static final String COL_GROUPING_ID = "GROUPING_ID";
+ public static final String GROUPING_PREFIX = "GROUPING_PREFIX_";
+ protected final List<Expression> groupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final Set<VirtualSlotReference> virtualSlotRefs;
+ protected final List<BitSet> groupingIdList;
+ protected final List<Expression> virtualGroupingExprs;
+ protected final List<List<Long>> groupingList;
+ protected final boolean isResolved;
+ protected final boolean changedOutput;
+ protected final boolean isNormalized;
+
+ /**
+ * initial construction method.
+ */
+ public LogicalGroupBy(
+ PlanType type,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties,
+ CHILD_TYPE child) {
+ super(type, groupExpression, logicalProperties, child);
+ this.groupByExpressions = ImmutableList.copyOf(groupByExpressions);
+ this.outputExpressions = ImmutableList.copyOf(outputExpressions);
+ this.groupingIdList = Lists.newArrayList();
+ this.virtualSlotRefs = new LinkedHashSet<>();
+ VirtualSlotReference virtualSlotReference = new VirtualSlotReference(
+ COL_GROUPING_ID, BigIntType.INSTANCE, new ArrayList<>(), false);
+ this.virtualSlotRefs.add(virtualSlotReference);
+ this.virtualGroupingExprs = Lists.newArrayList();
+ this.groupingList = Lists.newArrayList();
Review Comment:
List in TreeNode should be immutable. Instead of ImmutableList, ImmutableMap.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020020010
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so getUsedSortedSlots is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupExpr -> groupExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(outputExpr -> outputExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
Map<Boolean, Set<Slot>> whetherIsVirtualSlots = groupBySlot.stream()
.collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
LinkedHashMap::new, ImmutableSet.toImmutableSet()));
return ImmutableList.Builder()
.addAll(whetherIsVirtualSlots.get(false)) // non virtual slots
.addAll(whetherIsVirtualSlots.get(true)) // virtual slots
.build();
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
ditto
```java
private List<Expression> genPreExpressions(
List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupExpr -> groupExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(outputExpr -> outputExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
return allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference))
.collect(ImmutableList.toImmutableList());
}
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
Review Comment:
```suggestion
private void setNullableToNonVirtualSlotReference(TupleDescriptor tupleDescriptor, List<Slot> slots) {
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of time, so you can extract all used slot, and extract slots according to your needs.
```java
Set<Slot> usedSlots = extractUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> getUsedSortedSlots(Set<Slot> allSlot);
private Set<Expression> getUsedNonVirtualSlots(Set<Slot> allSlot);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not separate this method name to `containsGroupingScalarFunction` and `containsVirtualSlot`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not separate this method to `containsGroupingScalarFunction` and `containsVirtualSlot`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of times, so you can collect all used slot firstly, then collect slots according to your needs.
```java
Set<Slot> allUsedSlots = collectUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlots);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlots);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] morrySnow closed pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
morrySnow closed pull request #12583: [feature](Nereids) implement grouping sets
URL: https://github.com/apache/doris/pull/12583
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] wangshuo128 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
wangshuo128 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1013691586
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupingSets.java:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Grouping sets.
+ */
+public class LogicalGroupingSets<CHILD_TYPE extends Plan> extends LogicalRepeat<CHILD_TYPE> {
+ private final List<List<Expression>> groupingSets;
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<NamedExpression> outputExpressions,
+ CHILD_TYPE child) {
+ this(groupingSets, genOriginalGroupByExpressions(groupingSets), outputExpressions,
+ Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ CHILD_TYPE child) {
+ this(groupingSets, groupByExpressions, outputExpressions, groupingIdList,
+ virtualSlotRefs, virtualGroupingExprs, changedOutput,
+ isNormalized,
+ Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
+ super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions, outputExpressions,
+ groupExpression, logicalProperties, child);
+ this.groupingSets = groupingSets;
+ }
+
+ public LogicalGroupingSets(
+ List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
+ super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions, outputExpressions, groupingIdList,
+ virtualSlotRefs, virtualGroupingExprs, changedOutput,
+ isNormalized,
+ groupExpression, logicalProperties, child);
+ this.groupingSets = groupingSets;
+ }
+
+ @Override
+ public List<List<Expression>> getGroupingSets() {
+ return groupingSets;
+ }
+
+ @Override
+ public String toString() {
+ return Utils.toSqlString("LogicalGroupingSets",
+ "outputExpressions", outputExpressions,
+ "grouping sets", groupingSets,
+ "original groupByExpressions", groupByExpressions,
+ "groupByExpressions", getGroupByExpressions()
+ );
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitLogicalGroupingSets(this, context);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ LogicalGroupingSets that = (LogicalGroupingSets) o;
+ return Objects.equals(groupingSets, that.groupingSets)
+ && super.equals(that);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(groupingSets, groupByExpressions, outputExpressions, virtualSlotRefs,
+ groupingIdList, virtualGroupingExprs, changedOutput, isNormalized);
+ }
+
+ @Override
+ public List<Expression> getExpressions() {
+ return new ImmutableList.Builder<Expression>()
+ .addAll(getGroupByExpressions()).addAll(outputExpressions).build();
+ }
+
+ @Override
+ public List<Slot> computeOutput() {
+ return new ImmutableList.Builder<Slot>()
+ .addAll(child().getOutput())
+ .addAll(virtualGroupingExprs.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast).collect(Collectors.toList()))
+ .build();
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withChildren(List<Plan> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withGroupExpression(Optional<GroupExpression> groupExpression) {
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ groupExpression, Optional.of(getLogicalProperties()), children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withLogicalProperties(Optional<LogicalProperties> logicalProperties) {
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ Optional.empty(), logicalProperties, children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> replace(List<List<Expression>> groupByExprList,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressionList,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized) {
+ return new LogicalGroupingSets<>(groupByExprList, groupByExpressions, outputExpressionList,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized, child());
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> replaceWithChild(List<List<Expression>> groupByExprList,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressionList,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ Plan child) {
+ return new LogicalGroupingSets<>(groupByExprList, groupByExpressions, outputExpressionList,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized, child);
+ }
+
+ @Override
+ public List<BitSet> genGroupingIdList(List<Expression> groupingExprs, List<List<Expression>> groupingSets) {
+ List<BitSet> groupingIdList = new ArrayList<>();
+ groupingSets.forEach(expressions -> {
+ BitSet bitSet = new BitSet();
+ for (int i = 0; i < groupingExprs.size(); ++i) {
+ bitSet.set(i, expressions.contains(groupingExprs.get(i)));
+ }
+ if (!groupingIdList.contains(bitSet)) {
+ groupingIdList.add(bitSet);
+ }
+ });
+ return groupingIdList;
+ }
+
+ public static List<Expression> genOriginalGroupByExpressions(List<List<Expression>> groupingSets) {
Review Comment:
`genOriginalGroupByExpressions` -> `extractDistinctGroupByExpressions` ?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupingSets.java:
##########
@@ -0,0 +1,227 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.Slot;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.base.Preconditions;
+import com.google.common.collect.ImmutableList;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Grouping sets.
+ */
+public class LogicalGroupingSets<CHILD_TYPE extends Plan> extends LogicalRepeat<CHILD_TYPE> {
+ private final List<List<Expression>> groupingSets;
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<NamedExpression> outputExpressions,
+ CHILD_TYPE child) {
+ this(groupingSets, genOriginalGroupByExpressions(groupingSets), outputExpressions,
+ Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ CHILD_TYPE child) {
+ this(groupingSets, groupByExpressions, outputExpressions, groupingIdList,
+ virtualSlotRefs, virtualGroupingExprs, changedOutput,
+ isNormalized,
+ Optional.empty(), Optional.empty(), child);
+ }
+
+ public LogicalGroupingSets(List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
+ super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions, outputExpressions,
+ groupExpression, logicalProperties, child);
+ this.groupingSets = groupingSets;
+ }
+
+ public LogicalGroupingSets(
+ List<List<Expression>> groupingSets,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ Optional<GroupExpression> groupExpression,
+ Optional<LogicalProperties> logicalProperties, CHILD_TYPE child) {
+ super(PlanType.LOGICAL_GROUPING_SETS, groupByExpressions, outputExpressions, groupingIdList,
+ virtualSlotRefs, virtualGroupingExprs, changedOutput,
+ isNormalized,
+ groupExpression, logicalProperties, child);
+ this.groupingSets = groupingSets;
+ }
+
+ @Override
+ public List<List<Expression>> getGroupingSets() {
+ return groupingSets;
+ }
+
+ @Override
+ public String toString() {
+ return Utils.toSqlString("LogicalGroupingSets",
+ "outputExpressions", outputExpressions,
+ "grouping sets", groupingSets,
+ "original groupByExpressions", groupByExpressions,
+ "groupByExpressions", getGroupByExpressions()
+ );
+ }
+
+ @Override
+ public <R, C> R accept(PlanVisitor<R, C> visitor, C context) {
+ return visitor.visitLogicalGroupingSets(this, context);
+ }
+
+ @Override
+ public boolean equals(Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (o == null || getClass() != o.getClass()) {
+ return false;
+ }
+ LogicalGroupingSets that = (LogicalGroupingSets) o;
+ return Objects.equals(groupingSets, that.groupingSets)
+ && super.equals(that);
+ }
+
+ @Override
+ public int hashCode() {
+ return Objects.hash(groupingSets, groupByExpressions, outputExpressions, virtualSlotRefs,
+ groupingIdList, virtualGroupingExprs, changedOutput, isNormalized);
+ }
+
+ @Override
+ public List<Expression> getExpressions() {
+ return new ImmutableList.Builder<Expression>()
+ .addAll(getGroupByExpressions()).addAll(outputExpressions).build();
+ }
+
+ @Override
+ public List<Slot> computeOutput() {
+ return new ImmutableList.Builder<Slot>()
+ .addAll(child().getOutput())
+ .addAll(virtualGroupingExprs.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast).collect(Collectors.toList()))
+ .build();
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withChildren(List<Plan> children) {
+ Preconditions.checkArgument(children.size() == 1);
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withGroupExpression(Optional<GroupExpression> groupExpression) {
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ groupExpression, Optional.of(getLogicalProperties()), children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> withLogicalProperties(Optional<LogicalProperties> logicalProperties) {
+ return new LogicalGroupingSets<>(groupingSets, groupByExpressions, outputExpressions,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized,
+ Optional.empty(), logicalProperties, children.get(0));
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> replace(List<List<Expression>> groupByExprList,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressionList,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized) {
+ return new LogicalGroupingSets<>(groupByExprList, groupByExpressions, outputExpressionList,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized, child());
+ }
+
+ @Override
+ public LogicalGroupingSets<Plan> replaceWithChild(List<List<Expression>> groupByExprList,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressionList,
+ List<BitSet> groupingIdList,
+ Set<VirtualSlotReference> virtualSlotRefs,
+ List<Expression> virtualGroupingExprs,
+ boolean changedOutput,
+ boolean isNormalized,
+ Plan child) {
+ return new LogicalGroupingSets<>(groupByExprList, groupByExpressions, outputExpressionList,
+ groupingIdList, virtualSlotRefs, virtualGroupingExprs,
+ changedOutput, isNormalized, child);
+ }
+
+ @Override
+ public List<BitSet> genGroupingIdList(List<Expression> groupingExprs, List<List<Expression>> groupingSets) {
+ List<BitSet> groupingIdList = new ArrayList<>();
+ groupingSets.forEach(expressions -> {
+ BitSet bitSet = new BitSet();
+ for (int i = 0; i < groupingExprs.size(); ++i) {
+ bitSet.set(i, expressions.contains(groupingExprs.get(i)));
+ }
+ if (!groupingIdList.contains(bitSet)) {
+ groupingIdList.add(bitSet);
+ }
+ });
+ return groupingIdList;
+ }
+
+ public static List<Expression> genOriginalGroupByExpressions(List<List<Expression>> groupingSets) {
+ List<Expression> groupingExpression = new ArrayList<>();
+ groupingSets.forEach(groupingExpression::addAll);
+ return groupingExpression.stream().distinct().collect(ImmutableList.toImmutableList());
Review Comment:
```suggestion
return groupingSets.stream()
.flatMap(List::stream)
.distinct()
.collect(ImmutableList.toImmutableList());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1008814774
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/ResolveHaving.java:
##########
@@ -47,30 +49,50 @@
/**
* Resolve having clause to the aggregation.
*/
-public class ResolveHaving extends OneAnalysisRuleFactory {
+public class ResolveHaving implements AnalysisRuleFactory {
Review Comment:
In nereids, Resolve should rename to Bind, and Resolved should rename to Bound.
Please keep the naming consistent
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizeGroupBy.java:
##########
@@ -0,0 +1,265 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.rewrite.OneRewriteRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.grouping.GroupingSetsFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalGroupBy;
+import org.apache.doris.nereids.trees.plans.logical.LogicalPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalProject;
+import org.apache.doris.nereids.util.ExpressionUtils;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.Lists;
+import com.google.common.collect.Maps;
+
+import java.util.ArrayList;
+import java.util.List;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Set;
+import java.util.concurrent.atomic.AtomicBoolean;
+import java.util.stream.Collectors;
+
+/**
+ * normalize output, aggFunc and groupingFunc in grouping sets.
+ * This rule is executed after NormalizeAggregate.
+ *
+ * eg: select sum(k2 + 1), grouping(k1) from t1 group by grouping sets ((k1));
+ * Original Plan:
+ * Aggregate(
+ * keys:[k1#1, grouping_id()#0, grouping_prefix(k1#1)#7],
+ * outputs:[sum((k2 + 1)#8) AS `sum((k2 + 1))`#9, GROUPING_PREFIX_(k1#1)#7)
+ * +-- Project(projections: k1#1, (k2#2 + 1) as (k2 + 1)#8), grouping_prefix(k1#1)#7
+ * +-- GroupingSets(
+ * keys:[k1#1, grouping_id()#0, grouping_prefix(k1#1)#7]
+ * outputs:sum(k2#2 + 1) as `sum(k2 + 1)`#3, group(grouping_prefix(k1#1)#7) as `grouping(k1 + 1)`#4
+ *
+ * After rule:
+ * Aggregate(
+ * keys:[k1#1, SR#9]
+ * outputs:[sum((k2 + 1)#8) AS `sum((k2 + 1))`#9, GROUPING_PREFIX_(k1#1)#7))
+ * +-- Project(k1#1, (K2 + 1)#10 as `(k2 + 1)`#8, grouping_id()#0, grouping_prefix(k1#1)#7)
+ * +-- GropingSets(
+ * keys:[k1#1, grouping_id()#0, grouping_prefix(k1#1)#7]
+ * outputs:k1#1, (k2 + 1)#10, grouping_prefix(k1#1)#7
+ */
+public class NormalizeGroupBy extends OneRewriteRuleFactory {
+ @Override
+ public Rule build() {
+ return logicalAggregate(logicalProject(logicalGroupBy().whenNot(LogicalGroupBy::isNormalized)))
+ .then(agg -> {
Review Comment:
this rule is too complex, please extract main computation flow.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/GroupByResolve.java:
##########
@@ -0,0 +1,121 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.nereids.rules.Rule;
+import org.apache.doris.nereids.rules.RuleType;
+import org.apache.doris.nereids.rules.analysis.OneAnalysisRuleFactory;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.grouping.GroupingSetsFunction;
+import org.apache.doris.nereids.trees.expressions.visitor.DefaultExpressionRewriter;
+import org.apache.doris.nereids.trees.plans.GroupPlan;
+import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
+import org.apache.doris.nereids.trees.plans.logical.LogicalGroupBy;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.planner.PlannerContext;
+
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Parse the grouping sets operator,
+ * for the groupingFunc function, create a virtualSlotRefrance for a period of time.
+ *
+ * At the same time, synchronize the groupBy in groupingSets to aggregate.
+ */
+public class GroupByResolve extends OneAnalysisRuleFactory {
Review Comment:
Why not move this rule into the BindSlotReference? I think BindSlotReference should binding all slot in every plan.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupBy.java:
##########
@@ -0,0 +1,287 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.GroupBy;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Logical Grouping sets.
+ */
+public abstract class LogicalGroupBy<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements GroupBy {
Review Comment:
GroupBy look like AST, but nereids LogicalXxx is a Plan, using a mixture of naming conventions will cause complications later on.
And GroupBy is misleading, because it not contains the normal group by expressions.
I suggest rename it to GroupingElement, LogicalGroupingElement, PhysicalGroupingElement
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinFunctions.java:
##########
@@ -51,6 +53,11 @@ public class BuiltinFunctions implements FunctionHelper {
agg(Sum.class)
);
+ public final ImmutableList<GroupingSetsFunc> groupingSetsFunctions = ImmutableList.of(
+ groupingSets(Grouping.class),
+ groupingSets(GroupingId.class, "grouping_id")
+ );
Review Comment:
`grouping()` and `grouping_id()` are scalar function definitely, don't use special function list here, this will make others imitate wrongly and add more special function list.
The list separate by the function type of top abstract level:
- scalar function
- aggregate function
- table function
They describe how many rows would be consumed and how many rows would be produced, any function should be classified into one of them.
So I suggest rename GroupingSetsFunc to GroupingScalarFunc and extends ScalarFunc, then register in the scalar function list, without other special list.
```java
public final List<ScalarFunc> scalarFunctions = ImmutableList.builder()
.add(...) // other general functions
.add(groupingScalar(Grouping.class))
.add(groupingScalar(GroupingId.class, "groupinig_id"))
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not separate this method to `containsGroupingScalarFunction` and `containsVirtualSlot`?
Look the following suggestion.
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not separate this method to `containsGroupingScalarFunction` and `containsVirtualSlot`?
Look at the following suggestion.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151860
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private boolean needSetSlotToNullable(List<NamedExpression> slots) {
+ boolean hasVirtualSlotReference = slots.stream().anyMatch(VirtualSlotReference.class::isInstance);
+ boolean hasGroupingFunc = slots.stream()
+ .filter(Alias.class::isInstance)
+ .map(Alias.class::cast)
+ .anyMatch(s -> s.child().anyMatch(GroupingScalarFunction.class::isInstance));
+ return hasGroupingFunc || hasVirtualSlotReference;
+ }
Review Comment:
So why not separate method name to `containsGroupingScalarFunction` and `containsVirtualSlot`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020159143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -198,6 +220,10 @@ public PlanFragment visitPhysicalAggregate(
slotList.addAll(groupSlotList);
slotList.addAll(aggFunctionOutput);
outputTupleDesc = generateTupleDesc(slotList, null, context);
+
+ if (needSetSlotToNullable(outputExpressionList)) {
+ setSlotNullable(outputTupleDesc, slotList);
+ }
Review Comment:
please let the code reader understand the core logic without enter the method.
```suggestion
if (containsGroupingScalarFunctioncOrVirtualSlot(outputExpressionList)) {
setNullableToNonVirtualSlotReference(outputTupleDesc, slotList);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of time, so you can extract all used slot, and extract slots according to your needs.
```java
Set<Slot> usedSlots = extractUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlot);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlot);
```
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of time, so you can extract all used slot, and extract slots according to your needs.
```java
Set<Slot> usedSlots = collectUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlot);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlot);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of time, so you can extract all used slot, and extract slots according to your needs.
```java
Set<Slot> usedSlots = extractUsedSlots(groupByExpressions, outputExpressions);
private List<Slot> getUsedSortedSlots(Set<Slot> allSlot);
private List<Expression> getUsedNonVirtualSlots(Set<Slot> allSlot);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020034159
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
Look seems the used slots in aggregate/repeat will use lots of times, so you can collect all used slot firstly, then collect slots according to your needs.
```java
Set<Slot> usedSlots = collectUsedSlots(groupByExpressions, outputExpressions);
private Set<Slot> collectUsedSortedSlots(Set<Slot> allSlot);
private Set<Expression> collectUsedNonVirtualSlots(Set<Slot> allSlot);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1008815617
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/logical/LogicalGroupBy.java:
##########
@@ -0,0 +1,287 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.logical;
+
+import org.apache.doris.nereids.exceptions.AnalysisException;
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.GroupBy;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.types.BigIntType;
+import org.apache.doris.nereids.util.Utils;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.Lists;
+
+import java.util.ArrayList;
+import java.util.BitSet;
+import java.util.LinkedHashSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+import java.util.Set;
+
+/**
+ * Logical Grouping sets.
+ */
+public abstract class LogicalGroupBy<CHILD_TYPE extends Plan> extends LogicalUnary<CHILD_TYPE> implements GroupBy {
Review Comment:
GroupBy look like AST, but LogicalXxx is a Plan, using a mixture of naming conventions will cause complications later on.
And GroupBy is misleading, because it not contains the normal group by expressions.
I suggest rename it to GroupingElement, LogicalGroupingElement, PhysicalGroupingElement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1008811376
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/BuiltinFunctions.java:
##########
@@ -51,6 +53,11 @@ public class BuiltinFunctions implements FunctionHelper {
agg(Sum.class)
);
+ public final ImmutableList<GroupingSetsFunc> groupingSetsFunctions = ImmutableList.of(
+ groupingSets(Grouping.class),
+ groupingSets(GroupingId.class, "grouping_id")
+ );
Review Comment:
`grouping()` and `grouping_id()` are scalar function definitely, don't use special function list here, this will make others imitate wrongly and add more special function list.
The list separate by the function type of top abstract level:
- scalar function
- aggregate function
- table function
They describe how many rows would be consumed and how many rows would be produced, any function should be classified into one of them.
So I suggest rename GroupingSetsFunc to GroupingScalarFunc and extends ScalarFunc, rename groupingSets() to groupingScalar(), and then register in the scalar function list, without other special list.
```java
public final List<ScalarFunc> scalarFunctions = ImmutableList.builder()
.add(...) // other general functions
.add(groupingScalar(Grouping.class))
.add(groupingScalar(GroupingId.class, "groupinig_id"))
.build();
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019972143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,193 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so `getUsedSortedSlots` is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
.addAll(outputExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
).build();
Map<Boolean, Set<Slot>> whetherIsVirtualSlots = allSlots.stream()
.collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
LinkedHashMap::new, ImmutableSet.toImmutableSet());
return ImmutableList.Builder()
.addAll(whetherIsVirtualSlots.get(false)) // non virtual slots
.addAll(whetherIsVirtualSlots.get(true)); // virtual slots
.build();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019972143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,193 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so `getUsedSortedSlots` is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
.addAll(outputExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
).build();
Map<Boolean, Set<Slot>> whetherIsVirtualSlots = allSlots.stream()
.collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
LinkedHashMap::new, ImmutableSet.toImmutableSet());
return ImmutableList.Builder()
.addAll(whetherIsVirtualSlots.get(false)) // non virtual slots
.addAll(whetherIsVirtualSlots.get(true)); // virtual slots
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019972143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,193 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
Review Comment:
output maybe not slot, but contains slot, and you need sort it, so `getUsedSortedSlots` is more reasonable. And primary method should move to the tailing of the class.
And this method should be simplified to
```java
private List<Slot> getUsedSortedSlots(List<Expression> groupByExpressions, List<NamedExpression> outputExpressions)
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupByExpr -> groupByExpr.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
.addAll(outputExpressions.stream()
.flatMap(output -> output.getInputSlots().stream()))
.collect(ImmutableSet.toImmutableSet())
).build();
Map<Boolean, Set<Slot>> whetherIsVirtualSlots = allSlots.stream()
.collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
LinkedHashMap::new, ImmutableSet.toImmutableSet());
return ImmutableList.Builder()
.addAll(whetherIsVirtualSlots.get(false)) // non virtual slots
.addAll(whetherIsVirtualSlots.get(true)); // virtual slots
.build();
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1019915205
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -133,6 +140,26 @@ public PlanFragment translatePlan(PhysicalPlan physicalPlan, PlanTranslatorConte
return rootFragment;
}
+ private List<SlotReference> genGroupBySlotList(List<Expression> groupByExpressionList,
+ List<NamedExpression> outputExpressionList) {
+ List<SlotReference> groupSlotList = Lists.newArrayList();
+ Set<VirtualSlotReference> virtualSlotReferences = groupByExpressionList.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toSet());
+ for (Expression e : groupByExpressionList) {
+ if (e instanceof SlotReference && outputExpressionList.stream().anyMatch(o -> o.anyMatch(e::equals))) {
+ groupSlotList.add((SlotReference) e);
Review Comment:
```suggestion
if (outputExpressionList.contains(e))) {
groupSlotList.add((SlotReference) e);
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020020320
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
Review Comment:
ditto
```java
private List<Expression> getUsedNonVirtualSlots(
List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
Set <Slot> allSlots = ImmutableSet.builder()
.addAll(groupByExpressions.stream()
.flatMap(groupExpr -> groupExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
.addAll(outputExpressions.stream()
.flatMap(outputExpr -> outputExpr.getInputSlots().stream()))
.collect(ImmutableSet.toSet())
).build();
return allSlots.stream()
.filter(slot -> !(slot instanceof VirtualSlotReference))
.collect(ImmutableList.toImmutableList());
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020159143
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -198,6 +220,10 @@ public PlanFragment visitPhysicalAggregate(
slotList.addAll(groupSlotList);
slotList.addAll(aggFunctionOutput);
outputTupleDesc = generateTupleDesc(slotList, null, context);
+
+ if (needSetSlotToNullable(outputExpressionList)) {
+ setSlotNullable(outputTupleDesc, slotList);
+ }
Review Comment:
if the method is simple enough, please let the code reader understand the core logic without enter the method.
```suggestion
if (containsGroupingScalarFunctioncOrVirtualSlot(outputExpressionList)) {
setNullableToNonVirtualSlotReference(outputTupleDesc, slotList);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020151519
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/glue/translator/PhysicalPlanTranslator.java:
##########
@@ -257,6 +283,210 @@ public PlanFragment visitPhysicalAggregate(
return currentFragment;
}
+ /**
+ * Translate physicalRepeat to RepeatNode.
+ *
+ * eg: select k1 from t1 group by grouping sets ((k1), (k2, k3), (k3));
+ * groupingInfo:
+ * Contains:
+ * virtualTupleDesc: the tupleDescriptor generated for the virtual column
+ * preRepeatExprs: Expr used in repeatNode. eg: [k1, k2, k3]
+ *
+ * repeatSlotIdList: According to the bitmap corresponding to the original group by
+ * and the slotId information contained in the slotDescriptor,
+ * the bitmap corresponding to the slotId is obtained.
+ * eg: groupingSetsBitSet: [{0}, {1, 2}, {2}]. slotIdList:[5, 6, 7]
+ * ==> [{5}, {6, 7}, {7}].
+ *
+ * groupingList: The value of the virtual column generated by each group by.
+ * eg: [3, 4, 6]
+ *
+ * allSlotId: slotId of all used columns. eg: [5, 6, 7]
+ */
+ @Override
+ public PlanFragment visitPhysicalRepeat(PhysicalRepeat repeat, PlanTranslatorContext context) {
+ PlanFragment inputPlanFragment = repeat.child(0).accept(this, context);
+ List<Expression> groupByExpressions = repeat.getGroupByExpressions();
+ List<NamedExpression> outputExpressionList = repeat.getOutputExpressions();
+ List<Expression> virtualGroupByExpressions = repeat.getVirtualGroupByExpressions();
+ List<Expression> nonVirtualGroupByExpressions = repeat.getNonVirtualGroupByExpressions();
+
+ // 1.create GroupingInfo
+ /*
+ * finalSlots: grouping sets the final output column.
+ * preExpressions: Fields required by groupInfo, all fields that need to be used.
+ *
+ * eg: select sum(k2), grouping(k1) from t1 group by grouping sets((k1));
+ * finalSlots: k1, k2, GROUPING_ID(), GROUPING_PREFIX_k1
+ * prePressions: k1, k2
+ */
+ // create virtual tupleDesc
+ List<VirtualSlotReference> virtualSlotList = virtualGroupByExpressions.stream()
+ .filter(VirtualSlotReference.class::isInstance)
+ .map(VirtualSlotReference.class::cast)
+ .collect(Collectors.toList());
+ List<Slot> virtualSlots = Lists.newArrayList(virtualSlotList);
+ TupleDescriptor virtualTupleDesc = generateTupleDesc(virtualSlots, null, context);
+
+ // create repeat slots and PreExpressions
+ List<SlotReference> groupBySlot = genGroupBySlotList(groupByExpressions, outputExpressionList);
+ List<Slot> finalSlots = genOutputSlots(groupBySlot, outputExpressionList, virtualSlots);
+
+ List<Expression> preExpressions = genPreExpressions(nonVirtualGroupByExpressions, outputExpressionList);
+ ArrayList<Expr> preRepeatExprs = preExpressions.stream()
+ .map(e -> ExpressionTranslator.translate(e, context)).collect(Collectors.toCollection(ArrayList::new));
+
+ // create output TupleDesc
+ TupleResult tupleResult = genTupleDescAndDescList(finalSlots, null, context);
+ TupleDescriptor outputTupleDesc = tupleResult.getTupleDescriptor();
+ setSlotNullable(outputTupleDesc, finalSlots);
+
+ GroupingInfo groupingInfo = new GroupingInfo(ExpressionTranslator.translateGroupingType(repeat),
+ virtualTupleDesc, outputTupleDesc, preRepeatExprs);
+
+ // 2.create repeat node
+ // Replace the bitset of groupingsets with the bitset corresponding to slotId
+ List<Set<Integer>> repeatSlotIdList = genGroupingIdList(
+ repeat.useBitsetsToRepresentGroupingSets(), tupleResult.getSlotDescriptors());
+ Set<Integer> allSlotId = genAllSlotId(repeatSlotIdList);
+
+ RepeatNode repeatNode = new RepeatNode(context.nextPlanNodeId(),
+ inputPlanFragment.getPlanRoot(), groupingInfo, repeatSlotIdList,
+ allSlotId, repeat.getVirtualSlotValues());
+ repeatNode.setNumInstances(inputPlanFragment.getPlanRoot().getNumInstances());
+ inputPlanFragment.addPlanRoot(repeatNode);
+ inputPlanFragment.updateDataPartition(DataPartition.RANDOM);
+ return inputPlanFragment;
+ }
+
+ /**
+ * Generate outputSlot based on groupBy and output information.
+ * The generated outputSlot needs to guarantee the order,
+ * first slotReference and then virtualSlotReference.
+ *
+ * First collect the slot and then collect the virtualSlot.
+ * 1. Put the columns in groupBy into the slot
+ * 2. Put the columns in the output into the slot
+ * 3. Put slot and virtualSlot into finalSLots in turn.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * 1. slots:[k1]
+ * 2. slots:[k1, k2]
+ * 3. finalSlots: [k1, k2, GROUPING_PREFIX_k1]
+ */
+ private List<Slot> genOutputSlots(
+ List<SlotReference> groupBySlot,
+ List<NamedExpression> outputExpressionList,
+ List<Slot> virtualSlots) {
+
+ Map<Boolean, List<Slot>> allSlots = groupBySlot.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<Slot> nonVirtualSlots = allSlots.get(false);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+
+ nonVirtualSlots.addAll(nonVirtualOutput.stream()
+ .filter(e -> !nonVirtualSlots.contains(e))
+ .filter(SlotReference.class::isInstance)
+ .map(NamedExpression::toSlot)
+ .collect(Collectors.toSet()));
+ List<Slot> finalSlots = Lists.newArrayList();
+ finalSlots.addAll(nonVirtualSlots);
+ finalSlots.addAll(virtualSlots);
+ return finalSlots;
+ }
+
+ /**
+ * Get all the used columns that appear in repeatNode according to the column of groupBy
+ * and the column of aggFunc in output.
+ *
+ * eg: select sum(k2) grouping(k1) from t1 group by grouping sets((k1));
+ * preExpressions: [k1, k2].
+ */
+ private List<Expression> genPreExpressions(
+ List<Expression> nonVirtualGroupByExpressions, List<NamedExpression> outputExpressionList) {
+ List<Expression> preExpressions = Lists.newArrayList();
+ preExpressions.addAll(nonVirtualGroupByExpressions);
+ Map<Boolean, List<NamedExpression>> allOutput = outputExpressionList.stream()
+ .collect(Collectors.groupingBy(VirtualSlotReference.class::isInstance,
+ LinkedHashMap::new, Collectors.toList()));
+ List<NamedExpression> nonVirtualOutput = allOutput.get(false);
+ nonVirtualOutput.stream()
+ .filter(e -> !preExpressions.contains(e))
+ .forEach(preExpressions::add);
+ return preExpressions;
+ }
+
+ /**
+ * Generate bitSets corresponding to SlotId according to the original groupBy bitSets and the actual SlotId.
+ * eg:
+ * groupingSetsList: [(k1), (k2, k3), (k3)]
+ * originalGroupingIdList: [{0}, {1,2}, {2}]
+ * SlotIds in groupingSlotDescs: [3, 4, 5]
+ *
+ * return: [{3}, {4, 5}, {5}]
+ */
+ private List<Set<Integer>> genGroupingIdList(
+ List<BitSet> originalGroupingIdList, List<SlotDescriptor> groupingSlotDescs) {
+ List<Set<Integer>> groupingIdList = Lists.newArrayList();
+ for (BitSet bitSet : originalGroupingIdList) {
+ Set<Integer> slotIdSet = new HashSet<>();
+ for (int i = 0; i < groupingSlotDescs.size(); i++) {
+ if (bitSet.get(i)) {
+ slotIdSet.add(groupingSlotDescs.get(i).getId().asInt());
+ }
+ }
+ groupingIdList.add(slotIdSet);
+ }
+ return groupingIdList;
+ }
+
+ private static Set<Integer> genAllSlotId(List<Set<Integer>> repeatSlotIdList) {
+ Set<Integer> allSlotId = new LinkedHashSet<>();
+ for (Set<Integer> s : repeatSlotIdList) {
+ allSlotId.addAll(s);
+ }
+ return allSlotId;
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots) {
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!(slots.get(i) instanceof VirtualSlotReference)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
+
+ private void setSlotNullable(TupleDescriptor tupleDescriptor, List<Slot> slots, List<NamedExpression> projects) {
+ Preconditions.checkState(projects.size() == slots.size());
+ Set<Integer> notSetSlotNullable = new HashSet<>();
+ for (int i = 0; i < projects.size(); i++) {
+ if (projects.get(i) instanceof Alias
+ && ((Alias) projects.get(i)).child() instanceof GroupingScalarFunction) {
+ notSetSlotNullable.add(i);
+ }
+ }
+ for (int i = 0; i < slots.size(); ++i) {
+ if (!notSetSlotNullable.contains(i)) {
+ SlotDescriptor slotDescriptor = tupleDescriptor.getSlots().get(i);
+ slotDescriptor.setIsNullable(true);
+ }
+ }
+ }
Review Comment:
Again, can you rename method name to explain **what slot** should be set to nullable?
And, IMO, the slot int GroupingScalarFunction is VirtualSlotReference, is this method set nullable to non virtual slots?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [doris] 924060929 commented on a diff in pull request #12583: [feature](Nereids) implement grouping sets
Posted by GitBox <gi...@apache.org>.
924060929 commented on code in PR #12583:
URL: https://github.com/apache/doris/pull/12583#discussion_r1020218941
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/trees/plans/physical/PhysicalRepeat.java:
##########
@@ -0,0 +1,151 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.trees.plans.physical;
+
+import org.apache.doris.nereids.memo.GroupExpression;
+import org.apache.doris.nereids.properties.LogicalProperties;
+import org.apache.doris.nereids.properties.PhysicalProperties;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.plans.Plan;
+import org.apache.doris.nereids.trees.plans.PlanType;
+import org.apache.doris.nereids.trees.plans.algebra.Repeat;
+import org.apache.doris.nereids.trees.plans.logical.GroupingSetIdSlot;
+import org.apache.doris.nereids.trees.plans.visitor.PlanVisitor;
+import org.apache.doris.nereids.util.Utils;
+import org.apache.doris.statistics.StatsDeriveResult;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableList.Builder;
+
+import java.util.BitSet;
+import java.util.List;
+import java.util.Objects;
+import java.util.Optional;
+
+/**
+ * PhysicalRepeat.
+ */
+public abstract class PhysicalRepeat<CHILD_TYPE extends Plan>
+ extends PhysicalUnary<CHILD_TYPE> implements Repeat {
+ protected final List<Expression> groupByExpressions;
+ protected final List<NamedExpression> outputExpressions;
+ protected final List<GroupingSetIdSlot> groupingSetIdSlots;
+
+ /**
+ * initial construction method.
+ */
+ public PhysicalRepeat(
+ PlanType type,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<GroupingSetIdSlot> groupingSetIdSlots,
+ Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ CHILD_TYPE child) {
+ super(type, groupExpression, logicalProperties, child);
+ this.groupByExpressions = ImmutableList.copyOf(groupByExpressions);
+ this.outputExpressions = ImmutableList.copyOf(outputExpressions);
+ this.groupingSetIdSlots = ImmutableList.copyOf(groupingSetIdSlots);
+ }
+
+ /**
+ * Constructor with all parameters.
+ */
+ public PhysicalRepeat(
+ PlanType type,
+ List<Expression> groupByExpressions,
+ List<NamedExpression> outputExpressions,
+ List<GroupingSetIdSlot> groupingSetIdSlots,
+ Optional<GroupExpression> groupExpression, LogicalProperties logicalProperties,
+ PhysicalProperties physicalProperties, StatsDeriveResult statsDeriveResult, CHILD_TYPE child) {
+ super(type, groupExpression, logicalProperties, physicalProperties, statsDeriveResult, child);
+ this.groupByExpressions = ImmutableList.copyOf(groupByExpressions);
+ this.outputExpressions = ImmutableList.copyOf(outputExpressions);
+ this.groupingSetIdSlots = ImmutableList.copyOf(groupingSetIdSlots);
+ }
+
+ public List<NamedExpression> getOutputExpressions() {
+ return outputExpressions;
+ }
+
+ @Override
+ public List<Expression> getGroupByExpressions() {
+ return groupByExpressions;
+ }
+
+ public List<Expression> getNonVirtualGroupByExpressions() {
+ return groupByExpressions.stream().filter(e -> !(e instanceof VirtualSlotReference))
+ .collect(ImmutableList.toImmutableList());
+ }
+
+ public List<Expression> getVirtualGroupByExpressions() {
+ return groupByExpressions.stream().filter(VirtualSlotReference.class::isInstance)
+ .collect(ImmutableList.toImmutableList());
+ }
+
+ public List<GroupingSetIdSlot> getGroupingSetIdSlots() {
+ return groupingSetIdSlots;
+ }
+
+ public List<List<Long>> getVirtualSlotValues() {
+ Builder<List<Long>> wholeVirtualSlotValues = ImmutableList.builder();
+ groupingSetIdSlots.stream().forEach(e -> wholeVirtualSlotValues.add(e.toLongValue()));
+ return wholeVirtualSlotValues.build();
Review Comment:
RepeatNode.groupingList consists of
1. the big endian bits to long by the GroupingSetShape, that is your `groupingSetIdSlots.toLongValue()`
2. the big endian bits to long by other GroupingScalaFunction.toLongValue
Missing point 2?
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/NormalizePlan.java:
##########
@@ -0,0 +1,235 @@
+// Licensed to the Apache Software Foundation (ASF) under one
+// or more contributor license agreements. See the NOTICE file
+// distributed with this work for additional information
+// regarding copyright ownership. The ASF licenses this file
+// to you under the Apache License, Version 2.0 (the
+// "License"); you may not use this file except in compliance
+// with the License. You may obtain a copy of the License at
+//
+// http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing,
+// software distributed under the License is distributed on an
+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+// KIND, either express or implied. See the License for the
+// specific language governing permissions and limitations
+// under the License.
+
+package org.apache.doris.nereids.rules.rewrite.logical;
+
+import org.apache.doris.common.Pair;
+import org.apache.doris.nereids.trees.expressions.Alias;
+import org.apache.doris.nereids.trees.expressions.Expression;
+import org.apache.doris.nereids.trees.expressions.NamedExpression;
+import org.apache.doris.nereids.trees.expressions.SlotReference;
+import org.apache.doris.nereids.trees.expressions.VirtualSlotReference;
+import org.apache.doris.nereids.trees.expressions.functions.agg.AggregateFunction;
+import org.apache.doris.nereids.trees.expressions.functions.scalar.GroupingScalarFunction;
+import org.apache.doris.nereids.trees.expressions.literal.Literal;
+
+import com.google.common.collect.ImmutableList;
+import com.google.common.collect.ImmutableMap;
+import com.google.common.collect.Lists;
+import com.google.common.collect.Sets;
+
+import java.util.ArrayList;
+import java.util.HashMap;
+import java.util.LinkedHashMap;
+import java.util.List;
+import java.util.Map;
+import java.util.Set;
+import java.util.stream.Collectors;
+
+/**
+ * Normalize Plan.
+ */
+public interface NormalizePlan {
+ /**
+ * get AggFunc from outputExpressions
+ */
+ default Set<AggregateFunction> getAggregateFunctions(List<NamedExpression> outputs) {
+ Map<Boolean, List<NamedExpression>> partitionedOutputs = outputs.stream()
+ .collect(Collectors.groupingBy(e -> e.anyMatch(AggregateFunction.class::isInstance)
+ || e.anyMatch(GroupingScalarFunction.class::isInstance)));
+ return partitionedOutputs.containsKey(true)
+ ? partitionedOutputs.get(true).stream()
+ .flatMap(e -> e.<Set<AggregateFunction>>collect(
+ AggregateFunction.class::isInstance).stream())
+ .collect(Collectors.toSet())
+ : Sets.newHashSet();
+ }
Review Comment:
Contains AggregateFunction or GroupingScalarFunction, then collect AggregateFunction?
What is the function of GroupingScalarFunction here？
##########
fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/logical/PushdownFilterThroughAggregation.java:
##########
@@ -85,6 +91,9 @@ public Rule build() {
}
});
+ if (hasRepeat) {
+ return filter;
+ }
Review Comment:
add comment to explain why not pushdown when there is Repeat exists?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org