Change in asterixdb[master]: [ASTERIXDB-2483][COMP][FUN] Eliminate listify for distinct a...

["Dmitry Lychagin (Code Review)" <do...@asterixdb.incubator.apache.org>]

Hello Anon. E. Moose #1000171, Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/3293

to look at the new patch set (#2).

Change subject: [ASTERIXDB-2483][COMP][FUN] Eliminate listify for distinct aggregates
......................................................................

[ASTERIXDB-2483][COMP][FUN] Eliminate listify for distinct aggregates

- user model changes: no
- storage format changes: no
- interface changes: no

Details:
- Move distinct aggregate rewriting from SqlppQueryRewriter
  to RewriteDistinctAggregateRule in the optimizer
- Add runtime for scalar distinct aggregates
- Distinct operator should require input to be unpartitioned
  when it is executed in unpartitioned mode
- Fix ExtractCommonOperatorsRule handling of binary operators
- Additional tests for distinct aggregates

Change-Id: If13ea2696e9e0a8a639db684656e5642991c1f99
---
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/base/RuleCollections.java
A asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/RewriteDistinctAggregateRule.java
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.10.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.11.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.12.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.13.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.14.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.15.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.16.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.3.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.4.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.5.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.6.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.7.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.8.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/queries/aggregate-sql-sugar/distinct_mixed/distinct_mixed.9.sqlpp
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.10.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.11.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.12.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.13.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.14.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.15.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.16.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.3.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.4.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.5.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.6.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.7.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.8.plan
A asterixdb/asterix-app/src/test/resources/optimizerts/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.9.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/group-by/sugar-06-distinct.plan
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.1.ddl.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.10.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.11.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.12.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.13.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.14.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.15.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.16.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.2.update.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.3.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.4.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.5.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.6.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.7.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.8.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/aggregate-sql-sugar/distinct_mixed/distinct_mixed.9.query.sqlpp
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.10.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.11.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.12.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.13.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.14.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.15.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.16.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.3.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.4.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.5.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.6.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.7.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.8.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql-sugar/distinct_mixed/distinct_mixed.9.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/aggregate-sql/skewness_distinct/skewness_distinct.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/testsuite_sqlpp.xml
M asterixdb/asterix-geo/src/main/java/org/apache/asterix/geo/aggregates/ScalarSTUnionAggregateDescriptor.java
A asterixdb/asterix-geo/src/main/java/org/apache/asterix/geo/aggregates/ScalarSTUnionDistinctAggregateDescriptor.java
M asterixdb/asterix-geo/src/main/java/org/apache/asterix/geo/evaluators/GeoFunctionRegistrant.java
M asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/SqlppQueryRewriter.java
D asterixdb/asterix-lang-sqlpp/src/main/java/org/apache/asterix/lang/sqlpp/rewrites/visitor/SqlppDistinctAggregationSugarVisitor.java
M asterixdb/asterix-om/src/main/java/org/apache/asterix/om/functions/BuiltinFunctions.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/AbstractScalarAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/AbstractScalarDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/GenericScalarAggregateFunction.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/GenericScalarDistinctAggregateFunction.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarAvgAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarAvgDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarCountAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarCountDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarKurtosisAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarKurtosisDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarMaxAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarMaxDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarMinAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarMinDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSkewnessAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSkewnessDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlAvgAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlAvgDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlCountAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlCountDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlKurtosisAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlKurtosisDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlMaxAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlMaxDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlMinAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlMinDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlSkewnessAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlSkewnessDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlStddevAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlStddevDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlStddevPopAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlStddevPopDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlSumAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlSumDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlVarAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlVarDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlVarPopAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSqlVarPopDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarStddevAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarStddevDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarStddevPopAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarStddevPopDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSumAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarSumDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarVarAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarVarDistinctAggregateDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarVarPopAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/scalar/ScalarVarPopDistinctAggregateDescriptor.java
A asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/aggregates/utils/PointableHashSet.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/ArrayDistinctDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/ArrayIntersectDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/ArraySymDiffEval.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/ArrayUnionDescriptor.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/evaluators/functions/PointableHelper.java
M asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/functions/FunctionCollection.java
D asterixdb/asterix-runtime/src/main/java/org/apache/asterix/runtime/utils/ArrayFunctionsUtil.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/AbstractPreSortedDistinctByPOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/util/OperatorManipulationUtil.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/ExtractCommonOperatorsRule.java
128 files changed, 3,969 insertions(+), 526 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/93/3293/2
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/3293
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If13ea2696e9e0a8a639db684656e5642991c1f99
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Dmitry Lychagin <dm...@couchbase.com>
Gerrit-Reviewer: Anon. E. Moose #1000171
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>