Change in asterixdb[master]: Supports flexible degree of parallelism.

["Yingyi Bu (Code Review)" <do...@asterixdb.incubator.apache.org>]

Hello Jenkins,

I'd like you to reexamine a change.  Please visit

    https://asterix-gerrit.ics.uci.edu/1398

to look at the new patch set (#5).

Change subject: Supports flexible degree of parallelism.
......................................................................

Supports flexible degree of parallelism.

Detailed list of changes include:
- Add the IClusterInfoCollector interface and feed that to the compiler;
- Add partition calculation in APIFramework;
- Fix partitioning property requirements in binary operators such as Join and UnionAll;
- Fix PushSubplanIntoGroupByRule for type propagation;
- Fix InlineLeftNtsInSubplanJoinFlatteningVisitor for the returned top join operator;
- Add unit test for partition calculation in APIFrameworkTest
- Add integration test SqlppExecutionFullParallelismIT, SqlppExecutionLessParallelismIT,
  AqlExecutionFullParallelismIT, and AqlExecutionLessParallelismIT for different
  parallelism settings;
- Refactor AQLExecutionTest and SqlppExecutionTest;
- Separate one cluster state test out from AqlExecutionTest.

Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7
---
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/optimizer/rules/subplan/InlineLeftNtsInSubplanJoinFlatteningVisitor.java
M asterixdb/asterix-algebra/src/main/java/org/apache/asterix/translator/IStatementExecutor.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/api/common/APIFramework.java
M asterixdb/asterix-app/src/main/java/org/apache/asterix/app/translator/QueryTranslator.java
M asterixdb/asterix-app/src/main/resources/asterix-build-configuration.xml
A asterixdb/asterix-app/src/main/resources/asterix-build-configuration2.xml
A asterixdb/asterix-app/src/main/resources/asterix-build-configuration3.xml
A asterixdb/asterix-app/src/test/java/org/apache/asterix/api/common/APIFrameworkTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/ConnectorAPIServletTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/api/http/servlet/VersionAPIServletTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionFullParallelismIT.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionLessParallelismIT.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/AqlExecutionTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionFullParallelismTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionLessParallelismTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ClusterStateExecutionTest.java
D asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ExecutionTest.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/ExecutionTestUtil.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/LangExecutionUtil.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/RepeatedTest.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionFullParallelismIT.java
A asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionLessParallelismIT.java
M asterixdb/asterix-app/src/test/java/org/apache/asterix/test/runtime/SqlppExecutionTest.java
M asterixdb/asterix-app/src/test/resources/optimizerts/results/nonpure/query-ASTERIXDB-1608.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/open-index-enforced/inverted-index-join/word-jaccard-check-after-btree-access.plan
M asterixdb/asterix-app/src/test/resources/optimizerts/results/query-issue827-2.plan
A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state.xml
A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state_full.xml
A asterixdb/asterix-app/src/test/resources/runtimets/cluster_state_less.xml
A asterixdb/asterix-app/src/test/resources/runtimets/only_cluster_state.xml
M asterixdb/asterix-app/src/test/resources/runtimets/queries/api/APIQueries.xml
M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.4.asyncdefer.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at00/at00.5.async.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/flwor/at06/at06.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/open-closed/query-issue456/query-issue456.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue601/query-issue601.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue785-2/query-issue785-2.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch-sql-like/query-issue785/query-issue785.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue601/query-issue601.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue785-2/query-issue785-2.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries/tpch/query-issue785/query-issue785.3.query.aql
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.4.asyncdefer.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at00/at00.5.async.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/flwor/at06/at06.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/open-closed/query-issue456/query-issue456.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tinysocial/tinysocial-suite-open/group-by.26.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue601/query-issue601.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue785-2/query-issue785-2.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-like/query-issue785/query-issue785.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue601/query-issue601.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue785-2/query-issue785-2.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql-sugar/query-issue785/query-issue785.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue601/query-issue601.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue785-2/query-issue785-2.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-sql/query-issue785/query-issue785.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue601/query-issue601.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue785-2/query-issue785-2.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch-with-index/query-issue785/query-issue785.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue601/query-issue601.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue785-2/query-issue785-2.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/tpch/query-issue785/query-issue785.3.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/queries_sqlpp/types/any-object/any-object.2.query.sqlpp
M asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1/cluster_state_1.1.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1_full/cluster_state_1_full.1.adm
A asterixdb/asterix-app/src/test/resources/runtimets/results/api/cluster_state_1_less/cluster_state_1_less.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.2.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at00/at00.3.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/flwor/at06/at06.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tinysocial/tinysocial-suite-open/group-by.26.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue601/query-issue601.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue785-2/query-issue785-2.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-like/query-issue785/query-issue785.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue601/query-issue601.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue785-2/query-issue785-2.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql-sugar/query-issue785/query-issue785.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue601/query-issue601.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue785-2/query-issue785-2.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-sql/query-issue785/query-issue785.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue601/query-issue601.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue785-2/query-issue785-2.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch-with-index/query-issue785/query-issue785.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue601/query-issue601.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue785-2/query-issue785-2.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results/tpch/query-issue785/query-issue785.1.adm
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.4.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at00/at00.5.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/flwor/at06/at06.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/open-closed/query-issue456/query-issue456.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tinysocial/tinysocial-suite-open/tinysocial-suite.26.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue601/query-issue601.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue785-2/query-issue785-2.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch-sql-like/query-issue785/query-issue785.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue601/query-issue601.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue785-2/query-issue785-2.3.ast
M asterixdb/asterix-app/src/test/resources/runtimets/results_parser_sqlpp/tpch/query-issue785/query-issue785.3.ast
M asterixdb/asterix-common/src/main/java/org/apache/asterix/common/config/CompilerProperties.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/AbstractHashJoinPOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/NestedLoopJoinPOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/operators/physical/UnionAllPOperator.java
M hyracks-fullstack/algebricks/algebricks-core/src/main/java/org/apache/hyracks/algebricks/core/algebra/properties/StructuralPropertiesVector.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/EnforceStructuralPropertiesRule.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/ExtractFunctionsFromJoinConditionRule.java
M hyracks-fullstack/algebricks/algebricks-rewriter/src/main/java/org/apache/hyracks/algebricks/rewriter/rules/subplan/PushSubplanIntoGroupByRule.java
M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/HyracksConnection.java
A hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/IClusterInfoCollector.java
M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/IHyracksClientConnection.java
M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/client/NodeControllerInfo.java
M hyracks-fullstack/hyracks/hyracks-api/src/main/java/org/apache/hyracks/api/rewriter/runtime/SuperActivity.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/NodeControllerState.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-cc/src/main/java/org/apache/hyracks/control/cc/work/GetNodeControllersInfoWork.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-common/src/main/java/org/apache/hyracks/control/common/heartbeat/HeartbeatData.java
M hyracks-fullstack/hyracks/hyracks-control/hyracks-control-nc/src/main/java/org/apache/hyracks/control/nc/NodeControllerService.java
M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/union/UnionAllOperatorDescriptor.java
M hyracks-fullstack/hyracks/hyracks-hdfs/hyracks-hdfs-core/src/test/java/org/apache/hyracks/hdfs/scheduler/SchedulerTest.java
M hyracks-fullstack/hyracks/hyracks-hdfs/hyracks-hdfs-core/src/test/java/org/apache/hyracks/hdfs/utils/TestUtils.java
119 files changed, 3,764 insertions(+), 2,313 deletions(-)


  git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/98/1398/5
-- 
To view, visit https://asterix-gerrit.ics.uci.edu/1398
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: If70faf52bed995c7098cc343f2fabd1b6c8d96e7
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Yingyi Bu <bu...@gmail.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>