[Impala-ASF-CR] IMPALA-8834: Short-circuit partition key scan

["Tim Armstrong (Code Review)" <ge...@cloudera.org>]

Tim Armstrong has uploaded a new patch set (#11). ( http://gerrit.cloudera.org:8080/13993 )

Change subject: IMPALA-8834: Short-circuit partition key scan
......................................................................

IMPALA-8834: Short-circuit partition key scan

This generalises the OPTIMIZE_PARTITION_KEY_SCANS
to return correct results when files have zero rows.
It can therefore be safely enabled by default.

Instead of evaluating the query using metadata, the
planner instructs the backend to short-circuit HDFS
scans after a single row has been returned from each
file. This gives results equivalent to returning all
the rows from each file.

Planner cardinality estimates are adjusted accordingly
to enable potentially better plans and other optimisations
like disabling codegen.

We make some effort to avoid generated extra scan ranges
for remote scans by only generating one range per remote
file.

The backend optimisation is implemented by constructing a
row batch with capacity for a single row only and then
terminating each scan range once a single row has been
produced.  Both Parquet and ORC have optimized code paths
for zero slot table scans that mean this will only result
in a footer read. (Other file formats still need to read
some portion of the file, but can terminate early once
one row has been produced.)

This should be quite efficient in practice with file handle
caching and data caching enabled, because it then only
requires reading the footer from the cache for each file.

The partition key scan optimization is also slightly
generalised to apply to scans of unpartitioned tables
where no slots are materialized.

A limitation of the optimization where it did not apply
to multiple grouping classes was also fixed.

Limitations:
* This still scans every file in the partition. I.e. there is
  no short-circuiting if a row has already been found in the
  partition by the current scan node.
* Resource reservations and estimates for the scan node do
  not all take into account this optimisation, so are
  conservative - they assume the whole file is scanned.

Testing:
* Added end-to-end tests that execute the query on all
  HDFS file formats and verify that the correct number of rows
  flow through the plan.
* Added planner test based on the existing test partition key
  scan test.
* Added test to make sure single node optimisation kicks in
  when expected.
* Add test for cardinality estimates with and without stats
* Added test for unpartitioned tables.
* Added planner test that checks that optimisation is enabled
  for multiple aggregation classes.
* Added a targeted perf test.

Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c
---
M be/src/exec/hdfs-orc-scanner.cc
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scan-node-mt.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/runtime/row-batch.cc
M be/src/runtime/row-batch.h
M common/thrift/PlanNodes.thrift
M fe/src/main/java/org/apache/impala/analysis/MultiAggregateInfo.java
M fe/src/main/java/org/apache/impala/analysis/TupleDescriptor.java
M fe/src/main/java/org/apache/impala/planner/HdfsScanNode.java
M fe/src/main/java/org/apache/impala/planner/SingleNodePlanner.java
M fe/src/test/java/org/apache/impala/planner/CardinalityTest.java
M fe/src/test/java/org/apache/impala/planner/ExplainTest.java
M fe/src/test/java/org/apache/impala/planner/PlannerTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/disable-codegen.test
M testdata/workloads/functional-planner/queries/PlannerTest/distinct.test
M testdata/workloads/functional-planner/queries/PlannerTest/multiple-distinct-materialization.test
M testdata/workloads/functional-planner/queries/PlannerTest/parquet-stats-agg.test
A testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans-default.test
M testdata/workloads/functional-planner/queries/PlannerTest/partition-key-scans.test
M testdata/workloads/functional-planner/queries/PlannerTest/runtime-filter-propagation.test
M testdata/workloads/functional-planner/queries/PlannerTest/shuffle-by-distinct-exprs.test
M testdata/workloads/functional-planner/queries/PlannerTest/small-query-opt.test
A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans-with-joins.test
A testdata/workloads/functional-query/queries/QueryTest/partition-key-scans.test
M testdata/workloads/targeted-perf/queries/aggregation.test
M tests/query_test/test_nested_types.py
M tests/query_test/test_queries.py
30 files changed, 1,308 insertions(+), 370 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/93/13993/11
-- 
To view, visit http://gerrit.cloudera.org:8080/13993
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I26c87525a4f75ffeb654267b89948653b2e1ff8c
Gerrit-Change-Number: 13993
Gerrit-PatchSet: 11
Gerrit-Owner: Tim Armstrong <ta...@cloudera.com>