[Impala-ASF-CR] IMPALA-3481: Use Kudu ScanToken API for scan ranges

["Matthew Jacobs (Code Review)" <ge...@cloudera.org>]

Matthew Jacobs has uploaded a new patch set (#4).

Change subject: IMPALA-3481: Use Kudu ScanToken API for scan ranges
......................................................................

IMPALA-3481: Use Kudu ScanToken API for scan ranges

Switches the planner and KuduScanNode to use Kudu's new
ScanToken API instead of explicitly constructing scan ranges
for all tablets of a table, regardless of whether they were
needed. The ScanToken API allows Impala to specify the
projected columns and predicates during planning, and Kudu
returns a set of 'scan tokens' that represent a scanner for
each tablet that needs to be scanned. The scan tokens can
be serialized and distributed to the scan nodes, which can
then deserialize them into Kudu scanner objects. Upon
deserialization, the scan token has all scan parameters
already, including the 'pushed down' predicates. Impala no
longer needs to send the Kudu predicates to the BE and
convert them at the scan node.

This change also fixes:
1) IMPALA-4016: Avoid materializing slots only referenced
                by Kudu conjuncts
2) IMPALA-3874: Predicates are not always pushed to Kudu

TODO: Consider additional planning improvements.

Testing: Updated the existing tests, verified everything
works as expected. Some BE tests no longer make sense and
they were removed.

TODO: When KUDU-1065 is resolved, add tests that demonstrate pruning.

Change-Id: I160e5849d372755748ff5ba3c90a4651c804b220
---
M be/src/exec/kudu-scan-node-test.cc
M be/src/exec/kudu-scan-node.cc
M be/src/exec/kudu-scan-node.h
M be/src/exec/kudu-scanner.cc
M be/src/exec/kudu-scanner.h
M be/src/exec/kudu-util.cc
M be/src/exec/kudu-util.h
M be/src/scheduling/simple-scheduler.cc
M bin/impala-config.sh
M common/thrift/PlanNodes.thrift
M fe/pom.xml
M fe/src/main/java/com/cloudera/impala/analysis/Expr.java
M fe/src/main/java/com/cloudera/impala/analysis/LiteralExpr.java
M fe/src/main/java/com/cloudera/impala/catalog/KuduTable.java
M fe/src/main/java/com/cloudera/impala/catalog/delegates/KuduDdlDelegate.java
M fe/src/main/java/com/cloudera/impala/planner/KuduScanNode.java
M fe/src/main/java/com/cloudera/impala/util/KuduUtil.java
M fe/src/test/java/com/cloudera/impala/planner/KuduPlannerTest.java
M fe/src/test/java/com/cloudera/impala/planner/PlannerTestBase.java
M fe/src/test/java/com/cloudera/impala/util/KuduUtilTest.java
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-selectivity.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu-update.test
M testdata/workloads/functional-planner/queries/PlannerTest/kudu.test
M testdata/workloads/functional-planner/queries/PlannerTest/tpch-kudu.test
24 files changed, 521 insertions(+), 1,137 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/20/4120/4
-- 
To view, visit http://gerrit.cloudera.org:8080/4120
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I160e5849d372755748ff5ba3c90a4651c804b220
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Dan Burkert <da...@cloudera.com>
Gerrit-Reviewer: Dimitris Tsirogiannis <dt...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Todd Lipcon <to...@apache.org>