[jira] [Commented] (IMPALA-12818) Implement the initial framework for caching tuples

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-12818?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827244#comment-17827244 ] 

ASF subversion and git services commented on IMPALA-12818:
----------------------------------------------------------

Commit 4477398ae46415d3fb32db2a8fd5e6d2060cbd3f in impala's branch refs/heads/master from Kurt Deschler
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=4477398ae ]

IMPALA-12818: Intermediate Result Caching plan node framework

This patch adds a plan node framework for caching of intermediate result
tuples within a query. Actual caching of data will be implemented in
subsequent patches.

A new plan node type TupleCacheNode is introduced for brokering caching
decisions at runtime. If the result is in the cache, the TupleCacheNode will
return results from the cache and skip executing its child node. If the
result is not cached, the TupleCacheNode will execute its child node and
mirror the resulting RowBatches to the cache.

The TupleCachePlanner decides where to place the TupleCacheNodes. To
calculate eligibility and cache keys, the plan must be in a stable state
that will not change shape. TupleCachePlanner currently runs at the end
of planning after the DistributedPlanner and ParallelPlanner have run.
As a first cut, TupleCachePlanner places TupleCacheNodes at every
eligible location. Eligibility is currently restricted to immediately
above HdfsScanNodes. This implementation will need to incorporate cost
heuristics and other policies for placement.

Each TupleCacheNode has a hash key that is generated from the logical
plan below for the purpose of identifying results that have been cached
by semantically equivalent query subtrees. The initial implementation of
the subtree hash uses the plan Thrift to uniquely identify the subtree.

Tuple caching is enabled by setting the enable_tuple_cache query option
to true. As a safeguard during development, enable_tuple_cache can only
be set to true if the "allow_tuple_caching" startup option is set to
true. It defaults to false to minimize the impact for production clusters.
bin/start-impala-cluster.py sets allow_tuple_caching=true by default
to enable it in the development environment.

Testing:
 - This adds a frontend test that does basic checks for cache keys and
   eligibility
 - This verifies the presence of the caching information in the explain
   plan output.

Change-Id: Ia1f36a87dcce6efd5d1e1f0bc04009bf009b1961
Reviewed-on: http://gerrit.cloudera.org:8080/21035
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Michael Smith <mi...@cloudera.com>
Reviewed-by: Yida Wu <wy...@gmail.com>
Reviewed-by: Kurt Deschler <kd...@cloudera.com>


> Implement the initial framework for caching tuples
> --------------------------------------------------
>
>                 Key: IMPALA-12818
>                 URL: https://issues.apache.org/jira/browse/IMPALA-12818
>             Project: IMPALA
>          Issue Type: Task
>          Components: Backend, Frontend
>    Affects Versions: Impala 4.3.0
>            Reporter: Joe McDonnell
>            Priority: Major
>
> This tracks introducing the basic concepts of tuple caching. Specifically, it should introduce the TupleCacheNode, which is an ExecNode that can be placed in a plan to cache tuples between existing nodes in the plan. The TupleCachePlanner should place these nodes in locations that are eligible. The TupleCachePlanner is responsible for eligibility as well as producing a cache key.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org