[jira] [Commented] (IMPALA-7294) TABLESAMPLE clause allocates arrays based on total file count instead of selected partitions

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-7294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16851392#comment-16851392 ] 

ASF subversion and git services commented on IMPALA-7294:
---------------------------------------------------------

Commit 9f372f0d5df734c682c4c4cb237fc17e3e3f7bb0 in impala's branch refs/heads/2.x from Todd Lipcon
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=9f372f0 ]

IMPALA-7294. TABLESAMPLE should not allocate array based on total table file count

This changes HdfsTable.getFilesSample() to allocate its intermediate
sampling array based on the number of files in the selected
(post-pruning) partitions, rather than the total number of files in the
table. While the former behavior was correct (the total file count is of
course an upper bound on the pruned file count), it was an unnecessarily
large allocation, which has some downsides around garbage collection.

In addition, this is important for the LocalCatalog implementation of
table sampling, since we do not want to have to load all partition file
lists in order to compute a sample over a pruned subset of partitions.

The original code indicated that this was an optimization to avoid
looping over the partition list an extra time. However, typical
partition lists are relatively small even in the worst case (order of
100k) and looping over 100k in-memory Java objects is not likely to be
the bottleneck in planning any query. This is especially true
considering that we loop over that same list later in the function
anyway, so we probably aren't saving page faults or LLC cache misses
either.

In testing this change I noticed that the existing test for TABLESAMPLE
didn't test TABLESAMPLE when applied in conjunction with a predicate.
I added a new dimension to the test which employs a predicate which
prunes some partitions to ensure that the code works in that case.
I also added coverage of the "100%" sampling parameter as a sanity check
that it returns the same results as a non-sampled query.

Change-Id: I0248d89bcd9dd4ff8b4b85fef282c19e3fe9bdd5
Reviewed-on: http://gerrit.cloudera.org:8080/10936
Reviewed-by: Philip Zeyliger <ph...@cloudera.com>
Reviewed-by: Vuk Ercegovac <ve...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> TABLESAMPLE clause allocates arrays based on total file count instead of selected partitions
> --------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-7294
>                 URL: https://issues.apache.org/jira/browse/IMPALA-7294
>             Project: IMPALA
>          Issue Type: Bug
>    Affects Versions: Impala 3.0
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Minor
>             Fix For: Impala 3.1.0
>
>
> The HdfsTable.getFilesSample function takes a list of input partitions to sample files from, but then, when allocating an array to sample into, sizes that array based on the total file count across all partitions. This is an unnecessarily large array, which is expensive to allocate (may cause full GC when the heap is fragmented). The code claims this to be an optimization:
> {code}
>     // Use max size to avoid looping over inputParts for the exact size.
> {code}
> ...but I think the loop over inputParts is likely to be trivial here since we'll loop over them anyway later in the function and thus will already be pulled into CPU cache, etc. This is also necessary for fine-grained metadata loading in the impalad -- for a large table with many partitions, we don't want to load the file lists of all partitions just to tablesample from one partition.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org