[jira] [Commented] (IMPALA-5843) Use page index in Parquet files to skip pages

["ASF subversion and git services (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-5843?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16655907#comment-16655907 ] 

ASF subversion and git services commented on IMPALA-5843:
---------------------------------------------------------

Commit 48fb4902d4f28c9e4d327b80b00b33962c118c22 in impala's branch refs/heads/master from Zoltan Borok-Nagy
[ https://git-wip-us.apache.org/repos/asf?p=impala.git;h=48fb490 ]

IMPALA-7543: Enhance scan ranges to support sub-ranges

This commit enhances the ScanRange class to make it possible to only
read some smaller parts of the whole ScanRange. This functionality
is needed by IMPALA-5843.

A sub-range is an offset and length which is located within the scan
range. Sub-ranges can be added to a scan range when calling
ScanRange::Reset(). If done so, the ScanRange class will only read the
parts defined by the sub-ranges.

If we have sub-ranges for a cache read then the ScanRange won't
enqueue the whole cache buffer (which contains the whole ScanRange),
but memcpy() the sub-ranges to IO/client buffers.

Smaller refactorings needed to do:
 * remove scan_range_offset_ from BufferDescriptor
 * number of bytes read are bookkeeped by ScanRange again

Testing:
 * introduced CacheReaderTestStub to fake cache reads during testing
 * extended disk-io-mgr-test.cc with sub-ranges

Change-Id: Iea26ba386713990f7671aab5a372cf449b8d51e4
Reviewed-on: http://gerrit.cloudera.org:8080/11520
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>


> Use page index in Parquet files to skip pages
> ---------------------------------------------
>
>                 Key: IMPALA-5843
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5843
>             Project: IMPALA
>          Issue Type: New Feature
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Lars Volker
>            Assignee: Zoltán Borók-Nagy
>            Priority: Critical
>              Labels: parquet, performance
>
> Once IMPALA-5842 has been resolved, we should skip pages based on the page index in Parquet files.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org