You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Amogh Margoor (Jira)" <ji...@apache.org> on 2021/07/28 15:07:00 UTC

[jira] [Commented] (IMPALA-9874) Reduce or avoid I/O for pruned columns

    [ https://issues.apache.org/jira/browse/IMPALA-9874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17388832#comment-17388832 ] 

Amogh Margoor commented on IMPALA-9874:
---------------------------------------

Design for the same: https://docs.google.com/document/d/1QFu_Zu9nHuMpu5Pqb3qe62MbZPA88j_o7NtpZ2a2zSA/edit?usp=sharing.

> Reduce or avoid I/O for pruned columns
> --------------------------------------
>
>                 Key: IMPALA-9874
>                 URL: https://issues.apache.org/jira/browse/IMPALA-9874
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>            Reporter: Tim Armstrong
>            Priority: Major
>              Labels: parquet
>
> Skipping decoding of values may not be effective at reducing I/O in some cases, because we start the I/O in StartScans(). We don't wait for the I/O until we actually read the first data page from the column reader. So there is a race to determine whether the I/O happens in some cases.
> There are a couple of things we can do here.
> * The basic thing is to issue reads for the column readers in the order in which they are needed. We may be able to get this for free by ordering the column readers based on materialisation order.
> * We also want to avoid issuing I/O for columns that are not needed, if predicates are highly selective. This is maybe a bit harder and avoids more trade-offs, since delaying issuing of the reads may impact scan latency.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org