You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2017/07/24 00:29:00 UTC

[jira] [Resolved] (IMPALA-5676) BufferedTupleStreamV2::CheckConsistency() is too slow for large streams with small pages in Debug build

     [ https://issues.apache.org/jira/browse/IMPALA-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-5676.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0


  -----------
  IMPALA-5676: avoid expensive consistency checks in BTSv2

Doing an O(n) consistency check every time the read or write
page was advanced results in O(n^2) overall runtime.

The fix is to separate the O(1) and O(n) checks and only do the
O(n) checks if:
* The function does an an O(n) pass over the pages anyway (e.g.
  PinStream())
* The function is called only once per read or write pass over the
  stream.

This should make the cost of the checks O(n) (if we make the reasonable
assumption that PrepareForWrite(), PrepareForRead(), PinStream() and
UnpinStream() are called a bounded number of times per stream).

Testing:
Ran BufferedTupleStreamV2Test.

Change-Id: I8b380fcd0568cb73b36a490954bcd316db969ede
Reviewed-on: http://gerrit.cloudera.org:8080/7459
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins

> BufferedTupleStreamV2::CheckConsistency() is too slow for large streams with small pages in Debug build
> -------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-5676
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5676
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 2.10.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>             Fix For: Impala 2.10.0
>
>
> I saw a spilling query (with a bad plan) spending a huge amount of time in this function.
> We should probably split out the O(n) parts of CheckConsistency() into a separate function that is run less frequently, e.g. once per pass over the stream.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)