[jira] [Resolved] (IMPALA-5611) KuduPartitionExpr holds onto memory unnecessarily

["Thomas Tauber-Marshall (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Thomas Tauber-Marshall resolved IMPALA-5611.
--------------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

commit 4e17839033f931f98e0c3ec46d99b250b0bb4660
Author: Thomas Tauber-Marshall <tm...@cloudera.com>
Date:   Fri Jun 30 12:00:08 2017 -0700

    IMPALA-5611: KuduPartitionExpr holds onto memory unnecessarily
    
    IMPALA-3742 introduced KuduPartitionExpr, which takes a row and passes
    it to the Kudu client to determine what partitionit belongs to.
    
    The DataStreamSender never frees the local allocations for the Kudu
    partition exprs causing it to hang on to memory longer than it needs to.
    
    This patch also fixes two other related issues:
    - DataStreamSender was dropping the Status from AddRow in the Kudu
      branch. Adds 'RETURN_IF_ERROR' and 'WARN_UNUSED_RESULT'
    - Changes the HASH case in DataStreamSender to call FreeLocalAllocations
      on a per-batch basis, instead of a per-row basis.
    
    Testing:
    - Added an e2e test that runs a large insert with a mem limit that
      failed with oom previously.
    
    Change-Id: Ia661eb8bed114070728a1497ccf7ed6893237e5e
    Reviewed-on: http://gerrit.cloudera.org:8080/7346
    Reviewed-by: Dan Hecht <dh...@cloudera.com>
    Reviewed-by: Michael Ho <kw...@cloudera.com>
    Tested-by: Impala Public Jenkins

> KuduPartitionExpr holds onto memory unnecessarily
> -------------------------------------------------
>
>                 Key: IMPALA-5611
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5611
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>    Affects Versions: Impala 2.9.0
>            Reporter: Thomas Tauber-Marshall
>            Assignee: Thomas Tauber-Marshall
>            Priority: Critical
>             Fix For: Impala 2.10.0
>
>
> IMPALA-3742 introduced KuduPartitionExpr, which takes a row and passes it to the Kudu client to determine what partition it belongs to.
> KuduPartitionExpr never calls ScalarExprEvaluator::FreeLocalAllocations, causing it to hang on to memory longer than it needs it.
> Since we only need the value of the row for the call into the Kudu client, we can call FreeLocalAllocations after that.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)