You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Alexander Behm (JIRA)" <ji...@apache.org> on 2018/02/06 23:12:00 UTC

[jira] [Resolved] (IMPALA-5037) Change default Parquet array resolution according to Parquet standard.

     [ https://issues.apache.org/jira/browse/IMPALA-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Alexander Behm resolved IMPALA-5037.
------------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 3.0

commit dedff5e25c19da724ae032a52354cd4fd61ec40a
Author: Alex Behm <al...@cloudera.com>
Date:   Thu Feb 1 16:57:53 2018 -0800

    IMPALA-5037: Default PARQUET_ARRAY_RESOLUTION=THREE_LEVEL
    
    Changes the default value for the PARQUET_ARRAY_RESOLUTION
    query option to conform to the Parquet standard.
    Before: TWO_LEVEL_THEN_THREE_LEVEL
    After:  THREE_LEVEL
    
    For more information see:
    * IMPALA-4725
    * https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
    
    Testing:
    - expands and cleans up the existing tests for more coverage
      over the different resolution policies
    - private core/hdfs run passed
    
    Cherry-picks: not for 2.x.
    
    Change-Id: Ib8f7e9010c4354d667305d9df7b78862efb23fe1
    Reviewed-on: http://gerrit.cloudera.org:8080/9210
    Reviewed-by: Alex Behm <al...@cloudera.com>
    Tested-by: Impala Public Jenkins


> Change default Parquet array resolution according to Parquet standard.
> ----------------------------------------------------------------------
>
>                 Key: IMPALA-5037
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5037
>             Project: IMPALA
>          Issue Type: Improvement
>          Components: Backend
>    Affects Versions: Impala 2.8.0, Impala 2.9.0
>            Reporter: Alexander Behm
>            Assignee: Alexander Behm
>            Priority: Major
>              Labels: include-in-v3, incompatibility
>             Fix For: Impala 3.0
>
>
> With IMPALA-4725 we've introduced query options to control the field resolution behavior when scanning Parquet files with nested arrays. The current default strategy currently tries to auto-detect the array encoding within Parquet files, but this strategy can sometimes subtly go wrong and return incorrect results due to the inherent ambiguity of the 2/3-level encoding schemes in Parquet.
> We should switch the default resolution strategy according to the Parquet standard 3-level encoding, instead of the current auto-detect.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)