You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@impala.apache.org by "Alexander Behm (JIRA)" <ji...@apache.org> on 2018/02/06 23:12:00 UTC
[jira] [Resolved] (IMPALA-5037) Change default Parquet array
resolution according to Parquet standard.
[ https://issues.apache.org/jira/browse/IMPALA-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Alexander Behm resolved IMPALA-5037.
------------------------------------
Resolution: Fixed
Fix Version/s: Impala 3.0
commit dedff5e25c19da724ae032a52354cd4fd61ec40a
Author: Alex Behm <al...@cloudera.com>
Date: Thu Feb 1 16:57:53 2018 -0800
IMPALA-5037: Default PARQUET_ARRAY_RESOLUTION=THREE_LEVEL
Changes the default value for the PARQUET_ARRAY_RESOLUTION
query option to conform to the Parquet standard.
Before: TWO_LEVEL_THEN_THREE_LEVEL
After: THREE_LEVEL
For more information see:
* IMPALA-4725
* https://github.com/apache/parquet-format/blob/master/LogicalTypes.md
Testing:
- expands and cleans up the existing tests for more coverage
over the different resolution policies
- private core/hdfs run passed
Cherry-picks: not for 2.x.
Change-Id: Ib8f7e9010c4354d667305d9df7b78862efb23fe1
Reviewed-on: http://gerrit.cloudera.org:8080/9210
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
> Change default Parquet array resolution according to Parquet standard.
> ----------------------------------------------------------------------
>
> Key: IMPALA-5037
> URL: https://issues.apache.org/jira/browse/IMPALA-5037
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.8.0, Impala 2.9.0
> Reporter: Alexander Behm
> Assignee: Alexander Behm
> Priority: Major
> Labels: include-in-v3, incompatibility
> Fix For: Impala 3.0
>
>
> With IMPALA-4725 we've introduced query options to control the field resolution behavior when scanning Parquet files with nested arrays. The current default strategy currently tries to auto-detect the array encoding within Parquet files, but this strategy can sometimes subtly go wrong and return incorrect results due to the inherent ambiguity of the 2/3-level encoding schemes in Parquet.
> We should switch the default resolution strategy according to the Parquet standard 3-level encoding, instead of the current auto-detect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)