You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Kurt Deschler (Jira)" <ji...@apache.org> on 2021/03/12 14:48:00 UTC
[jira] [Commented] (IMPALA-10466) Handle deprecated TWO_LEVEL
Parquet arrays more gracefully
[ https://issues.apache.org/jira/browse/IMPALA-10466?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17300360#comment-17300360 ]
Kurt Deschler commented on IMPALA-10466:
----------------------------------------
Per the example in IMPALA-4725, it is not safe to auto-detect the format since there are cases that will successfully decode in both 2-level and 3-level. We should however error out instead of returning nulls (item b. the description). That should probably be the default behavior then have an option that turns off the null check so that data can be recovered if the formats get mixed.
> Handle deprecated TWO_LEVEL Parquet arrays more gracefully
> ----------------------------------------------------------
>
> Key: IMPALA-10466
> URL: https://issues.apache.org/jira/browse/IMPALA-10466
> Project: IMPALA
> Issue Type: Improvement
> Reporter: Csaba Ringhofer
> Priority: Minor
>
> The default of PARQUET_ARRAY_RESOLUTION was changed from TWO_LEVEL_THEN_THREE_LEVEL to THREE_LEVEL in IMPALA-4725. This solved incorrectly detecting some ambiguous cases, but now old TWO_LEVEL Parquet lists are not read correctly by default, replacing values with NULL without any error message.
> I would prefer a solution that:
> a. detects the correct resolution when possible
> b. returns a clear warning/error when resolution is not possible or ambiguous, and points the user toward the query option that needs to be set manually
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org