[jira] [Commented] (PARQUET-2221) [Format] Encoding spec incorrect for dictionary fallback

["Antoine Pitrou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-2221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17654025#comment-17654025 ] 

Antoine Pitrou commented on PARQUET-2221:
-----------------------------------------

cc [~julienledem] [~pnarang] [~rdblue] [~alexlevenson]

> [Format] Encoding spec incorrect for dictionary fallback
> --------------------------------------------------------
>
>                 Key: PARQUET-2221
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2221
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-format
>            Reporter: Antoine Pitrou
>            Priority: Critical
>             Fix For: format-2.10.0
>
>
> The spec for DICTIONARY_ENCODING states that:
> bq. If the dictionary grows too big, whether in size or number of distinct values, the encoding will fall back to the plain encoding. 
> https://github.com/apache/parquet-format/blob/master/Encodings.md#dictionary-encoding-plain_dictionary--2-and-rle_dictionary--8
> However, the parquet-mr implementation was deliberately changed to a different fallback mechanism in https://issues.apache.org/jira/browse/PARQUET-52
> I'm assuming the parquet-mr implementation is authoritative here. But then the spec is incorrect and should be fixed to reflect expected behavior.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)