[jira] [Assigned] (PARQUET-357) Parquet-thrift generates wrong schema for Thrift binary fields

["Nandor Kollar (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/PARQUET-357?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nandor Kollar reassigned PARQUET-357:
-------------------------------------

    Assignee: Nandor Kollar

> Parquet-thrift generates wrong schema for Thrift binary fields
> --------------------------------------------------------------
>
>                 Key: PARQUET-357
>                 URL: https://issues.apache.org/jira/browse/PARQUET-357
>             Project: Parquet
>          Issue Type: Bug
>          Components: parquet-mr
>    Affects Versions: 1.5.0, 1.6.0, 1.7.0, 1.8.0
>            Reporter: Cheng Lian
>            Assignee: Nandor Kollar
>
> Thrift [doesn't have true {{BINARY}} type|https://github.com/apache/thrift/blob/0.9.0/lib/java/src/org/apache/thrift/protocol/TType.java#L26-L39]. The {{BINARY}} type is actually just an unencoded {{STRING}}. Quoted from [Thrift Types|https://thrift.apache.org/docs/types] section of official Thrift documentation:
> {quote}
> binary: a sequence of unencoded bytes
> N.B.: This is currently a specialized form of the string type above, added to provide better interoperability with Java. The current plan-of-record is to elevate this to a base type at some point.
> {quote}
> The consequence is that, Thrift {{BINARY}} and {{STRING}} are both passed to {{parquet-thrift}} as {{STRING}}, and are [always encoded as {{BINARY (UTF8)}}|https://github.com/apache/parquet-mr/blob/apache-parquet-1.7.0/parquet-thrift/src/main/java/org/apache/parquet/thrift/ThriftSchemaConvertVisitor.java#L249-L252].
> This is really a problem on Thrift side. One possible workaround is to inspect binary fields in the actual generated Java classes to see whether the type is {{ByteBuffer}}.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)