Re: Thrift binary type in Parquet

[Cheng Lian <li...@gmail.com>]

Filed a JIRA ticket for this issue 
https://issues.apache.org/jira/browse/PARQUET-357

Cheng

On 7/15/15 1:10 AM, Ryan Blue wrote:
> This sounds like something we should fix. It may work for Thrift and 
> Scrooge because they have an external schema, but you're right that 
> this will cause buggy behavior for other object models.
>
> Alex and Tianshuo, any ideas about how to address this? It looks like 
> we need to update the ThriftSchemaConverter when converting the Thrift 
> object to Parquet's representation of its schema. That should detect 
> that the field is a binary (through reflection?) even though the 
> underlying Thrift metadata doesn't encode it.
>
> rb
>
> On 07/07/2015 04:00 PM, Cheng Lian wrote:
>> You may see that parquet-mr 1.7.0 can only handle Thrift STRING, and
>> always add UTF8 annotation:
>> https://github.com/apache/parquet-mr/blob/apache-parquet-1.7.0/parquet-thrift/src/main/java/org/apache/parquet/thrift/ThriftSchemaConvertVisitor.java#L249-L252 
>>
>>
>>
>> Because there’s just no |ThriftType.BinaryType|.
>>
>> On 7/7/15 3:56 PM, Cheng Lian wrote:
>>
>>> On 7/7/15 3:48 PM, Ryan Blue wrote:
>>>
>>>> On 07/07/2015 03:23 PM, Cheng Lian wrote:
>>>>> On 7/7/15 1:28 PM, Ashish Singh wrote:
>>>>>> I think you mean that we can’t treat Thrift BINARY type as UTF-8
>>>>>> string,
>>>>>> right?
>>>>> Yeah, it's possible that a Thrift BINARY contains illegal UTF-8 byte
>>>>> sequence(s) and I suppose this may cause problem. Trying to verify
>>>>> this.
>>>>
>>>> Isn't this the right behavior? As long as it isn't annotated as a
>>>> UTF8, then storing it as binary should be fine.
>>>
>>> Ah, it’s actually annotated as UTF8…
>>>
>>> Internally Thrift just maps BINARY to STRING and doesn’t have any
>>> annotation indicating that this field is a BINARY, so Parquet just
>>> assume it’s a normal UTF8 string and writes “BINARY (UTF8)”.
>>>
>>> Here are my testing Thrift schema and the Parquet schema extracted
>>> from the written Parquet file by |parquet-schema|:
>>>
>>> |struct ParquetThriftCompat { 1: binary binaryColumn; 2: string
>>> stringColumn; } message ParquetSchema { optional binary binaryColumn
>>> (UTF8); optional binary stringColumn (UTF8); } |
>>>>
>>>> rb
>>>>
>>>>
>>> ​
>>
>> ​
>>
>
>