[jira] [Commented] (ARROW-12201) [C++] [Parquet] Writing uint32 does not preserve parquet's LogicalType

["Antoine Pitrou (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/ARROW-12201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17314751#comment-17314751 ] 

Antoine Pitrou commented on ARROW-12201:
----------------------------------------

Hmm, that's bummer. Hopefully we can fix this for 4.0.0.

> [C++] [Parquet] Writing uint32 does not preserve parquet's LogicalType
> ----------------------------------------------------------------------
>
>                 Key: ARROW-12201
>                 URL: https://issues.apache.org/jira/browse/ARROW-12201
>             Project: Apache Arrow
>          Issue Type: Bug
>          Components: C++, Parquet
>    Affects Versions: 3.0.0
>            Reporter: Jorge Leitão
>            Priority: Minor
>
> When writing a `uint32` column, (parquet's) logical type is not written, limiting interoperability with other engines.
> Minimal Python
> ```
> import pyarrow as pa
> data = {"uint32", [1, None, 0]}
> schema = pa.schema([pa.field('uint32', pa.uint32())])
> t = pa.table(data, schema=schema)
> pa.parquet.write_table(t, "bla.parquet")
> ```
>  
> Inspecting it with spark:
> ```
> from pyspark.sql import SparkSession
> spark = SparkSession.builder.getOrCreate()
> df = spark.read.parquet("bla.parquet")
> print(df.select("uint32").schema)
> ```
> shows `StructType(List(StructField(uint32,LongType,true)))`. "LongType" indicates that the field is interpreted as a 64 bit integer. Further inspection of the metadata shows that both convertedType and logicalType are not being set. Note that this is independent of the arrow-specific schema written in the metadata.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)