You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@arrow.apache.org by "Matthew Topol (Jira)" <ji...@apache.org> on 2022/08/26 15:34:00 UTC

[jira] [Assigned] (ARROW-17453) Arrow-Parquet c++ - Parquet files show inconsistent data

     [ https://issues.apache.org/jira/browse/ARROW-17453?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Matthew Topol reassigned ARROW-17453:
-------------------------------------

    Assignee: Matthew Topol

> Arrow-Parquet c++ - Parquet files show inconsistent data 
> ---------------------------------------------------------
>
>                 Key: ARROW-17453
>                 URL: https://issues.apache.org/jira/browse/ARROW-17453
>             Project: Apache Arrow
>          Issue Type: Bug
>            Reporter: Nishanth
>            Assignee: Matthew Topol
>            Priority: Major
>         Attachments: athena_datatypes.gz.parquet
>
>
> Files generated from Athena using Iceberg and read via arrow-parquet c++ show inconsistent data, with row information missing. 
> To repro, create athena iceberg table with parquet as below. 
>  
> {code:java}
> create table datatypes_helper (d_int1 int, d_bigint1 bigint, d_float1 float, d_double1 double, d_decimal38_0_1 decimal(38,0), d_decimal19_10_1 decimal(19,10), d_date1 date, d_timestamp1 timestamp, d_string1 string, d_binary1 binary, d_decimal1 decimal, d_array1 array<string>, d_map1 map<string, int>)
> LOCATION 's3:/' 
> TBLPROPERTIES (
>   'table_type'='ICEBERG',
>   'format'='parquet'
> );
> insert into  datatypes_helper VALUES (11, CAST(11 as bigint), CAST(12.222222222100001 as real), 12.222222222222223, cast(11.0 as decimal(38,0)), cast(12.358024679 as decimal(19,10)), date('2022-01-12'), timestamp '2022-03-31 18:53:43', cast('IX1KNXR6KPF' as varchar), cast(X'0000011100' as varbinary), cast(11 as decimal(10,0)), array[cast('11' as varchar), cast('11' as varchar)], map(array[cast('rowNum' as varchar)], array[CAST(11 AS INT)]));
> insert into  datatypes_helper VALUES (12, CAST(12 as bigint), CAST(13.3333333332 as real), 13.333333333333334, cast(12.0 as decimal(38,0)), cast(13.481481468 as decimal(19,10)), date('2022-01-13'), timestamp '2022-03-31 18:53:43', cast('WDJC5KD74I0B' as varchar), cast(X'000101111100' as varbinary), cast(12 as decimal(10,0)), array[cast('12' as varchar), cast('12' as varchar)], map(array[cast('rowNum' as varchar)], array[CAST(12 AS INT)]));
> insert into  datatypes_helper VALUES (13, CAST(13 as bigint), CAST(14.4444444443 as real), 14.444444444444445, cast(13.0 as decimal(38,0)), cast(14.604938257 as decimal(19,10)), date('2022-01-14'), timestamp '2022-03-31 18:53:43', cast('Y8S9T7QOIWPS0' as varchar), cast(X'110011111110' as varbinary), cast(13 as decimal(10,0)),  array[cast('13' as varchar), cast('13' as varchar)], map(array[cast('rowNum' as varchar)], array[CAST(13 AS INT)]));
> insert into  datatypes_helper(d_int1) VALUES (null);
> create table datatypes (d_int1 int, d_bigint1 bigint, d_float1 float, d_double1 double, d_decimal38_0_1 decimal(38,0), d_decimal19_10_1 decimal(19,10), d_date1 date, d_timestamp1 timestamp, d_string1 string, d_binary1 binary, d_decimal1 decimal, d_array1 array<string>, d_map1 map<string, int>)
> LOCATION 's3://' 
> TBLPROPERTIES (
>   'table_type'='ICEBERG',
>   'format'='parquet'
>   );
> insert into  datatypes select * from datatypes_helper;
>  
> {code}
> The above inserts 4 rows. However querying the data using pyarrow return only 2 rows for some columns. Others parquet tools (parquet-tools-mr do not show this behavior)
> {{}}
> {code:java}
> >>> parquet_file.read_row_group(0)
> pyarrow.Table
> d_int1: int32
> d_bigint1: int64
> d_float1: float
> d_double1: double
> d_decimal38_0_1: decimal128(38, 0)
> d_decimal19_10_1: decimal128(19, 10)
> d_date1: date32[day]
> d_timestamp1: timestamp[us]
> d_string1: string
> d_binary1: binary
> d_decimal1: decimal128(10, 0)
> d_array1: list<element: string>
>   child 0, element: string
> d_map1: map<string, int32 ('d_map1')>
>   child 0, d_map1: struct<key: string not null, value: int32> not null
>       child 0, key: string not null
>       child 1, value: int32
> ----
> d_int1: [[12,11,null,null]]
> d_bigint1: [[12,11,null,null]]
> d_float1: [[13.333333,12.222222,null,null]]
> d_double1: [[13.333333333333334,12.222222222222223,null,null]]
> d_decimal38_0_1: [[12,11,null,null]]
> d_decimal19_10_1: [[13.4814814680,12.3580246790,null,null]]
> d_date1: [[2022-01-13,2022-01-12,null,null]]
> d_timestamp1: [[2022-03-31 18:53:43.000000,2022-03-31 18:53:43.000000,null,null]]
> d_string1: [["WDJC5KD74I0B","IX1KNXR6KPF",null,null]]
> d_binary1: [[000101111100,0000011100,null,null]]{code}
> The parquet file queried is attached to the JIRA. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)