You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@parquet.apache.org by "Uwe L. Korn (JIRA)" <ji...@apache.org> on 2018/08/03 14:16:00 UTC
[jira] [Moved] (PARQUET-1369) [Python] Unavailable Parquet column
statistics from Spark-generated file
[ https://issues.apache.org/jira/browse/PARQUET-1369?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Uwe L. Korn moved ARROW-2800 to PARQUET-1369:
---------------------------------------------
Fix Version/s: (was: 0.11.0)
cpp-1.5.0
Affects Version/s: (was: 0.9.0)
cpp-1.4.0
Component/s: (was: Python)
parquet-cpp
Workflow: patch-available, re-open possible (was: jira)
Key: PARQUET-1369 (was: ARROW-2800)
Project: Parquet (was: Apache Arrow)
> [Python] Unavailable Parquet column statistics from Spark-generated file
> ------------------------------------------------------------------------
>
> Key: PARQUET-1369
> URL: https://issues.apache.org/jira/browse/PARQUET-1369
> Project: Parquet
> Issue Type: Bug
> Components: parquet-cpp
> Affects Versions: cpp-1.4.0
> Reporter: Robert Gruener
> Assignee: Robert Gruener
> Priority: Major
> Labels: parquet
> Fix For: cpp-1.5.0
>
>
> I have a dataset generated by spark which shows it has statistics for the string column when using the java parquet-mr code (shown by using `parquet-tools meta`) however reading from pyarrow shows that the statistics for that column are not set. I should not the column only has a single value, though it still seems like a problem that pyarrow can't recognize it (it can recognize statistics set for the long and double types).
> See https://github.com/apache/arrow/files/2161147/metadata.zip for file example.
> Pyarrow Code To Check Statistics:
> {code}
> from pyarrow import parquet as pq
> meta = pq.read_metadata('/tmp/metadata.parquet')
> # No Statistics For String Column, prints false and statistics object is None
> print(meta.row_group(0).column(1).is_stats_set)
> {code}
> Example parquet-meta output:
> {code}
> file schema: spark_schema
> --------------------------------------------------------------------------------
> int: REQUIRED INT64 R:0 D:0
> string: OPTIONAL BINARY O:UTF8 R:0 D:1
> float: REQUIRED DOUBLE R:0 D:0
> row group 1: RC:8333 TS:76031 OFFSET:4
> --------------------------------------------------------------------------------
> int: INT64 SNAPPY DO:0 FPO:4 SZ:7793/8181/1.05 VC:8333 ENC:PLAIN_DICTIONARY,BIT_PACKED ST:[min: 0, max: 100, num_nulls: 0]
> string: BINARY SNAPPY DO:0 FPO:7797 SZ:1146/1139/0.99 VC:8333 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE ST:[min: hello, max: hello, num_nulls: 4192]
> float: DOUBLE SNAPPY DO:0 FPO:8943 SZ:66720/66711/1.00 VC:8333 ENC:PLAIN,BIT_PACKED ST:[min: 0.0057611096964338415, max: 99.99811053829232, num_nulls: 0]
> {code}
> I realize the column only has a single value though it still seems like pyarrow should be able to read the statistics set. I made this here and not a JIRA since I wanted to be sure this is actually an issue and there wasnt a ticket already made there (I couldnt find one but I wanted to be sure). Either way I would like to understand why this is
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)