You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/03/09 23:34:54 UTC

[GitHub] [arrow] dmgcodevil opened a new issue #12597: Does Pyarrow support Parquet 2.0

dmgcodevil opened a new issue #12597:
URL: https://github.com/apache/arrow/issues/12597


   I'm using pyarrow to read parquet files, however, it fails to read some files. 
   
   Error:
   
   ```
     table = self._dataset.to_table(
     File "pyarrow\_dataset.pyx", line 323, in pyarrow._dataset.Dataset.to_table
     File "pyarrow\_dataset.pyx", line 2311, in pyarrow._dataset.Scanner.to_table
     File "pyarrow\error.pxi", line 143, in pyarrow.lib.pyarrow_internal_check_status
     File "pyarrow\error.pxi", line 114, in pyarrow.lib.check_status
   OSError: Malformed levels. min: 60 max: 60 out of range.  Max Level: 1
   ```
   
   
   Based on my research the file that cannot be read uses a new 2.0 encoding
   
   Column meta from the file that pyarrow  reads successfully
   
   `id:             BINARY GZIP DO:0 FPO:630 SZ:101/63/0.62 VC:30 ENC:PLAIN_DICTIONARY,BIT_PACKED,RLE ST:[min: , max: D, num_nulls: 0]`
   
   Column meta from the file that pyarrow  fails to read
   
   `id:             BINARY GZIP DO:0 FPO:593 SZ:90/52/0.58 VC:30 ENC:RLE_DICTIONARY ST:[min: , max: D, num_nulls: 0]`
   
   Based on Parquet spec, `RLE_DICTIONARY` is a new encoding format:
   
   > Using the PLAIN_DICTIONARY enum value is deprecated in the Parquet 2.0 specification. Prefer using RLE_DICTIONARY in a data page and PLAIN in a dictionary page for Parquet 2.0+ files.
   
   Does Pyarrow support Parquet 2.0 format ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
emkornfield commented on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064631097


   Looking more closely at the error something else is going on.  If you can produce a minimal repro and open up a JIRA we can try to investigate further


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] dmgcodevil edited a comment on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
dmgcodevil edited a comment on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064360092


   @jorisvandenbossche , the files that Pyarrow successfully reads written by Spark/Iceberg data source and Iceberg's [ParquetWriter](https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java).
   
   The files that Pyarrow fails to read are written via Trino Iceberg catalog (connector). In theory, both Trino and Spark should use Iceberg ParquetWriter which internally uses Hadoop Parquet Writer. 
   
   What I've found is that some columns are encoded in `DELTA_BYTE_ARRAY`. Does Pyarrow support this encoding? I know that Fastparquet does not. I also found this [ticket](https://issues.apache.org/jira/browse/ARROW-6057?src=confmacro), is it still relevant? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] dmgcodevil commented on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
dmgcodevil commented on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064360092


   @jorisvandenbossche , the files that Pyarrow successfully reads written by Spark/Iceberg data source and Iceberg's [ParquetWriter].(https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java)
   
   The files that Pyarrow fails to read are written via Trino Iceberg catalog (connector). In theory, both Trino and Spark should use Iceberg ParquetWriter which internally uses Hadoop Parquet Writer. 
   
   What I've found is that some columns are encoded in DELTA_BYTE_ARRAY. Does Pyarrow support this encoding? I know that Fastparquet does not. I also found this [ticket](https://issues.apache.org/jira/browse/ARROW-6057?src=confmacro), is it still relevant? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] emkornfield commented on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
emkornfield commented on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064462511


   DELTA_BYTE_ARRAY has been implemented as of [pyarrow 7.0](https://issues.apache.org/jira/browse/PARQUET-492?jql=text%20~%20%22DELTA_BYTE_ARRAY%22).
   
   It [does not handle](https://issues.apache.org/jira/browse/ARROW-14089?jql=text%20~%20%22DELTA_BYTE_ARRAY%22) a bug for older writers of this encoding.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] dmgcodevil commented on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
dmgcodevil commented on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064507452


   @emkornfield Ok, I'll keep investigating the issue. Maybe you could give me some directions? One thing I know for sure: "bad" files are 20% smaller than "good" files, although they have identical content.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] jorisvandenbossche commented on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
jorisvandenbossche commented on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1063898187


   Yes, Arrow does generally support Parquet version 2.x (although not every feature), and specifically supports reading RLE_DICTIONARY encoding. See the overview of supported parquet features here: https://arrow.apache.org/docs/cpp/parquet.html#encodings
   
   Did you write the file yourself, or do you know how it was written? (with which software?) 
   Would you be able to share the file that reproduces the issue? (or a file with some dummy data, that if written in the same way also reproduces the issue)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [arrow] dmgcodevil edited a comment on issue #12597: Does Pyarrow support Parquet 2.0

Posted by GitBox <gi...@apache.org>.
dmgcodevil edited a comment on issue #12597:
URL: https://github.com/apache/arrow/issues/12597#issuecomment-1064360092


   @jorisvandenbossche , the files that Pyarrow successfully reads written by Spark/Iceberg data source and Iceberg's [ParquetWriter](https://github.com/apache/iceberg/blob/master/parquet/src/main/java/org/apache/iceberg/parquet/ParquetWriter.java).
   
   The files that Pyarrow fails to read are written via Trino Iceberg catalog (connector). In theory, both Trino and Spark should use Iceberg ParquetWriter which internally uses Hadoop Parquet Writer. 
   
   What I've found is that some columns are encoded in DELTA_BYTE_ARRAY. Does Pyarrow support this encoding? I know that Fastparquet does not. I also found this [ticket](https://issues.apache.org/jira/browse/ARROW-6057?src=confmacro), is it still relevant? 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org