[jira] [Commented] (PARQUET-2060) Parquet corruption can cause infinite loop with Snappy

["Gabor Szadovszky (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-2060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17368819#comment-17368819 ] 

Gabor Szadovszky commented on PARQUET-2060:
-------------------------------------------

[~mmeimaris], what do you think about simply returning the zero length BytesInput object (just like in the case of the codec is null)? This way we shall catch the error at same place if the data stream is empty. (We shall handle this case for uncompressed data as well.)
Are you willing to implement a PR about this? I'm happy to help/review.

> Parquet corruption can cause infinite loop with Snappy
> ------------------------------------------------------
>
>                 Key: PARQUET-2060
>                 URL: https://issues.apache.org/jira/browse/PARQUET-2060
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>            Reporter: Marios Meimaris
>            Priority: Major
>         Attachments: datapage_v2.snappy.parquet, datapage_v2.snappy.parquet1383
>
>
> I am attaching a valid and corrupt parquet file (datapageV2) that differ in one byte.
> We hit an infinite loop when trying to read the corrupt file in [https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/impl/ColumnReaderBase.java#L698] and specifically in the `page.getData().toInputStream()` call.  
> Stack trace of infinite loop:
> java.io.DataInputStream.readFully(DataInputStream.java:195)
>  java.io.DataInputStream.readFully(DataInputStream.java:169)
>  org.apache.parquet.bytes.BytesInput$StreamBytesInput.toByteArray(BytesInput.java:287)
>  org.apache.parquet.bytes.BytesInput.toByteBuffer(BytesInput.java:237)
>  org.apache.parquet.bytes.BytesInput.toInputStream(BytesInput.java:246)
>  org.apache.parquet.column.impl.ColumnReaderBase.readPageV2(ColumnReaderBase.java:698)
>  org.apache.parquet.column.impl.ColumnReaderBase.access$400(ColumnReaderBase.java:57)
>  org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:628)
>  org.apache.parquet.column.impl.ColumnReaderBase$3.visit(ColumnReaderBase.java:620)
>  org.apache.parquet.column.page.DataPageV2.accept(DataPageV2.java:192)
>  org.apache.parquet.column.impl.ColumnReaderBase.readPage(ColumnReaderBase.java:620)
>  org.apache.parquet.column.impl.ColumnReaderBase.checkRead(ColumnReaderBase.java:594)
>  
> The call to `readFully` will underneath go through `NonBlockedDecompressorStream` which will always hit this path: [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/NonBlockedDecompressorStream.java#L45]. This will cause `setInput` to not be called on the decompressor, and the subsequent calls to `decompress` will always hit this condition: [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/codec/SnappyDecompressor.java#L54]. Therefore, the 0 value will be returned by the read method, which will cause an infinite loop in [https://github.com/openjdk-mirror/jdk7u-jdk/blob/master/src/share/classes/java/io/DataInputStream.java#L198] 
>  This originates from the corruption, which causes the input stream of the data page to be of size 0, which makes `getCompressedData` always return -1. 
> I am wondering whether this can be caught earlier so that the read fails in case of such corruptions. 
> Since this happens in `BytesInput.toInputStream`, I don't think it's only relevant to DataPageV2. 
>  
> In [https://github.com/apache/parquet-mr/blob/master/parquet-hadoop/src/main/java/org/apache/parquet/hadoop/CodecFactory.java#L111,] if we call `bytes.toByteArray` and log its length, it is 0 in the case of the corrupt file, and 6 in the case of the valid file. 
> A potential fix is to check the array size there and fail early, but I am not sure if a zero-length byte array can ever be expected in the case of valid files.
>  
> Attached:
> Valid file: `datapage_v2_snappy.parquet`
> Corrupt file: `datapage_v2_snappy.parquet1383`



--
This message was sent by Atlassian Jira
(v8.3.4#803005)