[jira] [Commented] (PARQUET-1690) Integer Overflow of BinaryStatistics#isSmallerThan()

["Alexey Diomin (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/PARQUET-1690?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17701541#comment-17701541 ] 

Alexey Diomin commented on PARQUET-1690:
----------------------------------------

[~gszadovszky]  could you review the latest version of pr [https://github.com/apache/parquet-mr/pull/722]
or do we need to resubmit pr (we can add unit tests, but it will require at least 1G+ on disk for ci, don't know what the limits for your ci workers)?

_> Removing 1.11.0 target because this is an improvement only and not a regression since 1.10.0._
 
it's incorrect, the ticket was created with the incorrect *Type*
what we see in real life: # try to write large blobs with byte arrays
 # have overflowed in value in check do we need or not to write min/max statistics
 # writing it in final file + have some incorrect offsets 
 # have corrupted files and exceptions on read _java.lang.RuntimeException: corrupted file: the footer index is not within the file: 10687086773_

instead, file with 1 recond in 1GB we have at least = 1GB for data + metadata (1GB for min + 1GB for max)

file with 1gb blob, in reality, consume 5GB+ on disk and can't be read

 

reproducible example for spark

https://gist.github.com/REASY/98d80827d53e290d53a93f2bc3b76ab0

> Integer Overflow of BinaryStatistics#isSmallerThan()
> ----------------------------------------------------
>
>                 Key: PARQUET-1690
>                 URL: https://issues.apache.org/jira/browse/PARQUET-1690
>             Project: Parquet
>          Issue Type: Improvement
>          Components: parquet-mr
>    Affects Versions: 1.11.0
>            Reporter: Xinli Shang
>            Assignee: Xinli Shang
>            Priority: Major
>              Labels: pull-request-available
>
> "(min.length() + max.length()) < size" didn't handle integer overflow 
> [https://github.com/apache/parquet-mr/blob/master/parquet-column/src/main/java/org/apache/parquet/column/statistics/BinaryStatistics.java#L103]



--
This message was sent by Atlassian Jira
(v8.20.10#820010)