You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2022/08/12 03:16:00 UTC

[jira] [Commented] (IMPALA-11489) Async IO cannot handle >2GB ORC files

    [ https://issues.apache.org/jira/browse/IMPALA-11489?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17578766#comment-17578766 ] 

ASF subversion and git services commented on IMPALA-11489:
----------------------------------------------------------

Commit 0ce2403b58afd94877e36e1647ec34e8a9df7dba in impala's branch refs/heads/master from Csaba Ringhofer
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=0ce2403b5 ]

IMPALA-11489: Fix int overflow in >2GB ORC files

This can lead to returning an error "Invalid read len.", but can't lead
to other problems like reading corrupt data or crashing.

The fix is trivial, but creating automatic tests for it seems hard,
so no regression test is added.

Change-Id: Id8438d3512b0ebaafd319f369c1f6871d980a599
Reviewed-on: http://gerrit.cloudera.org:8080/18827
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>


> Async IO cannot handle >2GB ORC files
> -------------------------------------
>
>                 Key: IMPALA-11489
>                 URL: https://issues.apache.org/jira/browse/IMPALA-11489
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>            Reporter: Csaba Ringhofer
>            Priority: Major
>
> We assume that the size fits to an int:
> https://github.com/apache/impala/blob/308fda110758b0fc58e5b1f477d635aac29aea75/be/src/exec/hdfs-orc-scanner.cc#L253
> If the size overflows, then we can incorrectly hit the following error check (this check is meant to avoid crashing due to corrupt metadata). I see no other ways this could cause problems, if the catch still succeeds (because the overflow led to a valid looking length), then the data will be read correctly.
> This looks like a trivial fix, but I am concerned about lack of testing of >2GB files



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org