You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@hive.apache.org by "Konstantin Bereznyakov (Jira)" <ji...@apache.org> on 2025/12/15 20:25:00 UTC

[jira] [Work started] (HIVE-29367) ConvertJoinMapJoin.computeOnlineDataSizeGeneric() overflow on large tables

     [ https://issues.apache.org/jira/browse/HIVE-29367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Work on HIVE-29367 started by Konstantin Bereznyakov.
-----------------------------------------------------
> ConvertJoinMapJoin.computeOnlineDataSizeGeneric() overflow on large tables
> --------------------------------------------------------------------------
>
>                 Key: HIVE-29367
>                 URL: https://issues.apache.org/jira/browse/HIVE-29367
>             Project: Hive
>          Issue Type: Bug
>         Environment: Execute the attached query file [^mapjoin_negative_overflow.q] and see that its output [^mapjoin_negative_overflow.q.out] shows mapjoin conversion for *both* queries, even though t1 is massive in the second query.
>            Reporter: Konstantin Bereznyakov
>            Assignee: Konstantin Bereznyakov
>            Priority: Major
>         Attachments: mapjoin_negative_overflow.q, mapjoin_negative_overflow.q.out
>
>
> Once the value of onlineDataSize becomes really large, the following line of code:
> https://github.com/apache/hive/blob/2cd59de300e9dae3fe1d6d2538efdbf56f80c763/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L366
> and its two subsequent lines become unsafe for overflowing of Long.MAX_VALUE. As the result, the computed value could become negative.
> Therefore, the inputSize value assigned here: [https://github.com/apache/hive/blob/2cd59de300e9dae3fe1d6d2538efdbf56f80c763/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1200] becomes negative, and the large table is deemed to be fitting into the memory here: [https://github.com/apache/hive/blob/2cd59de300e9dae3fe1d6d2538efdbf56f80c763/ql/src/java/org/apache/hadoop/hive/ql/optimizer/ConvertJoinMapJoin.java#L1213]
> The ultimate outcome of the query is attempting to fit a giant table into RAM, and a failure with an OOM



--
This message was sent by Atlassian Jira
(v8.20.10#820010)