[jira] [Comment Edited] (HIVE-18269) LLAP: Fast llap io with slow processing pipeline can lead to OOM

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-18269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289886#comment-16289886 ] 

Sergey Shelukhin edited comment on HIVE-18269 at 12/13/17 8:51 PM:
-------------------------------------------------------------------

Is this actually going to work? Seems like the sync blocks inside which take and put are happening will obstruct each other, so if one blocks on the queue, the other cannot enter and unblock the first.


was (Author: sershe):
Is this actually going to work? Seems like the sync blocks inside which take and put are happening will obstruct each other, so if one blocks the other cannot enter and unblock the first.

> LLAP: Fast llap io with slow processing pipeline can lead to OOM
> ----------------------------------------------------------------
>
>                 Key: HIVE-18269
>                 URL: https://issues.apache.org/jira/browse/HIVE-18269
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 3.0.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>         Attachments: HIVE-18269.1.patch, Screen Shot 2017-12-13 at 1.15.16 AM.png
>
>
> pendingData linked list in Llap IO elevator (LlapRecordReader.java) may grow indefinitely when Llap IO is faster than processing pipeline. Since we don't have backpressure to slow down the IO, this can lead to indefinite growth of pending data leading to severe GC pressure and eventually lead to OOM.
> This specific instance of LLAP was running on HDFS on top of EBS volume backed by SSD. The query that triggered this is issue was ANALYZE STATISTICS .. FOR COLUMNS which also gather bitvectors. Fast IO and Slow processing case.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)