[jira] [Commented] (HIVE-9420) LLAP: separate decoding thread from read/uncompress thread

["Sergey Shelukhin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-9420?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14623121#comment-14623121 ] 

Sergey Shelukhin commented on HIVE-9420:
----------------------------------------

There is loss of ordering if this is done in a straightforward way (wrapper for consumer that offloads consumeData, the whole patch would be 30 lines)... setDone will potentially be called before some of the async consumeData-s on the consumer that is consuming from intermediate MTT consumer.

> LLAP: separate decoding thread from read/uncompress thread
> ----------------------------------------------------------
>
>                 Key: HIVE-9420
>                 URL: https://issues.apache.org/jira/browse/HIVE-9420
>             Project: Hive
>          Issue Type: Sub-task
>            Reporter: Sergey Shelukhin
>            Assignee: Sergey Shelukhin
>
> Right now reader is a callable that is dumped into threadpool, and all the data loading for one request happens on one thread (in parallel with operator pipeline).
> We might want to separate reading from HDFS/decompressing data, from decoding data and low-level row-level SARGs (that, and high level cache which is AWOL now, were the reasons for encoded data production separation from final VRB production)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)