You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@orc.apache.org by "Gang Wu (JIRA)" <ji...@apache.org> on 2019/04/03 05:12:00 UTC

[jira] [Resolved] (ORC-469) Add EncodedStringVectorBatch to expose string dictionary in VectorBatch

     [ https://issues.apache.org/jira/browse/ORC-469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Gang Wu resolved ORC-469.
-------------------------
    Resolution: Fixed

> Add EncodedStringVectorBatch to expose string dictionary in VectorBatch
> -----------------------------------------------------------------------
>
>                 Key: ORC-469
>                 URL: https://issues.apache.org/jira/browse/ORC-469
>             Project: ORC
>          Issue Type: Improvement
>          Components: Reader
>    Affects Versions: 1.5.5, 1.6.0, 2.0.0
>            Reporter: Yurui Zhou
>            Assignee: Yurui Zhou
>            Priority: Major
>          Time Spent: 20m
>  Remaining Estimate: 0h
>
> Propose to add EncodedStringVectorBatch to expose string dictionary in VectorBatch. Exposing string dictionary will benefits as bellow:
>  * Enable computation over encoded data. By exposing dictionary in vector batch we would be able to implement filter operator in a more efficient way. In our POC, by enabling encoded data based filter, we achieved 8% E2E perf improvement on tpch q1
>  * Make data serialization more efficient. Currently when serializing Orc Vectorbatch, we have to copy all the strings in a vector even though the string data is already dictionary encoded. Exposing String Dictionary will enable vector batch serializer to remove unnecessary string memcpy, which will greatly improve serialization efficiency.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)