[jira] [Commented] (DRILL-5846) Improve Parquet Reader Performance for Flat Data types

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/DRILL-5846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16349736#comment-16349736 ] 

ASF GitHub Bot commented on DRILL-5846:
---------------------------------------

Github user vrozov commented on a diff in the pull request:

    https://github.com/apache/drill/pull/1060#discussion_r165552608
  
    --- Diff: exec/memory/base/src/main/java/io/netty/buffer/DrillBuf.java ---
    @@ -703,7 +703,18 @@ protected void _setLong(int index, long value) {
     
       @Override
       public ByteBuf getBytes(int index, ByteBuf dst, int dstIndex, int length) {
    -    udle.getBytes(index + offset, dst, dstIndex, length);
    +    final int BULK_COPY_THR = 1024;
    --- End diff --
    
    My testing shows that `PlatformDependent.copyMemory` outperforms multiple calls to `PlatformDependent.getByte()`/`PlatformDependent.putByte()` starting with 16 bytes. Copying 1024 bytes buffer using `getByte()`/`putByte()` is 10x slower compared to `copyMemory()`.


> Improve Parquet Reader Performance for Flat Data types 
> -------------------------------------------------------
>
>                 Key: DRILL-5846
>                 URL: https://issues.apache.org/jira/browse/DRILL-5846
>             Project: Apache Drill
>          Issue Type: Improvement
>          Components: Storage - Parquet
>    Affects Versions: 1.11.0
>            Reporter: salim achouche
>            Assignee: salim achouche
>            Priority: Major
>              Labels: performance
>             Fix For: 1.13.0
>
>
> The Parquet Reader is a key use-case for Drill. This JIRA is an attempt to further improve the Parquet Reader performance as several users reported that Parquet parsing represents the lion share of the overall query execution. It tracks Flat Data types only as Nested DTs might involve functional and processing enhancements (e.g., a nested column can be seen as a Document; user might want to perform operations scoped at the document level that is no need to span all rows). Another JIRA will be created to handle the nested columns use-case.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)