[jira] [Resolved] (IMPALA-3205) Validate and fix spilling performance and memory usage of new buffer pool

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-3205.
-----------------------------------
       Resolution: Fixed
    Fix Version/s: Impala 2.10.0

See my comments for the validation that I've done here.

> Validate and fix spilling performance and memory usage of new buffer pool
> -------------------------------------------------------------------------
>
>                 Key: IMPALA-3205
>                 URL: https://issues.apache.org/jira/browse/IMPALA-3205
>             Project: IMPALA
>          Issue Type: Sub-task
>          Components: Backend
>    Affects Versions: Impala 2.6.0
>            Reporter: Tim Armstrong
>            Assignee: Tim Armstrong
>            Priority: Minor
>              Labels: resource-management
>             Fix For: Impala 2.10.0
>
>         Attachments: min_mem_buffer_pool_64kb_buffers.csv, min_mem_buffer_pool_default.csv, min_mem_old_aggs_joins.csv, min_mem_part_aggs_joins.csv, min_mem.py, min_mem_tpcds_5_bbm.csv, min_mem_tpcds_5_bbm.json, min_mem_tpcds_5_bp.csv, min_mem_tpcds_5_bp.json, min_mem_tpcds_5_bp_reduce_min_remaining.csv, min_mem_tpcds_5_bp_reduce_min_remaining.json, spilling-agg-bbm.txt, spilling-agg-bufferpool-8mb-buffers.txt, spilling-agg-bufferpool.txt, spilling-hj2-bbm.txt, spilling-hj2-bp-8mb-buffers.txt, spilling-hj2-bp.txt, spilling-sort-bbm.txt, spilling-sort-bufferpool.txt
>
>
> We want to ensure that performance doesn't regress:
> * Memory usage for the new buffer pool (with the old small buffers optimisation removed) should be similar or less than the old code for typical workloads. This will require reducing the buffer size - e.g. to 1MB or less.
> * Spill-to-disk performance should not regress more than is acceptable. This may be tricky to get right since the I/O size will be reduced from 8MB. We may need to allocate logically sequential blocks in contiguous disk and do some kind of prefetching in the buffer pool to issue multiple writes at a time.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)