[jira] [Resolved] (IMPALA-5161) TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch

["Tim Armstrong (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/IMPALA-5161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Tim Armstrong resolved IMPALA-5161.
-----------------------------------
    Resolution: Fixed

> TPC-DS Q78 with MEM_LIMIT=10GB fails with "Repartitioning did not reduce the size of a spilled partition" on BufferPool dev branch
> ----------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: IMPALA-5161
>                 URL: https://issues.apache.org/jira/browse/IMPALA-5161
>             Project: IMPALA
>          Issue Type: Sub-task
>    Affects Versions: Impala 2.9.0
>            Reporter: Mostafa Mokhtar
>            Assignee: Tim Armstrong
>            Priority: Critical
>         Attachments: tpcds_q78_memory_repartitioning_didnt_reduce_num_scanner_threads-1.txt, tpcds_q78_memory_repartitioning_didnt_reduce.txt
>
>
> I set the memory limit to 10GB which is fairly low for Q78 and it failed with:
> {code}
> Query (id=8d463de3722df36f:a2dae93a00000000):
>   Summary:
>     Session ID: b54cda644686da93:f1c20afc188e0fa0
>     Session Type: BEESWAX
>     Start Time: 2017-04-04 10:46:17.099246000
>     End Time: 2017-04-04 11:06:04.438426000
>     Query Type: QUERY
>     Query State: EXCEPTION
>     Query Status: 
> Cannot perform aggregation at node with id 30. Repartitioning did not reduce the size of a spilled partition. Repartitioning level 7. Number of rows 1.
>     Impala Version: impalad version 2.9.0-SNAPSHOT RELEASE (build 07021cc117c56e4bbc0744b342fee6f5a8cce964)
>     User: mmokhtar
>     Connected User: mmokhtar
>     Delegated User: 
>     Network Address: 10.17.234.19:45450
>     Default Db: tpcds_10000_parquet
>     Sql Statement: with ws as (select d_year AS ws_sold_year, ws_item_sk, ws_bill_customer_sk ws_customer_sk, sum(ws_quantity) ws_qty, sum(ws_wholesale_cost) ws_wc, sum(ws_sales_price) ws_sp from web_sales left join web_returns on wr_order_number=ws_order_number and ws_item_sk=wr_item_sk join date_dim on ws_sold_date_sk = d_date_sk where wr_order_number is null group by d_year, ws_item_sk, ws_bill_customer_sk ), cs as (select d_year AS cs_sold_year, cs_item_sk, cs_bill_customer_sk cs_customer_sk, sum(cs_quantity) cs_qty, sum(cs_wholesale_cost) cs_wc, sum(cs_sales_price) cs_sp from catalog_sales left join catalog_returns on cr_order_number=cs_order_number and cs_item_sk=cr_item_sk join date_dim on cs_sold_date_sk = d_date_sk where cr_order_number is null group by d_year, cs_item_sk, cs_bill_customer_sk ), ss as (select d_year AS ss_sold_year, ss_item_sk, ss_customer_sk, sum(ss_quantity) ss_qty, sum(ss_wholesale_cost) ss_wc, sum(ss_sales_price) ss_sp from store_sales left join store_returns on sr_ticket_number=ss_ticket_number and ss_item_sk=sr_item_sk join date_dim on ss_sold_date_sk = d_date_sk where sr_ticket_number is null group by d_year, ss_item_sk, ss_customer_sk ) select ss_item_sk, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) ratio, ss_qty store_qty, ss_wc store_wholesale_cost, ss_sp store_sales_price, coalesce(ws_qty,0)+coalesce(cs_qty,0) other_chan_qty, coalesce(ws_wc,0)+coalesce(cs_wc,0) other_chan_wholesale_cost, coalesce(ws_sp,0)+coalesce(cs_sp,0) other_chan_sales_price from ss left join ws on (ws_sold_year=ss_sold_year and ws_item_sk=ss_item_sk and ws_customer_sk=ss_customer_sk) left join cs on (cs_sold_year=ss_sold_year and cs_item_sk=cs_item_sk and cs_customer_sk=ss_customer_sk) where coalesce(ws_qty,0)>0 and coalesce(cs_qty, 0)>0 and ss_sold_year=2000 order by ss_item_sk, ss_qty desc, ss_wc desc, ss_sp desc, other_chan_qty, other_chan_wholesale_cost, other_chan_sales_price, round(ss_qty/(coalesce(ws_qty+cs_qty,1)),2) limit 100
>     Coordinator: vd1309.halxg.cloudera.com:22000
>     Query Options (non default): MEM_LIMIT=10737418240
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)