[jira] [Updated] (SPARK-7885) add config to control map aggregation in spark sql

["Hyukjin Kwon (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-7885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Hyukjin Kwon updated SPARK-7885:
--------------------------------
    Labels: bulk-closed  (was: )

> add config to control map aggregation in spark sql
> --------------------------------------------------
>
>                 Key: SPARK-7885
>                 URL: https://issues.apache.org/jira/browse/SPARK-7885
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 1.2.0, 1.2.2, 1.3.1
>            Reporter: jeanlyn
>            Priority: Minor
>              Labels: bulk-closed
>
> For now, *execution.HashAggregation* add the map aggregation in oder to decrease the shuffle data.However,we found gc problem when we use this optimization and finally the executor crash.For example,
> {noformat} 
> select sale_ord_id as order_id,
>       coalesce(sum(sku_offer_amount),0.0) as sku_offer_amount,
>       coalesce(sum(suit_offer_amount),0.0) as suit_offer_amount,
>       coalesce(sum(flash_gp_offer_amount),0.0) + coalesce(sum(gp_offer_amount),0.0) as gp_offer_amount,
>       coalesce(sum(flash_gp_offer_amount),0.0) as flash_gp_offer_amount,
>       coalesce(sum(full_minus_offer_amount),0.0) as full_rebate_offer_amount,
>       0.0 as telecom_point_offer_amount,
>       coalesce(sum(coupon_pay_amount),0.0) as dq_and_jq_pay_amount,
>       coalesce(sum(jq_pay_amount),0.0) + coalesce(sum(pop_shop_jq_pay_amount),0.0) + coalesce(sum(lim_cate_jq_pay_amount),0.0) as jq_pay_amount,
>       coalesce(sum(dq_pay_amount),0.0) + coalesce(sum(pop_shop_dq_pay_amount),0.0) + coalesce(sum(lim_cate_dq_pay_amount),0.0) as dq_pay_amount,
>       coalesce(sum(gift_cps_pay_amount),0.0) as gift_cps_pay_amount ,
>       coalesce(sum(mobile_red_packet_pay_amount),0.0) as mobile_red_packet_pay_amount,
>       coalesce(sum(acct_bal_pay_amount),0.0) as acct_bal_pay_amount,
>       coalesce(sum(jbean_pay_amount),0.0) as jbean_pay_amount,
>       coalesce(sum(sku_rebate_amount),0.0) as sku_rebate_amount,
>       coalesce(sum(yixun_point_pay_amount),0.0) as yixun_point_pay_amount,
>       coalesce(sum(sku_freight_coupon_amount),0.0) as freight_coupon_amount
> from        ord_at_det_di
> where       ds = '2015-05-20'
> group  by   sale_ord_id
> {noformat}
> the sql scan two text files and each file is 360MB,we use 6 executor, each executor has 8GB memory and 2 cpu.
> We can add a config control map aggregation to avoid it. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org