[jira] [Commented] (IMPALA-4530) Sort node after exchange should start sorting after first RowBatch is received

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/IMPALA-4530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17730387#comment-17730387 ] 

ASF subversion and git services commented on IMPALA-4530:
---------------------------------------------------------

Commit 80c1d2dbaabc78bdf1b6e4da5475bfa365cd375e in impala's branch refs/heads/master from noemi
[ https://gitbox.apache.org/repos/asf?p=impala.git;h=80c1d2dba ]

IMPALA-4530: Implement in-memory merge of quicksorted runs

This change aims to decrease back-pressure in the sorter. It offers an
alternative for the in-memory run formation strategy and sorting
algorithm by introducing a new in-memory merge level between the
in-memory quicksort and the external merge phase.
Instead of forming one big run, it produces many smaller in-memory runs
(called miniruns), sorts those with quicksort, then merges them
in memory, before spilling or serving GetNext().
The external merge phase remains the same.
Works with MAX_SORT_RUN_SIZE development query option that determines
the maximum number of pages in a 'minirun'. The default value of
MAX_SORT_RUN_SIZE is 0, which keeps the original implementation of 1
big initial in-memory run. Other options are integers of 2 and above.
The recommended value is 10 or more, to avoid high fragmentation
in case of large workloads and variable length data.

Testing:
- added MAX_SORT_RUN_SIZE as an additional test dimension to
  test_sort.py with values [0, 2, 20]
- additional partial sort test case (inserting into partitioned
  kudu table)
- manual E2E testing

Change-Id: I58c0ae112e279b93426752895ded7b1a3791865c
Reviewed-on: http://gerrit.cloudera.org:8080/18393
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
Tested-by: Csaba Ringhofer <cs...@cloudera.com>


> Sort node after exchange should start sorting after first RowBatch is received
> ------------------------------------------------------------------------------
>
>                 Key: IMPALA-4530
>                 URL: https://issues.apache.org/jira/browse/IMPALA-4530
>             Project: IMPALA
>          Issue Type: Bug
>          Components: Backend
>            Reporter: Mostafa Mokhtar
>            Assignee: Noemi Pap-Takacs
>            Priority: Major
>              Labels: performance
>
> Sort node after exchange doesn't start sorting until all data is received which add lots of latency to the query. 
> Not clear if this optimization would still make sense for a Scan followed by a sort run using the same thread. 
> Query
> {code}
> insert into tpcds_1000_parquet.store_sales_insert  partition(ss_sold_date_sk, ss_quantity)  /*+ clustered*/
> select
> ss_sold_time_sk,
>   ss_item_sk ,
>   ss_customer_sk,
>   ss_cdemo_sk,
>   ss_hdemo_sk,
>   ss_addr_sk,
>   ss_store_sk,
>   ss_promo_sk,
>   ss_ticket_number ,
>   ss_wholesale_cost ,
>   ss_list_price ,
>   ss_sales_price ,
>   ss_ext_discount_amt ,
>   ss_ext_sales_price ,
>   ss_ext_wholesale_cost ,
>   ss_ext_list_price ,
>   ss_ext_tax ,
>   ss_coupon_amt ,
>   ss_net_paid ,
>   ss_net_paid_inc_tax ,
>   ss_net_profit,
>   ss_sold_date_sk  , ss_quantity
> from   store_sales
> {code}
> Plan
> {code}
> WRITE TO HDFS [tpcds_1000_parquet.store_sales_insert, OVERWRITE=false, PARTITION-KEYS=(ss_sold_date_sk,ss_quantity)]
> |  partitions=180576
> |  hosts=15 per-host-mem=17.88GB
> |
> 02:SORT
> |  order by: ss_sold_date_sk DESC NULLS LAST, ss_quantity DESC NULLS LAST
> |  hosts=15 per-host-mem=1.45GB
> |  tuple-ids=1 row-size=100B cardinality=2879987999
> |
> 01:EXCHANGE [HASH(ss_sold_date_sk,ss_quantity)]
> |  hosts=15 per-host-mem=0B
> |  tuple-ids=0 row-size=100B cardinality=2879987999
> |
> 00:SCAN HDFS [tpcds_1000_parquet.store_sales, RANDOM]
>    partitions=1824/1824 files=1824 size=189.24GB
>    table stats: 2879987999 rows total
>    column stats: all
>    hosts=15 per-host-mem=88.00MB
>    tuple-ids=0 row-size=100B cardinality=2879987999
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org