You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org> on 2017/03/22 22:56:02 UTC
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Taras Bobrovytsky has uploaded a new change for review.
http://gerrit.cloudera.org:8080/6459
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 117 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/1
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#6).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 166 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/6
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 13: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 13: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#6).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 174 insertions(+), 51 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/6
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#5).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 147 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/5
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 12: Verified-1
Build failed: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/478/
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 12
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#6).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 174 insertions(+), 51 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/6
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#6).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 174 insertions(+), 51 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/6
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#7).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 186 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/7
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 10: Code-Review+1
A quick remark is that the simplification done in this patch may regress certain queries with union node containing expressions which are expensive to materialize. Previously, we would have stopped materializing once we reached the limit but now we would proceed until we hit a row batch limit.
Dan, can you please take one last pass ?
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 148 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 6: Code-Review+1
(4 comments)
Looks good. I'll let Michael finish his review - he can probably do the +2.
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 98:
Extra blank lines.
Line 235: if (ReachedLimit()) {
I don't think this can be taken now, right? Since we don't increment num_rows_returned_ until later.
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.h
File be/src/exec/union-node.h:
Line 116: /// call on the child.
Maybe mention that these GetNext* functions don't apply the limit?
PS6, Line 127: Increments 'num_rows_returned_' and
: /// 'child_row_idx_'
I think this needs an update.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 6:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS6, Line 29: dst_batch->tuple_data_pool()
is this where that mem-pool in the class is used when codegening? Why not use that mem-pool here to make that clear and keep the paths more consistent in their use of memory?
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS6, Line 113: abort codegen entirely.
this doesn't seem to match the code. wouldn't we have already filled in codegend_union_materialize_batch_fns_ for earlier children? do we have a test case for this path?
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS6, Line 68: Temporarily stores references to materialized tuples by a codegened function
i'm not sure what this means. MemPools don't store references, they contain chunks of memory that can be allocated.
Is this trying to say that this mem-pool is used by the codegen version of MaterializeExprs to materialize var-len slots? And if so, why is it specific to codegen?
PS6, Line 71: TODO: Remove this when no longer necessary in the future.
do we have a concrete jira for when this won't be necessary?
PS6, Line 96: Jitted
usually we say "codegen'ed"
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#8).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 183 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/8
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#6).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 174 insertions(+), 51 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/6
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 4:
(2 comments)
Hmm, I guess the per-row overhead is probably not as significant for the case when we're returning a bunch of columns. There might be a more pronounced effect if we're just returning a handful of columns.
I think we can still squeeze some more cycles out of this with minimal effort, but if we stop seeing a measurable improvement for a query that returns a smaller number of columns we could consider stopping then.
http://gerrit.cloudera.org:8080/#/c/6459/4/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
Line 28: while (!dst_batch->AtCapacity() && child_row_idx_ < child_batch_->num_rows()) {
The remaining optimisation is to pull all references to member variables out of the loop. I.e. child_row_idx_, child_batch_, child_row_idx_, child_expr_lists_, tuple_desc_->byte_size(). This will reduce the number of loads and stores quite a bit. E.g.
int child_row_idx = child_row_idx_;
int tuple_byte_size = tuple_desc_->byte_size;
while (...) {
}
child_row_idx_ = child_row_idx;
Currently it will do a load and store to variables like 'child_row_idx_' via the 'this' pointer on every loop iteration.
The compiler could do that automatically if it could deduce that the values aren't modified via a different pointer, but I don't think it's deducible in this case because the compiler has to generate code that's "correct" in a weird case like 'this' and *tuple_buf pointing to the same memory.
Line 34: if (ReachedLimit()) break;
We can avoid checking limits for each row if we check it at the end and truncate the batch using RowBatch::set_num_rows. E.g. see SortNode::GetNext().
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 148 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 5:
I reran the benchmark on patch 5 on a larger table where we select only 1 column:
SELECT
COUNT(c)
FROM (
select fnv_hash(ss_sold_time_sk) c from tpcds_10_parquet.store_sales_unpartitioned_big
union all
select fnv_hash(ss_sold_time_sk) c from tpcds_10_parquet.store_sales_unpartitioned_big
union all
select fnv_hash(ss_sold_time_sk) c from tpcds_10_parquet.store_sales_unpartitioned_big
union all
select fnv_hash(ss_sold_time_sk) c from tpcds_10_parquet.store_sales_unpartitioned_big
) t
Before: 17.6s
After: 9.98s
Not a huge difference. I think the bottleneck is scanning (not union), that's why the improvement is not as big. Maybe the difference will be more significant on a large cluster?
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 1:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS1, Line 28: TupleRow* dst_row = dst_batch->GetRow(dst_batch->AddRow());
Can this be not cross-compiled and we can make the caller pass dst_row directly instead ?
PS1, Line 32: dst_batch->tuple_data_pool()
Can the caller pass dst_batch->tuple_data_pool() to this function instead ? This avoid compiling code which won't provide much performance boost.
Line 34: dst_batch->CommitLastRow();
Also don't seem to be strictly needed to be cross-compiled. Please correct me if I misunderstood anything.
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS1, Line 40: NULL
nullptr
PS1, Line 108: DCHECK(codegen_status.ok());
Don't think it's safe to always assume codegen succeeded. It can fail for legitimate reason (e.g. we don't handle TYPE_CHAR). Please consider skipping this child expression if codegen for it failed.
PS1, Line 114: NULL
nullptr
PS1, Line 231: IsNodeCodegenDisabled()
Wouldn't that be implied by codegen_union_materialize_expr_fns_.size() ==0 ?
PS1, Line 233: codegend_union_materialize_exprs_fns_.data()[child_idx_] == NULL
Isn't this the only check we care about ? Are the other two conditions needed ?
PS1, Line 237: codegend_union_materialize_exprs_fns_.data()[child_idx_]
codegend_union_materialize_exprs_fns_[child_idx]
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 149 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#4).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 153 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/4
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#4).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 153 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/4
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 10:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/450/
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 1:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS1, Line 28: TupleRow* dst_row = dst_batch->GetRow(dst_batch->AddRow());
> Can this be not cross-compiled and we can make the caller pass dst_row dire
Done
PS1, Line 32: dst_batch->tuple_data_pool()
> Can the caller pass dst_batch->tuple_data_pool() to this function instead ?
Done
Line 34: dst_batch->CommitLastRow();
> Also don't seem to be strictly needed to be cross-compiled. Please correct
Done
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS1, Line 40: NULL
> nullptr
Done
PS1, Line 108: DCHECK(codegen_status.ok());
> Don't think it's safe to always assume codegen succeeded. It can fail for l
Done
PS1, Line 114: NULL
> nullptr
Done
PS1, Line 231: IsNodeCodegenDisabled()
> Wouldn't that be implied by codegen_union_materialize_expr_fns_.size() ==0
Done
PS1, Line 233: codegend_union_materialize_exprs_fns_.data()[child_idx_] == NULL
> Isn't this the only check we care about ? Are the other two conditions need
I rewrote this a little bit. I think we need at least 2 checks: that the size is not zero and that the current child is not null.
PS1, Line 237: codegend_union_materialize_exprs_fns_.data()[child_idx_]
> codegend_union_materialize_exprs_fns_[child_idx]
Done
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS1, Line 71: tuple_pool_;
> FWIW, this is probably not needed in the future as we avoid baking pointers
Ok, this can be removed in the future.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS2, Line 23: void UnionNode::MaterializeExprsInner(const vector<ExprContext*>& exprs,
: TupleRow* row, Tuple* dst_tuple, MemPool* pool) {
This seems like a thin-wrapper around Tuple::MaterializeExprs(). It seems unnecessary to cross-compile this function anymore. How about we just call the compiled IR function directly ?
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS2, Line 184: codegend_union_materialize_exprs_fns_.size() == 0
Would it be simpler to always initialize all entries of codegend_union_materialize_exprs_fns_ to nullptr ? Simpler logic is beneficial as this seems to be the critical path of union node.
PS2, Line 190: DCHECK(
DCHECK_LT
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 12:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/478/
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 12
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 8: Code-Review+1
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/6459/8/be/src/codegen/gen_ir_descriptions.py
File be/src/codegen/gen_ir_descriptions.py:
PS8, Line 183: ["UNION_MATERIALIZE_BATCH",
: "_ZN6impala9UnionNode16MaterializeBatchEPNS_8RowBatchEPPh"],
> nit: please consider putting it in a different entry so as not to break up
Done
http://gerrit.cloudera.org:8080/#/c/6459/8/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS8, Line 223: codegend_union_materialize_batch_fns_.data()
> Why not codegend_union_materialize_batch_fns_[child_idx_] ? Same below.
Done
Line 290: int num_rows_before = row_batch->num_rows();
> So, in that case, is it true that the rows in the non-empty batch are alrea
Yes, if a batch is not empty, the rows in it should already be counted towards num_rows_returned_ at this point. If on line 295, some rows get added the the batch, those are not added to num_rows_returned_ yet. num_rows_returned gets updated on line 306.
In other words, num_rows_returned gets updated only in one place: line 306.
PS8, Line 304: num_rows_added
> I probably misunderstood something here. Can you please explain why we need
Turns out this formula was wrong. I added a test case added a test case (which fails on Patch Set 8 and passes on the latest patch set).
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#4).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
5 files changed, 161 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/4
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#2).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 131 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 6:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS6, Line 29: dst_batch->tuple_data_pool()
> is this where that mem-pool in the class is used when codegening? Why not
Done.
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 98:
> Extra blank lines.
Done
PS6, Line 113: abort codegen entirely.
> this doesn't seem to match the code. wouldn't we have already filled in cod
Reprased. Added a test case.
Line 235: if (ReachedLimit()) {
> I don't think this can be taken now, right? Since we don't increment num_ro
Removed completely. I considered adding some extra logic to calculate if we reached the limit and the child can be closed, but I decided not to (to reduce complexity and it makes the code uglier for this special case). Let me know if you disagree with this decision.
Basically as a result of this, if we hit the limit, the child will be closed in UnionNode.Close() instead of here.
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS6, Line 68: Temporarily stores references to materialized tuples by a codegened function
> i'm not sure what this means. MemPools don't store references, they contain
Rewrote the comment. Also changed it so that mem-pool is used by the non-codegen version as well.
PS6, Line 71: TODO: Remove this when no longer necessary in the future.
> do we have a concrete jira for when this won't be necessary?
I couldn't find one, but I think Michael knows more about this.
PS6, Line 96: Jitted
> usually we say "codegen'ed"
Done.
I wrote it like this because I saw it written like that in other files such as (partitioned-aggregation-node.h).
Line 116: /// call on the child.
> Maybe mention that these GetNext* functions don't apply the limit?
Done. I wasn't really sure where to put the comment, so I put it at the top.
PS6, Line 127: Increments 'num_rows_returned_' and
: /// 'child_row_idx_'
> I think this needs an update.
Done
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6459/4/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
Line 28: while (!dst_batch->AtCapacity() && child_row_idx_ < child_batch_->num_rows()) {
> The remaining optimisation is to pull all references to member variables ou
Done. I think I took everything out. Let me know if you see any other ways to further improve performance.
Line 34: if (ReachedLimit()) break;
> We can avoid checking limits for each row if we check it at the end and tru
Good idea, done. (We already do a similar thing in the passthrough case)
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Dan Hecht, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#10).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 202 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/10
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
5 files changed, 163 insertions(+), 58 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#10).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 202 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/10
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#8).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 183 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/8
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#5).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 168 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/5
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 7:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6459/7/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS7, Line 34: RuntimeState* state
> Is this not used at all ?
Good catch, removed.
PS7, Line 45: child_row_idx < num_child_batch_rows) {
> There is a macro called FOREACH_ROW() which is useful for iterating through
Done
http://gerrit.cloudera.org:8080/#/c/6459/6/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS6, Line 71: the pointer to the mem pool is hard coded in the codegen'
> I couldn't find one, but I think Michael knows more about this.
Created IMPALA-5192
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 5:
(13 comments)
Getting pretty close, just minor cleanup at this point.
I also just wanted to check with Michael that the tuple_pool_ approach made the most sense for now - we'll need to clean that up as part of his codegen work but I don't think it makes sense to fix in this patch.
http://gerrit.cloudera.org:8080/#/c/6459/4/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
Line 34:
> We can avoid checking limits for each row if we check it at the end and tru
It looks like we already do this for the passthrough case so we might as well do it here.
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
Line 19: #include "runtime/tuple.h"
Do we need tuple.h? I don't think I see any references to Tuple* in here.
Line 21: #include "util/runtime-profile-counters.h"
Is this needed still?
Line 35: while (!dst_batch->AtCapacity() && child_row_idx < child_batch->num_rows()) {
Nice! We can maybe avoid a few more loads and stores via the child_batch and tuple_buf pointers. I.e.
int child_batch_rows = child_batch->num_rows().
uint8_t* curr_tuple = *tuple_buf;
...
*tuple_buf = curr_tuple.
Line 46: if (limit_ != -1 && num_rows_returned_ + dst_batch->num_rows() > limit_) {
We don't need to cross-compile this logic. Let's move it into the caller and save LLVM some work.
Although, see my comment about moving this logic to GetNext() and sharing it for all three codepaths.
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 168: if (limit_ != -1 && num_rows_returned_ + row_batch->num_rows() > limit_) {
How about we move this logic around num_rows_returned_ and limit_ into GetNext()? I believe the same logic can work for all three cases if we slightly extend it so that it handles the case when GetNext() is called with a non-empty batch, which can happen in a subplan.
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS1, Line 71: in this poo
> Ok, this can be removed in the future.
Michael, do you think it makes sense to use this approach for now? I'm not that familiar with CodegenMaterializeExprs() so not sure if there is a better way to do this.
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node.h
File be/src/exec/union-node.h:
Line 28: #include "runtime/tuple.h"
Do we need the tuple.h and tuple-row.h imports? Oh I guess for the inline MaterializeExprs function, but we can move that to the -ir.cc file anyway.
Line 72: /// each GetNext() call.
We should add a TODO to remove this. Maybe Michael knows if there's a JIRA that will allow us to remove it.
PS5, Line 99: Null
NULL here and below, just for consistency with other comments.
PS5, Line 128: row_batch
dst_batch.
Line 136: void IR_ALWAYS_INLINE MaterializeExprs(const std::vector<ExprContext*>& exprs,
Move this to the -ir.cc file? I don't think there's a reason we need to define it in the .h
Line 148: bool inline IsChildPassthrough(int child_idx) const {
I don't think any of the "inline" specifiers here and below do anything - if the function is defined in the class body it implicitly has an "inline" hint.
https://isocpp.org/wiki/faq/inline-functions#inline-member-fns-more
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#5).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 168 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/5
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, Michael Ho, Dan Hecht, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#13).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 203 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/13
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#7).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 187 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/7
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#8).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 185 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/8
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#13).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 203 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/13
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 10: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS1, Line 71: tuple_pool_;
FWIW, this is probably not needed in the future as we avoid baking pointers into generated IR. Looks like I missed this one case in Tuple::CodegenMaterializeExprs().
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#2).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 131 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 128 insertions(+), 22 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 2:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS2, Line 23: void UnionNode::MaterializeExprsInner(const vector<ExprContext*>& exprs,
: TupleRow* row, Tuple* dst_tuple, MemPool* pool) {
> This seems like a thin-wrapper around Tuple::MaterializeExprs(). It seems u
Changed it so that this function does more work (changed function name too).
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 26:
> unnecessary new line
Done
Line 110: if (!codegen_status.ok()) continue;
> We shouldn't drop this status - at the least we should display it in the pr
Done, aborting codegen in case of failure.
Line 129: runtime_profile()->AddCodegenMsg(codegen_status.ok(), codegen_status);
> How does this show up in the profile? If we're going to show the results of
Changed it so that only 1 message is displayed. If there is a failure, we indicate which child node caused it.
Line 184: if (UNLIKELY(codegend_union_materialize_exprs_fns_.size() == 0 ||
> Just to confirm my suspicion, it would be good to run "perf top" while your
-Initialize all entries to nullptr as suggested.
-Pushed the loop into into codegened code and got a few seconds improvement (25 sec vs 22)
I think the new implementation is equivalent to what you suggested (except we use a while loop instead of a for loop). Do you see any other ways to optimize the code further?
PS2, Line 190: DCHECK(
> DCHECK_LT
Code was reorganized and this was removed.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 13:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/490/
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 13
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
TODO: Benchmark
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 150 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 151 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#8).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 183 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/8
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Reviewed-on: http://gerrit.cloudera.org:8080/6459
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 203 insertions(+), 57 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Alex Behm: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 14
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 8:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/6459/8/be/src/codegen/gen_ir_descriptions.py
File be/src/codegen/gen_ir_descriptions.py:
PS8, Line 183: ["UNION_MATERIALIZE_BATCH",
: "_ZN6impala9UnionNode16MaterializeBatchEPNS_8RowBatchEPPh"],
nit: please consider putting it in a different entry so as not to break up READ_AVRO* sequence.
http://gerrit.cloudera.org:8080/#/c/6459/8/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
PS8, Line 223: codegend_union_materialize_batch_fns_.data()
Why not codegend_union_materialize_batch_fns_[child_idx_] ? Same below.
Line 290: int num_rows_before = row_batch->num_rows();
So, in that case, is it true that the rows in the non-empty batch are already counted towards num_rows_returned_ ? Line 303 seems to imply so but I could be wrong.
PS8, Line 304: num_rows_added
I probably misunderstood something here. Can you please explain why we need to subtract num_rows_added too ?
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#2).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 131 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#2).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 125 insertions(+), 22 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 1:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6459/7/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
PS7, Line 34: t_batch->CommitLast
Is this not used at all ?
PS7, Line 45:
There is a macro called FOREACH_ROW() which is useful for iterating through rows in a row batch. It will save some cycles as it avoids the multiplication in GetRow().
http://gerrit.cloudera.org:8080/#/c/6459/1/be/src/exec/union-node.h
File be/src/exec/union-node.h:
PS1, Line 71: tuple_pool_;
> Michael, do you think it makes sense to use this approach for now? I'm not
I agree that we should not mix in the fix for CodegenMaterializeExprs() in this patch. We probably need to fix other places such as TopNNode.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#12).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-subplan.test
M testdata/workloads/functional-query/queries/QueryTest/union.test
8 files changed, 202 insertions(+), 57 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/12
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 12
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 184: if (UNLIKELY(codegend_union_materialize_exprs_fns_.size() == 0 ||
> It would be great if we could avoid all of this per-row overhead - I'm coun
Just to confirm my suspicion, it would be good to run "perf top" while your benchmark is running. My guess is that it will spend a lot of time in MaterializeExprs()/GetNextMaterialized() . Ideally when we codegen something it should spend almost all of the time in the codegen'd code.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#7).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 168 insertions(+), 45 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/7
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Dan Hecht, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#9).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 196 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/9
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Dan Hecht, Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#9).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 196 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/9
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 150 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Hello Tim Armstrong,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6459
to look at the new patch set (#7).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 170 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/7
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 12: Code-Review+2
Rebased. Forwarding Dan's +2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 12
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 4:
I ran the query above on Patch Set 4 as well, complete results:
Apache/master: 17.60s
Patch set 4: 10.58s
Patch set 5: 9.98s
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 4:
Restructured the code a bit in patch 4 vs 3, and fixed some minor comment issues.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#3).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 149 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/3
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#7).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 187 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/7
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change.
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
Patch Set 2:
(4 comments)
The code looks generally good. I had two high-level comments: we should make sure that the output in the profile makes sense, and we should try to rearrange the inner perf-critical loop to squeeze out more perf (I think we can do much better than 50%).
http://gerrit.cloudera.org:8080/#/c/6459/2/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 26:
unnecessary new line
Line 110: if (!codegen_status.ok()) continue;
We shouldn't drop this status - at the least we should display it in the profile. It's also reasonable to abort codegen for the node entirely if it simplifies things - the cases when codegen fails are mainly corner cases like CHAR.
Line 129: runtime_profile()->AddCodegenMsg(codegen_status.ok(), codegen_status);
How does this show up in the profile? If we're going to show the results of codegening each union branch we should make sure that it is displayed in a reasonable way - e.g. mention the child number.
Line 184: if (UNLIKELY(codegend_union_materialize_exprs_fns_.size() == 0 ||
It would be great if we could avoid all of this per-row overhead - I'm counting a function call, three condition checks and an indirect function call via the function pointer before it does any actual work for the row, and that's just including this function, not the caller.
We usually try to push loops like this down into the codegen'd code. I think in this case the loop can be tightened up a lot and pushed down into cross-compiled code. If you have a for loop over (0 ..rows left in child batch), I think the only thing that really needs to be checked per row is the capacity of the destination batch - the other loop conditions can be converted into an iteration count before entering the loop.
HdfsParquetScanner::ProcessScratchBatch() is an example where all of this is pretty optimised.
I don't think that fully optimising it should block this getting in since a good improvement already, but there's so much low-hanging fruit that we're missing out on a lot of the potential benefit of your work.
Also if we leave code sitting around with inefficient patterns like this people tend to copy the bad patterns (e.g. the Kudu scan node inherited some bad patterns from the HBase scan node).
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#9).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s177ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 174.617us 174.617us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.693us 16.693us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s830ms 3s615ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s296ms 5s258ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s212ms 1s340ms 28.80M 28.80M 488.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s387ms 1s570ms 28.80M 28.80M 489.37 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s224ms 1s347ms 28.80M 28.80M 487.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s245ms 1s321ms 28.80M 28.80M 489.25 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s232ms 1s505ms 28.80M 28.80M 484.21 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s348ms 1s518ms 28.80M 28.80M 488.20 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s231ms 1s335ms 28.80M 28.80M 483.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s179ms 1s349ms 28.80M 28.80M 482.76 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s121ms 1s154ms 28.80M 28.80M 486.59 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s284ms 1s523ms 28.80M 28.80M 486.70 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
M testdata/workloads/functional-query/queries/QueryTest/union.test
7 files changed, 196 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/9
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#5).
Change subject: IMPALA-4883: Union Codegen
......................................................................
IMPALA-4883: Union Codegen
For each non-passthrough child of the Union node, codegen the loop that
does per row tuple materialization.
Testing:
Ran test_queries.py test locally in exchaustive mode.
Benchmark:
Ran a local benchmark on a local 10 GB TPCDS dataset on an unpartitioned
store_sales table.
SELECT
COUNT(c),
COUNT(ss_customer_sk),
COUNT(ss_cdemo_sk),
COUNT(ss_hdemo_sk),
COUNT(ss_addr_sk),
COUNT(ss_store_sk),
COUNT(ss_promo_sk),
COUNT(ss_ticket_number),
COUNT(ss_quantity),
COUNT(ss_wholesale_cost),
COUNT(ss_list_price),
COUNT(ss_sales_price),
COUNT(ss_ext_discount_amt),
COUNT(ss_ext_sales_price),
COUNT(ss_ext_wholesale_cost),
COUNT(ss_ext_list_price),
COUNT(ss_ext_tax),
COUNT(ss_coupon_amt),
COUNT(ss_net_paid),
COUNT(ss_net_paid_inc_tax),
COUNT(ss_net_profit),
COUNT(ss_sold_date_sk)
FROM (
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
union all
select fnv_hash(ss_sold_time_sk) c, * from tpcds_10_parquet.store_sales_unpartitioned
) t
Before: 39s704ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 194.504us 194.504us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 17.284us 17.284us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s202ms 2s934ms 3 1 115.00 KB 10.00 MB
00:UNION 3 32s514ms 34s926ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 158.373ms 216.085ms 28.80M 28.80M 489.71 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 167.002ms 171.738ms 28.80M 28.80M 489.74 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 125.331ms 145.496ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 148.478ms 194.311ms 28.80M 28.80M 489.69 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 143.995ms 162.781ms 28.80M 28.80M 489.57 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 169.731ms 250.201ms 28.80M 28.80M 489.58 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 164.110ms 254.374ms 28.80M 28.80M 489.61 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 135.631ms 162.117ms 28.80M 28.80M 489.63 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 138.736ms 167.778ms 28.80M 28.80M 489.67 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 202.015ms 248.728ms 28.80M 28.80M 489.68 MB 1.88 GB tpcds_10_parquet.store_sales
After: 20s664ms
Operator #Hosts Avg Time Max Time #Rows Est. #Rows Peak Mem Est. Peak Mem Detail
------------------------------------------------------------------------------------------------------------------------------
13:AGGREGATE 1 167.757us 167.757us 1 1 28.00 KB -1.00 B FINALIZE
12:EXCHANGE 1 16.592us 16.592us 3 1 0 -1.00 B UNPARTITIONED
11:AGGREGATE 3 2s924ms 3s715ms 3 1 115.00 KB 10.00 MB
00:UNION 3 4s971ms 6s082ms 288.01M 288.01M 3.08 MB 0
|--02:SCAN HDFS 3 1s189ms 1s588ms 28.80M 28.80M 483.82 MB 1.88 GB tpcds_10_parquet.store_sales
|--03:SCAN HDFS 3 1s117ms 1s157ms 28.80M 28.80M 484.85 MB 1.88 GB tpcds_10_parquet.store_sales
|--04:SCAN HDFS 3 1s226ms 1s454ms 28.80M 28.80M 483.00 MB 1.88 GB tpcds_10_parquet.store_sales
|--05:SCAN HDFS 3 1s141ms 1s380ms 28.80M 28.80M 486.90 MB 1.88 GB tpcds_10_parquet.store_sales
|--06:SCAN HDFS 3 1s180ms 1s292ms 28.80M 28.80M 488.22 MB 1.88 GB tpcds_10_parquet.store_sales
|--07:SCAN HDFS 3 1s227ms 1s391ms 28.80M 28.80M 481.99 MB 1.88 GB tpcds_10_parquet.store_sales
|--08:SCAN HDFS 3 1s169ms 1s277ms 28.80M 28.80M 483.67 MB 1.88 GB tpcds_10_parquet.store_sales
|--09:SCAN HDFS 3 1s192ms 1s366ms 28.80M 28.80M 484.92 MB 1.88 GB tpcds_10_parquet.store_sales
|--10:SCAN HDFS 3 1s189ms 1s298ms 28.80M 28.80M 487.83 MB 1.88 GB tpcds_10_parquet.store_sales
01:SCAN HDFS 3 1s239ms 1s314ms 28.80M 28.80M 481.33 MB 1.88 GB tpcds_10_parquet.store_sales
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 164 insertions(+), 41 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/5
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-4883: Union Codegen
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has posted comments on this change.
Change subject: IMPALA-4883: Union Codegen
......................................................................
Patch Set 5:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node-ir.cc
File be/src/exec/union-node-ir.cc:
Line 19: #include "runtime/tuple.h"
> Do we need tuple.h? I don't think I see any references to Tuple* in here.
Done
Line 21: #include "util/runtime-profile-counters.h"
> Is this needed still?
Done
Line 35: while (!dst_batch->AtCapacity() && child_row_idx < child_batch->num_rows()) {
> Nice! We can maybe avoid a few more loads and stores via the child_batch an
Great suggestions. Done.
Line 46: if (limit_ != -1 && num_rows_returned_ + dst_batch->num_rows() > limit_) {
> We don't need to cross-compile this logic. Let's move it into the caller an
Good point. Moved it out of here.
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node.cc
File be/src/exec/union-node.cc:
Line 168: if (limit_ != -1 && num_rows_returned_ + row_batch->num_rows() > limit_) {
> How about we move this logic around num_rows_returned_ and limit_ into GetN
Done
http://gerrit.cloudera.org:8080/#/c/6459/5/be/src/exec/union-node.h
File be/src/exec/union-node.h:
Line 28: #include "runtime/tuple.h"
> Do we need the tuple.h and tuple-row.h imports? Oh I guess for the inline M
Done
Line 72: /// each GetNext() call.
> We should add a TODO to remove this. Maybe Michael knows if there's a JIRA
Added a todo, couldn't find the relevant JIRA.
PS5, Line 99: Null
> NULL here and below, just for consistency with other comments.
Done
PS5, Line 128: row_batch
> dst_batch.
Done
Line 136: void IR_ALWAYS_INLINE MaterializeExprs(const std::vector<ExprContext*>& exprs,
> Move this to the -ir.cc file? I don't think there's a reason we need to def
Ok, moved it.
Line 148: bool inline IsChildPassthrough(int child_idx) const {
> I don't think any of the "inline" specifiers here and below do anything - i
Makes sense, removed all of them.
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4883: Implement Codegen for Union
Posted by "Taras Bobrovytsky (Code Review)" <ge...@cloudera.org>.
Taras Bobrovytsky has uploaded a new patch set (#2).
Change subject: IMPALA-4883: Implement Codegen for Union
......................................................................
IMPALA-4883: Implement Codegen for Union
For each non-passthrough child of the Union node, codegen
Tuple::MarializeExprs().
Testing:
Ran test_queries.py test locally in exchaustive mode.
Ran some hand crafted performance benchmark queries locally. There is
an up to 50% performance improvement.
Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/union-node-ir.cc
M be/src/exec/union-node.cc
M be/src/exec/union-node.h
6 files changed, 131 insertions(+), 23 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/59/6459/2
--
To view, visit http://gerrit.cloudera.org:8080/6459
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ib4107d27582ff5416172810364a6e76d3d93c439
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Taras Bobrovytsky <tb...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>