You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org> on 2017/10/02 23:17:57 UTC

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Bikramjeet Vig has uploaded this change for review. ( http://gerrit.cloudera.org:8080/8196


Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Preformance:
Query run on tpch_parquet.lineitem with perdicates of the form
[column_name > value AND]

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
6 files changed, 96 insertions(+), 27 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/1
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#9).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.
Improved test coverage for select node with limit.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M testdata/workloads/functional-query/queries/QueryTest/inline-view-limit.test
M tests/query_test/test_codegen.py
8 files changed, 131 insertions(+), 62 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/9
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 9
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 6:

(6 comments)

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node-ir.cc@29
PS6, Line 29: while (child_row_idx_ < child_batch_num_rows)
> Have you looked into using FOREACH_ROW_LIMIT() ? May have some complication
Done


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h
File be/src/exec/select-node.h:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@40
PS6, Line 40:   virtual void Codegen(RuntimeState* state);
> While you are at it, would you mind adding override to these functions ?
Done


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@63
PS6, Line 63: codegend_copy_rows_fn_;
> Feel free to ignore but I think we have recently switched to initializing f
I should do that change for all the member variables. Do you recommend I make that change in this commit?


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@69
PS6, Line 69: /// Codegen CopyRows().
> May help to explain that this is mostly codegen'ing the conjuncts evaluatio
Done


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc@91
PS6, Line 91:   if (ReachedLimit() || (child_row_idx_ == child_row_batch_->num_rows() && child_eos_)) {
> This is still a bit wonky (not your change but we should try to leave this 
Done


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc@107
PS6, Line 107:       child_row_batch_->TransferResourceOwnership(row_batch);
> Sorry for the additional churn but I think the resource transfer is overly 
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Oct 2017 21:46:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#7).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M tests/query_test/test_codegen.py
7 files changed, 121 insertions(+), 64 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/7
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 3:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc@113
PS3, Line 113: copy_rows_success
> (I'm not sure if that description I just gave is accurate or the best way t
true! the description is a bit more involved to capture it in a variable name. 
I think a generic name like "copy_row_result" is good enough and the function description in the header file is sufficient to understand that result.
what do u think?


http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc@126
PS3, Line 126:     if (child_eos_) {
> The control flow here is confusing to me. It took me a while to figure out 
Done


http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py
File tests/query_test/test_codegen.py:

http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py@46
PS3, Line 46: codegned
> missing an e
Done


http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py@51
PS3, Line 51:     assert len(exec_options)
> This is a matter of taste but generally we prefer being explcit about the c
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Oct 2017 00:46:19 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h
File be/src/exec/select-node.h:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@70
PS6, Line 70:   Status CodegenCopyRows(RuntimeState* state);
> WARN_UNUSED_RESULT for Status doesn't provide any additional coverage since
Good to know. Apparently missed this change while I was gone.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Oct 2017 01:01:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 1:

(11 comments)

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@8
PS1, Line 8: 
> Can you add a testing section? It would be good to mention what test covera
is there anyway we can make sure through a test that codegen was successful for this node?


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@9
PS1, Line 9: Preformance
> Performance
Done


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@10
PS1, Line 10: perdicates
> predicates
Done


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@11
PS1, Line 11: [column_name > value AND]
> Can you include a concrete example of the query to make it easier for other
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc@29
PS1, Line 29: child_row_batch_->num_rows()
> This probably doesn't matter, but you could save a few instructions by hois
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc@41
PS1, Line 41:       COUNTER_SET(rows_returned_counter_, num_rows_returned_);
> COUNTER_SET can be pretty expensive because of the atomic operation. We can
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@55
PS1, Line 55: NULL
> Use nullptr in new code (here and below).
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@65
PS1, Line 65:   if (codegen_status.ok()) {
> Instead of this branch the control flow may be easier to follow if you fact
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@71
PS1, Line 71:     DCHECK(copy_rows_fn != NULL);
> We should probably check if this is NULL and fail codegen if so. I'm not su
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@99
PS1, Line 99:   bool copy_rows_success = false;
> Just declare it inside the loop where it's needed? When I see it declared u
Done


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@114
PS1, Line 114:     copy_rows_success = false;
> It's assigned on both branches so this isn't necessary.
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2017 20:52:29 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#2).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
6 files changed, 99 insertions(+), 27 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/2
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 1:

(12 comments)

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@8
PS1, Line 8: 
Can you add a testing section? It would be good to mention what test coverage we have for this code path (it's certainly possible that we have some gaps).


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@9
PS1, Line 9: Preformance
Performance


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@10
PS1, Line 10: perdicates
predicates


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@11
PS1, Line 11: [column_name > value AND]
Can you include a concrete example of the query to make it easier for others to reproduce?


http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@18
PS1, Line 18: | Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
Nice!


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc@29
PS1, Line 29: child_row_batch_->num_rows()
This probably doesn't matter, but you could save a few instructions by hoisting this out of the loop. E.g.

  int child_batch_num_rows = ...;

Pretty sure the compiler can't infer that num_rows() is constant across loop iterations (since 'this' and 'this->child_row_batch_' could alias memory that's written in the loop so it will end up chasing the pointers on every iteration.


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node-ir.cc@41
PS1, Line 41:       COUNTER_SET(rows_returned_counter_, num_rows_returned_);
COUNTER_SET can be pretty expensive because of the atomic operation. We can actually move it out of this function entirely into the caller.


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@55
PS1, Line 55: NULL
Use nullptr in new code (here and below).


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@65
PS1, Line 65:   if (codegen_status.ok()) {
Instead of this branch the control flow may be easier to follow if you factor out the CopyRows codegen logic into a separate Status-returning function that returns early on an error.

This will be more true if you start checking FinalizeFunction() below.


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@71
PS1, Line 71:     DCHECK(copy_rows_fn != NULL);
We should probably check if this is NULL and fail codegen if so. I'm not sure if we do this everywhere in the code but we probably should be doing so (although it isn't super-important, since we shouldn't be failing verification in the first place).

(The invariants around this function are a bit wonky at the moment since we don't really have good testing around this, but FinalizeFunction() is documented as returning NULL in some cases).


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@99
PS1, Line 99:   bool copy_rows_success = false;
Just declare it inside the loop where it's needed? When I see it declared up here I tend to think that the value lives across loop iterations.


http://gerrit.cloudera.org:8080/#/c/8196/1/be/src/exec/select-node.cc@114
PS1, Line 114:     copy_rows_success = false;
It's assigned on both branches so this isn't necessary.

I think some people consider this good defensive coding practice but it seems unnecessary to me.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 1
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Oct 2017 23:40:05 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc@113
PS4, Line 113: copy_rows_result;
I think it would still be better to have a name that makes the below control flow more understandable, if we can come up with one. It is unnecessarily confusing right now.

After looking at this again with the benefit of some time, I think having CopyRows() returning a bool is unnecessarily confusing because it's telling the caller about the current state in a non-obvious way, yet the caller can check the state directly.


http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc@120
PS4, Line 120:     if (copy_rows_result) {
I think there's an equivalent but simpler way of expressing the control flow along the lines of:

  *eos = ReachedLimit() || (child_row_idx_ == child_row_batch->num_row() && child_eos_);
  if (*eos) {
    ... do the eos things ...
    return Status::OK();
  }
  if (row_batch->AtCapacity()) return Status::OK();

I don't think your change made this more confusing but I think it would be good to fix it so that the control flow is more obviously correct.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 04 Oct 2017 18:58:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 6:

(5 comments)

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node-ir.cc@29
PS6, Line 29: while (child_row_idx_ < child_batch_num_rows)
Have you looked into using FOREACH_ROW_LIMIT() ? May have some complication with bumping child_row_idx_ at the end but worth looking into it.


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h
File be/src/exec/select-node.h:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@40
PS6, Line 40:   virtual void Codegen(RuntimeState* state);
While you are at it, would you mind adding override to these functions ?


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@63
PS6, Line 63: codegend_copy_rows_fn_;
Feel free to ignore but I think we have recently switched to initializing fields directly in the class declaration here:
 - codegend_copy_rows_fn_ = nullptr;


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@69
PS6, Line 69: /// Codegen CopyRows().
May help to explain that this is mostly codegen'ing the conjuncts evaluation logic so as to provide more context.


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@70
PS6, Line 70:   Status CodegenCopyRows(RuntimeState* state);
WARN_UNUSED_RESULT



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Oct 2017 00:36:21 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#8).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.
Improved test coverage for select node with limit.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M testdata/workloads/functional-query/queries/QueryTest/inline-view-limit.test
M tests/query_test/test_codegen.py
8 files changed, 131 insertions(+), 64 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/8
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 8: Code-Review+1

Will let Michael take another look.


-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2017 21:41:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#3).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M tests/query_test/test_codegen.py
7 files changed, 110 insertions(+), 27 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/3
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 9: Code-Review+2

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@72
PS8, Line 72: Status(
> I believe finalize function does not fail frequently, so shouldn't affect p
It's probably quite clear what the call path is since this is the codegen function for an exec node (as opposed to an Expr). It's true that failure should rarely occur on this path so may be it's not that bad after all. If anything, I would print the IR function which fails to finalize if we aren't doing it already.


http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py
File tests/query_test/test_codegen.py:

http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py@52
PS9, Line 52:     assert_codegen_enabled(result.runtime_profile, [1])
Optionally, you can consider adding a test in which codegen fails. Not strictly needed for this patch.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 9
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Oct 2017 23:26:39 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@a69
PS8, Line 69: 
            : 
> Why are these not needed anymore ?
Thanks for catching this, I missed that it was removed.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Oct 2017 19:22:41 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 11:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py
File tests/query_test/test_codegen.py:

http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py@52
PS9, Line 52:     assert_codegen_enabled(result.runtime_profile, [1])
> Do we already have tests that check for codegen failed?
I will try something which involves a char column.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 11
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 26 Oct 2017 22:48:11 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 6:

(2 comments)

Your code looks good but I'd really like to simplify this code further so that it's easier to understand in future.

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc@91
PS6, Line 91:   if (ReachedLimit() || (child_row_idx_ == child_row_batch_->num_rows() && child_eos_)) {
This is still a bit wonky (not your change but we should try to leave this in a good state) - it's not clear when this will actually be taken. 


The invariant around GetNext() is that callers shouldn't call GetNext() after 'eos' is true. Given that invariant, I don't think it's possible that this is taken.

* ReachedLimit() should never be true here. limit_ == 0 should be caught by the frontend and converted into an EmptySetNode. If limit_ > 0, then we should have set *eos after copying the rows below.
* If we hit the end of the input below on line 120, we should already have set *eos.


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.cc@107
PS6, Line 107:       child_row_batch_->TransferResourceOwnership(row_batch);
Sorry for the additional churn but I think the resource transfer is overly complicated. This actually will often attach the resources to the batch after the last batch that references them. Instead it should attach resources to the batch as soon as the last row from the input batch is copied to the output.

I'm thinking removing the transfer and AtCapacity() check here and changing the logic below to something like:

  if (*eos || child_row_idx_ == child_row_batch_->num_rows()) {
    child_row_batch_->TransferResourceOwnership(row_batch);
    child_row_batch_->Reset();
  }
  if (*eos || row_batch->AtCapacity()) return Status::OK();

Or maybe it would then be clearer as a do/while() loop.

This would require some further changes, since it would reset child_row_batch_->num_rows() to 0 - but maybe child_row_batch_->num_rows() == 0 should actually just signal that the batch has been consumed.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 09 Oct 2017 18:56:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 4:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc@113
PS4, Line 113: copy_rows_result;
> I think it would still be better to have a name that makes the below contro
removed. see below comment.


http://gerrit.cloudera.org:8080/#/c/8196/4/be/src/exec/select-node.cc@120
PS4, Line 120:     if (copy_rows_result) {
> I think there's an equivalent but simpler way of expressing the control flo
You are absolutely right. I have refactored the code which now also make the bool returned by CopyRows() redundant. Therefore I've changed CopyRows()'s return type to void.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Sat, 07 Oct 2017 00:53:46 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/1382/


-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 10
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Oct 2017 21:18:55 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.
Improved test coverage for select node with limit.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Reviewed-on: http://gerrit.cloudera.org:8080/8196
Reviewed-by: Bikramjeet Vig <bi...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M testdata/workloads/functional-query/queries/QueryTest/inline-view-limit.test
M tests/query_test/test_codegen.py
8 files changed, 131 insertions(+), 62 deletions(-)

Approvals:
  Bikramjeet Vig: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 11
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h
File be/src/exec/select-node.h:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@70
PS6, Line 70:   Status CodegenCopyRows(RuntimeState* state);
> WARN_UNUSED_RESULT
WARN_UNUSED_RESULT for Status doesn't provide any additional coverage since the upgrade to Clang 3.9.1 which supports [[nodiscard]]. GCC 4.9.2 doesn't properly handle WARN_UNUSED_RESULT when returning structs/classes so we depend on clang-tidy to enforce it anyway. We could consider removing it in most places.



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Oct 2017 00:42:00 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@a69
PS8, Line 69: 
            : 
> Why are these not needed anymore ?
Done


http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@72
PS8, Line 72: Status(
> Will it be sufficient to use Status::Expected() for this ?
I believe finalize function does not fail frequently, so shouldn't affect perf if it writes to log. Also if it ever does fail, its probably worth printing the stacktrace to the logs.
What do you think?



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Oct 2017 23:03:37 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 10
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 25 Oct 2017 01:17:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 10: Code-Review+2

Rebased. carrying over Michael's +2


-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 10
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Oct 2017 21:17:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 3:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc@113
PS3, Line 113: copy_rows_success
> I did another pass over the code and realised this doesn't really mean "suc
(I'm not sure if that description I just gave is accurate or the best way to describe it - but we do need a better name).



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2017 23:19:04 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 9:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@72
PS8, Line 72: Status(
> It's probably quite clear what the call path is since this is the codegen f
both the message in the Status obj created and the information logged by VerifyFunction() provide context related to the failed finalize call.


http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py
File tests/query_test/test_codegen.py:

http://gerrit.cloudera.org:8080/#/c/8196/9/tests/query_test/test_codegen.py@52
PS9, Line 52:     assert_codegen_enabled(result.runtime_profile, [1])
> Optionally, you can consider adding a test in which codegen fails. Not stri
Do we already have tests that check for codegen failed?
Also is there a way I can induce a failure in codegen?



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 9
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 24 Oct 2017 20:45:53 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 8:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@a69
PS8, Line 69: 
            : 
Why are these not needed anymore ?


http://gerrit.cloudera.org:8080/#/c/8196/8/be/src/exec/select-node.cc@72
PS8, Line 72: Status(
Will it be sufficient to use Status::Expected() for this ?



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 8
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 23 Oct 2017 19:03:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#5).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M tests/query_test/test_codegen.py
7 files changed, 111 insertions(+), 36 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/5
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 5
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 2:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/8196/1//COMMIT_MSG@8
PS1, Line 8: 
> is there anyway we can make sure through a test that codegen was successful
as discussed, I have added the test to check the query profile



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 2
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2017 22:54:54 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#6).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M tests/query_test/test_codegen.py
7 files changed, 110 insertions(+), 36 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/6
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 6
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 7:

(4 comments)

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node-ir.cc@28
PS7, Line 28: limit_
This could return extra rows. E.g. if the limit for the node is 1025, we returned a batch of 1024 rows and are now processing another input batch of 1024 rows, then this would return another 1024 rows instead of only 1 more.

It seems like we might be missing some test coverage here of select nodes with limits (it may also require a single node plan if there's an exchange inserted otherwise).

I think the old ReachedLimit() check plus FOREACH_ROW would work.


http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h
File be/src/exec/select-node.h:

http://gerrit.cloudera.org:8080/#/c/8196/6/be/src/exec/select-node.h@70
PS6, Line 70:   Status CodegenCopyRows(RuntimeState* state);
> Good to know. Apparently missed this change while I was gone.
Yeah, there were actually two LLVM changes that might be of interest to you: https://gerrit.cloudera.org/#/q/status:merged+project:Impala-ASF+branch:master+topic:llvm-upgrade


http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc@91
PS7, Line 91:   DCHECK(*eos == false);
This isn't safe - the GetNext() API doesn't require that the caller initializes the output parameter. (IMO eos should be part of the ExecNode state rather than something transiently set as an output parameter, but that's a different discussion!)


http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc@112
PS7, Line 112: 
nit: remove extra vertical whitespace



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 10 Oct 2017 22:38:56 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Hello Michael Ho, Tim Armstrong, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/8196

to look at the new patch set (#4).

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................

IMPALA-4236: Codegen CopyRows() for select nodes

Testing:
Added test case to verify that CopyRows in select node is successfully
codegened.

Performance:
Queries used (num_nodes set to 1):
500 Predicates: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 and l_extendedprice > 10000 and
l_linenumber > 1 and l_comment >'foo0' .... and l_comment >'foo500'
order by l_orderkey limit 10;

1 Predicate: select * from (select * from tpch_parquet.lineitem
limit 6001215) t1 where l_partkey > 10 order by l_orderkey limit 10;

+--------------+-----------------------------------------------------+
|              |      500 Predicates      |       1 Predicate        |
|              +------------+-------------+------------+-------------+
|              |   After    |   Before    |   After    |   Before    |
+--------------+------------+-------------+------------+-------------+
| Select Node  | 12s385ms   | 1m1s        | 234ms      | 797ms       |
| Codegen time | 2s619ms    | 1s962ms     | 200ms      | 181ms       |
+--------------+------------+-------------+------------+-------------+

Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
---
M be/src/codegen/gen_ir_descriptions.py
M be/src/codegen/impala-ir.cc
M be/src/exec/CMakeLists.txt
A be/src/exec/select-node-ir.cc
M be/src/exec/select-node.cc
M be/src/exec/select-node.h
M tests/query_test/test_codegen.py
7 files changed, 111 insertions(+), 27 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/96/8196/4
-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 4
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 3: Code-Review+1

(4 comments)

I'll give Michael Ho a chance to at least skim this and decide if we wants to take a look.

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc@113
PS3, Line 113: copy_rows_success
I did another pass over the code and realised this doesn't really mean "success", it really means that no more rows should be added to the output batch. So I think we need to rename this.


http://gerrit.cloudera.org:8080/#/c/8196/3/be/src/exec/select-node.cc@126
PS3, Line 126:     if (child_eos_) {
The control flow here is confusing to me. It took me a while to figure out that !copy_rows_success implies that child_row_idx_ == child_row_batch_->num_rows(). I think that's what the comment below is trying to say. Can you add a DCHECK here to clarify and enforce it, e.g.

  DCHECK_EQ(child_row_idx_, child_row_batch_->num_rows());


http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py
File tests/query_test/test_codegen.py:

http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py@46
PS3, Line 46: codegned
missing an e


http://gerrit.cloudera.org:8080/#/c/8196/3/tests/query_test/test_codegen.py@51
PS3, Line 51:     assert len(exec_options)
This is a matter of taste but generally we prefer being explcit about the comparison versus zero.

  assert len(exec_options) > 0



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 3
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Oct 2017 23:15:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-4236: Codegen CopyRows() for select nodes

Posted by "Bikramjeet Vig (Code Review)" <ge...@cloudera.org>.
Bikramjeet Vig has posted comments on this change. ( http://gerrit.cloudera.org:8080/8196 )

Change subject: IMPALA-4236: Codegen CopyRows() for select nodes
......................................................................


Patch Set 7:

(3 comments)

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node-ir.cc
File be/src/exec/select-node-ir.cc:

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node-ir.cc@28
PS7, Line 28: limit_
> This could return extra rows. E.g. if the limit for the node is 1025, we re
Done.
also added a test case that checks if limit is enforced by select node. planner tests already exist that check if limit is pushed to a select node, so I have only added a query test.


http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc
File be/src/exec/select-node.cc:

http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc@91
PS7, Line 91:   DCHECK(*eos == false);
> This isn't safe - the GetNext() API doesn't require that the caller initial
Done


http://gerrit.cloudera.org:8080/#/c/8196/7/be/src/exec/select-node.cc@112
PS7, Line 112: 
> nit: remove extra vertical whitespace
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/8196
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie0d496d004418468e16b6f564f90f45ebbf87c1e
Gerrit-Change-Number: 8196
Gerrit-PatchSet: 7
Gerrit-Owner: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 17 Oct 2017 03:02:53 +0000
Gerrit-HasComments: Yes