You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Alex Rodoni (Code Review)" <ge...@cloudera.org> on 2018/06/26 21:31:34 UTC

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Alex Rodoni has uploaded this change for review. ( http://gerrit.cloudera.org:8080/10829


Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................

[DOCS] Clarification on admission control and DDL statements

Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
---
M docs/topics/impala_admission.xml
1 file changed, 9 insertions(+), 5 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10829/1
-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 3: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jul 2018 18:41:46 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 2:

To be clear, I meant the existing text about queueing and the order in which statements execute. The other text seems fine.


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Jun 2018 01:07:39 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................

[DOCS] Clarification on admission control and DDL statements

Removed the confusing example and paragraphs.

Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Reviewed-on: http://gerrit.cloudera.org:8080/10829
Reviewed-by: Tim Armstrong <ta...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M docs/topics/impala_admission.xml
1 file changed, 61 insertions(+), 85 deletions(-)

Approvals:
  Tim Armstrong: Looks good to me, approved
  Impala Public Jenkins: Verified

-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 4
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Balazs Jeszenszky (Code Review)" <ge...@cloudera.org>.
Balazs Jeszenszky has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 3:

Thanks Tim for clarifying the session-thing. I was thinking in impala-shell context only.


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jul 2018 10:55:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Alex Rodoni (Code Review)" <ge...@cloudera.org>.
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 2:

> I think this is somehow much more confusing than it should be. The
 > facts are:
 > * queries submitted to a given session are executed serially, DDL
 > or not
 > * DDLs are not part of admission control
 > * Naturally, this means that any query, including DDLs, that the
 > client submits in a session that has a query queued, will not
 > execute (but this behaviour has nothing to do with admission
 > control). The tricky part is that while the example query is
 > queued, it is possible to open a new session and drop the table the
 > queued query would refer to.
 > 
 > I think we should have a page somewhere (probably here?) to
 > separate these behaviours (queued due to serial exec vs queued due
 > to admission control), and spell out how this relates to DDLs
 > specifically (the scenario above is a good example IMO).

I created a new section at the end and made an attempt to address the confusion.


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Jun 2018 23:06:19 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Alex Rodoni (Code Review)" <ge...@cloudera.org>.
Alex Rodoni has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 2:

(6 comments)

Removed the confusing examples and paragraphs.

Added what are affected by admission control at the top.

http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml
File docs/topics/impala_admission.xml:

http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@822
PS2, Line 822:           Most write operations in Impala are not resource-intensive, but
> This bit about "most write operations" feels very tangential, not sure how 
Removed


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@847
PS2, Line 847:           With or without admission control, all queries submitted in a session
> Unfortunately this isn't even true - it depends on the driver, the APIs giv
Removed


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@853
PS2, Line 853:           Except for the <codeph>CREATE TABLE AS SELECT</codeph> statements,
> And compute stats
Done


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@855
PS2, Line 855: those DDL statements 
> which does "those DDL statements" mean? It's a little unclear. I think it's
Done


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@855
PS2, Line 855:  for serial execution
> remove "for serial execution" - AC doesn't guarantee anything whether state
Removed


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@859
PS2, Line 859:         <codeblock>-- This query is queued by admission control to avoid out-of-memory at times of heavy load.
             : SELECT * FROM huge_table JOIN enormous_table USING (id);
             : 
             : -- This subsequent statement in the same session is also queued, but for serial execution and
             : -- not for admission control, until the previous statement completes.
             : DROP TABLE huge_table;
             : </codeblock>
> I think this example is still misleading. I'm not sure what it's demonstrat
Done



-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 28 Jun 2018 21:54:29 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 3: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 02 Jul 2018 23:51:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 3:

Build started: https://jenkins.impala.io/job/gerrit-docs-submit/335/


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 03 Jul 2018 18:32:45 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Alex Rodoni (Code Review)" <ge...@cloudera.org>.
Hello Balazs Jeszenszky, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10829

to look at the new patch set (#2).

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................

[DOCS] Clarification on admission control and DDL statements

Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
---
M docs/topics/impala_admission.xml
1 file changed, 89 insertions(+), 83 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10829/2
-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Alex Rodoni (Code Review)" <ge...@cloudera.org>.
Hello Balazs Jeszenszky, Tim Armstrong, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/10829

to look at the new patch set (#3).

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................

[DOCS] Clarification on admission control and DDL statements

Removed the confusing example and paragraphs.

Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
---
M docs/topics/impala_admission.xml
1 file changed, 61 insertions(+), 85 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/29/10829/3
-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 3
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 2:

(7 comments)

Thanks for taking this on. I feel like there are still some remnants of some misconceptions here. Not sure exactly how best to fix it, but I feel like the misconceptions were so fundamental that we should just delete the existing text and figure out if there was a valid point to make.

http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml
File docs/topics/impala_admission.xml:

http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@820
PS2, Line 820:           primarily queries, but also include statements that write data such as
Maybe this could be clearer by stating that "Queries, DML statements and some DDL statements, including "CREATE TABLE AS SELECT" and "COMPUTE STATS" are affected by admission control.


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@822
PS2, Line 822:           Most write operations in Impala are not resource-intensive, but
This bit about "most write operations" feels very tangential, not sure how useful it is. If I was reading it I wouldn't know what it means. Does it mean that non-resource-intensive insert queries aren't affected by admission control (they are).

Anyway, this isn't the exact problem we're trying to fix but noticed it while reading through.


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@847
PS2, Line 847:           With or without admission control, all queries submitted in a session
Unfortunately this isn't even true - it depends on the driver, the APIs give a lot of flexibility about launching multiple queries in a session at the same time and interacting with them in different ways while running. Different drivers can do different things here.

I'm wondering if we could just omit this paragraph - admission control is orthogonal to sequencing of queries. All admission control can do is delay queries, which isn't really that different from them just running longer.


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@853
PS2, Line 853:           Except for the <codeph>CREATE TABLE AS SELECT</codeph> statements,
And compute stats


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@855
PS2, Line 855:  for serial execution
remove "for serial execution" - AC doesn't guarantee anything whether statements execute serially or concurrently. It can throttle statements but no reason 2, 10 or 100 statements can run concurrently.


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@855
PS2, Line 855: those DDL statements 
which does "those DDL statements" mean? It's a little unclear. I think it's actually wrong either way - maybe delete this sentence?


http://gerrit.cloudera.org:8080/#/c/10829/2/docs/topics/impala_admission.xml@859
PS2, Line 859:         <codeblock>-- This query is queued by admission control to avoid out-of-memory at times of heavy load.
             : SELECT * FROM huge_table JOIN enormous_table USING (id);
             : 
             : -- This subsequent statement in the same session is also queued, but for serial execution and
             : -- not for admission control, until the previous statement completes.
             : DROP TABLE huge_table;
             : </codeblock>
I think this example is still misleading. I'm not sure what it's demonstrating, because the behaviour is exactly the same with or without admission control. 

Is all of this trying to say "admission control is orthogonal to whether statements execute sequentially or not"? Maybe we should just find a better way to say that. E.g. "From the point of view of a client, queries that are queued by admission control behave the same as unqueued queries except that it takes longer to get results back."



-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 2
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Jun 2018 01:05:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] [DOCS] Clarification on admission control and DDL statements

Posted by "Balazs Jeszenszky (Code Review)" <ge...@cloudera.org>.
Balazs Jeszenszky has posted comments on this change. ( http://gerrit.cloudera.org:8080/10829 )

Change subject: [DOCS] Clarification on admission control and DDL statements
......................................................................


Patch Set 1:

I think this is somehow much more confusing than it should be. The facts are:
* queries submitted to a given session are executed serially, DDL or not
* DDLs are not part of admission control
* Naturally, this means that any query, including DDLs, that the client submits in a session that has a query queued, will not execute (but this behaviour has nothing to do with admission control). The tricky part is that while the example query is queued, it is possible to open a new session and drop the table the queued query would refer to.

I think we should have a page somewhere (probably here?) to separate these behaviours (queued due to serial exec vs queued due to admission control), and spell out how this relates to DDLs specifically (the scenario above is a good example IMO).


-- 
To view, visit http://gerrit.cloudera.org:8080/10829
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2e3e82bd34e88e7a13de1864aeb97f01023bc715
Gerrit-Change-Number: 10829
Gerrit-PatchSet: 1
Gerrit-Owner: Alex Rodoni <ar...@cloudera.com>
Gerrit-Reviewer: Balazs Jeszenszky <je...@gmail.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 26 Jun 2018 21:48:33 +0000
Gerrit-HasComments: No