You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2019/10/01 15:37:42 UTC
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/14332
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
17 files changed, 477 insertions(+), 152 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/1
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/common/status.cc
File be/src/common/status.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/common/status.cc@160
PS1, Line 160: apache::hive::service::cli::thrift::TStatusCode
> Maybe add a 'using TStatusCode' to this file so the formatting here isn't s
Done
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h@108
PS1, Line 108: (
> typo?
Done
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h@114
PS1, Line 114: wait for
> typo?
Done
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc@786
PS1, Line 786: void ClientRequestState::BlockOnWait() {
> Could we reduce some duplication by just calling BlockOnWait(0) here?
Done
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc@809
PS1, Line 809: block_on_wait_cv_.Wait(l);
> So if I understand correctly, here if 'timeout' is 0 we wait indefinitely b
Thanks for catching that. Have clarified the behavior in the docs. A value of 0 should cause the fetch request to wait indefinitely. Added a test as well.
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc@860
PS1, Line 860: return_val.status.__set_statusCode(thrift::TStatusCode::SUCCESS_STATUS);
> did you mean to make this change?
Yes, FetchInternal always sets the statusCode, unless it returns an error impala::Status, in which case the if loop above is triggered. If it reaches the end of the if without returning, than the statusCode is set to SUCCESS_STATUS.
The change is necessary otherwise SUCCESS_STATUS is always returned, even if the statusCode was already set to STILL_EXECUTING_STATUS
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Oct 2019 21:36:29 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 3:
(2 comments)
Nice tests!
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc@687
PS3, Line 687: parent_request_state_->fetch_rows_timeout_us()
: - parent_request_state_->block_on_wait_time_us
If by coincidence block_on_wait_time_us_ is the same as fetch_rows_timeout_us, can't this work out to 0 and incorrectly block indefinitely?
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h@353
PS3, Line 353: uint64_t block_on_wait_time_us_ = 0;
I think storing this per-query is incorrect since there could be multiple concurrent calls to BlockOnWait() and one thread might pick up the other thread's value. I think that might be an unimportant edge case and we could probably live with that but I think it still makes the logic unnecessarily confusing because this value really fits with the RPC request, not the query.
I could see either threading it through the function calls (probably the most obvious and explicit way) or using a thread-local variable.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2019 20:13:56 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Reviewed-on: http://gerrit.cloudera.org:8080/14332
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M common/thrift/ImpalaService.thrift
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
19 files changed, 562 insertions(+), 188 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 8
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 3:
(2 comments)
Addressed comments and made a few cosmetic changes.
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/runtime/coordinator.cc@687
PS3, Line 687: parent_request_state_->fetch_rows_timeout_us()
: - parent_request_state_->block_on_wait_time_us
> If by coincidence block_on_wait_time_us_ is the same as fetch_rows_timeout_
whoops, yeah that should be a max(1, ...) instead of max(0, ...). There was also a bug where timeout_us might not be set to 0, even if FETCH_ROWS_TIMEOUT_MS was 0. Fixed both issues.
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/14332/3/be/src/service/client-request-state.h@353
PS3, Line 353: uint64_t block_on_wait_time_us_ = 0;
> I think storing this per-query is incorrect since there could be multiple c
Done
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Oct 2019 00:19:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4692/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Oct 2019 22:00:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 6: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Oct 2019 21:48:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4693/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Oct 2019 22:15:29 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Oct 2019 21:47:24 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 1:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/common/status.cc
File be/src/common/status.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/common/status.cc@160
PS1, Line 160: apache::hive::service::cli::thrift::TStatusCode
Maybe add a 'using TStatusCode' to this file so the formatting here isn't so awful?
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h
File be/src/service/client-request-state.h:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h@108
PS1, Line 108: (
typo?
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.h@114
PS1, Line 114: wait for
typo?
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc
File be/src/service/client-request-state.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc@786
PS1, Line 786: void ClientRequestState::BlockOnWait() {
Could we reduce some duplication by just calling BlockOnWait(0) here?
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/client-request-state.cc@809
PS1, Line 809: block_on_wait_cv_.Wait(l);
So if I understand correctly, here if 'timeout' is 0 we wait indefinitely but in the (Buffered|Blocking)PlanRootSink::GetNext if 'timeout' is 0 we don't wait at all.
Is that intentional? It would be good to document the expected behavior for a '0' timeout
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc@860
PS1, Line 860: return_val.status.__set_statusCode(thrift::TStatusCode::SUCCESS_STATUS);
did you mean to make this change?
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Oct 2019 17:50:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4687/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 01 Oct 2019 16:18:37 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 7: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Oct 2019 19:13:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14332
to look at the new patch set (#2).
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M common/thrift/ImpalaService.thrift
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
18 files changed, 495 insertions(+), 174 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/2
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 7:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5068/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Oct 2019 14:59:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 5:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4727/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Oct 2019 03:31:05 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 7: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 7
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Oct 2019 14:59:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Tim Armstrong (Code Review)" <ge...@cloudera.org>.
Tim Armstrong has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h
File be/src/exec/blocking-plan-root-sink.h:
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h@72
PS4, Line 72: bool* eos, const uint64_t timeout_us) override;
I'd prefer non-const for arguments that are passed by value, it just adds noise.
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc@694
PS4, Line 694: max(static_cast<uint64_t>(1),
I think there's still a possibility of underflow cause it's unsigned. I think this would all be simpler if we used int64_t for the timeouts throughout, since we don't need the extra range of the unsigned type.
https://google.github.io/styleguide/cppguide.html#Integer_Types addresses this issue and says to use signed types in this situation.
We could address the underflow locally, but the choice of signed vs unsigned is kinda viral because anywhere you convert to signed you have to deal with overflow.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Oct 2019 17:27:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 3: Code-Review+1
(1 comment)
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
http://gerrit.cloudera.org:8080/#/c/14332/1/be/src/service/impala-hs2-server.cc@860
PS1, Line 860: return_val.status.__set_statusCode(thrift::TStatusCode::SUCCESS_STATUS);
> Yes, FetchInternal always sets the statusCode, unless it returns an error i
Got it, thanks.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Wed, 02 Oct 2019 01:28:34 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 6: Verified-1
Build failed: https://jenkins.impala.io/job/gerrit-verify-dryrun/5061/
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Tue, 08 Oct 2019 02:03:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 4:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h
File be/src/exec/blocking-plan-root-sink.h:
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/exec/blocking-plan-root-sink.h@72
PS4, Line 72: bool* eos, const uint64_t timeout_us) override;
> I'd prefer non-const for arguments that are passed by value, it just adds n
Done
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc
File be/src/runtime/coordinator.cc:
http://gerrit.cloudera.org:8080/#/c/14332/4/be/src/runtime/coordinator.cc@694
PS4, Line 694: max(static_cast<uint64_t>(1),
> I think there's still a possibility of underflow cause it's unsigned. I thi
Done. Changed everything to use int64_t.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Fri, 04 Oct 2019 02:50:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14332
to look at the new patch set (#5).
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M common/thrift/ImpalaService.thrift
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
19 files changed, 562 insertions(+), 188 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/5
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 5
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 6:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/5061/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 6
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Mon, 07 Oct 2019 21:48:48 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14332
to look at the new patch set (#4).
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/runtime/coordinator.h
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M common/thrift/ImpalaService.thrift
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
19 files changed, 560 insertions(+), 188 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/4
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Hello Thomas Tauber-Marshall, Tim Armstrong, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/14332
to look at the new patch set (#3).
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
IMPALA-7312 added the query option FETCH_ROWS_TIMEOUT_MS, but it only
applies to fetch requests against a query that has already transitioned
to the 'FINISHED' state. This patch changes the timeout so that it
applies to queries in the 'RUNNING' state as well. Before this patch,
fetch requests issued while a query was 'RUNNING' blocked until the query
transitioned to the 'FINISHED' state, and then it fetched results and
returned them. After this patch, fetch requests against queries in the
'RUNNING' state will block for 'FETCH_ROWS_TIMEOUT_MS' and then return.
For HS2 clients, fetch requests that return while a query is 'RUNNING'
set their TStatusCode to STILL_EXECUTING_STATUS. For Beeswax clients,
fetch requests that return while a query is 'RUNNING' set the 'ready'
flag to false. For both clients, hasMoreRows is set to true.
If the following sequence of events occurs:
* A fetch request is issued and blocks on a 'RUNNING' query
* The query transitions to the 'FINISHED' state
* The fetch request attempts to read multiple batches
Then the time spent waiting for the query to finish is deducted from
the timeout used when waiting for rows to be produced by the Coordinator
fragment.
Fixed a bug in the current usage of FETCH_ROWS_TIMEOUT_MS where the
time units for FETCH_ROWS_TIMEOUT_MS and MonotonicStopWatch were not
being converted properly.
Tests:
* Moved existing fetch timeout tests from hs2/test_fetch.py into a new
test file hs2/test_fetch_timeout.py.
* Added several new tests to hs2/test_fetch_timeout.py to validate that
the timeout is applied to 'RUNNING' queries and that the timeout applies
across a 'RUNNING' and 'FINISHED' query.
* Added new tests to query_test/test_fetch.py to validate the timeout
while using the Beeswax protocol.
Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
---
M be/src/common/status.cc
M be/src/exec/blocking-plan-root-sink.cc
M be/src/exec/blocking-plan-root-sink.h
M be/src/exec/buffered-plan-root-sink.cc
M be/src/exec/buffered-plan-root-sink.h
M be/src/exec/plan-root-sink.cc
M be/src/exec/plan-root-sink.h
M be/src/runtime/coordinator.cc
M be/src/service/client-request-state.cc
M be/src/service/client-request-state.h
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/util/error-util.cc
M common/thrift/ImpalaService.thrift
M tests/hs2/hs2_test_suite.py
M tests/hs2/test_fetch.py
A tests/hs2/test_fetch_timeout.py
M tests/query_test/test_fetch.py
18 files changed, 521 insertions(+), 174 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/14332/3
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-ASF-CR] IMPALA-8962: FETCH ROWS TIMEOUT MS should apply before rows are available
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/14332 )
Change subject: IMPALA-8962: FETCH_ROWS_TIMEOUT_MS should apply before rows are available
......................................................................
Patch Set 4:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/4708/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/14332
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I2cba6bf062dcc1af19471d21857caa797c1ea4a4
Gerrit-Change-Number: 14332
Gerrit-PatchSet: 4
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-Comment-Date: Thu, 03 Oct 2019 00:59:41 +0000
Gerrit-HasComments: No