You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org> on 2017/04/20 21:12:30 UTC
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock during planning
Bharath Vissapragada has uploaded a new change for review.
http://gerrit.cloudera.org:8080/6707
Change subject: IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock_ during planning
......................................................................
IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock_ during planning
- Prevents locking inside GetQueryExecState() method. It is confusing that some
callers choose to lock it inside the method and wait on it later. Now,
the locking is the responsibility of the callers.
- Removes holding of QueryExecState::lock_ during query planning. It
looks like it was done to prevent the callers from accessing
QueryExecState::result_metadata_. This unnecessarily blocks every
other access to the QES, especially blocking the web UIs. Instead
result_metadata() is now a blocking call on a separate promise that is
set after the ExecState is built.
This patch still requires some clean up. I was not fully sure if the
approach is correct, so I wanted to get opinions of others before
finalizing the approach.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/query-exec-state.h
6 files changed, 78 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/1
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 6:
> I haven't rebased this yet on to new QES changes so that you can
> diff it from previous versions. Please let me know if you need a
> rebased patch.
Please go ahead and rebase.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#7).
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 151 insertions(+), 47 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/7
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
Line 22: and run parallel queries.
> And can you first just try to reproduce the problem manually by doing those
Any luck with reproducing that scenario?
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 11: Code-Review+2
Carrying +2.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 11
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/9/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS9, Line 75: 30000
> I don't think it waits 15s everytime. It waits for 15s overall. So it still
What I'm saying is that with only 30s the test can still pass even on a build that contains the bug, because the second timeout expires at after the delay in impalad is over. So, the test doesn't work as a regression test with such a low timeout.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Reviewed-on: http://gerrit.cloudera.org:8080/6707
Reviewed-by: Bharath Vissapragada <bh...@cloudera.com>
Tested-by: Impala Public Jenkins
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 164 insertions(+), 54 deletions(-)
Approvals:
Impala Public Jenkins: Verified
Bharath Vissapragada: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 12
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 9:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6707/9/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 607: if (request_state->query_state() == beeswax::QueryState::CREATED) {
> as we discussed in person, here we'll return something like the caller does
Done
Line 607: if (request_state->query_state() == beeswax::QueryState::CREATED) {
> Additionally, let's exercise this case in the new test.
Done
http://gerrit.cloudera.org:8080/#/c/6707/9/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS9, Line 75: 30000
> that doesn't seem large enough. check_registered_queries() is allowed to wa
I don't think it waits 15s everytime. It waits for 15s overall. So it still returns before pause_injection time is hit (30s). Not sure I follow you. Did it fail for you locally? In any case, I made the pause 100s, we don't wait on it anyway.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/9/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS9, Line 75: 30000
that doesn't seem large enough. check_registered_queries() is allowed to wait 15 seconds each time, so the test can still pass even if the bug exists, right?
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 6:
(4 comments)
http://gerrit.cloudera.org:8080/#/c/6707/6/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 192: DEFINE_int64(metadata_loading_pause_injection_ms, 0, "Simulates metadata loading for a "
how about prefixing the flag with "stress_", like we do for the one in free-pool.h, to give more of a clue that this is just for stressing the system.
PS6, Line 822: // Note: this acquires the exec_state lock *before* the
: // query_exec_state_map_ lock. This is the opposite of
: // GetQueryExecState(..., true), and therefore looks like a
: // candidate for deadlock. The reason this works here is that
: // GetQueryExecState cannot find exec_state (under the exec state
: // map lock) and take it's lock until RegisterQuery has
: // finished. By that point, the exec state map lock will have been
: // given up, so the classic deadlock interleaving is not possible.
this comment is outdated now.
http://gerrit.cloudera.org:8080/#/c/6707/6/be/src/service/impala-server.h
File be/src/service/impala-server.h:
PS6, Line 85: /// 4. query_exec_state_map_lock_
i think this should be updated. with this change, do we ever hold this lock in conjunction with another lock?
http://gerrit.cloudera.org:8080/#/c/6707/6/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS6, Line 55: GetRuntimeProfileStr
This doesn't make sense. GetRuntimeProfileStr() doesn't hold the map lock while getting the QES::lock_. I think what's happening is that the call on line 76 to get_in_flight_queries() is hitting the QuerySummaryHandler() handler (please double check that), which then causes the lock cycle.
Without the fix, does the test ever get past line 76? I would expect the webserver to block on the QES::lock_ at that point.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 10: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/10/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS10, Line 83: result = impalad.service.read_debug_webpage("query_profile_encoded?query_id="\
: + inflight_query_ids[0]['query_id'])
: assert result.startswith("Could not obtain runtime profile")
This should be moved to after line 90 (after the original test case). otherwise, in the case without the fix, hitting this webpage causes the test to block until the planning delay is over and allows the test to pass
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 3:
(11 comments)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
PS2, Line 7: yExecState::lock_ contention
> let's not actually solve that problem right now. it doesn't seem necessary
Done
Line 11: The most common occurrence of this is while loading the webpage of a query
> Yes, but please double check this is safe. From code inspection, I believe
Ran a stress test and its green (Thanks Mike B.). Also I checked the callers manually and it looked ok to me too.
Line 18: side effects. Instead,
> As stated above, let's not do that.
Done
Line 22: - As an exception, we don't grab QES::lock_ while the query planning is
> As we discussed, please see if you can add a test that:
Not much success on this front. I'm not able to repro the scenario via tests. I added code to inject sleep simulating table loading in the exec request and loops that keep polling the web UI, but I'm not able to get the test failing on the asf-master (without this patch). May be I'm not generating enough load for lock contention to happen. Thoughts? (I've included the test in PS2).
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
Line 197: {
> Let's take the QES::lock_ here and not do the promise thing.
Done.
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
Line 741: lock_guard<mutex> l(*exec_state->lock());
> same
Done
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
Line 715: if (exec_state.get() != nullptr) {
> please fill in those details.
Done
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
PS2, Line 814:
: (*exec_state)->query_events()->MarkEvent("Query submitted");
:
Changes undone as discussed.
http://gerrit.cloudera.org:8080/#/c/6707/3/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
PS3, Line 603: // For queries in CREATED state, the profile information isn't populated yet.
: if (exec_state->query_state() == beeswax::QueryState::CREATED) return Status::OK();
Wanted to double check if this change is required. Reason being, with this change, the profile doesn't show up on the web UI till the metadata is loaded. That might affect some users/clients relying on the web UI profiles to track the query state?
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-server.h
File be/src/service/impala-server.h:
PS2, Line 296:
: /// goto http://msdn.microsoft.com/en-us/library/ms714687.aspx
> I think this comment just adds confusion. It's up to the QueryExecState cl
Done
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/query-exec-state.h
File be/src/service/query-exec-state.h:
PS2, Line 244:
> we won't need that.
Done
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Hello Dan Hecht,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/6707
to look at the new patch set (#11).
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 164 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/11
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 11
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 3:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
Line 22: - As an exception, we don't grab QES::lock_ while the query planning is
> Not much success on this front. I'm not able to repro the scenario via test
Load isn't relevant. The lock dependency is the problem, so as long as you hold the map lock, you should be able to demonstrate the problem.
Just to be clear, you tried to reproduce this using the test but without the fixes, right?
http://gerrit.cloudera.org:8080/#/c/6707/3/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
PS3, Line 603: // For queries in CREATED state, the profile information isn't populated yet.
: if (exec_state->query_state() == beeswax::QueryState::CREATED) return Status::OK();
> Wanted to double check if this change is required. Reason being, with this
Is it possible to get to this method for a query that hasn't returned from ExecStatement() yet? Where does the query_id come from along the paths that call into here?
Also, what info is in the profile up to this point?
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#3).
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
Holding query_exec_state_map_lock_ and QES::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a query
while the query planning is still in progress. Since we hold the
QES::lock_ during planning, it blocks the web page from loading which inturn
blocks incoming queries by holding query_exec_state_map_lock_.
While the actual fix for this is to not hold the QES::lock_ during
metadata load, this patch doesn't do it since it could have some unknown
side effects. Instead,
- We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
in hot paths.
- As an exception, we don't grab QES::lock_ while the query planning is
in progress.
Also, this patch removes locking inside the GetQueryExecState() method.
It is confusing that some callers choose to lock it inside the method and
wait on it later. Now, the locking is the responsibility of the callers.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 139 insertions(+), 36 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/3
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
Line 22: and run parallel queries.
> Load isn't relevant. The lock dependency is the problem, so as long as you
And can you first just try to reproduce the problem manually by doing those steps:
1) Executes a query and simulates getting stuck in planning (loop forever).
2) see if the debug webpage is stuck. It should be since you're holding a QES lock.
3) Execute a second query. Now this query should not even be able to register since the webserver is holding the map lock (while waiting for the QES::lock of #1).
http://gerrit.cloudera.org:8080/#/c/6707/3/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
PS3, Line 722: ate->query_status();
not sure if that's what we should put. Let's see what Henry suggests.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 7:
(8 comments)
The new test fails deterministically without the patch when run locally.
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
PS7, Line 296: NULL
> nit: change that one too to at least keep functions consistent
Done
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
PS7, Line 721: just return
> that's not what the code does (it also sets plan_metadata_unavailable), ple
Done
PS7, Line 730: adopt_lock_t
> shouldn't that be deleted?
Oops, missed that during rebase.
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 836: #endif
> actually, how about moving this to before UpdateQueryStatus() since that mo
Done
http://gerrit.cloudera.org:8080/#/c/6707/7/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS7, Line 32: The intention here is to check contention on the query_exec_state_map_lock_
> This is talking about how the old code worked, which won't make sense to pe
Rephrased.
PS7, Line 54: This creates lock contention on the coordinator by
: calling QuerySummaryHandler() method
> This is no longer true with your fix. How about saying:
Yea it doesn't make sense to someone reading it with the fix. . IMO, the test class comment conveys the intention already and this specific comment looks redundant. Removed it, please let me know if you disagree.
PS7, Line 74: time.sleep(2)
> I'm worried that this will be flaky, especially with ASAN. Instead of this
Yep good idea to reduce flakiness. But I don't think passing a large value to get_inflight_queries() would achieve that since the /inflight_queries end point never times out. Instead we should repeatedly poll that page in loop with timeout. Redid the logic accordingly.
PS7, Line 83: time.sleep(2)
> this delay is a bit harder to eliminate. How about we increase --stress_me
Same comment as above. I redid the logic, please let me know if that makes sense.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 10:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/10/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS10, Line 83: result = impalad.service.read_debug_webpage("query_profile_encoded?query_id="\
: + inflight_query_ids[0]['query_id'])
: assert result.startswith("Could not obtain runtime profile")
> This should be moved to after line 90 (after the original test case). other
Done
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock during planning
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock_ during planning
......................................................................
Patch Set 2:
(9 comments)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
PS2, Line 7: Don't hold QueryExecState::lock_ during planning
let's not actually solve that problem right now. it doesn't seem necessary for IMPALA-1972 which is the large pain.
Instead, the other two changes should be sufficient:
- Never block on QES::lock_ while holding the map lock. This breaks the cycle that leads to one query blocking another query.
- Don't try to take QES::lock_ from the webserver if the query is still planning. This deals with the webserver freezing when metadata loading takes a long time.
Line 11: the locking is the responsibility of the callers.
Yes, but please double check this is safe. From code inspection, I believe it is.
Line 18: set after the ExecState is built.
As stated above, let's not do that.
But please add a short snippet about the webserver change.
Line 22: and run parallel queries.
As we discussed, please see if you can add a test that:
1) Executes a query and simulates getting stuck in planning.
2) Hits the debug webserver. Before this fix, it will hang.
3) Executes a second query. Before this fix, this will also get stuck.
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
Line 197: // Convert TResultSetMetadata to Beeswax.ResultsMetadata
Let's take the QES::lock_ here and not do the promise thing.
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-hs2-server.cc
File be/src/service/impala-hs2-server.cc:
Line 741: // Convert TResultSetMetadata to TGetResultSetMetadataResp
same
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
Line 715: if (exec_state->query_state() == beeswax::QueryState::CREATED) return;
please fill in those details.
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/impala-server.h
File be/src/service/impala-server.h:
PS2, Line 296: Callers are responsible
: /// for acquiring QueryExecState::lock_ before performing any operations on it.
I think this comment just adds confusion. It's up to the QueryExecState class to explain its synchronization.
http://gerrit.cloudera.org:8080/#/c/6707/2/be/src/service/query-exec-state.h
File be/src/service/query-exec-state.h:
PS2, Line 244: planning_finished_promise_
we won't need that.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#10).
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 164 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/10
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 10
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 7:
(7 comments)
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-beeswax-server.cc
File be/src/service/impala-beeswax-server.cc:
PS7, Line 296: NULL
nit: change that one too to at least keep functions consistent
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-http-handler.cc
File be/src/service/impala-http-handler.cc:
PS7, Line 721: just return
that's not what the code does (it also sets plan_metadata_unavailable), please rephrase. Could rephrase the whole comment as:
If the query plan isn't generated, avoid waiting for the lock, which could take a while if catalog metadata is being loaded.
PS7, Line 730: adopt_lock_t
shouldn't that be deleted?
http://gerrit.cloudera.org:8080/#/c/6707/7/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS7, Line 32: The intention here is to check contention on the query_exec_state_map_lock_
This is talking about how the old code worked, which won't make sense to people reading the current code (after this change). It should say something like:
The intention is to check that the webserver does not hold any global locks or otherwise prevent impalad from servicing new requests.
PS7, Line 54: This creates lock contention on the coordinator by
: calling QuerySummaryHandler() method
This is no longer true with your fix. How about saying:
This is to verify that QuerySummaryHandler() does not hold any global locks that would, for example, prevent another query from starting.
PS7, Line 74: time.sleep(2)
I'm worried that this will be flaky, especially with ASAN. Instead of this delay, couldn't we just wait for in_flight_queries to become 1? And you could use the parameter to get_in_flight_queries() to do that by passing some largish value. That has the advantage that we'll wait only as long as necessary for the value to change to 1, so we can have a relatively long timeout (rather than delay).
PS7, Line 83: time.sleep(2)
this delay is a bit harder to eliminate. How about we increase --stress_metadata_loading_pause_injection_ms to something really large, say 1000 seconds (which doesn't matter -- we don't actually need the queries to finish planning to end the test, right?).
And then we can use a larger timeout here, but we don't need to delay for it. We can just do:
inflight_query_ids = impalad.service.get_in_flight_queries(30)
which will poll the webui once per second and give up after 30 seconds.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 7:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/7/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 836: #endif
actually, how about moving this to before UpdateQueryStatus() since that more closely mimics what happens when planning takes a long time (in case of failure, the status won't be updated until after the delay).
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 7
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#5).
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
Holding query_exec_state_map_lock_ and QES::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
QES::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding query_exec_state_map_lock_.
While the actual fix for this is to not hold the QES::lock_ during
metadata load, this patch doesn't do it since it could have some unknown
side effects. Instead,
- We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
in hot paths.
- As an exception, we don't grab QES::lock_ while the query planning is
in progress.
Also, this patch removes locking inside the GetQueryExecState() method.
It is confusing that some callers choose to lock it inside the method
and wait on it later. Now, the locking is the responsibility of the
callers.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 144 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/5
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 5:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/6707/2//COMMIT_MSG
Commit Message:
Line 22: - As an exception, we don't grab QES::lock_ while the query planning is
> Any luck with reproducing that scenario?
The bug was exactly what we are discussing in the other comment. My test was waiting on ExecuteInternal() to return and by that time, the metadata pause has already completed. I rewrote the test to scrape the inflight_running_queryids page to pickup the id and then poll the webpage. It consistently fails without the patch.
http://gerrit.cloudera.org:8080/#/c/6707/3/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
PS3, Line 603: // For queries in CREATED state, the profile information isn't populated yet.
: if (exec_state->query_state() == beeswax::QueryState::CREATED) return Status::OK();
> Is it possible to get to this method for a query that hasn't returned from
I think it is possible. Reason being, the /queries page polls the query_exec_state_map_ to get the running queries and display on the UI. So, as long as the RegisterQuery() returns, it shows up on the UI (the metadata loading happens right after that). Beeswax/HS2 clients though would block on the ExecuteInternal() call and wait for the metadata to be loaded to get the query_handle.
http://gerrit.cloudera.org:8080/#/c/6707/3/www/query_plan.tmpl
File www/query_plan.tmpl:
PS3, Line 42: {{?plan_metadata_unavailable}}
: <h3> Query plan information is not yet available since the planning hasn't finished.</h3>
: {{/plan_metadata_unavailable}}
:
: {{^plan_metadata_unavailable}}
> Since the webpage refreshes every few seconds, it would be best to have thi
Done
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 5
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#9).
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 162 insertions(+), 53 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/9
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 11:
Build started: http://jenkins.impala.io:8080/job/gerrit-verify-dryrun/611/
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 11
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 6:
I haven't rebased this yet on to new QES changes so that you can diff it from previous versions. Please let me know if you need a rebased patch.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 6:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/6707/6//COMMIT_MSG
Commit Message:
PS6, Line 16: While the actual fix for this is to not hold the QES::lock_ during
: metadata load,
I disagree. what you are doing is the actual fix for the lock acquisition order. Not holding QES::lock_ while planning is a partial fix for a different issue (not being able to cancel while planning).
PS6, Line 20: We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
: in hot paths.
After this change, isn't it the case that we'll never try to acquire another lock while holding the map lock? If so, this should say make query_exec_state_map_lock_ a terminal lock.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock during planning
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#2).
Change subject: IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock_ during planning
......................................................................
IMPALA-1972/IMPALA-3882: [DRAFT] Don't hold QueryExecState::lock_ during planning
- Prevents locking inside GetQueryExecState() method. It is confusing that some
callers choose to lock it inside the method and wait on it later. Now,
the locking is the responsibility of the callers.
- Removes holding of QueryExecState::lock_ during query planning. It
looks like it was done to prevent the callers from accessing
QueryExecState::result_metadata_. This unnecessarily blocks every
other access to the QES, especially blocking the web UIs. Instead
result_metadata() is now a blocking call on a separate promise that is
set after the ExecState is built.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries.
This patch still requires some clean up. I was not fully sure if the
approach is correct, so I wanted to get opinions of others before
finalizing the approach.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
M be/src/service/query-exec-state.h
6 files changed, 78 insertions(+), 74 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/2
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Henry Robinson (Code Review)" <ge...@cloudera.org>.
Henry Robinson has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/3/www/query_plan.tmpl
File www/query_plan.tmpl:
PS3, Line 42: {{?plan_metadata_unavailable}}
: <h3> Query plan information is not yet available since the planning hasn't finished.</h3>
: {{/plan_metadata_unavailable}}
:
: {{^plan_metadata_unavailable}}
Since the webpage refreshes every few seconds, it would be best to have this say something like "Plan not yet available. Page will update when query planning completes".
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 11: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 11
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/9/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 607: if (request_state->query_state() == beeswax::QueryState::CREATED) {
as we discussed in person, here we'll return something like the caller does. Actually, maybe we should just return an error Status with the message, and then let the caller put that string in the profile?
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#4).
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
Holding query_exec_state_map_lock_ and QES::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
QES::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding query_exec_state_map_lock_.
While the actual fix for this is to not hold the QES::lock_ during
metadata load, this patch doesn't do it since it could have some unknown
side effects. Instead,
- We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
in hot paths.
- As an exception, we don't grab QES::lock_ while the query planning is
in progress.
Also, this patch removes locking inside the GetQueryExecState() method.
It is confusing that some callers choose to lock it inside the method
and wait on it later. Now, the locking is the responsibility of the
callers.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 139 insertions(+), 36 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/4
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#8).
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
Holding client_request_state_map_lock_ and CRS::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
CRS::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding client_request_state_map_lock_.
This patch makes client_request_state_map_lock_ a terminal lock so that
we don't have interleaving locking with CRS::lock_.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries. Also added a custom cluster test that does the
same sequence of actions by injecting a metadata loading pause.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 164 insertions(+), 54 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/8
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 8
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 6:
(6 comments)
http://gerrit.cloudera.org:8080/#/c/6707/6//COMMIT_MSG
Commit Message:
PS6, Line 16: While the actual fix for this is to not hold the QES::lock_ during
: metadata load,
> I disagree. what you are doing is the actual fix for the lock acquisition o
Done
PS6, Line 20: We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
: in hot paths.
> After this change, isn't it the case that we'll never try to acquire anothe
This was stale. Updated.
http://gerrit.cloudera.org:8080/#/c/6707/6/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 192: DEFINE_int64(metadata_loading_pause_injection_ms, 0, "Simulates metadata loading for a "
> how about prefixing the flag with "stress_", like we do for the one in free
Done
PS6, Line 822: // Note: this acquires the exec_state lock *before* the
: // query_exec_state_map_ lock. This is the opposite of
: // GetQueryExecState(..., true), and therefore looks like a
: // candidate for deadlock. The reason this works here is that
: // GetQueryExecState cannot find exec_state (under the exec state
: // map lock) and take it's lock until RegisterQuery has
: // finished. By that point, the exec state map lock will have been
: // given up, so the classic deadlock interleaving is not possible.
> this comment is outdated now.
Removed.
http://gerrit.cloudera.org:8080/#/c/6707/6/be/src/service/impala-server.h
File be/src/service/impala-server.h:
PS6, Line 85: /// 4. query_exec_state_map_lock_
> i think this should be updated. with this change, do we ever hold this lock
No. Checked the usage of in the callers. Updated the comment.
http://gerrit.cloudera.org:8080/#/c/6707/6/tests/custom_cluster/test_query_concurrency.py
File tests/custom_cluster/test_query_concurrency.py:
PS6, Line 55: GetRuntimeProfileStr
> This doesn't make sense. GetRuntimeProfileStr() doesn't hold the map lock w
Ah this comment is wrong. My intention was exactly to call QuerySummaryHandler() which is place that calls with the interleaving locking that causes the hang. Corrected it.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix client request state map lock contention
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-1972/IMPALA-3882: Fix client_request_state_map_lock_ contention
......................................................................
Patch Set 9:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/6707/9/be/src/service/impala-server.cc
File be/src/service/impala-server.cc:
Line 607: if (request_state->query_state() == beeswax::QueryState::CREATED) {
> as we discussed in person, here we'll return something like the caller does
Additionally, let's exercise this case in the new test.
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 9
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock contention
Posted by "Bharath Vissapragada (Code Review)" <ge...@cloudera.org>.
Bharath Vissapragada has uploaded a new patch set (#6).
Change subject: IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
......................................................................
IMPALA-1972/IMPALA-3882: Fix QueryExecState::lock_ contention
Holding query_exec_state_map_lock_ and QES::lock_ together in certain
paths could potentially block the impalad from registering new queries.
The most common occurrence of this is while loading the webpage of a
query while the query planning is still in progress. Since we hold the
QES::lock_ during planning, it blocks the web page from loading which
inturn blocks incoming queries by holding query_exec_state_map_lock_.
While the actual fix for this is to not hold the QES::lock_ during
metadata load, this patch doesn't do it since it could have some unknown
side effects. Instead,
- We make sure that QES::lock_ doesn't block query_exec_state_map_lock_
in hot paths.
- As an exception, we don't grab QES::lock_ while the query planning is
in progress.
Also, this patch removes locking inside the GetQueryExecState() method.
It is confusing that some callers choose to lock it inside the method
and wait on it later. Now, the locking is the responsibility of the
callers.
Testing: Tested it locally by adding a long sleep in
JniFrontend.createExecRequest() and still was able to refresh the web UI
and run parallel queries.
Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
---
M be/src/service/impala-beeswax-server.cc
M be/src/service/impala-hs2-server.cc
M be/src/service/impala-http-handler.cc
M be/src/service/impala-server.cc
M be/src/service/impala-server.h
A tests/custom_cluster/test_query_concurrency.py
M www/query_plan.tmpl
7 files changed, 144 insertions(+), 37 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/07/6707/6
--
To view, visit http://gerrit.cloudera.org:8080/6707
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie44daa93e3ae4d04d091261f3ec4891caffe8026
Gerrit-PatchSet: 6
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Bharath Vissapragada <bh...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Henry Robinson <he...@cloudera.com>