You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org> on 2016/12/02 23:26:00 UTC
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Thomas Tauber-Marshall has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5334
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
IMPALA-4566: Kudu client glog contention can cause timeouts
Under stressful workloads, there appears to be significant contention
in glog resulting from Kudu logging, causing timeouts and failed
queries.
An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
WARNINGs grab a lock to flush the log.
Testing: Manually verified that the patch redirects Kudu WARNINGs
to the INFO log. Not tested under stress to verify if this actually
solves the contention problem.
Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
---
M be/src/exec/kudu-util.cc
1 file changed, 3 insertions(+), 1 deletion(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/5334/1
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2: Code-Review+2
I think this addressed the issue, thanks!
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 3: Verified-1
Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/628/
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2:
(2 comments)
http://gerrit.cloudera.org:8080/#/c/5334/1/be/src/exec/kudu-util.cc
File be/src/exec/kudu-util.cc:
PS1, Line 81: us
> remove
Done
PS1, Line 88: Log
> remove We
Done
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Dan Hecht (Code Review)" <ge...@cloudera.org>.
Dan Hecht has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 1:
I saw Todd has a change to address this on the client side.. is this just a bandaid until that change comes in, or do we plan to keep this change?
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2: -Code-Review
Hm maybe the issue is still there, I thought it was fixed but Todd said he's still seeing it. I'll look further today.
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 1:
Thanks, I'll test this on the stress cluster over the weekend
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 1:
(3 comments)
> I saw Todd has a change to address this on the client side.. is
> this just a bandaid until that change comes in, or do we plan to
> keep this change?
I'm not sure yet, I was going to try porting his patch over after we branch for the release. His patch is cool, but it's more moving pieces and it's generally not as useful for Impala since we don't use WARNING frequently, and thus maybe not worth the added complexity. Really it should just be fixed in upstream glog, Todd filed a ticket upstream.
TL;DR, wait and see.
http://gerrit.cloudera.org:8080/#/c/5334/1//COMMIT_MSG
Commit Message:
PS1, Line 13: An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
: WARNINGs grab a lock to flush the log.
and mention that Kudu warns much more frequently than Impala does, for things that we'd consider to be INFO-level typically.
http://gerrit.cloudera.org:8080/#/c/5334/1/be/src/exec/kudu-util.cc
File be/src/exec/kudu-util.cc:
PS1, Line 81: we
remove
PS1, Line 88: We l
remove We
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2:
FYI I've sent Todd an e-mail (he's in Singapore) asking why he thinks this isn't working. If we don't hear anything by tomorrow let's get this in.
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2: Code-Review+2
can you submit this today?
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2: Verified-1
Build failed: http://sandbox.jenkins.cloudera.com/job/impala-external-gerrit-verify-merge-ASF/626/
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Thomas Tauber-Marshall (Code Review)" <ge...@cloudera.org>.
Thomas Tauber-Marshall has uploaded a new patch set (#2).
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
IMPALA-4566: Kudu client glog contention can cause timeouts
Under stressful workloads, there appears to be significant contention
in glog resulting from Kudu logging, causing timeouts and failed
queries.
An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
WARNINGs grab a lock to flush the log. This is appropriate as Kudu
logs WARNINGs much more frequently than Impala and for things that
Impala would normally consider INFO-level.
Testing: Manually verified that the patch redirects Kudu WARNINGs
to the INFO log. Not tested under stress to verify if this actually
solves the contention problem.
Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
---
M be/src/exec/kudu-util.cc
1 file changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/34/5334/2
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
Patch Set 2:
can you rebase and resubmit?
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4566: Kudu client glog contention can cause timeouts
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4566: Kudu client glog contention can cause timeouts
......................................................................
IMPALA-4566: Kudu client glog contention can cause timeouts
Under stressful workloads, there appears to be significant contention
in glog resulting from Kudu logging, causing timeouts and failed
queries.
An easy solution for now is to downgrade Kudu WARNINGs to INFOs as
WARNINGs grab a lock to flush the log. This is appropriate as Kudu
logs WARNINGs much more frequently than Impala and for things that
Impala would normally consider INFO-level.
Testing: Manually verified that the patch redirects Kudu WARNINGs
to the INFO log. Not tested under stress to verify if this actually
solves the contention problem.
Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Reviewed-on: http://gerrit.cloudera.org:8080/5334
Reviewed-by: Matthew Jacobs <mj...@cloudera.com>
Tested-by: Internal Jenkins
---
M be/src/exec/kudu-util.cc
1 file changed, 4 insertions(+), 2 deletions(-)
Approvals:
Matthew Jacobs: Looks good to me, approved
Internal Jenkins: Verified
--
To view, visit http://gerrit.cloudera.org:8080/5334
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ie535d89ec2525232d4f6a29dd44f51cd6e18a0d2
Gerrit-PatchSet: 4
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Thomas Tauber-Marshall <tm...@cloudera.com>
Gerrit-Reviewer: Dan Hecht <dh...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Thomas Tauber-Marshall <tm...@cloudera.com>