You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Matthew Jacobs (Code Review)" <ge...@cloudera.org> on 2016/11/23 19:40:12 UTC
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Matthew Jacobs has uploaded a new change for review.
http://gerrit.cloudera.org:8080/5205
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
IMPALA-4522: Bound Kudu client threads to avoid stress crash
In stress testing on physical boxes (80 cores, 200gb ram) we
discovered that the Kudu Java client creates a huge number
of threads (2x the #cores) per Kudu client, and this was
causing the impalad to crash when the JVM couldn't create
more threads.
This addresses the issue by setting the number of Kudu
client worker threads rather than letting the Kudu client
pick the default (2 * #cores). The number set here was
suggested by the Kudu team as being sufficient for Impala's
FE usage and this has been tested for 8+ hours on the stress
cluster where the crash was previously observed quickly.
In the future, Impala should probably be sharing a single
Kudu client (it is multithreaded), but additional support
from Kudu may be needed to ensure this usage is correct
(e.g. client metadata may need invalidation after some
operations).
Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
---
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
1 file changed, 15 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/5205/1
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has posted comments on this change.
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/5205/1/fe/src/main/java/org/apache/impala/util/KuduUtil.java
File fe/src/main/java/org/apache/impala/util/KuduUtil.java:
Line 56: // sufficient for the Frontend/Catalog use. It has been tested in Impala stress tests.
> Mention how many threads a client created by default even if the client doe
Done
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Matthew Jacobs (Code Review)" <ge...@cloudera.org>.
Matthew Jacobs has uploaded a new patch set (#2).
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
IMPALA-4522: Bound Kudu client threads to avoid stress crash
In stress testing on physical boxes (80 cores, 200gb ram) we
discovered that the Kudu Java client creates a huge number
of threads (2x the #cores) per Kudu client, and this was
causing the impalad to crash when the JVM couldn't create
more threads.
This addresses the issue by setting the number of Kudu
client worker threads rather than letting the Kudu client
pick the default (2 * #cores). The number set here was
suggested by the Kudu team as being sufficient for Impala's
FE usage and this has been tested for 8+ hours on the stress
cluster where the crash was previously observed quickly.
In the future, Impala should probably be sharing a single
Kudu client (it is multithreaded), but additional support
from Kudu may be needed to ensure this usage is correct
(e.g. client metadata may need invalidation after some
operations).
Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
---
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
1 file changed, 17 insertions(+), 11 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/05/5205/2
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has submitted this change and it was merged.
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
IMPALA-4522: Bound Kudu client threads to avoid stress crash
In stress testing on physical boxes (80 cores, 200gb ram) we
discovered that the Kudu Java client creates a huge number
of threads (2x the #cores) per Kudu client, and this was
causing the impalad to crash when the JVM couldn't create
more threads.
This addresses the issue by setting the number of Kudu
client worker threads rather than letting the Kudu client
pick the default (2 * #cores). The number set here was
suggested by the Kudu team as being sufficient for Impala's
FE usage and this has been tested for 8+ hours on the stress
cluster where the crash was previously observed quickly.
In the future, Impala should probably be sharing a single
Kudu client (it is multithreaded), but additional support
from Kudu may be needed to ensure this usage is correct
(e.g. client metadata may need invalidation after some
operations).
Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Reviewed-on: http://gerrit.cloudera.org:8080/5205
Reviewed-by: Alex Behm <al...@cloudera.com>
Tested-by: Internal Jenkins
---
M fe/src/main/java/org/apache/impala/util/KuduUtil.java
1 file changed, 17 insertions(+), 11 deletions(-)
Approvals:
Internal Jenkins: Verified
Alex Behm: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 3
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Alex Behm (Code Review)" <ge...@cloudera.org>.
Alex Behm has posted comments on this change.
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
Patch Set 1:
(1 comment)
Good that we are catching issues like this!
http://gerrit.cloudera.org:8080/#/c/5205/1/fe/src/main/java/org/apache/impala/util/KuduUtil.java
File fe/src/main/java/org/apache/impala/util/KuduUtil.java:
Line 56: // sufficient for the Frontend/Catalog use. It has been tested in Impala stress tests.
Mention how many threads a client created by default even if the client does not need that many threads. Also mention, that Impala operations on Kudu table typically do not share clients, then it becomes clear that the number of threads can blow up pretty quickly.
Just to make the problem a little clearer for readers :)
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 1
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-4522: Bound Kudu client threads to avoid stress crash
Posted by "Internal Jenkins (Code Review)" <ge...@cloudera.org>.
Internal Jenkins has posted comments on this change.
Change subject: IMPALA-4522: Bound Kudu client threads to avoid stress crash
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/5205
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: I3940df776eaa5ad22e1bbb572559afcc8990bf1d
Gerrit-PatchSet: 2
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-Owner: Matthew Jacobs <mj...@cloudera.com>
Gerrit-Reviewer: Alex Behm <al...@cloudera.com>
Gerrit-Reviewer: Internal Jenkins
Gerrit-Reviewer: Matthew Jacobs <mj...@cloudera.com>
Gerrit-HasComments: No